From patchwork Sat Nov 5 15:43:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Kjellerstedt X-Patchwork-Id: 14955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7C6FC4332F for ; Sat, 5 Nov 2022 15:43:14 +0000 (UTC) Received: from smtp1.axis.com (smtp1.axis.com [195.60.68.17]) by mx.groups.io with SMTP id smtpd.web11.7208.1667662988552932470 for ; Sat, 05 Nov 2022 08:43:09 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@axis.com header.s=axis-central1 header.b=FQ1kJvva; spf=pass (domain: axis.com, ip: 195.60.68.17, mailfrom: peter.kjellerstedt@axis.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axis.com; q=dns/txt; s=axis-central1; t=1667662989; x=1699198989; h=from:to:subject:date:message-id:mime-version: content-transfer-encoding; bh=d0KlWDUc9ANjXfOwNarWrDQ6nZYpzSYO8PeCRy4czto=; b=FQ1kJvvaQn/5Bbrio7yvpQXoNYo0IIbPuiC3fEzBjSRHJEOksxsScj07 qBeg9N1Lf2YvWEsUXAd0BwGCLGN2owExbHovnZgIQ5b77/b2A2WDcnQoE /fRh7R2z0l5dQO4tTpUD4dRR4UAEnDFtrAMyacfRWCkaHLWQYm0FxoHU3 emYXwsCMOJipFZXSdS9JqJ7JkqLECgHbC9RwWsfu48PJzXITVSwyZX65Z pwQzJxTeb2CFEBCxkmr3pCzdoCE6Nu8eP7qsc1X/cHAnj7lsdB7eMH8vW 5CqpGb2RZNuFkRUrA9mL5nwZfgNgEHScJXx0IqT6KNbb674buGeIkRx+H w==; From: Peter Kjellerstedt To: Subject: [langdale][PATCH 1/3] externalsrc: git submodule--helper list unsupported Date: Sat, 5 Nov 2022 16:43:01 +0100 Message-ID: <20221105154303.274085-1-pkj@axis.com> X-Mailer: git-send-email 2.38.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 05 Nov 2022 15:43:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/172749 From: John Edward Broadbent Git has removed support for "git submodule--helper list". https://github.com/git/git/commit/31955475d1c283120d5d84247eb3fd55d9f5fdd9 This change provides an alternate method for gathering the submodules information. Tested: Build recipes with and without submodules Signed-off-by: Carson Labrado Signed-off-by: John Edward Broadbent Signed-off-by: Richard Purdie Signed-off-by: Peter Kjellerstedt --- meta/classes/externalsrc.bbclass | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass index 8136d25cb1..766c4979c5 100644 --- a/meta/classes/externalsrc.bbclass +++ b/meta/classes/externalsrc.bbclass @@ -229,15 +229,16 @@ def srctree_hash_files(d, srcdir=None): env['GIT_INDEX_FILE'] = tmp_index.name subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env) git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8") - submodule_helper = subprocess.check_output(['git', 'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8") - for line in submodule_helper.splitlines(): - module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1]) - if os.path.isdir(module_dir): - proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - proc.communicate() - proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL) - stdout, _ = proc.communicate() - git_sha1 += stdout.decode("utf-8") + if os.path.exists(".gitmodules"): + submodule_helper = subprocess.check_output(["git", "config", "--file", ".gitmodules", "--get-regexp", "path"], cwd=s_dir, env=env).decode("utf-8") + for line in submodule_helper.splitlines(): + module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1]) + if os.path.isdir(module_dir): + proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + proc.communicate() + proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL) + stdout, _ = proc.communicate() + git_sha1 += stdout.decode("utf-8") sha1 = hashlib.sha1(git_sha1.encode("utf-8")).hexdigest() with open(oe_hash_file, 'w') as fobj: fobj.write(sha1) From patchwork Sat Nov 5 15:43:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Kjellerstedt X-Patchwork-Id: 14953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D162CC433FE for ; Sat, 5 Nov 2022 15:43:14 +0000 (UTC) Received: from smtp1.axis.com (smtp1.axis.com [195.60.68.17]) by mx.groups.io with SMTP id smtpd.web11.7208.1667662988552932470 for ; Sat, 05 Nov 2022 08:43:10 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@axis.com header.s=axis-central1 header.b=FtQ1Vg0+; spf=pass (domain: axis.com, ip: 195.60.68.17, mailfrom: peter.kjellerstedt@axis.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axis.com; q=dns/txt; s=axis-central1; t=1667662990; x=1699198990; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=jJ0iTEXgSfw4UfShldmMw1xUNHJn1pATmQ/HyMK43VI=; b=FtQ1Vg0+gu6ZPnPs1J5uTTzQ2qzUjbf2/1SU3apcEKC4r/WAZRU5rg9+ 16amWTZE3uSwvLhqqB2s14gO5NOy95JOv4xQtRefqUpeQev9y8tIQVwlM 21D6QIUAk9Z3NMwwLY9kO2TaJog0/1OCUkzTUN6HrnjjqfAOj5J4+qvT9 CD4XnnkX/v3muaH1Rs4MNX7crWSsL79Kv3puc8NrG4wReD7djaq+H5G6y b8Tp81M/HAuG7n6j9GuhuRuVInqvOXZutYksL+V/cJMdNZoqzwe12HvG/ YCc6FC0EkOwv2QGoqKDEoTlUc7nPgQSdfXQ3R5LEBCZjkbMz7kBroaWYR w==; From: Peter Kjellerstedt To: Subject: [langdale][PATCH 2/3] externalsrc.bbclass: fix git repo detection Date: Sat, 5 Nov 2022 16:43:02 +0100 Message-ID: <20221105154303.274085-2-pkj@axis.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221105154303.274085-1-pkj@axis.com> References: <20221105154303.274085-1-pkj@axis.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 05 Nov 2022 15:43:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/172751 From: Martin Jansa * fix issue introduced in: https://git.openembedded.org/openembedded-core/commit/?id=95fbac8dcad6c93f4c9737e9fe13e92ab6befa09 * it added check for s_dir + git-dir (typically '.git') isn't the same as ${TOPDIR} + git-dir, but due to copy-paste issue it was just comparing it with s_dir + git-dir again, resulting in most external repos (where git-dir is '.git') to be processed as regular directory (not taking advantage of git write-tree). * normally this wouldn't be an issue, but for big repo with a lot of files this added a lot of checksums in: d.setVarFlag('do_compile', 'file-checksums', '${@srctree_hash_files(d)}') and I mean *a lot, e.g. in chromium build it was 380227 paths which still wouldn't that bad, but the checksum processing in siggen.py isn't trivial and just looping through all these checksums takes very long time (over 1000sec on fast NVME drive with warm cache) and then https://git.openembedded.org/bitbake/commit/?id=b4975d2ecf615ac4c240808fbc5a3f879a93846b made the processing a bit more complicated and the loop in get_taskhash() function took 6448sec and to make things worse there was no output from bitbake during that time, so even with -DDD it looks like this: DEBUG: virtual/libgles2 resolved to: mesa (langdale/oe-core/meta/recipes-graphics/mesa/mesa_22.2.0.bb) Bitbake still alive (no events for 600s). Active tasks: Bitbake still alive (no events for 1200s). Active tasks: Bitbake still alive (no events for 1800s). Active tasks: Bitbake still alive (no events for 2400s). Active tasks: Bitbake still alive (no events for 3000s). Active tasks: Bitbake still alive (no events for 3600s). Active tasks: Bitbake still alive (no events for 4200s). Active tasks: Bitbake still alive (no events for 4800s). Active tasks: Bitbake still alive (no events for 5400s). Active tasks: Bitbake still alive (no events for 6000s). Active tasks: DEBUG: Starting bitbake-worker without -DDD it will get stuck for almost 2 hours in: "Initialising tasks..." before it finally writes sstate summary like: "Sstate summary: Wanted 3102 Local 0 Mirrors 0 Missed 3102 Current 1483 (0% match, 32% complete)" * fix the copy&paste typo to use git work-tree in most cases, but be aware that this issue still exists for huge local source trees not in git [YOCTO #14942] Signed-off-by: Martin Jansa Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie Signed-off-by: Peter Kjellerstedt --- meta/classes/externalsrc.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass index 766c4979c5..686553f900 100644 --- a/meta/classes/externalsrc.bbclass +++ b/meta/classes/externalsrc.bbclass @@ -211,8 +211,8 @@ def srctree_hash_files(d, srcdir=None): try: git_dir = os.path.join(s_dir, subprocess.check_output(['git', '-C', s_dir, 'rev-parse', '--git-dir'], stderr=subprocess.DEVNULL).decode("utf-8").rstrip()) - top_git_dir = os.path.join(s_dir, subprocess.check_output(['git', '-C', d.getVar("TOPDIR"), 'rev-parse', '--git-dir'], - stderr=subprocess.DEVNULL).decode("utf-8").rstrip()) + top_git_dir = os.path.join(d.getVar("TOPDIR"), + subprocess.check_output(['git', '-C', d.getVar("TOPDIR"), 'rev-parse', '--git-dir'], stderr=subprocess.DEVNULL).decode("utf-8").rstrip()) if git_dir == top_git_dir: git_dir = None except subprocess.CalledProcessError: From patchwork Sat Nov 5 15:43:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Kjellerstedt X-Patchwork-Id: 14954 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D189CC43219 for ; Sat, 5 Nov 2022 15:43:14 +0000 (UTC) Received: from smtp2.axis.com (smtp2.axis.com [195.60.68.18]) by mx.groups.io with SMTP id smtpd.web12.7243.1667662989383294807 for ; Sat, 05 Nov 2022 08:43:10 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@axis.com header.s=axis-central1 header.b=BFTfl/NQ; spf=pass (domain: axis.com, ip: 195.60.68.18, mailfrom: peter.kjellerstedt@axis.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axis.com; q=dns/txt; s=axis-central1; t=1667662990; x=1699198990; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=U8Uwx7iDblKbv/DWiAq2KfIU/1OI9vUkBuEaFB0jWAM=; b=BFTfl/NQqm8o00GBP4TMPYKzbk6gvMQIK/1rw4OksTKF8sk7/v89vqTi /7QgLWXsoYL9cP9A7i1UTxTJZZra1lkD7K4E4GKfi/W0J5UHDPH1aBKUC mrg47jQSAT6Rm2mlgbk5DFfm98FSnWyqB6QuXOFW42XmZ9WOGMlM4PYvZ PbWC39yUO2zrqWrqoxWeftZDLNoy9h4oxWENWfXymamobzGC36pu4Go2/ 2WaDW2He2aduZgHN3LznGjoStbnT3ouV8mwB7BTeDGpjgwVdfOke8llz3 8inWz60ZrxSv8YDq7BeNqb873pBQNW9RUssmvIu07IhTJojC3MXDPiaPJ Q==; From: Peter Kjellerstedt To: Subject: [langdale][PATCH 3/3] externalsrc.bbclass: Remove a trailing slash from ${B} Date: Sat, 5 Nov 2022 16:43:03 +0100 Message-ID: <20221105154303.274085-3-pkj@axis.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221105154303.274085-1-pkj@axis.com> References: <20221105154303.274085-1-pkj@axis.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 05 Nov 2022 15:43:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/172750 From: Peter Kjellerstedt The trailing slash in ${B} caused -fdebug-prefix-map=${B}=... to not match as intended, resulting in ${TMPDIR} ending up in files in ${PN}-dbg when externalsrc was in use, which in turn triggered buildpath QA warnings. Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie Signed-off-by: Peter Kjellerstedt --- meta/classes/externalsrc.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass index 686553f900..a6a8ca6318 100644 --- a/meta/classes/externalsrc.bbclass +++ b/meta/classes/externalsrc.bbclass @@ -60,7 +60,7 @@ python () { if externalsrcbuild: d.setVar('B', externalsrcbuild) else: - d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') + d.setVar('B', '${WORKDIR}/${BPN}-${PV}') local_srcuri = [] fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d)