From patchwork Fri Jul 26 16:22:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Hatle X-Patchwork-Id: 46883 X-Patchwork-Delegate: steve@sakoman.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 275BEC3DA49 for ; Fri, 26 Jul 2024 16:22:50 +0000 (UTC) Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) by mx.groups.io with SMTP id smtpd.web11.65421.1722010960429916309 for ; Fri, 26 Jul 2024 09:22:40 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: kernel.crashing.org, ip: 63.228.1.57, mailfrom: mark.hatle@kernel.crashing.org) Received: from kernel.crashing.org.net (70-99-78-136.nuveramail.net [70.99.78.136] (may be forged)) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id 46QGMcnk026618 for ; Fri, 26 Jul 2024 11:22:39 -0500 From: Mark Hatle To: openembedded-core@lists.openembedded.org Subject: [scarthgap][PATCH 1/8] package.py: Fix static debuginfo split Date: Fri, 26 Jul 2024 11:22:31 -0500 Message-Id: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> X-Mailer: git-send-email 1.8.3.1 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Jul 2024 16:22:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202550 From: Mark Hatle Fix: NameError: name 'shutil' is not defined Signed-off-by: Mark Hatle Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 13bdd750ae54d57a5f459e4b7d8636c864978241) Signed-off-by: Mark Hatle --- meta/lib/oe/package.py | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index 1511ba4..ffe5a21 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -14,6 +14,7 @@ import glob import stat import mmap import subprocess +import shutil import oe.cachedpath From patchwork Fri Jul 26 16:22:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Hatle X-Patchwork-Id: 46885 X-Patchwork-Delegate: steve@sakoman.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36DF0C52CDA for ; Fri, 26 Jul 2024 16:22:50 +0000 (UTC) Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) by mx.groups.io with SMTP id smtpd.web10.64840.1722010960609210796 for ; Fri, 26 Jul 2024 09:22:40 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: kernel.crashing.org, ip: 63.228.1.57, mailfrom: mark.hatle@kernel.crashing.org) Received: from kernel.crashing.org.net (70-99-78-136.nuveramail.net [70.99.78.136] (may be forged)) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id 46QGMcnl026618 for ; Fri, 26 Jul 2024 11:22:39 -0500 From: Mark Hatle To: openembedded-core@lists.openembedded.org Subject: [scarthgap][PATCH 2/8] package.py: Fix static library processing Date: Fri, 26 Jul 2024 11:22:32 -0500 Message-Id: <1722010958-23635-2-git-send-email-mark.hatle@kernel.crashing.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> References: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Jul 2024 16:22:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202551 From: Mark Hatle When PACKAGE_STRIP_STATIC is enabled the system did not pay attention to hardlinks. This could trigger a race condition during stripping of static libraries where multiple strips (through hardlinks) could run at the same time triggering a truncated or modified file error. The hardlink breaking code is based on the existing code for elf files, but due to the nature of the symlinks needed to be done in a separate block of code. Add support for static-library debugfs hardlinking through the existing inode processing code. Print a note to the logs if the link target can't be found. This isn't strictly an error, but may be useful for debugging an issue where a file isn't present. Signed-off-by: Mark Hatle Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit ff371d69f60a1529ed456acb7d8e9305242e74bd) Signed-off-by: Mark Hatle --- meta/lib/oe/package.py | 56 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 47 insertions(+), 9 deletions(-) diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index ffe5a21..af0923a 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -1065,6 +1065,7 @@ def process_split_and_strip_files(d): d.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT') != '1'): checkelf = {} checkelflinks = {} + checkstatic = {} for root, dirs, files in cpath.walk(dvar): for f in files: file = os.path.join(root, f) @@ -1078,10 +1079,6 @@ def process_split_and_strip_files(d): if file in skipfiles: continue - if oe.package.is_static_lib(file): - staticlibs.append(file) - continue - try: ltarget = cpath.realpath(file, dvar, False) s = cpath.lstat(ltarget) @@ -1093,6 +1090,13 @@ def process_split_and_strip_files(d): continue if not s: continue + + if oe.package.is_static_lib(file): + # Use a reference of device ID and inode number to identify files + file_reference = "%d_%d" % (s.st_dev, s.st_ino) + checkstatic[file] = (file, file_reference) + continue + # Check its an executable if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) \ or (s[stat.ST_MODE] & stat.S_IXOTH) \ @@ -1157,6 +1161,27 @@ def process_split_and_strip_files(d): # Modified the file so clear the cache cpath.updatecache(file) + # Do the same hardlink processing as above, but for static libraries + results = list(checkstatic.keys()) + + # As above, sort the results. + results.sort(key=lambda x: x[0]) + + for file in results: + # Use a reference of device ID and inode number to identify files + file_reference = checkstatic[file][1] + if file_reference in inodes: + os.unlink(file) + os.link(inodes[file_reference][0], file) + inodes[file_reference].append(file) + else: + inodes[file_reference] = [file] + # break hardlink + bb.utils.break_hardlinks(file) + staticlibs.append(file) + # Modified the file so clear the cache + cpath.updatecache(file) + def strip_pkgd_prefix(f): nonlocal dvar @@ -1195,11 +1220,24 @@ def process_split_and_strip_files(d): dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(target) + dv["append"] fpath = dvar + dest ftarget = dvar + dv["libdir"] + os.path.dirname(target) + dv["dir"] + "/" + os.path.basename(target) + dv["append"] - bb.utils.mkdirhier(os.path.dirname(fpath)) - # Only one hardlink of separated debug info file in each directory - if not os.access(fpath, os.R_OK): - #bb.note("Link %s -> %s" % (fpath, ftarget)) - os.link(ftarget, fpath) + if os.access(ftarget, os.R_OK): + bb.utils.mkdirhier(os.path.dirname(fpath)) + # Only one hardlink of separated debug info file in each directory + if not os.access(fpath, os.R_OK): + #bb.note("Link %s -> %s" % (fpath, ftarget)) + os.link(ftarget, fpath) + elif (d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'): + deststatic = dv["staticlibdir"] + os.path.dirname(src) + dv["staticdir"] + "/" + os.path.basename(file) + dv["staticappend"] + fpath = dvar + deststatic + ftarget = dvar + dv["staticlibdir"] + os.path.dirname(target) + dv["staticdir"] + "/" + os.path.basename(target) + dv["staticappend"] + if os.access(ftarget, os.R_OK): + bb.utils.mkdirhier(os.path.dirname(fpath)) + # Only one hardlink of separated debug info file in each directory + if not os.access(fpath, os.R_OK): + #bb.note("Link %s -> %s" % (fpath, ftarget)) + os.link(ftarget, fpath) + else: + bb.note("Unable to find inode link target %s" % (target)) # Create symlinks for all cases we were able to split symbols for file in symlinks: From patchwork Fri Jul 26 16:22:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Hatle X-Patchwork-Id: 46890 X-Patchwork-Delegate: steve@sakoman.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F7CAC52D1F for ; Fri, 26 Jul 2024 16:22:50 +0000 (UTC) Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) by mx.groups.io with SMTP id smtpd.web11.65422.1722010960647780957 for ; Fri, 26 Jul 2024 09:22:40 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: kernel.crashing.org, ip: 63.228.1.57, mailfrom: mark.hatle@kernel.crashing.org) Received: from kernel.crashing.org.net (70-99-78-136.nuveramail.net [70.99.78.136] (may be forged)) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id 46QGMcnm026618 for ; Fri, 26 Jul 2024 11:22:40 -0500 From: Mark Hatle To: openembedded-core@lists.openembedded.org Subject: [scarthgap][PATCH 3/8] selftest-hardlink: Add additional test cases Date: Fri, 26 Jul 2024 11:22:33 -0500 Message-Id: <1722010958-23635-3-git-send-email-mark.hatle@kernel.crashing.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> References: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Jul 2024 16:22:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202552 From: Mark Hatle Additional test cases for debug symlink generation both binaries and static libraries. This also has the side effect of testing for race conditions in the hardlink debug generation and stripping. Signed-off-by: Mark Hatle Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 7171f41c07a39a7543bb64f075d38b8e74563089) Signed-off-by: Mark Hatle --- .../selftest-hardlink/selftest-hardlink.bb | 13 +++++++++++ meta/lib/oeqa/selftest/cases/package.py | 26 ++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb b/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb index be346b8..052bf0c 100644 --- a/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb +++ b/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb @@ -10,6 +10,9 @@ S = "${WORKDIR}" do_compile () { ${CC} hello.c -o hello1 ${CFLAGS} ${LDFLAGS} + + ${CC} hello.c -c -o hello.o ${CFLAGS} + ${AR} rcs libhello.a hello.o } do_install () { @@ -22,9 +25,19 @@ do_install () { ln ${D}${bindir}/hello1 ${D}${libexecdir}/hello3 ln ${D}${bindir}/hello1 ${D}${libexecdir}/hello4 + # We need so many hardlink copies to look for specific race conditions + install -d ${D}${libdir} + install -m 0644 libhello.a ${D}${libdir} + for num in `seq 1 100` ; do + ln ${D}${libdir}/libhello.a ${D}${libdir}/libhello-${num}.a + done + dd if=/dev/zero of=${D}${bindir}/sparsetest bs=1 count=0 seek=1M } RDEPENDS:${PN}-gdb += "gdb" PACKAGES =+ "${PN}-gdb" FILES:${PN}-gdb = "${bindir}/gdb.sh" + +PACKAGE_STRIP_STATIC = "1" +PACKAGE_DEBUG_STATIC_SPLIT = "1" diff --git a/meta/lib/oeqa/selftest/cases/package.py b/meta/lib/oeqa/selftest/cases/package.py index 1aa6c03..38ed717 100644 --- a/meta/lib/oeqa/selftest/cases/package.py +++ b/meta/lib/oeqa/selftest/cases/package.py @@ -103,11 +103,37 @@ class PackageTests(OESelftestTestCase): dest = get_bb_var('PKGDEST', 'selftest-hardlink') bindir = get_bb_var('bindir', 'selftest-hardlink') + libdir = get_bb_var('libdir', 'selftest-hardlink') + libexecdir = get_bb_var('libexecdir', 'selftest-hardlink') def checkfiles(): # Recipe creates 4 hardlinked files, there is a copy in package/ and a copy in packages-split/ # so expect 8 in total. self.assertEqual(os.stat(dest + "/selftest-hardlink" + bindir + "/hello1").st_nlink, 8) + self.assertEqual(os.stat(dest + "/selftest-hardlink" + libexecdir + "/hello3").st_nlink, 8) + + # Check dbg version + # 2 items, a copy in both package/packages-split so 4 + self.assertEqual(os.stat(dest + "/selftest-hardlink-dbg" + bindir + "/.debug/hello1").st_nlink, 4) + self.assertEqual(os.stat(dest + "/selftest-hardlink-dbg" + libexecdir + "/.debug/hello1").st_nlink, 4) + + # Even though the libexecdir name is 'hello3' or 'hello4', that isn't the debug target name + self.assertEqual(os.path.exists(dest + "/selftest-hardlink-dbg" + libexecdir + "/.debug/hello3"), False) + self.assertEqual(os.path.exists(dest + "/selftest-hardlink-dbg" + libexecdir + "/.debug/hello4"), False) + + # Check the staticdev libraries + # 101 items, a copy in both package/packages-split so 202 + self.assertEqual(os.stat(dest + "/selftest-hardlink-staticdev" + libdir + "/libhello.a").st_nlink, 202) + self.assertEqual(os.stat(dest + "/selftest-hardlink-staticdev" + libdir + "/libhello-25.a").st_nlink, 202) + self.assertEqual(os.stat(dest + "/selftest-hardlink-staticdev" + libdir + "/libhello-50.a").st_nlink, 202) + self.assertEqual(os.stat(dest + "/selftest-hardlink-staticdev" + libdir + "/libhello-75.a").st_nlink, 202) + + # Check static dbg + # 101 items, a copy in both package/packages-split so 202 + self.assertEqual(os.stat(dest + "/selftest-hardlink-dbg" + libdir + "/.debug-static/libhello.a").st_nlink, 202) + self.assertEqual(os.stat(dest + "/selftest-hardlink-dbg" + libdir + "/.debug-static/libhello-25.a").st_nlink, 202) + self.assertEqual(os.stat(dest + "/selftest-hardlink-dbg" + libdir + "/.debug-static/libhello-50.a").st_nlink, 202) + self.assertEqual(os.stat(dest + "/selftest-hardlink-dbg" + libdir + "/.debug-static/libhello-75.a").st_nlink, 202) # Test a sparse file remains sparse sparsestat = os.stat(dest + "/selftest-hardlink" + bindir + "/sparsetest") From patchwork Fri Jul 26 16:22:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Hatle X-Patchwork-Id: 46888 X-Patchwork-Delegate: steve@sakoman.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD12BC52D1E for ; Fri, 26 Jul 2024 16:22:50 +0000 (UTC) Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) by mx.groups.io with SMTP id smtpd.web11.65423.1722010960775681089 for ; Fri, 26 Jul 2024 09:22:41 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: kernel.crashing.org, ip: 63.228.1.57, mailfrom: mark.hatle@kernel.crashing.org) Received: from kernel.crashing.org.net (70-99-78-136.nuveramail.net [70.99.78.136] (may be forged)) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id 46QGMcnn026618 for ; Fri, 26 Jul 2024 11:22:40 -0500 From: Mark Hatle To: openembedded-core@lists.openembedded.org Subject: [scarthgap][PATCH 4/8] create-spdx-*: Support multilibs via SPDX_MULTILIB_SSTATE_ARCHS Date: Fri, 26 Jul 2024 11:22:34 -0500 Message-Id: <1722010958-23635-4-git-send-email-mark.hatle@kernel.crashing.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> References: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Jul 2024 16:22:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202553 From: Mark Hatle When a create-spdx-* classes is processing documents, it needs to find the document in a path that is related to the SSTATE_ARCH when a packge is generated. The SSTATE_ARCH can be affected by multilib configurations, resulting is something like armv8a-mlib. When the image (or SDK) is being generated and the components are collected, the system has no knowledge of the multilib arch and will fail to find it, such as: ERROR: meta-toolchain-1.0-r0 do_populate_sdk: No SPDX file found for package libilp32-libgcc-dbg, False sstate:libilp32-libgcc:armv8a-ilp32-mllibilp32-elf:14.1.0:r0:armv8a-ilp32:12: sstate:libilp32-libgcc::14.1.0:r0::12: Adding in the new SPDX_MULTILIB_SSTATE_ARCHS will provide a full set of SSTATE_ARCHS including ones that contain the multilib extension which will allow create-spdx-* to correctly find the document it is looking for. This would also be valuable to any other function doing a similar search through SSTATE_ARCH that may have been extended with multilib configurations. Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit f1499c36c1054fc90f7b7268cc95285f2eca72f7) spdx-3.0 items are not application and were removed. spdx-common.bbclass item was moved into create-sdpx-2.2.bbclass. Signed-off-by: Mark Hatle --- meta/classes-recipe/populate_sdk_base.bbclass | 4 ++++ meta/classes/create-spdx-2.2.bbclass | 14 ++++++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/meta/classes-recipe/populate_sdk_base.bbclass b/meta/classes-recipe/populate_sdk_base.bbclass index 81896d8..6cb43ad 100644 --- a/meta/classes-recipe/populate_sdk_base.bbclass +++ b/meta/classes-recipe/populate_sdk_base.bbclass @@ -6,6 +6,10 @@ PACKAGES = "" +# This exists as an optimization for SPDX processing to only run in image and +# SDK processing context. This class happens to be common to these usages. +SPDX_MULTILIB_SSTATE_ARCHS = "${@all_multilib_tune_values(d, 'SSTATE_ARCHS')}" + inherit image-postinst-intercepts image-artifact-names # Wildcards specifying complementary packages to install for every package that has been explicitly diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 4ea91f6..d104668 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -35,6 +35,8 @@ SPDX_LICENSES ??= "${COREBASE}/meta/files/spdx-licenses.json" SPDX_CUSTOM_ANNOTATION_VARS ??= "" +SPDX_MULTILIB_SSTATE_ARCHS ??= "${SSTATE_ARCHS}" + SPDX_ORG ??= "OpenEmbedded ()" SPDX_SUPPLIER ??= "Organization: ${SPDX_ORG}" SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field for SPDX packages created from \ @@ -349,7 +351,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) spdx_deps_file = Path(d.getVar("SPDXDEPS")) - package_archs = d.getVar("SSTATE_ARCHS").split() + package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() dep_recipes = [] @@ -389,7 +391,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): return dep_recipes -collect_dep_recipes[vardepsexclude] = "SSTATE_ARCHS" +collect_dep_recipes[vardepsexclude] = "SPDX_MULTILIB_SSTATE_ARCHS" def collect_dep_sources(d, dep_recipes): import oe.sbom @@ -763,7 +765,7 @@ python do_create_runtime_spdx() { providers = collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - package_archs = d.getVar("SSTATE_ARCHS").split() + package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() if not is_native: @@ -869,7 +871,7 @@ python do_create_runtime_spdx() { oe.sbom.write_doc(d, runtime_doc, pkg_arch, "runtime", spdx_deploy, indent=get_json_indent(d)) } -do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS" +do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SPDX_MULTILIB_SSTATE_ARCHS" addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work SSTATETASKS += "do_create_runtime_spdx" @@ -1004,7 +1006,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx import bb.compress.zstd providers = collect_package_providers(d) - package_archs = d.getVar("SSTATE_ARCHS").split() + package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") @@ -1155,4 +1157,4 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx tar.addfile(info, fileobj=index_str) -combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SSTATE_ARCHS" +combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SPDX_MULTILIB_SSTATE_ARCHS" From patchwork Fri Jul 26 16:22:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Hatle X-Patchwork-Id: 46887 X-Patchwork-Delegate: steve@sakoman.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E812C52D1D for ; Fri, 26 Jul 2024 16:22:50 +0000 (UTC) Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) by mx.groups.io with SMTP id smtpd.web10.64841.1722010961012389009 for ; Fri, 26 Jul 2024 09:22:41 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: kernel.crashing.org, ip: 63.228.1.57, mailfrom: mark.hatle@kernel.crashing.org) Received: from kernel.crashing.org.net (70-99-78-136.nuveramail.net [70.99.78.136] (may be forged)) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id 46QGMcno026618 for ; Fri, 26 Jul 2024 11:22:40 -0500 From: Mark Hatle To: openembedded-core@lists.openembedded.org Subject: [scarthgap][PATCH 5/8] create-spdx-3.0/populate_sdk_base: Add SDK_CLASSES inherit mechanism to fix tarball SPDX manifests Date: Fri, 26 Jul 2024 11:22:35 -0500 Message-Id: <1722010958-23635-5-git-send-email-mark.hatle@kernel.crashing.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> References: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Jul 2024 16:22:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202554 From: Richard Purdie Currently, "tarball" sdk based recipes don't generate SPDX manifests as they don't include the rootfs generation classes. Split the SPDX 3.0 image class into two so the SDK components can be included where needed. To do this, introduce an SDK_CLASSES variable similar to IMAGE_CLASSES which the SDK code can use. Migrate testsdk usage to this. Also move the image/sdk spdx classes to classes-recipe rather than the general classes directory since they'd never be included on a global level. For buildtools-tarball, it has its own testsdk functions so disable the class there as a deferred inherit would overwrite it. Signed-off-by: Richard Purdie (cherry picked from commit 662396533177b72cc1d83e95841b27f7e42dcb20) Eliminate spdx-3.0 items, not applicable to Scarthgap. Signed-off-by: Mark Hatle --- meta/classes-recipe/populate_sdk_base.bbclass | 3 +++ meta/classes-recipe/testimage.bbclass | 2 -- meta/recipes-core/meta/buildtools-tarball.bb | 3 +++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/meta/classes-recipe/populate_sdk_base.bbclass b/meta/classes-recipe/populate_sdk_base.bbclass index 6cb43ad..a103e7b 100644 --- a/meta/classes-recipe/populate_sdk_base.bbclass +++ b/meta/classes-recipe/populate_sdk_base.bbclass @@ -4,6 +4,9 @@ # SPDX-License-Identifier: MIT # +SDK_CLASSES += "${@bb.utils.contains("IMAGE_CLASSES", "testimage", "testsdk", "", d)}" +inherit_defer ${SDK_CLASSES} + PACKAGES = "" # This exists as an optimization for SPDX processing to only run in image and diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass index ed0d87b..2f68f83 100644 --- a/meta/classes-recipe/testimage.bbclass +++ b/meta/classes-recipe/testimage.bbclass @@ -483,5 +483,3 @@ python () { if oe.types.boolean(d.getVar("TESTIMAGE_AUTO") or "False"): bb.build.addtask("testimage", "do_build", "do_image_complete", d) } - -inherit testsdk diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb index 92fbda3..e2ce5b3 100644 --- a/meta/recipes-core/meta/buildtools-tarball.bb +++ b/meta/recipes-core/meta/buildtools-tarball.bb @@ -112,6 +112,9 @@ INHIBIT_DEFAULT_DEPS = "1" # Directory in testsdk that contains testcases TESTSDK_CASES = "buildtools-cases" +# We have our own code, avoid deferred inherit +SDK_CLASSES:remove = "testsdk" + python do_testsdk() { import oeqa.sdk.testsdk testsdk = oeqa.sdk.testsdk.TestSDK() From patchwork Fri Jul 26 16:22:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Hatle X-Patchwork-Id: 46889 X-Patchwork-Delegate: steve@sakoman.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44841C52CDC for ; Fri, 26 Jul 2024 16:22:50 +0000 (UTC) Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) by mx.groups.io with SMTP id smtpd.web11.65425.1722010961262244021 for ; Fri, 26 Jul 2024 09:22:41 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: kernel.crashing.org, ip: 63.228.1.57, mailfrom: mark.hatle@kernel.crashing.org) Received: from kernel.crashing.org.net (70-99-78-136.nuveramail.net [70.99.78.136] (may be forged)) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id 46QGMcnp026618 for ; Fri, 26 Jul 2024 11:22:40 -0500 From: Mark Hatle To: openembedded-core@lists.openembedded.org Subject: [scarthgap][PATCH 6/8] oeqa sdk cases: Skip SDK test cases when TCLIBC is newlib Date: Fri, 26 Jul 2024 11:22:36 -0500 Message-Id: <1722010958-23635-6-git-send-email-mark.hatle@kernel.crashing.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> References: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Jul 2024 16:22:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202555 From: Mark Hatle Newlib generally requires additional components to function. Skip the cases where newlib is known to not work. Signed-off-by: Mark Hatle Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit b9934755554e40d9980b90c3d541f4c702203561) Signed-off-by: Mark Hatle --- meta/lib/oeqa/sdk/cases/assimp.py | 4 ++++ meta/lib/oeqa/sdk/cases/buildcpio.py | 5 +++++ meta/lib/oeqa/sdk/cases/buildepoxy.py | 4 ++++ meta/lib/oeqa/sdk/cases/buildgalculator.py | 4 ++++ meta/lib/oeqa/sdk/cases/buildlzip.py | 5 +++++ meta/lib/oeqa/sdk/cases/gcc.py | 4 ++++ 6 files changed, 26 insertions(+) diff --git a/meta/lib/oeqa/sdk/cases/assimp.py b/meta/lib/oeqa/sdk/cases/assimp.py index d990b1e..4cc30f2 100644 --- a/meta/lib/oeqa/sdk/cases/assimp.py +++ b/meta/lib/oeqa/sdk/cases/assimp.py @@ -19,6 +19,10 @@ class BuildAssimp(OESDKTestCase): """ def setUp(self): + libc = self.td.get("TCLIBC") + if libc in [ 'newlib' ]: + raise unittest.SkipTest("CMakeTest class: SDK doesn't contain a supported C library") + if not (self.tc.hasHostPackage("nativesdk-cmake") or self.tc.hasHostPackage("cmake-native")): raise unittest.SkipTest("Needs cmake") diff --git a/meta/lib/oeqa/sdk/cases/buildcpio.py b/meta/lib/oeqa/sdk/cases/buildcpio.py index 51003b1..ab8fc41 100644 --- a/meta/lib/oeqa/sdk/cases/buildcpio.py +++ b/meta/lib/oeqa/sdk/cases/buildcpio.py @@ -17,6 +17,11 @@ class BuildCpioTest(OESDKTestCase): """ Check that autotools will cross-compile correctly. """ + def setUp(self): + libc = self.td.get("TCLIBC") + if libc in [ 'newlib' ]: + raise unittest.SkipTest("AutotoolsTest class: SDK doesn't contain a supported C library") + def test_cpio(self): with tempfile.TemporaryDirectory(prefix="cpio-", dir=self.tc.sdk_dir) as testdir: tarball = self.fetch(testdir, self.td["DL_DIR"], "https://ftp.gnu.org/gnu/cpio/cpio-2.15.tar.gz") diff --git a/meta/lib/oeqa/sdk/cases/buildepoxy.py b/meta/lib/oeqa/sdk/cases/buildepoxy.py index 147ee3e..5b9c36f 100644 --- a/meta/lib/oeqa/sdk/cases/buildepoxy.py +++ b/meta/lib/oeqa/sdk/cases/buildepoxy.py @@ -18,6 +18,10 @@ class EpoxyTest(OESDKTestCase): Test that Meson builds correctly. """ def setUp(self): + libc = self.td.get("TCLIBC") + if libc in [ 'newlib' ]: + raise unittest.SkipTest("MesonTest class: SDK doesn't contain a supported C library") + if not (self.tc.hasHostPackage("nativesdk-meson") or self.tc.hasHostPackage("meson-native")): raise unittest.SkipTest("EpoxyTest class: SDK doesn't contain Meson") diff --git a/meta/lib/oeqa/sdk/cases/buildgalculator.py b/meta/lib/oeqa/sdk/cases/buildgalculator.py index 178f074..2818743 100644 --- a/meta/lib/oeqa/sdk/cases/buildgalculator.py +++ b/meta/lib/oeqa/sdk/cases/buildgalculator.py @@ -18,6 +18,10 @@ class GalculatorTest(OESDKTestCase): Test that autotools and GTK+ 3 compiles correctly. """ def setUp(self): + libc = self.td.get("TCLIBC") + if libc in [ 'newlib' ]: + raise unittest.SkipTest("GTK3Test class: SDK doesn't contain a supported C library") + if not (self.tc.hasTargetPackage("gtk+3", multilib=True) or \ self.tc.hasTargetPackage("libgtk-3.0", multilib=True)): raise unittest.SkipTest("GalculatorTest class: SDK don't support gtk+3") diff --git a/meta/lib/oeqa/sdk/cases/buildlzip.py b/meta/lib/oeqa/sdk/cases/buildlzip.py index b4b7d85..afedc25 100644 --- a/meta/lib/oeqa/sdk/cases/buildlzip.py +++ b/meta/lib/oeqa/sdk/cases/buildlzip.py @@ -13,6 +13,11 @@ class BuildLzipTest(OESDKTestCase): """ Test that "plain" compilation works, using just $CC $CFLAGS etc. """ + def setUp(self): + libc = self.td.get("TCLIBC") + if libc in [ 'newlib' ]: + raise unittest.SkipTest("MakefileTest class: SDK doesn't contain a supported C library") + def test_lzip(self): with tempfile.TemporaryDirectory(prefix="lzip", dir=self.tc.sdk_dir) as testdir: tarball = self.fetch(testdir, self.td["DL_DIR"], "http://downloads.yoctoproject.org/mirror/sources/lzip-1.19.tar.gz") diff --git a/meta/lib/oeqa/sdk/cases/gcc.py b/meta/lib/oeqa/sdk/cases/gcc.py index fc28b9c..e810d2c 100644 --- a/meta/lib/oeqa/sdk/cases/gcc.py +++ b/meta/lib/oeqa/sdk/cases/gcc.py @@ -26,6 +26,10 @@ class GccCompileTest(OESDKTestCase): os.path.join(self.tc.sdk_dir, f)) def setUp(self): + libc = self.td.get("TCLIBC") + if libc in [ 'newlib' ]: + raise unittest.SkipTest("GccCompileTest class: SDK doesn't contain a supported C library") + machine = self.td.get("MACHINE") if not (self.tc.hasHostPackage("packagegroup-cross-canadian-%s" % machine) or self.tc.hasHostPackage("^gcc-", regex=True)): From patchwork Fri Jul 26 16:22:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Hatle X-Patchwork-Id: 46886 X-Patchwork-Delegate: steve@sakoman.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35506C3DA4A for ; Fri, 26 Jul 2024 16:22:50 +0000 (UTC) Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) by mx.groups.io with SMTP id smtpd.web10.64842.1722010961472792667 for ; Fri, 26 Jul 2024 09:22:41 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: kernel.crashing.org, ip: 63.228.1.57, mailfrom: mark.hatle@kernel.crashing.org) Received: from kernel.crashing.org.net (70-99-78-136.nuveramail.net [70.99.78.136] (may be forged)) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id 46QGMcnq026618 for ; Fri, 26 Jul 2024 11:22:40 -0500 From: Mark Hatle To: openembedded-core@lists.openembedded.org Subject: [scarthgap][PATCH 7/8] pseudo: Fix to work with glibc 2.40 Date: Fri, 26 Jul 2024 11:22:37 -0500 Message-Id: <1722010958-23635-7-git-send-email-mark.hatle@kernel.crashing.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> References: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Jul 2024 16:22:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202556 From: Richard Purdie glibc 2.40 renames some internal header variables. Update our hack to work with the new version. These kinds of problems illustrate we need to address the issue properly. Signed-off-by: Richard Purdie (cherry picked from commit 35021d650de3eecc3f42000181b39a5db5a8eaa0) Signed-off-by: Mark Hatle --- meta/recipes-devtools/pseudo/files/glibc238.patch | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/meta/recipes-devtools/pseudo/files/glibc238.patch b/meta/recipes-devtools/pseudo/files/glibc238.patch index da4b8ca..dfb5c28 100644 --- a/meta/recipes-devtools/pseudo/files/glibc238.patch +++ b/meta/recipes-devtools/pseudo/files/glibc238.patch @@ -9,7 +9,7 @@ Index: git/pseudo_wrappers.c =================================================================== --- git.orig/pseudo_wrappers.c +++ git/pseudo_wrappers.c -@@ -6,6 +6,15 @@ +@@ -6,6 +6,18 @@ * SPDX-License-Identifier: LGPL-2.1-only * */ @@ -21,6 +21,9 @@ Index: git/pseudo_wrappers.c +#undef __GLIBC_USE_ISOC2X +#undef __GLIBC_USE_C2X_STRTOL +#define __GLIBC_USE_C2X_STRTOL 0 ++#undef __GLIBC_USE_ISOC23 ++#undef __GLIBC_USE_C23_STRTOL ++#define __GLIBC_USE_C23_STRTOL 0 + #include #include @@ -29,7 +32,7 @@ Index: git/pseudo_util.c =================================================================== --- git.orig/pseudo_util.c +++ git/pseudo_util.c -@@ -8,6 +8,14 @@ +@@ -8,6 +8,17 @@ */ /* we need access to RTLD_NEXT for a horrible workaround */ #define _GNU_SOURCE @@ -41,6 +44,9 @@ Index: git/pseudo_util.c +#undef __GLIBC_USE_ISOC2X +#undef __GLIBC_USE_C2X_STRTOL +#define __GLIBC_USE_C2X_STRTOL 0 ++#undef __GLIBC_USE_ISOC23 ++#undef __GLIBC_USE_C23_STRTOL ++#define __GLIBC_USE_C23_STRTOL 0 #include #include From patchwork Fri Jul 26 16:22:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Hatle X-Patchwork-Id: 46884 X-Patchwork-Delegate: steve@sakoman.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3909AC3DA7E for ; Fri, 26 Jul 2024 16:22:50 +0000 (UTC) Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) by mx.groups.io with SMTP id smtpd.web10.64843.1722010961654962818 for ; Fri, 26 Jul 2024 09:22:41 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: kernel.crashing.org, ip: 63.228.1.57, mailfrom: mark.hatle@kernel.crashing.org) Received: from kernel.crashing.org.net (70-99-78-136.nuveramail.net [70.99.78.136] (may be forged)) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id 46QGMcnr026618 for ; Fri, 26 Jul 2024 11:22:41 -0500 From: Mark Hatle To: openembedded-core@lists.openembedded.org Subject: [scarthgap][PATCH 8/8] pseudo: Update to include open symlink handling bugfix Date: Fri, 26 Jul 2024 11:22:38 -0500 Message-Id: <1722010958-23635-8-git-send-email-mark.hatle@kernel.crashing.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> References: <1722010958-23635-1-git-send-email-mark.hatle@kernel.crashing.org> List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 26 Jul 2024 16:22:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202557 From: Richard Purdie Update to a new revision which includes "Bugfix for Linux open(O_CREAT|O_EXCL)" Signed-off-by: Richard Purdie (cherry picked from commit 92a9710ec88c8729fa3d83baa2e63dd74d95cdf8) Signed-off-by: Mark Hatle --- meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb index 5f32b37..7d8f71f 100644 --- a/meta/recipes-devtools/pseudo/pseudo_git.bb +++ b/meta/recipes-devtools/pseudo/pseudo_git.bb @@ -14,7 +14,7 @@ SRC_URI:append:class-nativesdk = " \ file://older-glibc-symbols.patch" SRC_URI[prebuilt.sha256sum] = "ed9f456856e9d86359f169f46a70ad7be4190d6040282b84c8d97b99072485aa" -SRCREV = "e11ae91da7d0711f5e33ea9dfbf1875dde3c1734" +SRCREV = "374089f2ed83da4d0d4e58df067142ff99c7eb12" S = "${WORKDIR}/git" PV = "1.9.0+git"