From patchwork Fri Jun 2 13:34:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 744CCC7EE2C for ; Fri, 2 Jun 2023 13:34:58 +0000 (UTC) Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by mx.groups.io with SMTP id smtpd.web10.13227.1685712896792220177 for ; Fri, 02 Jun 2023 06:34:57 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@linuxfoundation.org header.s=google header.b=NllwaFsL; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.47, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-30789a4c537so1856802f8f.0 for ; Fri, 02 Jun 2023 06:34:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712895; x=1688304895; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ZElALzCmcRkMWLq2T39r4KxAFcQxJ6GIHmKrm9U6lEQ=; b=NllwaFsLU7Zb62H+FPZv3Hyz7tCoA+5MPXdZa7pLrT8ud1U5ClukDlE3Csksg4CBgt PLkO6+ALJgD/kxpQ+2IA3PtWJJswgScyQIP19MMX9OMZ1pNf7Pdauln0BjlzBVN5uGCW o+C1igCu2eSwQpHocKYg+xGbSpshd8ItYhoIg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712895; x=1688304895; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ZElALzCmcRkMWLq2T39r4KxAFcQxJ6GIHmKrm9U6lEQ=; b=ZRF42EZ4XTeftjWRAl4qOwh6EjKJttcsiZ7JnJwY7/aM2mctfuGQbPJns8xWrQMpR0 HZhfrDFx0FbAqg+dV2vnc8T4poQUNSkZsbC7W5/y6KAISvS3EfowOrmgK9nx5y7ixtmt L2rs9xo55puJx5lj+gucPBXdl9YyNkUbpjMG4QcmMcYgJtoSY76jStW+qaEbqlkRW+Gc c1Rn7MD8eFub+D+9F7XNz4tMl2ug77IcDbEOfka+RTnXzJa9p9m9CUqr37Tr1BpDNArd a3/gnzEsONLmiNG9vKrivQ8ph5z+lKxC9jyW1fX6MNBkM5/AStvryq0WgHwotzeTzvWA cgeg== X-Gm-Message-State: AC+VfDyV6nG2XSYV+vJpQJN+WG2zPRl5bW29igU7qPZVD39bQaKWg84M 45ICfmO8Cqm5M1IZxAMopuXbYnIxOkpL38vyilQ= X-Google-Smtp-Source: ACHHUZ4hT7UU6QLX7cRaHEo++YbflzBwmp3Pu6Syo7PvshA6XI3zXDvGBuYa4FXrOytjO6wbWuKEKw== X-Received: by 2002:a5d:61ce:0:b0:306:3b39:9a2a with SMTP id q14-20020a5d61ce000000b003063b399a2amr3899897wrv.42.1685712894909; Fri, 02 Jun 2023 06:34:54 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.34.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:34:54 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Cc: Joshua Watt Subject: [PATCH 01/11] classes/create-spdx-2.2: Use hashfn from BB_TASKDEPDATA instead of MACHINE Date: Fri, 2 Jun 2023 14:34:43 +0100 Message-Id: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:34:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182319 From: Joshua Watt Enabling the SPDX class and running two builds, one with SDKMACHINE=i686 and then again with SDKMACHINE=x86_64 would lead to errors since the output was stored per MACHINE and the overlapping files would cause manifest errors. The hashfn data from bitbake has SSTATE_PKGARCH encoded into it which is how sstate separates out it's targets and SDPX should be using the same structure. Therefore switch to using this. Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie --- meta/classes/create-spdx-2.2.bbclass | 61 +++++++++++++++++----------- meta/lib/oe/sbom.py | 22 ++++++++-- 2 files changed, 57 insertions(+), 26 deletions(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index e0f62a43a2d..6ec0c1465e7 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -4,7 +4,7 @@ # SPDX-License-Identifier: GPL-2.0-only # -DEPLOY_DIR_SPDX ??= "${DEPLOY_DIR}/spdx/${MACHINE}" +DEPLOY_DIR_SPDX ??= "${DEPLOY_DIR}/spdx" # The product name that the CVE database uses. Defaults to BPN, but may need to # be overriden per recipe (for example tiff.bb sets CVE_PRODUCT=libtiff). @@ -337,6 +337,20 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag package_doc.add_relationship(pkg_file, "GENERATED_FROM", ref_id, comment=debugsrc) +def collect_deps(d): + current_task = "do_" + d.getVar("BB_CURRENTTASK") + + taskdepdata = d.getVar("BB_TASKDEPDATA", False) + deps = sorted(set( + (dep[0], dep[7]) for dep in taskdepdata.values() if + dep[1] == current_task and dep[0] != d.getVar("PN") + )) + + return deps + +collect_deps[vardepsexclude] += "BB_TASKDEPDATA" +collect_deps[vardeps] += "DEPENDS" + def collect_dep_recipes(d, doc, spdx_recipe): from pathlib import Path import oe.sbom @@ -345,13 +359,9 @@ def collect_dep_recipes(d, doc, spdx_recipe): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) dep_recipes = [] - taskdepdata = d.getVar("BB_TASKDEPDATA", False) - deps = sorted(set( - dep[0] for dep in taskdepdata.values() if - dep[1] == "do_create_spdx" and dep[0] != d.getVar("PN") - )) - for dep_pn in deps: - dep_recipe_path = deploy_dir_spdx / "recipes" / ("recipe-%s.spdx.json" % dep_pn) + + for dep_pn, dep_hashfn in collect_deps(d): + dep_recipe_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, "recipe-" + dep_pn, dep_hashfn) spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_recipe_path) @@ -380,8 +390,6 @@ def collect_dep_recipes(d, doc, spdx_recipe): return dep_recipes -collect_dep_recipes[vardepsexclude] += "BB_TASKDEPDATA" -collect_dep_recipes[vardeps] += "DEPENDS" def collect_dep_sources(d, dep_recipes): import oe.sbom @@ -571,7 +579,7 @@ python do_create_spdx() { dep_recipes = collect_dep_recipes(d, doc, recipe) - doc_sha1 = oe.sbom.write_doc(d, doc, "recipes", indent=get_json_indent(d)) + doc_sha1 = oe.sbom.write_doc(d, doc, d.getVar("SSTATE_PKGARCH"), "recipes", indent=get_json_indent(d)) dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe)) recipe_ref = oe.spdx.SPDXExternalDocumentRef() @@ -636,7 +644,7 @@ python do_create_spdx() { add_package_sources_from_debug(d, package_doc, spdx_package, package, package_files, sources) - oe.sbom.write_doc(d, package_doc, "packages", indent=get_json_indent(d)) + oe.sbom.write_doc(d, package_doc, d.getVar("SSTATE_PKGARCH"), "packages", indent=get_json_indent(d)) } # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source addtask do_create_spdx after do_package do_packagedata do_unpack before do_populate_sdk do_build do_rm_work @@ -667,11 +675,11 @@ def collect_package_providers(d): taskdepdata = d.getVar("BB_TASKDEPDATA", False) deps = sorted(set( - dep[0] for dep in taskdepdata.values() if dep[0] != d.getVar("PN") + (dep[0], dep[7]) for dep in taskdepdata.values() if dep[0] != d.getVar("PN") )) - deps.append(d.getVar("PN")) + deps.append((d.getVar("PN"), d.getVar("BB_HASHFILENAME"))) - for dep_pn in deps: + for dep_pn, dep_hashfn in deps: recipe_data = oe.packagedata.read_pkgdata(dep_pn, d) for pkg in recipe_data.get("PACKAGES", "").split(): @@ -681,7 +689,7 @@ def collect_package_providers(d): rprovides.add(pkg) for r in rprovides: - providers[r] = pkg + providers[r] = (pkg, dep_hashfn) return providers @@ -717,7 +725,7 @@ python do_create_runtime_spdx() { if not oe.packagedata.packaged(package, localdata): continue - pkg_spdx_path = deploy_dir_spdx / "packages" / (pkg_name + ".spdx.json") + pkg_spdx_path = oe.sbom.doc_path(deploy_dir_spdx, pkg_name, d.getVar("SSTATE_PKGARCH"), "packages") package_doc, package_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path) @@ -761,7 +769,7 @@ python do_create_runtime_spdx() { if dep not in providers: continue - dep = providers[dep] + (dep, dep_hashfn) = providers[dep] if not oe.packagedata.packaged(dep, localdata): continue @@ -772,7 +780,7 @@ python do_create_runtime_spdx() { if dep in dep_package_cache: (dep_spdx_package, dep_package_ref) = dep_package_cache[dep] else: - dep_path = deploy_dir_spdx / "packages" / ("%s.spdx.json" % dep_pkg) + dep_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, dep_pkg, dep_hashfn) spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_path) @@ -800,7 +808,7 @@ python do_create_runtime_spdx() { ) seen_deps.add(dep) - oe.sbom.write_doc(d, runtime_doc, "runtime", spdx_deploy, indent=get_json_indent(d)) + oe.sbom.write_doc(d, runtime_doc, d.getVar("SSTATE_PKGARCH"), "runtime", spdx_deploy, indent=get_json_indent(d)) } addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work @@ -933,6 +941,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx import tarfile import bb.compress.zstd + providers = collect_package_providers(d) + creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) source_date_epoch = d.getVar("SOURCE_DATE_EPOCH") @@ -956,7 +966,12 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx doc.packages.append(image) for name in sorted(packages.keys()): - pkg_spdx_path = deploy_dir_spdx / "packages" / (name + ".spdx.json") + if name not in providers: + bb.fatal("Unable to find provider for '%s'" % name) + + pkg_name, pkg_hashfn = providers[name] + + pkg_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, pkg_name, pkg_hashfn) pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path) for p in pkg_doc.packages: @@ -973,7 +988,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx else: bb.fatal("Unable to find package with name '%s' in SPDX file %s" % (name, pkg_spdx_path)) - runtime_spdx_path = deploy_dir_spdx / "runtime" / ("runtime-" + name + ".spdx.json") + runtime_spdx_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, "runtime-" + name, pkg_hashfn) runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path) runtime_ref = oe.spdx.SPDXExternalDocumentRef() @@ -1045,7 +1060,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx }) for ref in doc.externalDocumentRefs: - ref_path = deploy_dir_spdx / "by-namespace" / ref.spdxDocument.replace("/", "_") + ref_path = oe.sbom.doc_path_by_namespace(deploy_dir_spdx, ref.spdxDocument) collect_spdx_document(ref_path) collect_spdx_document(image_spdx_path) diff --git a/meta/lib/oe/sbom.py b/meta/lib/oe/sbom.py index 22ed5070ea0..1130fa668bd 100644 --- a/meta/lib/oe/sbom.py +++ b/meta/lib/oe/sbom.py @@ -38,18 +38,34 @@ def get_sdk_spdxid(sdk): return "SPDXRef-SDK-%s" % sdk -def write_doc(d, spdx_doc, subdir, spdx_deploy=None, indent=None): +def doc_path_by_namespace(spdx_deploy, doc_namespace): + return spdx_deploy / "by-namespace" / doc_namespace.replace("/", "_") + + +def doc_path_by_hashfn(spdx_deploy, doc_name, hashfn): + return spdx_deploy / "by-hash" / hashfn.split()[1] / (doc_name + ".spdx.json") + + +def doc_path(spdx_deploy, doc_name, arch, subdir): + return spdx_deploy / arch/ subdir / (doc_name + ".spdx.json") + + +def write_doc(d, spdx_doc, arch, subdir, spdx_deploy=None, indent=None): from pathlib import Path if spdx_deploy is None: spdx_deploy = Path(d.getVar("SPDXDEPLOY")) - dest = spdx_deploy / subdir / (spdx_doc.name + ".spdx.json") + dest = doc_path(spdx_deploy, spdx_doc.name, arch, subdir) dest.parent.mkdir(exist_ok=True, parents=True) with dest.open("wb") as f: doc_sha1 = spdx_doc.to_json(f, sort_keys=True, indent=indent) - l = spdx_deploy / "by-namespace" / spdx_doc.documentNamespace.replace("/", "_") + l = doc_path_by_namespace(spdx_deploy, spdx_doc.documentNamespace) + l.parent.mkdir(exist_ok=True, parents=True) + l.symlink_to(os.path.relpath(dest, l.parent)) + + l = doc_path_by_hashfn(spdx_deploy, spdx_doc.name, d.getVar("BB_HASHFILENAME")) l.parent.mkdir(exist_ok=True, parents=True) l.symlink_to(os.path.relpath(dest, l.parent)) From patchwork Fri Jun 2 13:34:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25052 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 737F9C7EE29 for ; Fri, 2 Jun 2023 13:34:58 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.web10.13230.1685712897632323475 for ; Fri, 02 Jun 2023 06:34:57 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@linuxfoundation.org header.s=google header.b=EUbs3DSe; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-3f6da07feb2so20186705e9.0 for ; Fri, 02 Jun 2023 06:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712896; x=1688304896; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OxkAeJnKydq7+yPLQnDuLTGIVNtBIvXIWIsAarOGi90=; b=EUbs3DSemokOjfT2r8sFlI8R7F2K23osteZRKyZW+ixN6+UoqUb0puLBwuHdgJsZBX 1CdVuSrOlTk8Rit/ch8Lee50tsVEvWMMR/DfOSauilivr19xlLCDOxrXYKIc5HvPL0mq aaaq064yI+88jm9Ck+Lw9xcuhvspikouf3nXc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712896; x=1688304896; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OxkAeJnKydq7+yPLQnDuLTGIVNtBIvXIWIsAarOGi90=; b=af1/mAyJwagdBEu+PYyhj3mDu72EKB0FPFIuP5eEmCW3VE+n0i3BP8cpHwIxDrP7Ba DjGhHPiBIRuItfrdjgOx0kT9qrUdlwBM/l0cBJtkYTTlKn3QL89iTYvBUTGMj7yRayTo yYsL26fXU+Hm4YjKQHHUQ0UXsLlX3+OskgPazQ7wcq4NL5QEK0fAX86IAZBwsY+/v/zf RK5TTLZRELND2Yt74tVr89o047m/JIc41c4IpRD6AU9rYa+ZaGBF2xyKFnOV/uImMjBu PnAlBJijHxzGk6g2F1Nqk2DcYUbpJxJe9LB/G8mXcVF33bF05Rq+0qauPh9vJTN8+y1f 9EDg== X-Gm-Message-State: AC+VfDxykHgTpY/78cVpSuaxRtdkpyPikKZ8c0It9jEi4u6qcyRvqY/g UAKEZOie6vzvWqG9+3Dylya0EmJY5MK/9MwJmvs= X-Google-Smtp-Source: ACHHUZ7jRP/YIpYi8sMBuC1Ia0fKuGHs0iEZMy17MnV3GGi2Aig3kQCxx0Qub9IDoA98DPn7rXcpaQ== X-Received: by 2002:a7b:ce88:0:b0:3f6:11e9:de8e with SMTP id q8-20020a7bce88000000b003f611e9de8emr2043223wmj.4.1685712895759; Fri, 02 Jun 2023 06:34:55 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.34.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:34:55 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Cc: Joshua Watt Subject: [PATCH 02/11] classes/create-spdx-2.2: Respect PKG for providers Date: Fri, 2 Jun 2023 14:34:44 +0100 Message-Id: <20230602133453.229023-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:34:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182320 From: Joshua Watt If a package renames itself using PKG, the new name should be respected as a name that the recipe provides. Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie --- meta/classes/create-spdx-2.2.bbclass | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 6ec0c1465e7..f12e155f6ba 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -688,6 +688,10 @@ def collect_package_providers(d): rprovides = set(n for n, _ in bb.utils.explode_dep_versions2(pkg_data.get("RPROVIDES", "")).items()) rprovides.add(pkg) + if "PKG" in pkg_data: + pkg = pkg_data["PKG"] + rprovides.add(pkg) + for r in rprovides: providers[r] = (pkg, dep_hashfn) From patchwork Fri Jun 2 13:34:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68CFAC7EE2C for ; Fri, 2 Jun 2023 13:35:08 +0000 (UTC) Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by mx.groups.io with SMTP id smtpd.web11.13170.1685712898162251362 for ; Fri, 02 Jun 2023 06:34:58 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@linuxfoundation.org header.s=google header.b=WaLiaWyg; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.53, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-30d181952a2so105061f8f.0 for ; Fri, 02 Jun 2023 06:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712896; x=1688304896; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RfGKcIK7/sAlMPLWI87IL1gQoI6ib42pUk2rr3KAN/g=; b=WaLiaWygzbpalaSH07UPF95PS2xvLD3X3mOAsYmegG46ZkPCLq/4A7iqGXU1ddNrIh wv/GD6oYUxO1mLbAxSOkCq7acjuI86pr9GYZdrMKO1ba93gJ2coa4wDNjZaMm4kUI7jr YGo5vVU6L62ry+c9YCJ6uIiE3eox7tqWry10w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712896; x=1688304896; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RfGKcIK7/sAlMPLWI87IL1gQoI6ib42pUk2rr3KAN/g=; b=KPe1c4YKndrICcSSqUGP/vZUD0kSrZEUA3AsiuEyTutoI+cu2TIboiziw9UdeRFX4q jgaK3QDEgehHyu+Ex4A4Mxk7VI8jM2KqFmXv1iypVOViPDxEeThCUMTxvM4WSNAGAeRO eUTBj6vAGh4i0Go/6V4hvn+DqtRUP68zKZ6r5jwPBI5qubtddIv7JCvIdjA8nHAcjMyW GKNW6T+cW7YEqryCr2unwzDA10W46lrt8n8UGeKfXhDbALsaWxaOtkOPu2ISRBelqoJl RnBgfy+57fCoDbr5Rte7kcogBKlLkfzvonGSsRJi0JrW4yDqGqUC81JCm5LmxAZjrr+A CvFg== X-Gm-Message-State: AC+VfDw4UwjAJA6lNRS2lmUTaXz0XANZhayV/bza6KnFZZ/eMt3HleZI WwrRNIk0Wo9o3YyHz2AbES2dQpV9HsuChXrh7xU= X-Google-Smtp-Source: ACHHUZ6XPwXVTAPFQUpPbkarjWcAFitZeDGuHiPngrSYzKiHt6VVjTz+U91sD9iteXdp0x901D7z0Q== X-Received: by 2002:a5d:4524:0:b0:30a:e495:2544 with SMTP id j4-20020a5d4524000000b0030ae4952544mr7936wra.23.1685712896469; Fri, 02 Jun 2023 06:34:56 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.34.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:34:56 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Cc: Joshua Watt Subject: [PATCH 03/11] classes/create-spdx-2.2: Fix build time dependency calculations Date: Fri, 2 Jun 2023 14:34:45 +0100 Message-Id: <20230602133453.229023-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182321 From: Joshua Watt Build time dependencies were not being correctly calculated for SPDX documents because while a task can `deptask` itself (as do_create_spdx did), those dependencies do not appear in BB_TASKDEPDATA (to avoid circular dependencies). To fix this, an intermediate task called do_collect_sdpx_deps is created that does the 'deptask' on do_create_spdx and records the recipe dependencies. do_create_spdx then runs after this new task. This breaks the circular dependency and thus all of the do_create_spdx tasks correctly show up as dependencies of do_collect_spdx_deps. In addition, the dependency collection logic was improved to handle the case of transitive dependencies (that is, a dependency of a dependency) SPDX documents missing and causing an error. These transitive dependencies don't actually need to be included anyway since one can follow the relationship of the direct dependency to find them. As such, the code is reworked to find the current task in BB_TASKDEPDATA, and then only collect the immediate dependencies of the current task. Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie --- meta/classes/create-spdx-2.2.bbclass | 71 +++++++++++++++++++++------- 1 file changed, 54 insertions(+), 17 deletions(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index f12e155f6ba..c461c77744a 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -16,6 +16,7 @@ SPDXDEPLOY = "${SPDXDIR}/deploy" SPDXWORK = "${SPDXDIR}/work" SPDXIMAGEWORK = "${SPDXDIR}/image-work" SPDXSDKWORK = "${SPDXDIR}/sdk-work" +SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" @@ -337,30 +338,21 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag package_doc.add_relationship(pkg_file, "GENERATED_FROM", ref_id, comment=debugsrc) -def collect_deps(d): - current_task = "do_" + d.getVar("BB_CURRENTTASK") - - taskdepdata = d.getVar("BB_TASKDEPDATA", False) - deps = sorted(set( - (dep[0], dep[7]) for dep in taskdepdata.values() if - dep[1] == current_task and dep[0] != d.getVar("PN") - )) - - return deps - -collect_deps[vardepsexclude] += "BB_TASKDEPDATA" -collect_deps[vardeps] += "DEPENDS" - def collect_dep_recipes(d, doc, spdx_recipe): + import json from pathlib import Path import oe.sbom import oe.spdx deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + spdx_deps_file = Path(d.getVar("SPDXDEPS")) dep_recipes = [] - for dep_pn, dep_hashfn in collect_deps(d): + with spdx_deps_file.open("r") as f: + deps = json.load(f) + + for dep_pn, dep_hashfn in deps: dep_recipe_path = oe.sbom.doc_path_by_hashfn(deploy_dir_spdx, "recipe-" + dep_pn, dep_hashfn) spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_recipe_path) @@ -462,6 +454,52 @@ def add_download_packages(d, doc, recipe): # but this should be sufficient for now doc.add_relationship(package, "BUILD_DEPENDENCY_OF", recipe) +def collect_deps(d, dep_task): + current_task = "do_" + d.getVar("BB_CURRENTTASK") + pn = d.getVar("PN") + + taskdepdata = d.getVar("BB_TASKDEPDATA", False) + + for this_dep in taskdepdata.values(): + if this_dep[0] == pn and this_dep[1] == current_task: + break + else: + bb.fatal(f"Unable to find this {pn}:{current_task} in taskdepdata") + + deps = set() + for dep_name in this_dep[3]: + dep_data = taskdepdata[dep_name] + if dep_data[1] == dep_task and dep_data[0] != pn: + deps.add((dep_data[0], dep_data[7])) + + return sorted(deps) + +collect_deps[vardepsexclude] += "BB_TASKDEPDATA" +collect_deps[vardeps] += "DEPENDS" + +python do_collect_spdx_deps() { + # This task calculates the build time dependencies of the recipe, and is + # required because while a task can deptask on itself, those dependencies + # do not show up in BB_TASKDEPDATA. To work around that, this task does the + # deptask on do_create_spdx and writes out the dependencies it finds, then + # do_create_spdx reads in the found dependencies when writing the actual + # SPDX document + import json + from pathlib import Path + + spdx_deps_file = Path(d.getVar("SPDXDEPS")) + + deps = collect_deps(d, "do_create_spdx") + + with spdx_deps_file.open("w") as f: + json.dump(deps, f) +} +# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source +addtask do_collect_spdx_deps after do_unpack +do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" +do_collect_spdx_deps[deptask] = "do_create_spdx" +do_collect_spdx_deps[dirs] = "${SPDXDIR}" + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom @@ -647,7 +685,7 @@ python do_create_spdx() { oe.sbom.write_doc(d, package_doc, d.getVar("SSTATE_PKGARCH"), "packages", indent=get_json_indent(d)) } # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -661,7 +699,6 @@ addtask do_create_spdx_setscene do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += "${PATCHDEPENDENCY}" -do_create_spdx[deptask] = "do_create_spdx" def collect_package_providers(d): from pathlib import Path From patchwork Fri Jun 2 13:34:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B55BC7EE3A for ; Fri, 2 Jun 2023 13:35:08 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web10.13232.1685712900813745672 for ; Fri, 02 Jun 2023 06:35:01 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=IbVLmXV6; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-30aa1eb95a0so2099625f8f.1 for ; Fri, 02 Jun 2023 06:35:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712899; x=1688304899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U3yeS4RdBIE6jJ9bW2aOkbiHNkgwrOmJSuKeV9jYuwk=; b=IbVLmXV61LFNLcFdO1TzILazSCDarTdvpvVodzjRLrazP8qk8JVQDaXKBjLbauKhqN Nwwlr8e+RaLy/U/1wVLz2y5SUgeOh/WMIPIvV6KNQWfnb+FTRd+4F9D4qtWH+wlk+/Ly muXglu3n7Wjihfyet3svK2wjWqG21GY7sHouY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712899; x=1688304899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U3yeS4RdBIE6jJ9bW2aOkbiHNkgwrOmJSuKeV9jYuwk=; b=MTYTdTnyluUQVS77+2MvWa/7QW6lKysRrMtrqbkZs3jIeiLjuB6Ng2ZH1XY6C8Tolu OJ8lxMjRWzdcAN2FDHI0C/+vRZCYQiOPh0Kk3vt7IARfnUEafDnUKUOrEVoXOoyUguIy VVB0EXC+RwB7QuQmwK7+Ymqsg6INDZzXZmDEkE72zho1gSkxx+U+6aZ9Q+sQBdNoh6MU vMVKK3hp9/FdZ9LzoBhqdwqnaCVFYRFyr6l1Jgp8VXVeZvbl2aP1DWoaRvmExS4QJXlt iPWwyUN+2pNP6mKdaHIfgqwjnbjiRZ9SqTwiZNotGaFsdWjxywswfsM76hfD7Uk7kdMa 0quA== X-Gm-Message-State: AC+VfDxX5rVE/6PPrgrrfj4NtzeJXKHtdyAYp1XOvjD09FHy2yY/7iZ4 RF2TxzdfyBMpK3H0wnEC1ZqEsE+nW73vETjSqdg= X-Google-Smtp-Source: ACHHUZ58KEVfitx+lgxTY2GIyZBxg2zMhQNLnvHxZCoY6tcyeYJWLDgs5KQ5LJwEwFGI4csNkPB+jQ== X-Received: by 2002:adf:f9c9:0:b0:30a:bda7:f16f with SMTP id w9-20020adff9c9000000b0030abda7f16fmr54301wrr.0.1685712897187; Fri, 02 Jun 2023 06:34:57 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.34.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:34:56 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Cc: Joshua Watt Subject: [PATCH 04/11] classes/create-spdx-2.2: Fix runtime dependency calculations Date: Fri, 2 Jun 2023 14:34:46 +0100 Message-Id: <20230602133453.229023-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182322 From: Joshua Watt Fixes the runtime dependency calculations to also only follow direct dependencies of the current task instead of all transient dependencies in a similar way as the build time dependencies. Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie --- meta/classes/create-spdx-2.2.bbclass | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index c461c77744a..f9cb3add387 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -454,7 +454,7 @@ def add_download_packages(d, doc, recipe): # but this should be sufficient for now doc.add_relationship(package, "BUILD_DEPENDENCY_OF", recipe) -def collect_deps(d, dep_task): +def collect_direct_deps(d, dep_task): current_task = "do_" + d.getVar("BB_CURRENTTASK") pn = d.getVar("PN") @@ -474,8 +474,8 @@ def collect_deps(d, dep_task): return sorted(deps) -collect_deps[vardepsexclude] += "BB_TASKDEPDATA" -collect_deps[vardeps] += "DEPENDS" +collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" +collect_direct_deps[vardeps] += "DEPENDS" python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is @@ -489,7 +489,7 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = collect_deps(d, "do_create_spdx") + deps = collect_direct_deps(d, "do_create_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) @@ -710,10 +710,7 @@ def collect_package_providers(d): providers = {} - taskdepdata = d.getVar("BB_TASKDEPDATA", False) - deps = sorted(set( - (dep[0], dep[7]) for dep in taskdepdata.values() if dep[0] != d.getVar("PN") - )) + deps = collect_direct_deps(d, "do_create_spdx") deps.append((d.getVar("PN"), d.getVar("BB_HASHFILENAME"))) for dep_pn, dep_hashfn in deps: From patchwork Fri Jun 2 13:34:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BF59C7EE2E for ; Fri, 2 Jun 2023 13:35:08 +0000 (UTC) Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by mx.groups.io with SMTP id smtpd.web11.13173.1685712901398271936 for ; Fri, 02 Jun 2023 06:35:01 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=dOlNC6Ev; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.45, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-3f6077660c6so19163225e9.0 for ; Fri, 02 Jun 2023 06:35:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712899; x=1688304899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=rPFnWLHW9tEySVJQtiAkcBOVVoDy5/D5nm4j4zuE23M=; b=dOlNC6EvJkZa9AvAjLDaC3W63cuG6QVClxLg9msS9dfe453b0VsDvoR3ExbkK2OIxM P/AcPjGbmHg4hyTN86JaeVYBF09av5Mnt2w8XF1sGHI008rspQNg8CMG0ULc1UAqGwiX dVHt8ZhlJIVYQRDqY7qziN8l33FT8KujREaVk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712899; x=1688304899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rPFnWLHW9tEySVJQtiAkcBOVVoDy5/D5nm4j4zuE23M=; b=IGffFdXM+ay9Ph8SOjcWQ+R/FgmNk/wF0EdHMl0yYKmAL+hrvDnKMfq07m6RdmuOTf yQA6tZdeYwPfae51kf8ItZZujjWRD+ASg2Nvgy0MsQf06aZl9ZKPRZdbMZ5QLH0Yx3LV GH+IGiKGXz/qLuHoq26cxh2yNCgKdiIEY20pVtJfnz2EZGQNyxe2UHYszYv6+Sok2f+8 Z7e7xoUVdLgiVC/AE6HYwaCmUNXb6a751VCKtynaM/QJy3PVZvW/6VrvJmXmVml74QXC aY3LFxPLopYzwxUknsOWA3hz2Wtb61SUVSV0vLq9yFdes9eT8ChLX3QlbR4cbELlv+ed N/9w== X-Gm-Message-State: AC+VfDxyD/xll/DrlIn8vjgKuSyBtdqNibOdrhy0UeKMZXnenY2KvdGd 5oiat2zRbFaDZjLFU8L4tPjNAqsxSj1AIPLV06M= X-Google-Smtp-Source: ACHHUZ53eKDuSC921dXCDDltOh8eyN/Fjoki59iOiUOhwnpQmqhWKL21/veGBdKtltI7rSzgJEcSyg== X-Received: by 2002:a7b:c7ca:0:b0:3f6:cfc7:8bce with SMTP id z10-20020a7bc7ca000000b003f6cfc78bcemr2151803wmk.22.1685712899686; Fri, 02 Jun 2023 06:34:59 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.34.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:34:59 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 05/11] create-spdx-2-2: Fix packagedata usage to work with SDK packages Date: Fri, 2 Jun 2023 14:34:47 +0100 Message-Id: <20230602133453.229023-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182323 There are two seperate PKGDATA_DIR directories, one for the target and one for the SDK. Rather than fail when a package can't be found, try the SDK first. We use a datastore copy to keep the code simple, rather than havng to parameterise all the packagedata functions. Signed-off-by: Richard Purdie --- meta/classes/create-spdx-2.2.bbclass | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index f9cb3add387..b8728c5705b 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -714,11 +714,16 @@ def collect_package_providers(d): deps.append((d.getVar("PN"), d.getVar("BB_HASHFILENAME"))) for dep_pn, dep_hashfn in deps: - recipe_data = oe.packagedata.read_pkgdata(dep_pn, d) + localdata = d + recipe_data = oe.packagedata.read_pkgdata(dep_pn, localdata) + if not recipe_data: + localdata = bb.data.createCopy(d) + localdata.setVar("PKGDATA_DIR", "${PKGDATA_DIR_SDK}") + recipe_data = oe.packagedata.read_pkgdata(dep_pn, localdata) for pkg in recipe_data.get("PACKAGES", "").split(): - pkg_data = oe.packagedata.read_subpkgdata_dict(pkg, d) + pkg_data = oe.packagedata.read_subpkgdata_dict(pkg, localdata) rprovides = set(n for n, _ in bb.utils.explode_dep_versions2(pkg_data.get("RPROVIDES", "")).items()) rprovides.add(pkg) From patchwork Fri Jun 2 13:34:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85174C7EE39 for ; Fri, 2 Jun 2023 13:35:08 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web10.13234.1685712901995388716 for ; Fri, 02 Jun 2023 06:35:02 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@linuxfoundation.org header.s=google header.b=iUkd7/6D; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-3078a3f3b5fso2047761f8f.0 for ; Fri, 02 Jun 2023 06:35:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712900; x=1688304900; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=+07BB2F06dmMl33PwubqMzCPIremYoAqfE9RxL6LqGU=; b=iUkd7/6DtCwMjA+Kz8zLNasFQbEMGWsnVuUtT7X/PSUxVCFnOTliRxjYALTj+On+UR XQ9TWCOD0xKPUMZ5gRG9iGIHeh6mRGBY+jMQd3Nwjjk5eMuxzkeyTCggExAFwTrUWzoY 49MYr/+AA/eyTGyQzBZlDQySdJeyXZsSQHSXQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712900; x=1688304900; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+07BB2F06dmMl33PwubqMzCPIremYoAqfE9RxL6LqGU=; b=LJ7vubMCxl4EHJDimlcXIGv2TI0msiDOy+UA2YYtOt7DbBP/7EXRkdRkupC5B0Qntg +HKEXEgKROO6gPwNSurqjsay49ejnocw7UuFgsvs6EStVRcAtuf/JFWml1qOBsWHOE0p Ef3qgJbkH9uZUlwHNUaOYBKB/P7rG2CNyUtZ8hyechm7vxf96PlTAotqHle2RDaiuZcM CXcgJvjgn7sCFsZS0aGo0vW0YRgj3s8+2rFCo2UJMfBSmfQBegRCmZ8484tS5L1cLQh2 Up5k/Qf6jKoJD3qvHWC/dvA/ojIxPNmoU8RM6dD0VMLVncE9RpF4W0aZEem+hNsdu1il rN9w== X-Gm-Message-State: AC+VfDw5QCPHkZRU5fed5kWp+uG9tTL1EC46LFgqEjckOv4J3WdC/u2k a97ZZv7zqeryRXfGDGHwpssUIyEd1Bq2Pf6Daz0= X-Google-Smtp-Source: ACHHUZ7fdWLXphDx3yJcAcG0Gf6TZF7teDQWnm8Lha57cV4ascsFCe/LG+qvucO42I3CyYgSqSD01w== X-Received: by 2002:a5d:61ca:0:b0:2fb:1d3a:93ff with SMTP id q10-20020a5d61ca000000b002fb1d3a93ffmr4956720wrv.61.1685712900382; Fri, 02 Jun 2023 06:35:00 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.34.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:35:00 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 06/11] create-spdx-2.2: Add missing variable exclusions Date: Fri, 2 Jun 2023 14:34:48 +0100 Message-Id: <20230602133453.229023-6-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182324 Some variables cause issues with taskhashes and are accounted for in the output in other ways, or don't affect it. Exclude these to improve the taskhashes with create-spdx and allow the sstate selftests to work better. Signed-off-by: Richard Purdie --- meta/classes/create-spdx-2.2.bbclass | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index b8728c5705b..7569f8792b2 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -338,6 +338,8 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag package_doc.add_relationship(pkg_file, "GENERATED_FROM", ref_id, comment=debugsrc) +add_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR" + def collect_dep_recipes(d, doc, spdx_recipe): import json from pathlib import Path @@ -684,6 +686,7 @@ python do_create_spdx() { oe.sbom.write_doc(d, package_doc, d.getVar("SSTATE_PKGARCH"), "packages", indent=get_json_indent(d)) } +do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work @@ -854,6 +857,8 @@ python do_create_runtime_spdx() { oe.sbom.write_doc(d, runtime_doc, d.getVar("SSTATE_PKGARCH"), "runtime", spdx_deploy, indent=get_json_indent(d)) } +do_create_runtime_spdx[vardepsexclude] += "OVERRIDES" + addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work SSTATETASKS += "do_create_runtime_spdx" do_create_runtime_spdx[sstate-inputdirs] = "${SPDXRUNTIMEDEPLOY}" @@ -920,6 +925,8 @@ def spdx_get_src(d): finally: d.setVar("WORKDIR", workdir) +spdx_get_src[vardepsexclude] += "STAGING_KERNEL_DIR" + do_rootfs[recrdeptask] += "do_create_spdx do_create_runtime_spdx" do_rootfs[cleandirs] += "${SPDXIMAGEWORK}" @@ -1125,3 +1132,5 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx info.gname = "root" tar.addfile(info, fileobj=index_str) + +combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS" From patchwork Fri Jun 2 13:34:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8330AC7EE37 for ; Fri, 2 Jun 2023 13:35:08 +0000 (UTC) Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by mx.groups.io with SMTP id smtpd.web10.13235.1685712902605356221 for ; Fri, 02 Jun 2023 06:35:02 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@linuxfoundation.org header.s=google header.b=DtYfj4Ct; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.53, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-30ad99fa586so2021405f8f.2 for ; Fri, 02 Jun 2023 06:35:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712901; x=1688304901; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=otsYrUAmudVjB8p6IUNTE7UsKKXIDTHuSnsiXyyIOh8=; b=DtYfj4CtiuwruG09S8z7a3leXkXsPGWO2TeujRMd6OXqMyAVbNCiYBJxd+c8s7C6+0 nvA6SJfYwunyGg+bLJPjSBJJLo93Lte1yFfZI+vnL9srFiq05y2H4vR2BvGsUhPX6MBR Sns7TzAPEbLVPWuafhBS+WQK/5M4cmJyxQy60= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712901; x=1688304901; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=otsYrUAmudVjB8p6IUNTE7UsKKXIDTHuSnsiXyyIOh8=; b=il9A9MuXFFcv017rmf4wfIrmgMijEKV10clGCVTTXuhEeN6+PSkslY6mLe8cWJfHVr w8GmBSpoLu94Ygb75sX6/iKQ3v8g9dAtz7G9pdXPJFxq6d24hE/j3qRzrpsQqIoJMJtR b7aRb+Rp8aBAd8+mzmFJhD5f0fCLHlmbjMP6AcgGbHY4MnO3vbGvDtBdi11RsTKQFgBK AG17loddLNTec5Oyep+enheP8jccRSaYFijbox11fKsPxoRk90rHhj44I1y7uF7I73fO tR3nc75zsopKoqVSA9x8a4OQYToB2ieL0oMxagk2lyzMCe7iZIxbaXkNMwjgcj0e2NKb qWXQ== X-Gm-Message-State: AC+VfDwLgEhSV4jNiwaVONQ87A6ZMRC4KEXGr6YmXc9E9L8GeTFBIzWz yfugRlfJWz3KuTzLXZXA9Uqk0tYGkh5RAoEbGzA= X-Google-Smtp-Source: ACHHUZ5PckdgZQMnHvJfsgjKKNYa1fiVv9w7DNWyKe/QAlJEmlZcSSrAUzPqMHL18faadoiUoJSf8A== X-Received: by 2002:adf:ef45:0:b0:30a:eada:8b9b with SMTP id c5-20020adfef45000000b0030aeada8b9bmr3943004wrp.41.1685712900967; Fri, 02 Jun 2023 06:35:00 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.35.00 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:35:00 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 07/11] layer.conf: Add missing dependency exclusion Date: Fri, 2 Jun 2023 14:34:49 +0100 Message-Id: <20230602133453.229023-7-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182325 Add a dependency which should have been in this list but wasn't, found when debugging create-spdx hash issues. Signed-off-by: Richard Purdie --- meta/conf/layer.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf index 948ded667e8..2cc7ed84155 100644 --- a/meta/conf/layer.conf +++ b/meta/conf/layer.conf @@ -69,6 +69,7 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \ initramfs-module-install->grub \ initramfs-module-install->parted \ initramfs-module-install->util-linux \ + initramfs-module-setup-live->udev-extraconf \ grub-efi->grub-bootconf \ liberation-fonts->fontconfig \ cantarell-fonts->fontconfig \ From patchwork Fri Jun 2 13:34:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A532C7EE2F for ; Fri, 2 Jun 2023 13:35:08 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web11.13175.1685712903196777016 for ; Fri, 02 Jun 2023 06:35:03 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@linuxfoundation.org header.s=google header.b=My6tzrx+; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-307d20548adso1944683f8f.0 for ; Fri, 02 Jun 2023 06:35:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712901; x=1688304901; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=0LgxHd6PD+IqNEOvdsjKmG15gImbtP5ccRmmNrZEPYk=; b=My6tzrx+dzeYiyKSCvp/5TP58UCS2EaDCY5hbobhpylwqZxhzc4F/lcs3PnttYQLk9 jfzy8AyO1jOMOQq7heEOidPlEw78JUh/UIpedhrsz0NtVfU4paxewpywBd+FQ0ja2Lrf Cc5Pb3GRip3mgKhHBmIOc5UyeJWqqmNxF6MMU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712901; x=1688304901; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0LgxHd6PD+IqNEOvdsjKmG15gImbtP5ccRmmNrZEPYk=; b=HtiVxVG58xUpzqt2V+DvtZEfm043CSjz0OvLz0GUhlQ/oTTpFRvQpy6NbQ57QkvSep 45db5LF558+2EoY2mcBT1c3ILSBfpIXbUkaRr5lUb35beZHvsRAoSVvcndHFPVWE0+zj PvbU5lsuwwDJU/+BhyrCwzvZT4tnv75dt6TaQfFWx9rl1BWRQQL5NVA9pxpggJbsPNpj I1fwIt91oDHVgzITX58hDRyVZVAEXRCmY86d4dx4WKqRI6CVqJcuv4rdSCROYz3mqwlI 0n3TpQGwxIsKybiRF0mQZ82V1xXp0HFdWpukj3TeKwSFV21rZv6m7CqXFFeyZknzNW0M pP2A== X-Gm-Message-State: AC+VfDwE1xfXJavt/Uifw7CwbEURA0gVRicLcLw7dDys3Ar256Q7DjSw EfD8dhiqcD3GGoC4bezZWQZUE52JSEwMbOd/pwI= X-Google-Smtp-Source: ACHHUZ66KbpTg8Tt39k7Iwy18uK7IRkor8psUkEiZIptKWlMBtjtxlU3rK/GiWSmV/Li6C5+7Xp39Q== X-Received: by 2002:adf:f187:0:b0:2f4:4e1a:bea2 with SMTP id h7-20020adff187000000b002f44e1abea2mr4390178wro.59.1685712901599; Fri, 02 Jun 2023 06:35:01 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.35.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:35:01 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 08/11] sstatesig: Drop SPDX special casing Date: Fri, 2 Jun 2023 14:34:50 +0100 Message-Id: <20230602133453.229023-8-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182326 Other changes in create-spdx code mean we shouldn't need to do this now. We need the various exclusions to allow the task hashes to behave correctly for the SPDX tasks too. Signed-off-by: Richard Purdie --- meta/lib/oe/sstatesig.py | 9 --------- 1 file changed, 9 deletions(-) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index ae7ef14453f..f943df181e6 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -26,8 +26,6 @@ def sstate_rundepfilter(siggen, fn, recipename, task, dep, depname, dataCaches): return "/allarch.bbclass" in inherits def isImage(mc, fn): return "/image.bbclass" in " ".join(dataCaches[mc].inherits[fn]) - def isSPDXTask(task): - return task in ("do_create_spdx", "do_create_runtime_spdx") depmc, _, deptaskname, depmcfn = bb.runqueue.split_tid_mcfn(dep) mc, _ = bb.runqueue.split_mc(fn) @@ -38,13 +36,6 @@ def sstate_rundepfilter(siggen, fn, recipename, task, dep, depname, dataCaches): if task == "do_rm_work": return False - # Keep all dependencies between SPDX tasks in the signature. SPDX documents - # are linked together by hashes, which means if a dependent document changes, - # all downstream documents must be re-written (even if they are "safe" - # dependencies). - if isSPDXTask(task) and isSPDXTask(deptaskname): - return True - # (Almost) always include our own inter-task dependencies (unless it comes # from a mcdepends). The exception is the special # do_kernel_configme->do_unpack_and_patch dependency from archiver.bbclass. From patchwork Fri Jun 2 13:34:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25060 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82247C7EE32 for ; Fri, 2 Jun 2023 13:35:08 +0000 (UTC) Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by mx.groups.io with SMTP id smtpd.web11.13176.1685712903921560385 for ; Fri, 02 Jun 2023 06:35:04 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@linuxfoundation.org header.s=google header.b=baMVgqaT; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.49, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-30ad752f433so1855136f8f.3 for ; Fri, 02 Jun 2023 06:35:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712902; x=1688304902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=NdIaRcCWonAN8QTLc7Tm1d7IedxlZLR631LNirdd9oI=; b=baMVgqaTZXiqkjLZlO6FbfOSUN0u3Xz0gVG9A87dLe3QKM/ZZ+uwEkbRJeqhtHjveI 7tswi6+Dh4yIhVZtwKguV5/HX0lyges7KWmFtAVh7hE+rej/+FkEIrvpD2+Z4NFcqld4 KnGHS0ylxzkprC0o53H6qqdrN6SqMA0DE3pfA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712902; x=1688304902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NdIaRcCWonAN8QTLc7Tm1d7IedxlZLR631LNirdd9oI=; b=SjJnBCTEymQ8sHJLXxVceH1u1SbiEgUZ401uuklyWLSYZvVk2EuvWCoAOpOpGqeE+/ q/LJzccAZKl0T227qszKagOylPth1a2yNdy33ThOytZ4xZAcFClSULYNMkByRHfvQRz7 Pc67zxwRqoARV5IbmyLap0HQmyYIfWr1SmfkeL8ue4kHCcmyzekHrWAshYhoS/XweVnu WNp0g1JihzF1DUp53fPjtunuc0B0OlpKdP3X+uvX+Q/f762t4bZa1lEz8TUryMMWvdNM d178aQE2uypcBcP4w7DA/32Cp6c9fUgaDX8BlulDoIzwY10I1fJBzv/4Lkpx35KopIOs Sbmw== X-Gm-Message-State: AC+VfDwulyI7bn35g9MJRxK8v0L9ZAJBOCP3RNXrrOVqV0pvePJenxWv KKjKNsklFTQKtidBqcDC2EBugjYt9FbdmpQLf5E= X-Google-Smtp-Source: ACHHUZ7sN+lYDAO6QPNxs9g3fLoV0ocd3v8oBAprzK4IoBs7PT2Y1z2DjcukhEQbB8OpCHtN/WcIjw== X-Received: by 2002:adf:f08a:0:b0:30a:d4e3:8a08 with SMTP id n10-20020adff08a000000b0030ad4e38a08mr4155429wro.71.1685712902302; Fri, 02 Jun 2023 06:35:02 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.35.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:35:01 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 09/11] selftest/incompatible_lic: Ensure create_sdpx isn't used with the tests Date: Fri, 2 Jun 2023 14:34:51 +0100 Message-Id: <20230602133453.229023-9-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182327 The SPDX class doens't get on well with non-standard licenses. Disable it for the purposes of this test to avoid errors. Add a new helper function to the core test code to allow this to be done easily. Signed-off-by: Richard Purdie --- meta/lib/oeqa/selftest/case.py | 7 +++++++ meta/lib/oeqa/selftest/cases/incompatible_lic.py | 2 ++ 2 files changed, 9 insertions(+) diff --git a/meta/lib/oeqa/selftest/case.py b/meta/lib/oeqa/selftest/case.py index dcad4f76ecd..54d90c78ac5 100644 --- a/meta/lib/oeqa/selftest/case.py +++ b/meta/lib/oeqa/selftest/case.py @@ -249,6 +249,13 @@ class OESelftestTestCase(OETestCase): self.logger.debug("Writing to: %s\n%s\n" % (self.machineinc_path, data)) ftools.write_file(self.machineinc_path, data) + def disable_class(self, classname): + destfile = "%s/classes/%s.bbclass" % (self.builddir, classname) + os.makedirs(os.path.dirname(destfile), exist_ok=True) + self.track_for_cleanup(destfile) + self.logger.debug("Creating empty class: %s\n" % (destfile)) + ftools.write_file(destfile, "") + # check does path exist def assertExists(self, expr, msg=None): if not os.path.exists(expr): diff --git a/meta/lib/oeqa/selftest/cases/incompatible_lic.py b/meta/lib/oeqa/selftest/cases/incompatible_lic.py index 4edf60fc55f..1597d30ab03 100644 --- a/meta/lib/oeqa/selftest/cases/incompatible_lic.py +++ b/meta/lib/oeqa/selftest/cases/incompatible_lic.py @@ -113,6 +113,7 @@ INCOMPATIBLE_LICENSE:pn-core-image-minimal = "GPL-3.0* LGPL-3.0*" raise AssertionError(result.output) def test_bash_and_license(self): + self.disable_class("create-spdx") self.write_config(self.default_config() + '\nLICENSE:append:pn-bash = " & SomeLicense"') error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash cannot be installed into the image because it has incompatible license(s): GPL-3.0-or-later" @@ -121,6 +122,7 @@ INCOMPATIBLE_LICENSE:pn-core-image-minimal = "GPL-3.0* LGPL-3.0*" raise AssertionError(result.output) def test_bash_or_license(self): + self.disable_class("create-spdx") self.write_config(self.default_config() + '\nLICENSE:append:pn-bash = " | SomeLicense"') bitbake('core-image-minimal') From patchwork Fri Jun 2 13:34:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 517CDC7EE29 for ; Fri, 2 Jun 2023 13:35:08 +0000 (UTC) Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by mx.groups.io with SMTP id smtpd.web10.13238.1685712904695153327 for ; Fri, 02 Jun 2023 06:35:05 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@linuxfoundation.org header.s=google header.b=OoqhZsqX; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.42, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-30ae5f2ac94so2033679f8f.1 for ; Fri, 02 Jun 2023 06:35:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712903; x=1688304903; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=vA+Mypp4vBTiQuWyXoFmJYzNn/bVcn4eLXRGUhSct7Q=; b=OoqhZsqXYTCjioqyiJ9TI71DORVQZKDDDIfZ4pYCQN81VrFx9psfCeH3DQmqo8jIBn ghEX571Np+G2NeU3xGtD/arm+s+ykB7hn3ebnNdSh70XvDPgQT7breBqTrUGwpcqtyvY ec6J2h/o+BB+YEsBvoIDtzlfqrf4KEnCOy21g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712903; x=1688304903; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vA+Mypp4vBTiQuWyXoFmJYzNn/bVcn4eLXRGUhSct7Q=; b=JyLXPKHvrMK4QI2xwV6UUCo+S4ScXDyyBoaavZnt8R8vpxec4H4TyFsZWvp4mBNE4y rmSmTmUIySE9W5KPVXvi4TxeXWs+h3+K+BcvN+bF9fBGmpAFtFmBnEcneFbQiO0DIlEj GrBSmFsSmjORF+dL9QqVeakMW6veKZC8GpPvNUps8CSl1MwVFcnjO0iHsOhQKLh3UR2e lvgvajeoY3SCn4QWCltdE8QC1jWo6VQ/zcztJxl6eD2uLwFTrseuoBF10tAHMmcJcU3D QHd6S2ta/oi0TgcSZp7azXMZAGfseOU8dcwp0Fw2tJKQ14Lj5SLVoAcJJq2fRXLl4tWY tg3Q== X-Gm-Message-State: AC+VfDw4XJk0nZto30KGI8TDx5V/OTb50w7yRxFsOy4VvQA71m0CorOf bYUi94V52auEbLivv7tUjp+T46np93PsovgkFFI= X-Google-Smtp-Source: ACHHUZ5In44HMLE/1EGx6XWzJ4+OrruLoxHgRFfV/+nR9sJm7rUjEJkiJdUpNGfzwPJlN+nguTAYgw== X-Received: by 2002:a5d:6210:0:b0:30a:f17e:e02e with SMTP id y16-20020a5d6210000000b0030af17ee02emr19196wru.12.1685712903001; Fri, 02 Jun 2023 06:35:03 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.35.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:35:02 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 10/11] oeqa/selftest/sstatetests: Add easier debug option Date: Fri, 2 Jun 2023 14:34:52 +0100 Message-Id: <20230602133453.229023-10-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182328 In order to debug these selftest failures you end up having to comment out the file cleanup. Make this an option at the top of the file to make it a bit easier, I've had to do this too many times now. Signed-off-by: Richard Purdie --- meta/lib/oeqa/selftest/cases/sstatetests.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index e978313c61f..febafdb2f7a 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -18,6 +18,9 @@ from oeqa.selftest.case import OESelftestTestCase import oe import bb.siggen +# Set to True to preserve stamp files after test execution for debugging failures +keep_temp_files = False + class SStateBase(OESelftestTestCase): def setUpLocal(self): @@ -35,6 +38,10 @@ class SStateBase(OESelftestTestCase): self.target_os = bb_vars['TARGET_OS'] self.distro_specific_sstate = os.path.join(self.sstate_path, self.hostdistro) + def track_for_cleanup(self, path): + if not keep_temp_files: + super().track_for_cleanup(path) + # Creates a special sstate configuration with the option to add sstate mirrors def config_sstate(self, temp_sstate_location=False, add_local_mirrors=[]): self.temp_sstate_location = temp_sstate_location From patchwork Fri Jun 2 13:34:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 25057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59AB7C7EE24 for ; Fri, 2 Jun 2023 13:35:08 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web11.13177.1685712905287007370 for ; Fri, 02 Jun 2023 06:35:05 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="signature has expired" header.i=@linuxfoundation.org header.s=google header.b=TmRS1jA3; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-3f6e72a1464so19617425e9.1 for ; Fri, 02 Jun 2023 06:35:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1685712903; x=1688304903; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mXqUane1bpcOW4xXS00buX+/aurD+4C8L6Y41LdznoM=; b=TmRS1jA3ombBf+2zFxKr/HN64BLJMJ44uV4KpGv5VBRqVs6DBLAhA0cL1odqsIs8Ow GQQ9gB1JRUFHGzS+ulHquHc0/JzNs4IyU4k8V75+FOZt4jILjBBLfU+95zcmpnmFDO5t kBbF3wRWzmbrYatkuQToxgGBzon3KwOkLuk2w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685712903; x=1688304903; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mXqUane1bpcOW4xXS00buX+/aurD+4C8L6Y41LdznoM=; b=dP4CkbpxVi3jx2fOcEqLSZ/gJwhn2MEyvRJcOpnWjCCzxEiGz2q5CAuvI3ntPglh8G ZVc4xumXUiYJw9slZwNloVliWbi6yDZkNZUK2wzgbLw/BAQkMrQnNMmUpztdWeDsXQtW h4QMNUWIxMuw2mLIfR07jBUqq3s8ILegmxywnIkOJf5QHmMWjBFIvNdS468mDVp17okH thfuBH/koa2x+3WS08FONqOxG07wRVTgYDMpgdYxz8EJzdwGMBX6eZdSpNGEa0+bbZ54 babQeiLLaa/4xuLVNwCzzzjeYRNnrTf/uJpN8x85CJSQGVr5TCZ3JprHFZIc/0OI9XGh oHtQ== X-Gm-Message-State: AC+VfDx7DK1AAc8xQb4EY2dOYkt/gpluhIAJ5hoffr9ohe8J6a5Xptzv VN40ScXr5rkUhMUKKW8ElfktA0OMcuEr1AqiFRk= X-Google-Smtp-Source: ACHHUZ7bGkPDby8ReP/Xs/t8gfslLqQBm6aHM55XCyf7F910yXMbrQCiulsenfsBAOAwAcCbL9wySA== X-Received: by 2002:adf:fe43:0:b0:30a:e86d:17a3 with SMTP id m3-20020adffe43000000b0030ae86d17a3mr4400569wrs.46.1685712903647; Fri, 02 Jun 2023 06:35:03 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:527f:ecab:4831:c523]) by smtp.gmail.com with ESMTPSA id z10-20020a056000110a00b003063a92bbf5sm1734262wrw.70.2023.06.02.06.35.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 06:35:03 -0700 (PDT) From: Richard Purdie To: openembedded-core@lists.openembedded.org Cc: Joshua Watt Subject: [PATCH 11/11] classes/create-spdx-2.2: Make license errors fatal Date: Fri, 2 Jun 2023 14:34:53 +0100 Message-Id: <20230602133453.229023-11-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> References: <20230602133453.229023-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 13:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182329 From: Joshua Watt Otherwise the task is still put into sstate, which means the error can effectively be hidden by ignoring it. Signed-off-by: Joshua Watt --- meta/classes/create-spdx-2.2.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 7569f8792b2..844b674f6f1 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -138,7 +138,7 @@ def convert_license_to_spdx(lic, document, d, existing={}): with open(filename, errors="replace") as f: extracted_info.extractedText = f.read() else: - bb.error("Cannot find any text for license %s" % name) + bb.fatal("Cannot find any text for license %s" % name) extracted[name] = extracted_info document.hasExtractedLicensingInfos.append(extracted_info)