From patchwork Tue Mar 24 13:29:55 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Tondo X-Patchwork-Id: 84225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE8E2F54AB8 for ; Tue, 24 Mar 2026 13:30:17 +0000 (UTC) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.19485.1774359013493423746 for ; Tue, 24 Mar 2026 06:30:13 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=EwOjgUs6; spf=pass (domain: gmail.com, ip: 209.85.128.42, mailfrom: stondo@gmail.com) Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-48540d21f7dso45449375e9.0 for ; Tue, 24 Mar 2026 06:30:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1774359011; x=1774963811; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SiV4/un9QN6UMofI9DjH4M2mmIjT0G0QJ08PrF9lFoU=; b=EwOjgUs6CEZBVgjY/vFYbHIaxKZp/ZHliGv1zlGFfhJW4I70uoN+56m3b1ZR6lwDp6 ZHVHLdyf7FFjGCTMXAh90GTcqxWdwqFNU+Twz96sB9Y7EtVRXyP3kpPuw/p7ahMd0jDE 11uTb3PhHYbZ8cfg4IoNGFCT4dtwATaOmVaSEYF+qHqH2m1TAj4xjNQG6/oHp6UuLjiO mwWBLAMSBFs3aQnKzvy/HYWv4CV0JYVj5XPoAkbU42kHCbyTTC2cGRSI+g1FB2roU1Du PO+3s4Ra/jCTwBm5aT/g7F8GrUUy77FQqBWtdA6NXkivMe/TSSsYKDY2orl9R+sXSZSN tqWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774359011; x=1774963811; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=SiV4/un9QN6UMofI9DjH4M2mmIjT0G0QJ08PrF9lFoU=; b=cBYtB1NFqbU08TTgEa92+gb5uGvH7NTxy12822GkIqYL5KfvyLZ879ANPHWoEt1WR5 /TKgFuAhE6hPc+TfTmqZceHR9RbvEYZf7v0bYQ0NhW9uKMt6tHu+oHj4CA6fddKwQR8T 8XXnyB/u9NdH5PlNG8hU7TC1eVnZ9nhTGPqb5qEoiySIBIuQI5JH3H+CtRLMYeIptqoh ImVvmxjvLOUmFzb/FWoOD96JWLLLkY0e62UafCCbc+1D2+Ozp1N11wO29CmZ+5ps6/FU wpJoHL9nqHuoUiEd9I9WY8xKOFXMHQcl8oc260QNf7efAuWIJeOLFz09XkcyoUfcXyfb PBTg== X-Gm-Message-State: AOJu0YxlCh8Yj88jRjYwWaLYwl4BijQuNqIclEcSrHz49I3aAxm4/EOS qPFAZ6SiiH1Erdoj5Ro8oaBJiXM7FwrmnyI4fmJv84kweTVFY/YrRBhgvBXsd6fs X-Gm-Gg: ATEYQzxzd6EC/EnslWQTXOs+KZUForc/xfTveMM+BjIrFgrZmvq5Y+MkKBgzzQab2KE F4tgHDI38oBR1R9EhTAn7j69ZTqWJm5P8l6Df6DwJZIE6Hb5Pk5zXSZB+dZ8eMNTKLhwSH48uiO 81grxMExvTjGh6Yh08lvdO+2+GNY1xgJr7s6VkTtlmXFuiH6XSLnsDshb/ZPQwIOmNBWqC/ksic Xh3zPkA+1uUmUkKllLCzcssUOtdQ1vfsz/HlGx9NgZjPOC4ZOiKNms8C3SWu0AUUXApLDNrlxOV 9oToJYyMQZDXpdV1VMt4MgEKLJ0KGMDVt8BU0Gqk7s7ucjR08aIDls67rs5fAyT6ajaEwb2lIJj Tgm8a7wJNz2YxpbBimknQiJLr+4S75/Etrc9bEisP5UX/DLN7V4kKhzriTbr8xz6xRzQGFeoZ4R GdPBx5bOahFaKxa/YqnmT/TnGYK3bnDllvLsu9UNCCHZd9cdDHskJXhGCC2MjtwA1ouJI5KsLfz HA0HmZf X-Received: by 2002:a05:600c:c083:b0:485:3f72:324d with SMTP id 5b1f17b1804b1-486fee0481amr175698435e9.14.1774359011097; Tue, 24 Mar 2026 06:30:11 -0700 (PDT) Received: from fedora (mob-194-230-148-205.cgn.sunrise.net. [194.230.148.205]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48710fa0e35sm47494875e9.3.2026.03.24.06.30.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2026 06:30:10 -0700 (PDT) From: stondo@gmail.com To: openembedded-core@lists.openembedded.org Cc: richard.purdie@linuxfoundation.org, ross.burton@arm.com, jpewhacker@gmail.com, stefano.tondo.ext@siemens.com, peter.marko@siemens.com, adrian.freihofer@siemens.com, mathieu.dubois-briand@bootlin.com Subject: [OE-core][PATCH v14 1/4] spdx30: Add configurable file exclusion pattern support Date: Tue, 24 Mar 2026 14:29:55 +0100 Message-ID: <20260324132958.2316491-2-stondo@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260324132958.2316491-1-stondo@gmail.com> References: <20260323210745.1337169-1-stefano.tondo.ext@siemens.com> <20260324132958.2316491-1-stondo@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Mar 2026 13:30:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/233798 From: Stefano Tondo Add SPDX_FILE_EXCLUDE_PATTERNS variable that allows filtering files from SPDX output by regex matching. The variable accepts a space-separated list of Python regular expressions; files whose paths match any pattern (via re.search) are excluded. When empty (the default), no filtering is applied and all files are included, preserving existing behavior. This enables users to reduce SBOM size by excluding files that are not relevant for compliance (e.g., test files, object files, patches). Excluded files are tracked in a set returned from add_package_files() and passed to get_package_sources_from_debug(), which uses the set for precise cross-checking rather than re-evaluating patterns. Signed-off-by: Stefano Tondo Reviewed-by: Joshua Watt --- meta/classes/spdx-common.bbclass | 7 +++ meta/lib/oe/spdx30_tasks.py | 80 +++++++++++++++++++++----------- 2 files changed, 60 insertions(+), 27 deletions(-) diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 83f05579b6..40701730a6 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -82,6 +82,13 @@ SPDX_MULTILIB_SSTATE_ARCHS[doc] = "The list of sstate architectures to consider when collecting SPDX dependencies. This includes multilib architectures when \ multilib is enabled. Defaults to SSTATE_ARCHS." +SPDX_FILE_EXCLUDE_PATTERNS ??= "" +SPDX_FILE_EXCLUDE_PATTERNS[doc] = "Space-separated list of Python regular \ + expressions to exclude files from SPDX output. Files whose paths match \ + any pattern (via re.search) will be filtered out. Defaults to empty \ + (no filtering). Example: \ + SPDX_FILE_EXCLUDE_PATTERNS = '\\.patch$ \\.diff$ /test/ \\.pyc$ \\.o$'" + python () { from oe.cve_check import extend_cve_status extend_cve_status(d) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 353d783fa2..68ed821a8c 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -13,6 +13,7 @@ import oe.spdx30 import oe.spdx_common import oe.sdk import os +import re from contextlib import contextmanager from datetime import datetime, timezone @@ -157,17 +158,27 @@ def add_package_files( file_counter = 1 if not os.path.exists(topdir): bb.note(f"Skip {topdir}") - return spdx_files + return spdx_files, set() check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1" if check_compiled_sources: compiled_sources, types = oe.spdx_common.get_compiled_sources(d) bb.debug(1, f"Total compiled files: {len(compiled_sources)}") + exclude_patterns = [ + re.compile(pattern) + for pattern in (d.getVar("SPDX_FILE_EXCLUDE_PATTERNS") or "").split() + ] + excluded_files = set() + for subdir, dirs, files in os.walk(topdir, onerror=walk_error): - dirs[:] = [d for d in dirs if d not in ignore_dirs] + dirs[:] = [directory for directory in dirs if directory not in ignore_dirs] if subdir == str(topdir): - dirs[:] = [d for d in dirs if d not in ignore_top_level_dirs] + dirs[:] = [ + directory + for directory in dirs + if directory not in ignore_top_level_dirs + ] dirs.sort() files.sort() @@ -177,14 +188,19 @@ def add_package_files( continue filename = str(filepath.relative_to(topdir)) + + if exclude_patterns and any( + pattern.search(filename) for pattern in exclude_patterns + ): + excluded_files.add(filename) + continue + file_purposes = get_purposes(filepath) - # Check if file is compiled - if check_compiled_sources: - if not oe.spdx_common.is_compiled_source( - filename, compiled_sources, types - ): - continue + if check_compiled_sources and not oe.spdx_common.is_compiled_source( + filename, compiled_sources, types + ): + continue spdx_file = objset.new_file( get_spdxid(file_counter), @@ -218,12 +234,15 @@ def add_package_files( bb.debug(1, "Added %d files to %s" % (len(spdx_files), objset.doc._id)) - return spdx_files + return spdx_files, excluded_files def get_package_sources_from_debug( - d, package, package_files, sources, source_hash_cache + d, package, package_files, sources, source_hash_cache, excluded_files=None ): + if excluded_files is None: + excluded_files = set() + def file_path_match(file_path, pkg_file): if file_path.lstrip("/") == pkg_file.name.lstrip("/"): return True @@ -256,6 +275,12 @@ def get_package_sources_from_debug( continue if not any(file_path_match(file_path, pkg_file) for pkg_file in package_files): + if file_path.lstrip("/") in excluded_files: + bb.debug( + 1, + f"Skipping debug source lookup for excluded file {file_path} in {package}", + ) + continue bb.fatal( "No package file found for %s in %s; SPDX found: %s" % (str(file_path), package, " ".join(p.name for p in package_files)) @@ -737,7 +762,7 @@ def create_spdx(d): bb.debug(1, "Adding source files to SPDX") oe.spdx_common.get_patched_src(d) - files = add_package_files( + files, _ = add_package_files( d, build_objset, spdx_workdir, @@ -909,7 +934,7 @@ def create_spdx(d): ) bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) - package_files = add_package_files( + package_files, excluded_files = add_package_files( d, pkg_objset, pkgdest / package, @@ -932,7 +957,8 @@ def create_spdx(d): if include_sources: debug_sources = get_package_sources_from_debug( - d, package, package_files, dep_sources, source_hash_cache + d, package, package_files, dep_sources, source_hash_cache, + excluded_files=excluded_files, ) debug_source_ids |= set( oe.sbom30.get_element_link_id(d) for d in debug_sources @@ -944,7 +970,7 @@ def create_spdx(d): if include_sources: bb.debug(1, "Adding sysroot files to SPDX") - sysroot_files = add_package_files( + sysroot_files, _ = add_package_files( d, build_objset, d.expand("${COMPONENTS_DIR}/${PACKAGE_ARCH}/${PN}"), @@ -1326,18 +1352,18 @@ def create_image_spdx(d): image_filename = image["filename"] image_path = image_deploy_dir / image_filename if os.path.isdir(image_path): - a = add_package_files( - d, - objset, - image_path, - lambda file_counter: objset.new_spdxid( - "imagefile", str(file_counter) - ), - lambda filepath: [], - license_data=None, - ignore_dirs=[], - ignore_top_level_dirs=[], - archive=None, + a, _ = add_package_files( + d, + objset, + image_path, + lambda file_counter: objset.new_spdxid( + "imagefile", str(file_counter) + ), + lambda filepath: [], + license_data=None, + ignore_dirs=[], + ignore_top_level_dirs=[], + archive=None, ) artifacts.extend(a) else: From patchwork Tue Mar 24 13:29:56 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Tondo X-Patchwork-Id: 84224 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A99A0F54AB7 for ; Tue, 24 Mar 2026 13:30:17 +0000 (UTC) Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.19486.1774359015897368253 for ; Tue, 24 Mar 2026 06:30:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=KxstllSW; spf=pass (domain: gmail.com, ip: 209.85.128.54, mailfrom: stondo@gmail.com) Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-4838c15e3cbso44652385e9.3 for ; Tue, 24 Mar 2026 06:30:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1774359014; x=1774963814; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=msv1QujP3/OtyqVWNxdCYkHOiHANGiaQ6fHKsgSMM5M=; b=KxstllSWdjwanmdBA56wvp/x3m9TlJTYPWil4miatr4+26ND/QyfGzS3DecV2bJzNw Ireh7iFbGhZO8DUPOM3rp7l8eKuwoI+VAVexLObzjfRTrDz96zOqfQjaCB1Z7nzjTmF2 uGArs32eJIWEOk3mDDqGk9grJUHdzJkES/2gIRFkNNd0J/737PEEfMEv52+1jP8eVTcn A8hfG/UZT9SciP6s3KdPYJ/K4za91lfr/WIknqHE985fJuUByouhnUHFUxy3OKOvlmwv 8sKNQ6GaUmhGLcCtiETKL+7daFbthz4zi3jfAwvLvyX+CGjfCubfdqWMxcvhuAMfp7M0 a8OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774359014; x=1774963814; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=msv1QujP3/OtyqVWNxdCYkHOiHANGiaQ6fHKsgSMM5M=; b=NoPdXPSlzMDsqg7riuxpPVRSqmX7fuO+CaNURfcqSRosvA7qvU30WF8QUSJVEYltWH 4Rsfcrn5OzW0QSqh5wby7drslhsZgz/QYRrkVtYLHWVlyDEfU6VOlYVdBw+zNYCGTLBe fT5WShwqzd/7oPagzJK2cFf8aDKtRFwUFWpaLlAtuFKhN4kiduVfVxlP1lLagPLoHK7v ShytXeHjMnhXJeHSr+hJ/QE/vF2iYIhF2Z6DjlCYchjfpgtNtVRjrUNETj9Env94pD2Q 5tdPeER/p5aGRt/QGgYalyZSnEUg12vS4ml5xfrLWwV531Li/6cJ5EVSjhQg2dt9dOFA SYkQ== X-Gm-Message-State: AOJu0YzwU9AhZN9lK6kPZZ1iFCoSrlXmYNOh4G2H/QdbvMyF+/kjYdN0 CDKn2xv33IEcEZThpb9A7HmEK5OlPyqOuXYq8Pt76UxTiC44usW86bEzGBCcV6mo X-Gm-Gg: ATEYQzxuVmqZyUDhM9QilayfJCeDDc1AyzPP3+Cy20TQ9BdnW73rOyo/VkRIPC92OAh s+7Gg4T3j+FQYKv92GS1gwQXt7f5kR67KwjufQECTOKVv6+V7xyV5fgMmWkoCj41CJfvIWAYOKo 8pVIoAx69kR31YC7QwSTH2uWgc/JEDWAwMuh9vSAQURUujn0x5ZejG1NV6CQusKJKiUvzFNoggS EgDmhV4CIPVW1XZ49mjiHjIvOLyZB5vsikNf+WMTxTSeEzMVuSA64BuaaxSFnUOZXwHBr+RE9FC 7YIpV9hN8dYuMHENbEPu06jFdTwNf1mulzKpBukLQKa2YL/twidofIcpnffjTeAAGXLSMkaYX9o k2/9jJVJJshcGZY0/WBJeeM2nUYSL5dzJXlvRH+zYx8kVLCDb5FWdwHdAk11bW0gGFZ0JrTGZQV O+PJELHXoF1WFIJtXx1K09Vro5SDY+yOeRGL4CPdtOgYP8xcYU3zDAfRCQC5qrWOJkKZ2M/ftQp /5/NKrm X-Received: by 2002:a05:600c:8219:b0:485:ae14:8187 with SMTP id 5b1f17b1804b1-486febb5996mr215368085e9.2.1774359013422; Tue, 24 Mar 2026 06:30:13 -0700 (PDT) Received: from fedora (mob-194-230-148-205.cgn.sunrise.net. [194.230.148.205]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48710fa0e35sm47494875e9.3.2026.03.24.06.30.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2026 06:30:12 -0700 (PDT) From: stondo@gmail.com To: openembedded-core@lists.openembedded.org Cc: richard.purdie@linuxfoundation.org, ross.burton@arm.com, jpewhacker@gmail.com, stefano.tondo.ext@siemens.com, peter.marko@siemens.com, adrian.freihofer@siemens.com, mathieu.dubois-briand@bootlin.com, Joshua Watt Subject: [OE-core][PATCH v14 2/4] spdx30: Add supplier support for image and SDK SBOMs Date: Tue, 24 Mar 2026 14:29:56 +0100 Message-ID: <20260324132958.2316491-3-stondo@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260324132958.2316491-1-stondo@gmail.com> References: <20260323210745.1337169-1-stefano.tondo.ext@siemens.com> <20260324132958.2316491-1-stondo@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Mar 2026 13:30:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/233799 From: Stefano Tondo Add SPDX_IMAGE_SUPPLIER and SPDX_SDK_SUPPLIER variables that allow setting a supplier agent on image and SDK SBOM root elements using the suppliedBy property. These follow the existing SPDX_PACKAGE_SUPPLIER pattern and use the standard agent variable system to define supplier information. Signed-off-by: Stefano Tondo Reviewed-by: Joshua Watt --- meta/classes/create-spdx-3.0.bbclass | 10 ++++++++++ meta/lib/oe/spdx30_tasks.py | 23 ++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 7515f460c3..9a6606dce6 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -124,6 +124,16 @@ SPDX_ON_BEHALF_OF[doc] = "The base variable name to describe the Agent on who's SPDX_PACKAGE_SUPPLIER[doc] = "The base variable name to describe the Agent who \ is supplying artifacts produced by the build" +SPDX_IMAGE_SUPPLIER[doc] = "The base variable name to describe the Agent who \ + is supplying the image SBOM. The supplier will be set on all root elements \ + of the image SBOM using the suppliedBy property. If not set, no supplier \ + information will be added to the image SBOM." + +SPDX_SDK_SUPPLIER[doc] = "The base variable name to describe the Agent who \ + is supplying the SDK SBOM. The supplier will be set on all root elements \ + of the SDK SBOM using the suppliedBy property. If not set, no supplier \ + information will be added to the SDK SBOM." + SPDX_PACKAGE_VERSION ??= "${PV}" SPDX_PACKAGE_VERSION[doc] = "The version of a package, software_packageVersion \ in software_Package" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 68ed821a8c..62a00069df 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1449,6 +1449,16 @@ def create_image_sbom_spdx(d): objset, sbom = oe.sbom30.create_sbom(d, image_name, root_elements) + # Set supplier on root elements if SPDX_IMAGE_SUPPLIER is defined + supplier = objset.new_agent("SPDX_IMAGE_SUPPLIER", add=False) + if supplier is not None: + supplier_id = supplier if isinstance(supplier, str) else supplier._id + if not isinstance(supplier, str): + objset.add(supplier) + for elem in sbom.rootElement: + if hasattr(elem, "suppliedBy"): + elem.suppliedBy = supplier_id + oe.sbom30.write_jsonld_doc(d, objset, spdx_path) def make_image_link(target_path, suffix): @@ -1560,12 +1570,19 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): d, toolchain_outputname, sorted(list(files)), [rootfs_objset] ) + # Set supplier on root elements if SPDX_SDK_SUPPLIER is defined + supplier = objset.new_agent("SPDX_SDK_SUPPLIER", add=False) + if supplier is not None: + supplier_id = supplier if isinstance(supplier, str) else supplier._id + if not isinstance(supplier, str): + objset.add(supplier) + for elem in sbom.rootElement: + if hasattr(elem, "suppliedBy"): + elem.suppliedBy = supplier_id + oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) - - -def create_recipe_sbom(d, deploydir): sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") recipe, recipe_objset = load_recipe_spdx(d) From patchwork Tue Mar 24 13:29:57 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Tondo X-Patchwork-Id: 84227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9F2FF54AB8 for ; Tue, 24 Mar 2026 13:30:27 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.19720.1774359017624828247 for ; Tue, 24 Mar 2026 06:30:17 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=VYZtK49g; spf=pass (domain: gmail.com, ip: 209.85.128.50, mailfrom: stondo@gmail.com) Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-48540d21f7dso45450165e9.0 for ; Tue, 24 Mar 2026 06:30:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1774359016; x=1774963816; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bIcS2AS99ZawaKufZMobTzyJo5sWrrdUM1JjF99pEYo=; b=VYZtK49gKihXosPzVDn5wb3vKAkblxPjxbYC6Nu2fyHtrqrpqcvvQle/9NhkrCV4TA uQFWo3YQulBzDX4yWMXSwZoy3mS/u6F6XrrnaAuUq01gjCwEy6INxuTQ5SniNyCK/YRV OlBQzySmgtIqn8aA1fTVl74yFH0QY5mh2C5wTB9r1Rx6u+B702YP80WwwD5577mwrofr dNjS3WYOM0GoyNmKnBx3seSE78l43PHFUTSBa+CJ8ApxOaFWp1hebPcClpswlGiKXZnC uSIfh4BZ0Vl1TQS+vR/tjXHZUw35BSOWDELqYdzFFqy460qGKPTOqYMKDtlNnMenbd4r uY8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774359016; x=1774963816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=bIcS2AS99ZawaKufZMobTzyJo5sWrrdUM1JjF99pEYo=; b=W3rOYsH4Qu6hJ8qUf8x08GzFBmdDtB9lYNEDwxJyk+FB9KOZKcQirhKILzKTQQuxpn 7eZEPsA9KZGY/94gXAkvzych3mJYh6qJX4qDWgaichNRW7gjkW8zhERlupSTkJiOozCG HZ1HihqXoW6oPsAb7DQkfpbYpFAS3uDhxjZ1F0Hr6M8IjhOdls3xm7EJgAkAykmd0pkk NGp03MOsM7ftl0rPxM43od6H7+/SoGLN44dxkTA6KkKEKJL/WYBxW9ZdVvrgbzk8YcVS qaQERtTtTGN8DftXbs1IWgGpHS21DdssTf9M5jLFoK+/NqX/tqi9As4mqXcXG+llQSEh KfUw== X-Gm-Message-State: AOJu0YxYHi11HkwMsrIEGcSz4ZxYBTPoHqXWgTMjSedI8BfomyDzpKoS po49gYZD4x0s6UX9pZW4tmRAUED0gn3HIEungoOp3g2M+bqkG4YC294SXmp2ZFey X-Gm-Gg: ATEYQzw/J+87VBz1nzFxLinxRCL7ENHN4T/xMl6XUN4TzHyerUWe+CUebyvYHMTCG7w H1kZWESkapL9TCGcKt/W3KV54CjiFJHanB8f3NpFFM3vwa6hu69rOdVeqXG/2fj79EQZW3cJ/h5 TFdICogC2l9qg4uJ7+5lykuClKv9BvxKypWXQ7+fEm3m28Ip+OH1PN73LCh75uh5n5C/RoOeawl /k70I5PC5f53wfYDARSRoEpJ+MbYe/i/KowMSR+e9ypnn3lRp9SOpkt6ToOJLftIJkzrSXydcO4 od7EUcjO9jHdNSXs6X/7Na51S5zkD9Mtl78pLLmqs4Pv4QiUdtT+4PVtY6Qw+vVfO4Nd0VEKcTN v9mJ/sx/OyHQycpqb+1NxkKqg+b0sl+f2R6ODHlW4Cb5xqy8mB9M9tlEfqG8iNzNE7CvjqY37AV ZMLZ9UlVO36HdMTTXUNK5UKd0Zh83P2xAhxdD2JxEDloGWBdIW+K58b+eKCkkRcFUJRv79xjUU7 DSv2v5nOWaM/nrQyQQ= X-Received: by 2002:a05:600c:1383:b0:485:3d00:efd with SMTP id 5b1f17b1804b1-486fede7363mr241609305e9.7.1774359015269; Tue, 24 Mar 2026 06:30:15 -0700 (PDT) Received: from fedora (mob-194-230-148-205.cgn.sunrise.net. [194.230.148.205]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48710fa0e35sm47494875e9.3.2026.03.24.06.30.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2026 06:30:14 -0700 (PDT) From: stondo@gmail.com To: openembedded-core@lists.openembedded.org Cc: richard.purdie@linuxfoundation.org, ross.burton@arm.com, jpewhacker@gmail.com, stefano.tondo.ext@siemens.com, peter.marko@siemens.com, adrian.freihofer@siemens.com, mathieu.dubois-briand@bootlin.com Subject: [OE-core][PATCH v14 3/4] spdx30: Enrich source downloads with version and PURL Date: Tue, 24 Mar 2026 14:29:57 +0100 Message-ID: <20260324132958.2316491-4-stondo@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260324132958.2316491-1-stondo@gmail.com> References: <20260323210745.1337169-1-stefano.tondo.ext@siemens.com> <20260324132958.2316491-1-stondo@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Mar 2026 13:30:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/233800 From: Stefano Tondo Add version extraction, PURL generation, and external references to source download packages in SPDX 3.0 SBOMs: - Extract version from SRCREV for Git sources (full SHA-1) - Generate PURLs for Git sources on github.com by default - Support custom mappings via SPDX_GIT_PURL_MAPPINGS variable (format: "domain:purl_type", split(':', 1) for parsing) - Use ecosystem PURLs from SPDX_PACKAGE_URLS for non-Git - Add VCS external references for Git downloads - Add distribution external references for tarball downloads - Parse Git URLs using urllib.parse - Extract logic into _generate_git_purl() and _enrich_source_package() helpers For non-Git sources, version is not set from PV since the recipe version does not necessarily reflect the version of individual downloaded files. Ecosystem PURLs (which include version) from SPDX_PACKAGE_URLS are still used when available. The SPDX_GIT_PURL_MAPPINGS variable allows configuring PURL generation for self-hosted Git services (e.g., GitLab). github.com is always mapped to pkg:github by default. Add ecosystem-specific SPDX_PACKAGE_URLS to recipe classes: - cargo_common.bbclass: pkg:cargo - cpan.bbclass: pkg:cpan (with prefix stripping) - go-mod.bbclass: pkg:golang - npm.bbclass: pkg:npm (with prefix stripping) - pypi.bbclass: pkg:pypi (with normalization) Signed-off-by: Stefano Tondo --- meta/classes-recipe/cargo_common.bbclass | 3 + meta/classes-recipe/cpan.bbclass | 11 ++ meta/classes-recipe/go-mod.bbclass | 6 + meta/classes-recipe/npm.bbclass | 7 + meta/classes-recipe/pypi.bbclass | 6 +- meta/classes/create-spdx-3.0.bbclass | 7 + meta/lib/oe/spdx30_tasks.py | 175 +++++++++++++++++------ 7 files changed, 172 insertions(+), 43 deletions(-) diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass index bc44ad7918..0d3edfe4a7 100644 --- a/meta/classes-recipe/cargo_common.bbclass +++ b/meta/classes-recipe/cargo_common.bbclass @@ -240,3 +240,6 @@ EXPORT_FUNCTIONS do_configure # https://github.com/rust-lang/libc/issues/3223 # https://github.com/rust-lang/libc/pull/3175 INSANE_SKIP:append = " 32bit-time" + +# Generate ecosystem-specific Package URL for SPDX +SPDX_PACKAGE_URLS =+ "pkg:cargo/${BPN}@${PV} " diff --git a/meta/classes-recipe/cpan.bbclass b/meta/classes-recipe/cpan.bbclass index bb76a5b326..dbf44da9d2 100644 --- a/meta/classes-recipe/cpan.bbclass +++ b/meta/classes-recipe/cpan.bbclass @@ -68,4 +68,15 @@ cpan_do_install () { done } +# Generate ecosystem-specific Package URL for SPDX +def cpan_spdx_name(d): + bpn = d.getVar('BPN') + if bpn.startswith('perl-'): + return bpn[5:] + elif bpn.startswith('libperl-'): + return bpn[8:] + return bpn + +SPDX_PACKAGE_URLS =+ "pkg:cpan/${@cpan_spdx_name(d)}@${PV} " + EXPORT_FUNCTIONS do_configure do_compile do_install diff --git a/meta/classes-recipe/go-mod.bbclass b/meta/classes-recipe/go-mod.bbclass index a15dda8f0e..5b3cb2d8b9 100644 --- a/meta/classes-recipe/go-mod.bbclass +++ b/meta/classes-recipe/go-mod.bbclass @@ -32,3 +32,9 @@ do_compile[dirs] += "${B}/src/${GO_WORKDIR}" # Make go install unpack the module zip files in the module cache directory # before the license directory is polulated with license files. addtask do_compile before do_populate_lic + +# Generate ecosystem-specific Package URL for SPDX +SPDX_PACKAGE_URLS =+ "pkg:golang/${GO_IMPORT}@${PV} " + +# Generate ecosystem-specific Package URL for SPDX +SPDX_PACKAGE_URLS =+ "pkg:golang/${GO_IMPORT}@${PV} " diff --git a/meta/classes-recipe/npm.bbclass b/meta/classes-recipe/npm.bbclass index 344e8b4bec..7bb791d543 100644 --- a/meta/classes-recipe/npm.bbclass +++ b/meta/classes-recipe/npm.bbclass @@ -354,4 +354,11 @@ FILES:${PN} += " \ ${nonarch_libdir} \ " +# Generate ecosystem-specific Package URL for SPDX +def npm_spdx_name(d): + bpn = d.getVar('BPN') + return bpn[5:] if bpn.startswith('node-') else bpn + +SPDX_PACKAGE_URLS =+ "pkg:npm/${@npm_spdx_name(d)}@${PV} " + EXPORT_FUNCTIONS do_configure do_compile do_install diff --git a/meta/classes-recipe/pypi.bbclass b/meta/classes-recipe/pypi.bbclass index 9d46c035f6..e2d054af6d 100644 --- a/meta/classes-recipe/pypi.bbclass +++ b/meta/classes-recipe/pypi.bbclass @@ -43,7 +43,8 @@ SECTION = "devel/python" SRC_URI:prepend = "${PYPI_SRC_URI} " S = "${UNPACKDIR}/${PYPI_PACKAGE}-${PV}" -UPSTREAM_CHECK_PYPI_PACKAGE ?= "${PYPI_PACKAGE}" +# Replace any '_' characters in the pypi URI with '-'s to follow the PyPi website naming conventions +UPSTREAM_CHECK_PYPI_PACKAGE ?= "${@pypi_normalize(d)}" # Use the simple repository API rather than the potentially unstable project URL # More information on the pypi API specification is avaialble here: @@ -54,3 +55,6 @@ UPSTREAM_CHECK_URI ?= "https://pypi.org/simple/${@pypi_normalize(d)}/" UPSTREAM_CHECK_REGEX ?= "${UPSTREAM_CHECK_PYPI_PACKAGE}-(?P(\d+[\.\-_]*)+).(tar\.gz|tgz|zip|tar\.bz2)" CVE_PRODUCT ?= "python:${PYPI_PACKAGE}" + +# Generate ecosystem-specific Package URL for SPDX +SPDX_PACKAGE_URLS =+ "pkg:pypi/${@pypi_normalize(d)}@${PV} " diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 9a6606dce6..265dc525bc 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -156,6 +156,13 @@ SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom" SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \ create_recipe_sbom" +SPDX_GIT_PURL_MAPPINGS ??= "" +SPDX_GIT_PURL_MAPPINGS[doc] = "A space separated list of domain:purl_type \ + mappings to configure PURL generation for Git source downloads. \ + For example, "gitlab.example.com:pkg:gitlab" maps repositories hosted \ + on gitlab.example.com to the pkg:gitlab PURL type. \ + github.com is always mapped to pkg:github by default." + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 62a00069df..6f0bdba975 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -14,6 +14,7 @@ import oe.spdx_common import oe.sdk import os import re +import urllib.parse from contextlib import contextmanager from datetime import datetime, timezone @@ -384,6 +385,120 @@ def collect_dep_sources(dep_objsets, dest): index_sources_by_hash(e.to, dest) +def _generate_git_purl(d, download_location, srcrev): + """Generate a Package URL for a Git source from its download location. + + Parses the Git URL to identify the hosting service and generates the + appropriate PURL type. Supports github.com by default and custom + mappings via SPDX_GIT_PURL_MAPPINGS. + + Returns the PURL string or None if no mapping matches. + """ + if not download_location or not download_location.startswith('git+'): + return None + + git_url = download_location[4:] # Remove 'git+' prefix + + # Default handler: github.com + git_purl_handlers = { + 'github.com': 'pkg:github', + } + + # Custom PURL mappings from SPDX_GIT_PURL_MAPPINGS + # Format: "domain1:purl_type1 domain2:purl_type2" + custom_mappings = d.getVar('SPDX_GIT_PURL_MAPPINGS') + if custom_mappings: + for mapping in custom_mappings.split(): + parts = mapping.split(':', 1) + if len(parts) == 2: + git_purl_handlers[parts[0]] = parts[1] + bb.debug(2, f"Added custom Git PURL mapping: {parts[0]} -> {parts[1]}") + else: + bb.warn(f"Invalid SPDX_GIT_PURL_MAPPINGS entry: {mapping} (expected format: domain:purl_type)") + + try: + parsed = urllib.parse.urlparse(git_url) + except Exception: + return None + + hostname = parsed.hostname + if not hostname: + return None + + for domain, purl_type in git_purl_handlers.items(): + if hostname == domain: + path = parsed.path.strip('/') + path_parts = path.split('/') + if len(path_parts) >= 2: + owner = path_parts[0] + repo = path_parts[1].replace('.git', '') + return f"{purl_type}/{owner}/{repo}@{srcrev}" + break + + return None + + +def _enrich_source_package(d, dl, fd, file_name, primary_purpose): + """Enrich a source download package with version, PURL, and external refs. + + Extracts version from SRCREV for Git sources, generates PURLs for + known hosting services, and adds external references for VCS, + distribution URLs, and homepage. + """ + version = None + purl = None + + if fd.type == "git": + # Use full SHA-1 from fd.revision + srcrev = getattr(fd, 'revision', None) + if srcrev and srcrev not in {'${AUTOREV}', 'AUTOINC', 'INVALID'}: + version = srcrev + + # Generate PURL for Git hosting services + download_location = getattr(dl, 'software_downloadLocation', None) + if version and download_location: + purl = _generate_git_purl(d, download_location, version) + else: + # Use ecosystem PURL from SPDX_PACKAGE_URLS if available + package_urls = (d.getVar('SPDX_PACKAGE_URLS') or '').split() + for url in package_urls: + if not url.startswith('pkg:yocto'): + purl = url + break + + if version: + dl.software_packageVersion = version + + if purl: + dl.software_packageUrl = purl + + # Add external references + download_location = getattr(dl, 'software_downloadLocation', None) + if download_location and isinstance(download_location, str): + dl.externalRef = dl.externalRef or [] + + if download_location.startswith('git+'): + # VCS reference for Git repositories + git_url = download_location[4:] + if '@' in git_url: + git_url = git_url.split('@')[0] + + dl.externalRef.append( + oe.spdx30.ExternalRef( + externalRefType=oe.spdx30.ExternalRefType.vcs, + locator=[git_url], + ) + ) + elif download_location.startswith(('http://', 'https://', 'ftp://')): + # Distribution reference for tarball/archive downloads + dl.externalRef.append( + oe.spdx30.ExternalRef( + externalRefType=oe.spdx30.ExternalRefType.altDownloadLocation, + locator=[download_location], + ) + ) + + def add_download_files(d, objset): inputs = set() @@ -447,10 +562,14 @@ def add_download_files(d, objset): ) ) + _enrich_source_package(d, dl, fd, file_name, primary_purpose) + if fd.method.supports_checksum(fd): # TODO Need something better than hard coding this for checksum_id in ["sha256", "sha1"]: - expected_checksum = getattr(fd, "%s_expected" % checksum_id, None) + expected_checksum = getattr( + fd, "%s_expected" % checksum_id, None + ) if expected_checksum is None: continue @@ -506,7 +625,6 @@ def get_is_native(d): def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) pn = d.getVar("PN") license_data = oe.spdx_common.load_spdx_license_data(d) @@ -541,20 +659,6 @@ def create_recipe_spdx(d): set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) - # TODO: This doesn't work before do_unpack because the license text has to - # be available for recipes with NO_GENERIC_LICENSE - # recipe_spdx_license = add_license_expression( - # d, - # recipe_objset, - # d.getVar("LICENSE"), - # license_data, - # ) - # recipe_objset.new_relationship( - # [recipe], - # oe.spdx30.RelationshipType.hasDeclaredLicense, - # [oe.sbom30.get_element_link_id(recipe_spdx_license)], - # ) - if val := d.getVar("HOMEPAGE"): recipe.software_homePage = val @@ -588,7 +692,6 @@ def create_recipe_spdx(d): sorted(oe.sbom30.get_element_link_id(dep) for dep in dep_recipes), ) - # Add CVEs cve_by_status = {} if include_vex != "none": patched_cves = oe.cve_check.get_patched_cves(d) @@ -598,8 +701,6 @@ def create_recipe_spdx(d): description = patched_cve.get("justification", None) resources = patched_cve.get("resource", []) - # If this CVE is fixed upstream, skip it unless all CVEs are - # specified. if include_vex != "all" and detail in ( "fixed-version", "cpe-stable-backport", @@ -692,7 +793,6 @@ def create_recipe_spdx(d): def load_recipe_spdx(d): - return oe.sbom30.find_root_obj_in_jsonld( d, "static", @@ -717,10 +817,8 @@ def create_spdx(d): pn = d.getVar("PN") deploydir = Path(d.getVar("SPDXDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" - pkg_arch = d.getVar("SSTATE_PKGARCH") is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d) @@ -783,7 +881,6 @@ def create_spdx(d): dep_objsets, dep_builds = collect_dep_objsets( d, direct_deps, "builds", "build-", oe.spdx30.build_Build ) - if dep_builds: build_objset.new_scoped_relationship( [build], @@ -919,9 +1016,7 @@ def create_spdx(d): # Add concluded license relationship if manually set # Only add when license analysis has been explicitly performed - concluded_license_str = d.getVar( - "SPDX_CONCLUDED_LICENSE:%s" % package - ) or d.getVar("SPDX_CONCLUDED_LICENSE") + concluded_license_str = d.getVar("SPDX_CONCLUDED_LICENSE:%s" % package) or d.getVar("SPDX_CONCLUDED_LICENSE") if concluded_license_str: concluded_spdx_license = add_license_expression( d, build_objset, concluded_license_str, license_data @@ -1011,13 +1106,12 @@ def create_spdx(d): status = "enabled" if feature in enabled else "disabled" build.build_parameter.append( oe.spdx30.DictionaryEntry( - key=f"PACKAGECONFIG:{feature}", value=status + key=f"PACKAGECONFIG:{feature}", + value=status ) ) - bb.note( - f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled" - ) + bb.note(f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled") oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "builds", deploydir) @@ -1025,9 +1119,7 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") - providers = oe.spdx_common.collect_package_providers(d, direct_deps) pkg_arch = d.getVar("SSTATE_PKGARCH") @@ -1205,15 +1297,15 @@ def write_bitbake_spdx(d): def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None): import oe.sbom30 - direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") - + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_package_spdx") providers = oe.spdx_common.collect_package_providers(d, direct_deps) build_deps = set() + missing_providers = set() for name in sorted(packages.keys()): if name not in providers: - bb.note(f"Unable to find SPDX provider for '{name}'") + missing_providers.add(name) continue pkg_name, pkg_hashfn = providers[name] @@ -1232,6 +1324,11 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) for h, f in pkg_objset.by_sha256_hash.items(): files_by_hash.setdefault(h, set()).update(f) + if missing_providers: + bb.fatal( + f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}" + ) + if build_deps: objset.new_scoped_relationship( [build], @@ -1390,6 +1487,7 @@ def create_image_spdx(d): set_timestamp_now(d, a, "builtTime") + if artifacts: objset.new_scoped_relationship( [image_build], @@ -1583,10 +1681,3 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) - sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") - - recipe, recipe_objset = load_recipe_spdx(d) - - objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) - - oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json")) From patchwork Tue Mar 24 13:29:58 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Tondo X-Patchwork-Id: 84226 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7989F54ABA for ; Tue, 24 Mar 2026 13:30:27 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.19721.1774359018918558739 for ; Tue, 24 Mar 2026 06:30:19 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=EdbNYfl/; spf=pass (domain: gmail.com, ip: 209.85.128.48, mailfrom: stondo@gmail.com) Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-48704db565eso31051555e9.1 for ; Tue, 24 Mar 2026 06:30:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1774359017; x=1774963817; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5UZZzXQzurrs8J8UMbL8fHyl6r4+dsVbTu7T2utEzjo=; b=EdbNYfl/jspdITq50BeXZqZBDZgyKFB7hpSqQdhonRHGusb3rKjx9nHY0UDSU6nhjJ Vg991XY4whJAkLDt6y08e3qrOh5mSOEiYT5M4SvQB4jOktskBIGnwetbryRADY8EMmwF YcIOSmWo5WAiaoEhpojNvsYconhMd1NFawab8+VBgXq7ZDiCjRMY4ViOlqF20JS/1FIZ UZGn7b9vxynJ5wy1qRKVkleD6kWZb/dIpBELIIsF89cq6qFS0qirAXf8CwarcIs8lExZ Pj4zlMJLHrw40DHOgiYGrCOcYonrQkEfB4CUOaZFGotn6wnkc//o5/mPgzCEBo/g9oLV hDEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774359017; x=1774963817; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5UZZzXQzurrs8J8UMbL8fHyl6r4+dsVbTu7T2utEzjo=; b=l+40kx1gfSKHe6GdqtrC4LQUVW4lQUAkJm3J4AWXPZSV4wR8AFy+KCjw1LRTAe5H8T EEtAIIpJQEdv6wQiyQgDFu56UD7RX4zRqFavpHA1VaULzlUaTJJm+qMHBFa2ncAAD84S gJnCncNkX5NjpCKry0h8ibpm2LdGNKiYtdQdq5ic4Ok6lxRaMCU4C5X/cCtGphj5rns0 whTRIq+UchGY0loCh7mtiTdsc1RHJMJoBOe44q/HlvIN0NKOdA0lm28Wn039DmQce3oo AOwOX2EC1q4KLASKVcp3u7YPjeUjXJ1lk+TfIIYCAOCETzYQvwmfIfaZ6Bu/0SgLcQu7 1/Uw== X-Gm-Message-State: AOJu0YzW0duxptqVYS6sHIOJntvC26pLMnaSXWZrTqVTJYfZlI/Vz38J Sn0oNL5E76MHjVTixn+wJ/wG1HJp4aZX4q9SiaDFGYnySNbWopkC197Br34D0A8e X-Gm-Gg: ATEYQzzlQwvUmQc/e8EqHNoOasRC9lRVaHScoySzZT03yoUFn81MD03iWWtkXGaP2L+ D8/2rAP2GToszuXf5RtyMJOd056Lrlq34upMTKg0mGViG9T0LyaYoFkA2gtru1wVNoBMLqrKYVH Zp9jz30Muy9QfI1ZYsECczrex0HRY6TgqsF2tFHeAzBDocgmGQKNjrf8y18LLI5KNrfsy+bw4kJ vtFhRmDw5Hkj1pOG3ovnwWSTFcSCCGsSAYEVDYDFAde9CBspqRn0dV2KLSyBS22TV8eVFWh09f5 KamKOtuc+JpdiFrm0LixBHUA9sNqtzPdYZc7wL+JBBSPu9V1lro1ENbzr8/9TBwt+VUFjFuw8Go u3ynRjVS4zMJ7ydo8zsJmWXiOR4WSN0VOyVSh+VTxaEEXR/aicizI+Ghcc/hMXUiJXkVRxCWnnX PxwoLtdLWM2gcVgptWA8vQI3Di9+64lmZAv6+70sgrgjmKB0aBWSpWoUtdxqNaFpbacD1G9DQ9V UDgrrMj X-Received: by 2002:a05:600c:1c0a:b0:485:40c6:f528 with SMTP id 5b1f17b1804b1-486fee2bb38mr249454285e9.30.1774359016828; Tue, 24 Mar 2026 06:30:16 -0700 (PDT) Received: from fedora (mob-194-230-148-205.cgn.sunrise.net. [194.230.148.205]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48710fa0e35sm47494875e9.3.2026.03.24.06.30.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2026 06:30:16 -0700 (PDT) From: stondo@gmail.com To: openembedded-core@lists.openembedded.org Cc: richard.purdie@linuxfoundation.org, ross.burton@arm.com, jpewhacker@gmail.com, stefano.tondo.ext@siemens.com, peter.marko@siemens.com, adrian.freihofer@siemens.com, mathieu.dubois-briand@bootlin.com Subject: [OE-core][PATCH v14 4/4] oeqa/selftest: Add tests for source download enrichment Date: Tue, 24 Mar 2026 14:29:58 +0100 Message-ID: <20260324132958.2316491-5-stondo@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260324132958.2316491-1-stondo@gmail.com> References: <20260323210745.1337169-1-stefano.tondo.ext@siemens.com> <20260324132958.2316491-1-stondo@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Mar 2026 13:30:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/233801 From: Stefano Tondo Add two tests for the new source download SPDX features: test_download_location_defensive_handling: Verify that packages with no download location (e.g. packagegroups, images, virtual providers) are handled gracefully without crashing the SPDX generation pipeline. test_version_extraction_patterns: Verify that Git source packages get SRCREV as their version in the SPDX output, rather than the recipe PV. Signed-off-by: Stefano Tondo --- meta/lib/oeqa/selftest/cases/spdx.py | 76 ++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index af1144c1e5..9347e0bf7b 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -428,3 +428,79 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): value, ["enabled", "disabled"], f"Unexpected PACKAGECONFIG value '{value}' for {key}" ) + + def test_download_location_defensive_handling(self): + """Test that download_location handling is defensive. + + Verifies SPDX generation succeeds and external references are + properly structured when download_location retrieval works. + """ + objset = self.check_recipe_spdx( + "m4", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-m4.spdx.json", + ) + + found_external_refs = False + for pkg in objset.foreach_type(oe.spdx30.software_Package): + if pkg.externalRef: + found_external_refs = True + for ref in pkg.externalRef: + self.assertIsNotNone(ref.externalRefType) + self.assertIsNotNone(ref.locator) + self.assertGreater(len(ref.locator), 0, "Locator should have at least one entry") + for loc in ref.locator: + self.assertIsInstance(loc, str) + break + + self.logger.info( + f"External references {'found' if found_external_refs else 'not found'} " + f"in SPDX output (defensive handling verified)" + ) + + def test_version_extraction_patterns(self): + """Test that version extraction works for various package formats. + + Verifies that Git source downloads carry extracted versions and that + the reported version strings are well-formed. + """ + objset = self.check_recipe_spdx( + "opkg-utils", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-opkg-utils.spdx.json", + ) + + # Collect all packages with versions + packages_with_versions = [] + for pkg in objset.foreach_type(oe.spdx30.software_Package): + if pkg.software_packageVersion: + packages_with_versions.append((pkg.name, pkg.software_packageVersion)) + + self.assertGreater( + len(packages_with_versions), 0, + "Should find packages with extracted versions" + ) + + for name, version in packages_with_versions: + self.assertRegex( + version, + r"^[0-9a-f]{40}$", + f"Expected Git source version for {name} to be a full SHA-1", + ) + + self.logger.info(f"Found {len(packages_with_versions)} packages with versions") + + # Log some examples for debugging + for name, version in packages_with_versions[:5]: + self.logger.info(f" {name}: {version}") + + # Verify that versions follow expected patterns + for name, version in packages_with_versions: + # Version should not be empty + self.assertIsNotNone(version) + self.assertNotEqual(version, "") + + # Version should contain digits + self.assertRegex( + version, + r'\d', + f"Version '{version}' for package '{name}' should contain digits" + ) From patchwork Tue Mar 24 17:12:46 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Tondo X-Patchwork-Id: 84235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21B82FC72DF for ; Tue, 24 Mar 2026 17:14:55 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.2732.1774372493537378542 for ; Tue, 24 Mar 2026 10:14:53 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=gn8/2R72; spf=pass (domain: gmail.com, ip: 209.85.128.43, mailfrom: stondo@gmail.com) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-486fe2024a9so32780635e9.0 for ; Tue, 24 Mar 2026 10:14:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1774372492; x=1774977292; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lle3j5ejD55GWoR3QxdxmnApS09jZT3GXcZ5Cv6BXeo=; b=gn8/2R72XjORFZDtynNUXNx69e2+jlNfcior4t5bNiL21WXCUsEw+zNYtfk+S1awKn S+uqSOvjJlRjSA++zG9RjedunabpVYK3G8W+udoePHRHIdFEFm6MrWNqei76lc2/wQUr /NNnMFgFlgCGNKArLrMIY3mjW8R7Hdkg8ScLxYPJVFLp6ke8sc8knTkQEjlsHO48FOzM KVGKX9aLAFFkmUWoOtgDEyuRkIK04owLNEzkJZU5Xu5GmDPkjSpyD8H2qgTscGcgk2lM O8beVmiX4rjOw1/SsQKa6E/DbwAW7+pO4/MVckt8WRkz2iWjRz+DMc5bgfnVL03f4naP suzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774372492; x=1774977292; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=lle3j5ejD55GWoR3QxdxmnApS09jZT3GXcZ5Cv6BXeo=; b=eLruiJg0CfsiH7ln7maKl4wOiGwRhAxm54R03RcI5+O1hj3kH+pK0DeOX8vSK6ffj3 d2Q5kNt41De1dh/kUMxf24awZAjIP1Ytlm6AwD+V+qayRfpIBui6HFxpeH9Vp+elvvgy BBAWkAikhdhZ54nAuAobWO3Qkf+3IB7N6g29RNL8mNWXZMrXqSs4lrDN7o9vXrHnMqaZ ZbYUvMMzc9lZcliTLv3irrAGi/5xVfkGJOCI/8C0zoKArjbD097XfPn3tx+TsjR7y/1J p47yEooejRBKNgDKJcd0hh1IYrvrHqfWz0HE7+5Hfuk39BI4r29eiQnMqvP+x4MaXTFb 30/w== X-Gm-Message-State: AOJu0YxXDnYCulPGX5C2oNtuTAfp5sxZ83K+/Ag8ahBz2MU2vzhsvPXw eA5PiH8NyW75Q4vTRLQbQi9jc9Jin7tRl+TTBdVGhEBcsgCL6WiJ3OIvSxXdl8kZ X-Gm-Gg: ATEYQzxATxP4kYVMzD5SqQuDqPVdchr0LVLnZWlnY0QFVLOO+YwTsLnjXCkVE0iq3Zm 0i6ECQBqJp0aVB2ILi0wrGJtbD7xaF+Z+py5e7Ix2o5H+NaEpXYh8DmSnAkur2ptUMs8+i04rH0 qV/QxrRoJKyjJTsioIzui/E1leHr1ND30ZxCI5jnox9mI2jiWffgWZ/9BHBD8q8GyjqDOJw3ZWK mTseJB0sqMqDyCXIYPiVLr6ppw/avCOwSwjxnPfQPW9qdXtR4rTvJTzhctjpWDwprG7tHYsdaxQ Cfsq6gSUU2+bOBKnLRaj+CQkFE9JNgDxY2l4u1F+5j98ow00JXS3jPGoznFsBGzw4d5qu+tKZ2Q gCbXnmCU7Qv0uyFNOtBG0MUD2G5YScy4ng6Uuf1NLLYBxPgO0Sq31onAQ8ShVUUquuMOje58qXK TZHleYFbFFaWvDVsrIe6HVTDO7Szdf/O33LRKCLs2elGwacrqdfeqlBg23V1Yj7tVEbRuDCxxyp 4RmA6vs+IOZv1gh4DtkOtzx9w== X-Received: by 2002:a05:600c:190e:b0:486:ffa3:55f with SMTP id 5b1f17b1804b1-48716050973mr8630225e9.27.1774372491439; Tue, 24 Mar 2026 10:14:51 -0700 (PDT) Received: from fedora (mob-194-230-148-205.cgn.sunrise.net. [194.230.148.205]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-487165e8e0csm1089275e9.1.2026.03.24.10.14.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2026 10:14:50 -0700 (PDT) From: Stefano Tondo X-Google-Original-From: Stefano Tondo To: openembedded-core@lists.openembedded.org Cc: richard.purdie@linuxfoundation.org, ross.burton@arm.com, jpewhacker@gmail.com, stefano.tondo.ext@siemens.com, peter.marko@siemens.com, adrian.freihofer@siemens.com, mathieu.dubois-briand@bootlin.com Subject: [PATCH v16 5/5] oeqa/selftest: Add tests for source download enrichment Date: Tue, 24 Mar 2026 18:12:46 +0100 Message-ID: <20260324171246.686862-6-stefano.tondo.ext@siemens.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260324132958.2316491-1-stondo@gmail.com> References: <20260324132958.2316491-1-stondo@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Mar 2026 17:14:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/233813 Add two new test methods to SPDX30Check: test_download_location_defensive_handling: Builds m4 and verifies that SPDX generation succeeds and any external references present are properly structured with valid types and locator strings. test_version_extraction_patterns: Builds opkg-utils (a Git-based recipe) and verifies that source download packages carry the full SHA-1 commit hash as their software_packageVersion. Signed-off-by: Stefano Tondo --- meta/lib/oeqa/selftest/cases/spdx.py | 76 ++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index af1144c1e5..9347e0bf7b 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -428,3 +428,79 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): value, ["enabled", "disabled"], f"Unexpected PACKAGECONFIG value '{value}' for {key}" ) + + def test_download_location_defensive_handling(self): + """Test that download_location handling is defensive. + + Verifies SPDX generation succeeds and external references are + properly structured when download_location retrieval works. + """ + objset = self.check_recipe_spdx( + "m4", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-m4.spdx.json", + ) + + found_external_refs = False + for pkg in objset.foreach_type(oe.spdx30.software_Package): + if pkg.externalRef: + found_external_refs = True + for ref in pkg.externalRef: + self.assertIsNotNone(ref.externalRefType) + self.assertIsNotNone(ref.locator) + self.assertGreater(len(ref.locator), 0, "Locator should have at least one entry") + for loc in ref.locator: + self.assertIsInstance(loc, str) + break + + self.logger.info( + f"External references {'found' if found_external_refs else 'not found'} " + f"in SPDX output (defensive handling verified)" + ) + + def test_version_extraction_patterns(self): + """Test that version extraction works for various package formats. + + Verifies that Git source downloads carry extracted versions and that + the reported version strings are well-formed. + """ + objset = self.check_recipe_spdx( + "opkg-utils", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-opkg-utils.spdx.json", + ) + + # Collect all packages with versions + packages_with_versions = [] + for pkg in objset.foreach_type(oe.spdx30.software_Package): + if pkg.software_packageVersion: + packages_with_versions.append((pkg.name, pkg.software_packageVersion)) + + self.assertGreater( + len(packages_with_versions), 0, + "Should find packages with extracted versions" + ) + + for name, version in packages_with_versions: + self.assertRegex( + version, + r"^[0-9a-f]{40}$", + f"Expected Git source version for {name} to be a full SHA-1", + ) + + self.logger.info(f"Found {len(packages_with_versions)} packages with versions") + + # Log some examples for debugging + for name, version in packages_with_versions[:5]: + self.logger.info(f" {name}: {version}") + + # Verify that versions follow expected patterns + for name, version in packages_with_versions: + # Version should not be empty + self.assertIsNotNone(version) + self.assertNotEqual(version, "") + + # Version should contain digits + self.assertRegex( + version, + r'\d', + f"Version '{version}' for package '{name}' should contain digits" + )