From patchwork Sun Nov 10 03:07:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongxu Jia X-Patchwork-Id: 52256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 643D5D5E390 for ; Sun, 10 Nov 2024 03:07:50 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web10.30296.1731208064088616583 for ; Sat, 09 Nov 2024 19:07:44 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=10446b0cf6=hongxu.jia@windriver.com) Received: from pps.filterd (m0250809.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 4AA30C04012513; Sat, 9 Nov 2024 19:07:42 -0800 Received: from ala-exchng01.corp.ad.wrs.com (ala-exchng01.wrs.com [147.11.82.252]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 42t84prdk8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Sat, 09 Nov 2024 19:07:42 -0800 (PST) Received: from ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) by ala-exchng01.corp.ad.wrs.com (147.11.82.252) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sat, 9 Nov 2024 19:07:41 -0800 Received: from ala-lpggp7.wrs.com (147.11.136.210) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Sat, 9 Nov 2024 19:07:41 -0800 From: Hongxu Jia To: , Subject: [PATCH 1/3] sbom30.py: reduce redundant spdxid-hash symlinks to save inode on host Date: Sat, 9 Nov 2024 19:07:39 -0800 Message-ID: <20241110030741.4108407-1-hongxu.jia@windriver.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Authority-Analysis: v=2.4 cv=IrVMc6/g c=1 sm=1 tr=0 ts=6730237e cx=c_pps a=/ZJR302f846pc/tyiSlYyQ==:117 a=/ZJR302f846pc/tyiSlYyQ==:17 a=VlfZXiiP6vEA:10 a=24AZYWMyAAAA:8 a=t7CeM3EgAAAA:8 a=CDkeZTuGm8h2x8QjWH8A:9 a=bG88sKzkDEFeXWNnvthB:22 a=FdTzh2GWekK77mhwV6Dw:22 X-Proofpoint-ORIG-GUID: QXQZjLCUMt1z4rFaPfzYHiI9YpE76Jlz X-Proofpoint-GUID: QXQZjLCUMt1z4rFaPfzYHiI9YpE76Jlz X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-11-09_25,2024-11-08_01,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 clxscore=1015 malwarescore=0 priorityscore=1501 suspectscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 lowpriorityscore=0 impostorscore=0 classifier=spam authscore=0 adjust=0 reason=mlx scancount=1 engine=8.21.0-2409260000 definitions=main-2411100026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 10 Nov 2024 03:07:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206912 In order to support all in-scope SPDX data within a single JSON-LD file for SPDX 3.0.1, Yocto's SBOM: - In native/target/nativesdk recipe, created spdxid-hash symlink for each element to point to the JSON-LD file that contains element details; - In image recipe, use spdxid-hash symlink to collect element details from varies of JSON-LD files While SPDX_INCLUDE_SOURCES = "1", it adds sources to JSON-LD file and create 2N+ spdxid-hash symlinks for N source files. (N for software_File, N for hasDeclaredLicense's Relationship) For large numbers of source files, adding an extra symlink -> real file will occupy one more inode (per file), which will need a slot in the OS's inode cache. In this situation, disk performance is slow and inode is used up quickly While using function add_package_files to add source files to JSON-LD file, the spdxid-hash symlinks for source files point to the same JSON-LD file, then according to the format of spdxId - spdxId of souce file: http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/0838759b8d71923d250a0813dda7356ffd309576115bbf8ed7e266cf4aed86a5/sourcefile/1 Remove the count number ('/1') from spdxId suffix, then all source files in one recipe will share one spdxid-hash symlink. The same reason to sysroot and package files - spdxId of sysroot file: http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/0838759b8d71923d250a0813dda7356ffd309576115bbf8ed7e266cf4aed86a5/sysroot/1 - spdxId of pacakge file: http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/0838759b8d71923d250a0813dda7356ffd309576115bbf8ed7e266cf4aed86a5/package/shadow-src/file/1 Build core-image-minimal with/without this commit, comparing the spdxid-hash number, 7 281 824 -> 70 508 echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf With this commit: $ time bitbake core-image-minimal real 95m6.960s user 0m22.832s sys 0m4.087s $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash/ -name "*.spdx.json" |wc -l 70508 Without this commit: $ time bitbake core-image-minimal real 100m17.769s user 0m24.516s sys 0m4.334s $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l 7281824 Signed-off-by: Hongxu Jia --- meta/lib/oe/sbom30.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index e3a9428668..4efeaae3a0 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -911,6 +911,10 @@ def jsonld_arch_path(d, arch, subdir, name, deploydir=None): def jsonld_hash_path(_id): + # For the spdId added by add_package_files, remove suffix count number + if re.match(r".*/(sourcefile|sysroot|file)/\w+$", _id): + _id = os.path.dirname(_id) + h = hashlib.sha256(_id.encode("utf-8")).hexdigest() return Path("by-spdxid-hash") / h[:2], h @@ -992,6 +996,11 @@ def write_recipe_jsonld_doc( *hash_path, deploydir=deploydir, ) + + # Return if expected symlink exists + if link_name.is_symlink() and link_name.resolve() == dest: + return hash_path[-1] + try: link_name.parent.mkdir(exist_ok=True, parents=True) link_name.symlink_to(os.path.relpath(dest, link_name.parent)) From patchwork Sun Nov 10 03:07:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongxu Jia X-Patchwork-Id: 52257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AD17D5E392 for ; Sun, 10 Nov 2024 03:07:50 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.30167.1731208064053570605 for ; Sat, 09 Nov 2024 19:07:44 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=10446b0cf6=hongxu.jia@windriver.com) Received: from pps.filterd (m0250809.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 4AA30C05012513; Sat, 9 Nov 2024 19:07:42 -0800 Received: from ala-exchng01.corp.ad.wrs.com (ala-exchng01.wrs.com [147.11.82.252]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 42t84prdk8-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Sat, 09 Nov 2024 19:07:42 -0800 (PST) Received: from ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) by ala-exchng01.corp.ad.wrs.com (147.11.82.252) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sat, 9 Nov 2024 19:07:41 -0800 Received: from ala-lpggp7.wrs.com (147.11.136.210) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Sat, 9 Nov 2024 19:07:41 -0800 From: Hongxu Jia To: , Subject: [PATCH 2/3] sbom30.py: use file-license-relationship as spdxid_name of hasDeclaredLicense relationship Date: Sat, 9 Nov 2024 19:07:40 -0800 Message-ID: <20241110030741.4108407-2-hongxu.jia@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241110030741.4108407-1-hongxu.jia@windriver.com> References: <20241110030741.4108407-1-hongxu.jia@windriver.com> MIME-Version: 1.0 X-Authority-Analysis: v=2.4 cv=IrVMc6/g c=1 sm=1 tr=0 ts=6730237e cx=c_pps a=/ZJR302f846pc/tyiSlYyQ==:117 a=/ZJR302f846pc/tyiSlYyQ==:17 a=VlfZXiiP6vEA:10 a=24AZYWMyAAAA:8 a=t7CeM3EgAAAA:8 a=EHEOu2Xr1FpDUlZMKjkA:9 a=bG88sKzkDEFeXWNnvthB:22 a=FdTzh2GWekK77mhwV6Dw:22 X-Proofpoint-ORIG-GUID: 8mt3IKimH_5tCh-boe2ScYqIk8AxBtkV X-Proofpoint-GUID: 8mt3IKimH_5tCh-boe2ScYqIk8AxBtkV X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-11-09_25,2024-11-08_01,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 clxscore=1015 malwarescore=0 priorityscore=1501 suspectscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=822 adultscore=0 lowpriorityscore=0 impostorscore=0 classifier=spam authscore=0 adjust=0 reason=mlx scancount=1 engine=8.21.0-2409260000 definitions=main-2411100026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 10 Nov 2024 03:07:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206913 In order to distinguish the spdxId of hasDeclaredLicense relationship with other relationship, use file-license-relationship as spdxid_name, and filter it out for spdxid-hash symlink creation While SPDX_INCLUDE_SOURCES = "1", N hasDeclaredLicense's Relationship for N source files in one recipe will share one spdxid-hash symlink - spdxId of hasDeclaredLicense relationship: http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/0838759b8d71923d250a0813dda7356ffd309576115bbf8ed7e266cf4aed86a5/file-license-relationship/00395f9e72c7f872e29a4a5e6a22981e - spdxId of other relationship: http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/0838759b8d71923d250a0813dda7356ffd309576115bbf8ed7e266cf4aed86a5/relationship/1808c755cc8a41ddaa39d708a4de1022 Signed-off-by: Hongxu Jia --- meta/lib/oe/sbom30.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 4efeaae3a0..64b357ea92 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -548,8 +548,8 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): return ret - def new_relationship(self, from_, typ, to): - return self._new_relationship(oe.spdx30.Relationship, from_, typ, to) + def new_relationship(self, from_, typ, to, spdxid_name="relationship"): + return self._new_relationship(oe.spdx30.Relationship, from_, typ, to, spdxid_name=spdxid_name) def new_scoped_relationship(self, from_, typ, scope, to): return self._new_relationship( @@ -624,6 +624,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): [spdx_file], oe.spdx30.RelationshipType.hasDeclaredLicense, file_licenses, + spdxid_name="file-license-relationship" ) spdx_file.extension.append(OELicenseScannedExtension()) @@ -912,7 +913,7 @@ def jsonld_arch_path(d, arch, subdir, name, deploydir=None): def jsonld_hash_path(_id): # For the spdId added by add_package_files, remove suffix count number - if re.match(r".*/(sourcefile|sysroot|file)/\w+$", _id): + if re.match(r".*/(sourcefile|sysroot|file|file-license-relationship)/\w+$", _id): _id = os.path.dirname(_id) h = hashlib.sha256(_id.encode("utf-8")).hexdigest() From patchwork Sun Nov 10 03:07:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongxu Jia X-Patchwork-Id: 52255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 675CFD5E391 for ; Sun, 10 Nov 2024 03:07:50 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.30168.1731208064713092567 for ; Sat, 09 Nov 2024 19:07:44 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=10446b0cf6=hongxu.jia@windriver.com) Received: from pps.filterd (m0250809.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 4AA30C06012513; Sat, 9 Nov 2024 19:07:43 -0800 Received: from ala-exchng01.corp.ad.wrs.com (ala-exchng01.wrs.com [147.11.82.252]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 42t84prdk8-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Sat, 09 Nov 2024 19:07:43 -0800 (PST) Received: from ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) by ala-exchng01.corp.ad.wrs.com (147.11.82.252) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sat, 9 Nov 2024 19:07:41 -0800 Received: from ala-lpggp7.wrs.com (147.11.136.210) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Sat, 9 Nov 2024 19:07:41 -0800 From: Hongxu Jia To: , Subject: [PATCH 3/3] oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal build Date: Sat, 9 Nov 2024 19:07:41 -0800 Message-ID: <20241110030741.4108407-3-hongxu.jia@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241110030741.4108407-1-hongxu.jia@windriver.com> References: <20241110030741.4108407-1-hongxu.jia@windriver.com> MIME-Version: 1.0 X-Authority-Analysis: v=2.4 cv=IrVMc6/g c=1 sm=1 tr=0 ts=6730237f cx=c_pps a=/ZJR302f846pc/tyiSlYyQ==:117 a=/ZJR302f846pc/tyiSlYyQ==:17 a=VlfZXiiP6vEA:10 a=t7CeM3EgAAAA:8 a=X7M7THNEruu-k4-VEk8A:9 a=FdTzh2GWekK77mhwV6Dw:22 X-Proofpoint-ORIG-GUID: 9Fzfds3lXxXEB3_fVoN7mT7MEAtJ_Eui X-Proofpoint-GUID: 9Fzfds3lXxXEB3_fVoN7mT7MEAtJ_Eui X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-11-09_25,2024-11-08_01,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 clxscore=1015 malwarescore=0 priorityscore=1501 suspectscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 lowpriorityscore=0 impostorscore=0 classifier=spam authscore=0 adjust=0 reason=mlx scancount=1 engine=8.21.0-2409260000 definitions=main-2411100026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 10 Nov 2024 03:07:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206914 $ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source 2024-11-09 09:17:54,600 - oe-selftest - INFO - Adding layer libraries: 2024-11-09 09:17:54,601 - oe-selftest - INFO - path-to/poky/meta/lib 2024-11-09 09:17:54,601 - oe-selftest - INFO - path-to/poky/meta-yocto-bsp/lib 2024-11-09 09:17:54,601 - oe-selftest - INFO - path-to/poky/meta-selftest/lib 2024-11-09 09:17:54,601 - oe-selftest - INFO - path-to/meta-openembedded/meta-oe/lib 2024-11-09 09:17:54,602 - oe-selftest - INFO - Checking base configuration is valid/parsable 2024-11-09 09:17:56,653 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf 2024-11-09 09:17:56,653 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2024-11-09 09:17:56,653 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source) 2024-11-09 10:41:16,654 - oe-selftest - INFO - Keepalive message 2024-11-09 11:37:53,091 - oe-selftest - INFO - ... ok 2024-11-09 11:55:18,638 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-11-09 11:55:18,638 - oe-selftest - INFO - Ran 1 test in 9442.187s 2024-11-09 11:55:18,638 - oe-selftest - INFO - OK 2024-11-09 11:55:35,453 - oe-selftest - INFO - RESULTS: 2024-11-09 11:55:35,453 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (8396.65s) 2024-11-09 11:55:35,490 - oe-selftest - INFO - SUMMARY: 2024-11-09 11:55:35,490 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 9442.187s 2024-11-09 11:55:35,490 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) Signed-off-by: Hongxu Jia --- meta/lib/oeqa/selftest/cases/spdx.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index 8384070219..c785f5445f 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -174,6 +174,20 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): # Document should be fully linked self.check_objset_missing_ids(objset) + def test_core_image_minimal_include_source(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json", + extraconf=textwrap.dedent( + """\ + SPDX_INCLUDE_SOURCES = "1" + """ + ), + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + def test_core_image_minimal_sdk(self): objset = self.check_recipe_spdx( "core-image-minimal",