From patchwork Sat Oct 26 03:25:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongxu Jia X-Patchwork-Id: 51340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A21A5D11706 for ; Sat, 26 Oct 2024 03:25:32 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.4378.1729913123299613067 for ; Fri, 25 Oct 2024 20:25:23 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=002900eeaf=hongxu.jia@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49Q3OaKG000751; Fri, 25 Oct 2024 20:25:21 -0700 Received: from ala-exchng02.corp.ad.wrs.com (ala-exchng02.wrs.com [147.11.82.254]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 42c8240wrm-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 25 Oct 2024 20:25:21 -0700 (PDT) Received: from ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 25 Oct 2024 20:25:19 -0700 Received: from ala-lpggp7.wrs.com (147.11.136.210) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 25 Oct 2024 20:25:19 -0700 From: Hongxu Jia To: CC: Subject: [oe-core][PATCH 1/6] meta/lib/oe/sbom30.py: correct python list comprehension Date: Fri, 25 Oct 2024 20:25:14 -0700 Message-ID: <20241026032519.1968518-2-hongxu.jia@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241026032519.1968518-1-hongxu.jia@windriver.com> References: <20241026032519.1968518-1-hongxu.jia@windriver.com> MIME-Version: 1.0 X-Authority-Analysis: v=2.4 cv=UrgxNPwB c=1 sm=1 tr=0 ts=671c6121 cx=c_pps a=K4BcnWQioVPsTJd46EJO2w==:117 a=K4BcnWQioVPsTJd46EJO2w==:17 a=DAUX931o1VcA:10 a=24AZYWMyAAAA:8 a=t7CeM3EgAAAA:8 a=kB6j27Rbq3LGHQ-lY6oA:9 a=bG88sKzkDEFeXWNnvthB:22 a=FdTzh2GWekK77mhwV6Dw:22 X-Proofpoint-ORIG-GUID: ZGTaQhdgKD4r4ZKZEHpGOmm4YTTrEYJJ X-Proofpoint-GUID: ZGTaQhdgKD4r4ZKZEHpGOmm4YTTrEYJJ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-25_14,2024-10-25_02,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 bulkscore=0 impostorscore=0 spamscore=0 clxscore=1015 adultscore=0 mlxlogscore=933 priorityscore=1501 lowpriorityscore=0 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam authscore=0 adjust=0 reason=mlx scancount=1 engine=8.21.0-2409260000 definitions=main-2410260026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Oct 2024 03:25:32 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206380 The python list comprehension is not right for list: >>> license_text_map = {'LicenseRef-FSF-Unlimited': 'http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/8d31e22acc4a8979f24dc24042692fb548fc8fc8d85d775ddac406abb122ceea/license-text/FSF-Unlimited'} >>> license_text = ((k, license_text_map[k]) for k in sorted(license_text_map.keys())) >>> print(license_text) at 0x7f8575173270> >>> [(k,v) for k, v in license_text] [] After apply this commit, as expected >>> license_text = [] >>> for k in sorted(license_text_map.keys()): ... license_text.append((k, license_text_map[k])) ... >>> license_text [('LicenseRef-FSF-Unlimited', 'http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/8d31e22acc4a8979f24dc24042692fb548fc8fc8d85d775ddac406abb122ceea/license-text/FSF-Unlimited')] Signed-off-by: Hongxu Jia --- meta/lib/oe/sbom30.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 27ab5e45ac..73a957a20c 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -577,9 +577,9 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): re.sub(r"[^a-zA-Z0-9_-]", "_", license_expression), ] - license_text = ( - (k, license_text_map[k]) for k in sorted(license_text_map.keys()) - ) + license_text = [] + for k in sorted(license_text_map.keys()): + license_text.append((k, license_text_map[k])) if not license_text: lic = self.find_filter( From patchwork Sat Oct 26 03:25:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongxu Jia X-Patchwork-Id: 51344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4844D10BE8 for ; Sat, 26 Oct 2024 03:25:32 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.4380.1729913124587639262 for ; Fri, 25 Oct 2024 20:25:24 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=002900eeaf=hongxu.jia@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49Q3OaKH000751; Fri, 25 Oct 2024 20:25:21 -0700 Received: from ala-exchng02.corp.ad.wrs.com (ala-exchng02.wrs.com [147.11.82.254]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 42c8240wrm-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 25 Oct 2024 20:25:21 -0700 (PDT) Received: from ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 25 Oct 2024 20:25:19 -0700 Received: from ala-lpggp7.wrs.com (147.11.136.210) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 25 Oct 2024 20:25:19 -0700 From: Hongxu Jia To: CC: Subject: [oe-core][PATCH 2/6] meta/lib/oe/sbom30.py: correct typo Date: Fri, 25 Oct 2024 20:25:15 -0700 Message-ID: <20241026032519.1968518-3-hongxu.jia@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241026032519.1968518-1-hongxu.jia@windriver.com> References: <20241026032519.1968518-1-hongxu.jia@windriver.com> MIME-Version: 1.0 X-Authority-Analysis: v=2.4 cv=UrgxNPwB c=1 sm=1 tr=0 ts=671c6121 cx=c_pps a=K4BcnWQioVPsTJd46EJO2w==:117 a=K4BcnWQioVPsTJd46EJO2w==:17 a=DAUX931o1VcA:10 a=t7CeM3EgAAAA:8 a=uo5YQZFxS8OF35rn94wA:9 a=FdTzh2GWekK77mhwV6Dw:22 X-Proofpoint-ORIG-GUID: IhdN_VMcXioQ9fJCVm3jdvKoE8XktKtv X-Proofpoint-GUID: IhdN_VMcXioQ9fJCVm3jdvKoE8XktKtv X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-25_14,2024-10-25_02,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 bulkscore=0 impostorscore=0 spamscore=0 clxscore=1015 adultscore=0 mlxlogscore=895 priorityscore=1501 lowpriorityscore=0 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam authscore=0 adjust=0 reason=mlx scancount=1 engine=8.21.0-2409260000 definitions=main-2410260026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Oct 2024 03:25:32 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206383 The isinstance expected 2 arguments Signed-off-by: Hongxu Jia --- meta/lib/oe/sbom30.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 73a957a20c..f4cc1f49dd 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -631,7 +631,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): sha256_hash = bb.utils.sha256_file(path) for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(oe.spdx30.software_File): + if not isinstance(f, oe.spdx30.software_File): continue if purposes: From patchwork Sat Oct 26 03:25:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongxu Jia X-Patchwork-Id: 51343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B78E6D10BE3 for ; Sat, 26 Oct 2024 03:25:32 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.4379.1729913123808476961 for ; Fri, 25 Oct 2024 20:25:23 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=002900eeaf=hongxu.jia@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49Q3OaKI000751; Fri, 25 Oct 2024 20:25:22 -0700 Received: from ala-exchng02.corp.ad.wrs.com (ala-exchng02.wrs.com [147.11.82.254]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 42c8240wrm-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 25 Oct 2024 20:25:21 -0700 (PDT) Received: from ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 25 Oct 2024 20:25:19 -0700 Received: from ala-lpggp7.wrs.com (147.11.136.210) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 25 Oct 2024 20:25:19 -0700 From: Hongxu Jia To: CC: Subject: [oe-core][PATCH 3/6] meta/lib/oe/sbom30.py: create hasDeclaredLicense relationship conditionally Date: Fri, 25 Oct 2024 20:25:16 -0700 Message-ID: <20241026032519.1968518-4-hongxu.jia@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241026032519.1968518-1-hongxu.jia@windriver.com> References: <20241026032519.1968518-1-hongxu.jia@windriver.com> MIME-Version: 1.0 X-Authority-Analysis: v=2.4 cv=UrgxNPwB c=1 sm=1 tr=0 ts=671c6122 cx=c_pps a=K4BcnWQioVPsTJd46EJO2w==:117 a=K4BcnWQioVPsTJd46EJO2w==:17 a=DAUX931o1VcA:10 a=24AZYWMyAAAA:8 a=t7CeM3EgAAAA:8 a=R4ATSylRSYF2y5OD84UA:9 a=bG88sKzkDEFeXWNnvthB:22 a=FdTzh2GWekK77mhwV6Dw:22 X-Proofpoint-ORIG-GUID: wc31tVtEGSlzBC-Re_0uxriEucNTaM2u X-Proofpoint-GUID: wc31tVtEGSlzBC-Re_0uxriEucNTaM2u X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-25_14,2024-10-25_02,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 bulkscore=0 impostorscore=0 spamscore=0 clxscore=1015 adultscore=0 mlxlogscore=733 priorityscore=1501 lowpriorityscore=0 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam authscore=0 adjust=0 reason=mlx scancount=1 engine=8.21.0-2409260000 definitions=main-2410260026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Oct 2024 03:25:32 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206381 If file_licenses is empty, the hasDeclaredLicense relationship from sourcefile to NoneElement which makes no sense. Such as { "type": "Relationship", ... "from": "http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb/sourcefile/3323", "relationshipType": "hasDeclaredLicense", "to": [ "NoneElement" ] }, Create hasDeclaredLicense relationship only if file_licenses is not empty Signed-off-by: Hongxu Jia --- meta/lib/oe/sbom30.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index f4cc1f49dd..6df5759596 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -620,11 +620,12 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): for extracted_lic in oe.spdx_common.extract_licenses(filepath): file_licenses.add(self.new_license_expression(extracted_lic, license_data)) - self.new_relationship( - [spdx_file], - oe.spdx30.RelationshipType.hasDeclaredLicense, - file_licenses, - ) + if file_licenses: + self.new_relationship( + [spdx_file], + oe.spdx30.RelationshipType.hasDeclaredLicense, + file_licenses, + ) spdx_file.extension.append(OELicenseScannedExtension()) def new_file(self, _id, name, path, *, purposes=[]): From patchwork Sat Oct 26 03:25:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongxu Jia X-Patchwork-Id: 51342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B850BD10BE4 for ; Sat, 26 Oct 2024 03:25:32 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.4382.1729913125363708125 for ; Fri, 25 Oct 2024 20:25:25 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=002900eeaf=hongxu.jia@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49Q3OaKJ000751; Fri, 25 Oct 2024 20:25:22 -0700 Received: from ala-exchng02.corp.ad.wrs.com (ala-exchng02.wrs.com [147.11.82.254]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 42c8240wrm-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 25 Oct 2024 20:25:22 -0700 (PDT) Received: from ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 25 Oct 2024 20:25:20 -0700 Received: from ala-lpggp7.wrs.com (147.11.136.210) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 25 Oct 2024 20:25:20 -0700 From: Hongxu Jia To: CC: Subject: [oe-core][PATCH 4/6] meta/lib/oe/spdx30_tasks.py: improve debug log in add_package_files Date: Fri, 25 Oct 2024 20:25:17 -0700 Message-ID: <20241026032519.1968518-5-hongxu.jia@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241026032519.1968518-1-hongxu.jia@windriver.com> References: <20241026032519.1968518-1-hongxu.jia@windriver.com> MIME-Version: 1.0 X-Authority-Analysis: v=2.4 cv=UrgxNPwB c=1 sm=1 tr=0 ts=671c6122 cx=c_pps a=K4BcnWQioVPsTJd46EJO2w==:117 a=K4BcnWQioVPsTJd46EJO2w==:17 a=DAUX931o1VcA:10 a=24AZYWMyAAAA:8 a=iGHA9ds3AAAA:8 a=t7CeM3EgAAAA:8 a=O3SGjoYcfhoxR4A708MA:9 a=aAh4nC1LCUkA:10 a=bG88sKzkDEFeXWNnvthB:22 a=nM-MV4yxpKKO9kiQg6Ot:22 a=FdTzh2GWekK77mhwV6Dw:22 X-Proofpoint-ORIG-GUID: geB5D1_wp0wOywip20pFCq9m4q2Ea_nD X-Proofpoint-GUID: geB5D1_wp0wOywip20pFCq9m4q2Ea_nD X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-25_14,2024-10-25_02,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 bulkscore=0 impostorscore=0 spamscore=0 clxscore=1015 adultscore=0 mlxlogscore=999 priorityscore=1501 lowpriorityscore=0 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam authscore=0 adjust=0 reason=mlx scancount=1 engine=8.21.0-2409260000 definitions=main-2410260026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Oct 2024 03:25:32 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206385 While SPDX_INCLUDE_SOURCES = "1" [1], there are mess of `Adding file' in debug log ''' DEBUG: Adding file tmp/work/x86_64-linux/gettext-minimal-native/0.22.5/spdx/3.0.1/work/sources-unpack/COPYING to http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/e2c2366654a818397af8b8ddb45fda88c2c71aa2d71695861f82376a658d8e66/document/gettext-minimal-native DEBUG: Adding file tmp/work/x86_64-linux/gettext-minimal-native/0.22.5/spdx/3.0.1/work/gettext-0.22.5/.tarball-version to http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/e2c2366654a818397af8b8ddb45fda88c2c71aa2d71695861f82376a658d8e66/document/gettext-minimal-native ''' Summary the total number other than print for each file. ''' DEBUG: Start adding files to http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/f5e0e04913ac4c595be791fc001d545a77519ed6ee8c743deef721ca0898bc94/document/gettext-minimal-native DEBUG: Added 7201 files to http://spdx.org/spdxdocs/gettext-minimal-native-1fa0d5cb-2bb8-5631-9fab-cd219801733f/f5e0e04913ac4c595be791fc001d545a77519ed6ee8c743deef721ca0898bc94/document/gettext-minimal-native ''' Correct file_counter to start with 0 [1] https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SPDX_INCLUDE_SOURCES Signed-off-by: Hongxu Jia --- meta/lib/oe/spdx30_tasks.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index e0b656d81f..575b66ea5d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -144,19 +144,19 @@ def add_package_files( spdx_files = set() - file_counter = 1 + bb.debug(1, "Start adding files to %s" % (objset.doc._id)) + file_counter = 0 for subdir, dirs, files in os.walk(topdir): dirs[:] = [d for d in dirs if d not in ignore_dirs] if subdir == str(topdir): dirs[:] = [d for d in dirs if d not in ignore_top_level_dirs] + for file in files: filepath = Path(subdir) / file if filepath.is_symlink() or not filepath.is_file(): continue - bb.debug(1, "Adding file %s to %s" % (filepath, objset.doc._id)) - filename = str(filepath.relative_to(topdir)) file_purposes = get_purposes(filepath) @@ -187,6 +187,8 @@ def add_package_files( file_counter += 1 + bb.debug(1, "Added %d files to %s" % (file_counter, objset.doc._id)) + return spdx_files From patchwork Sat Oct 26 03:25:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongxu Jia X-Patchwork-Id: 51345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4175D10BE7 for ; Sat, 26 Oct 2024 03:25:32 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web10.4422.1729913124561061051 for ; Fri, 25 Oct 2024 20:25:24 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=002900eeaf=hongxu.jia@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49Q3OaKK000751; Fri, 25 Oct 2024 20:25:23 -0700 Received: from ala-exchng02.corp.ad.wrs.com (ala-exchng02.wrs.com [147.11.82.254]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 42c8240wrm-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 25 Oct 2024 20:25:22 -0700 (PDT) Received: from ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 25 Oct 2024 20:25:20 -0700 Received: from ala-lpggp7.wrs.com (147.11.136.210) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 25 Oct 2024 20:25:20 -0700 From: Hongxu Jia To: CC: Subject: [oe-core][PATCH 5/6] create-spdx-{2.2,3.0}: support SPDX include source for work-share directory Date: Fri, 25 Oct 2024 20:25:18 -0700 Message-ID: <20241026032519.1968518-6-hongxu.jia@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241026032519.1968518-1-hongxu.jia@windriver.com> References: <20241026032519.1968518-1-hongxu.jia@windriver.com> MIME-Version: 1.0 X-Authority-Analysis: v=2.4 cv=UrgxNPwB c=1 sm=1 tr=0 ts=671c6122 cx=c_pps a=K4BcnWQioVPsTJd46EJO2w==:117 a=K4BcnWQioVPsTJd46EJO2w==:17 a=DAUX931o1VcA:10 a=iGHA9ds3AAAA:8 a=t7CeM3EgAAAA:8 a=FOIc7NzRO18f_obxkkgA:9 a=aAh4nC1LCUkA:10 a=nM-MV4yxpKKO9kiQg6Ot:22 a=FdTzh2GWekK77mhwV6Dw:22 X-Proofpoint-ORIG-GUID: VLzZtMJ5l_PZTWnxtW4zxQ7ZC-bItblF X-Proofpoint-GUID: VLzZtMJ5l_PZTWnxtW4zxQ7ZC-bItblF X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-25_14,2024-10-25_02,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 bulkscore=0 impostorscore=0 spamscore=0 clxscore=1015 adultscore=0 mlxlogscore=999 priorityscore=1501 lowpriorityscore=0 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam authscore=0 adjust=0 reason=mlx scancount=1 engine=8.21.0-2409260000 definitions=main-2410260026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Oct 2024 03:25:32 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206382 Originally, while SPDX_INCLUDE_SOURCES = "1" [1], there is bug in scan for gcc, libgcc in which the sources locates in work-share directory. Copy source from ${WORKDIR} to ${SPDXWORK} did not satisfy the situation while ${S} was not included in ${WORKDIR} This commit aim to support SPDX include source for work-share directory 1. If is_work_shared_spdx, Copy source from ${S} to ${SPDXWORK}, normally the dest dir in ${SPDXWORK} has the same basename dir of ${S}; but for kernel source, rename basename dir 'kernel-source' to ${BP} (${BPN}-${PV}) 2. For SPDX source copy, do hard link copy to save copy time 3. Move do_patch to no work shared situation along with do_unpack 4. Tweak task do_create_spdx dependencies to assure the patched source in work share is ready for SPDX source copy 5. Remove bb.data.inherits_class('kernel', d) from is_work_shared_spdx, the kernel source locates in 'work-shared', test kernel.bbclass is not necessary [1] https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SPDX_INCLUDE_SOURCES Signed-off-by: Hongxu Jia --- meta/classes/create-spdx-2.2.bbclass | 20 ++++++++++++++ meta/classes/create-spdx-3.0.bbclass | 19 +++++++++++++ meta/lib/oe/spdx_common.py | 41 +++++++++++----------------- 3 files changed, 55 insertions(+), 25 deletions(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index cd1d6819bf..900595a7e9 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -947,3 +947,23 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx tar.addfile(info, fileobj=index_str) combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SPDX_MULTILIB_SSTATE_ARCHS" + +python () { + import oe.spdx_common + + # Assure the patched source in work share is ready for SPDX source copy + if oe.spdx_common.is_work_shared_spdx(d) and \ + d.getVar("SPDX_INCLUDE_SOURCES") == "1" and \ + oe.spdx_common.process_sources(d): + pn = d.getVar('PN') + # There is a corner case with "gcc-source-${PV}" recipes, they use "do_preconfigure" + # to modify source file after do_patch + if oe.spdx_common.has_task(d, "do_preconfigure"): + d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_preconfigure' % pn) + elif oe.spdx_common.has_task(d, "do_patch"): + d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_patch' % pn) + # There is a corner case with "gcc-cross-x86_64" recipes, it has "do_configure" + # but no do_patch + elif oe.spdx_common.has_task(d, "do_configure"): + d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_configure' % pn) +} diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 5f0590198f..284e0ff91c 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -190,3 +190,22 @@ python spdx30_build_started_handler () { addhandler spdx30_build_started_handler spdx30_build_started_handler[eventmask] = "bb.event.BuildStarted" +python () { + import oe.spdx_common + + # Assure the patched source in work share is ready for SPDX source copy + if oe.spdx_common.is_work_shared_spdx(d) and \ + d.getVar("SPDX_INCLUDE_SOURCES") == "1" and \ + oe.spdx_common.process_sources(d): + pn = d.getVar('PN') + # There is a corner case with "gcc-source-${PV}" recipes, they use "do_preconfigure" + # to modify source file after do_patch + if oe.spdx_common.has_task(d, "do_preconfigure"): + d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_preconfigure' % pn) + elif oe.spdx_common.has_task(d, "do_patch"): + d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_patch' % pn) + # There is a corner case with "gcc-cross-x86_64" recipes, it has "do_configure" + # but no do_patch + elif oe.spdx_common.has_task(d, "do_configure"): + d.appendVarFlag('do_create_spdx', 'depends', ' %s:do_configure' % pn) +} diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 1ea55419ae..ab037b4088 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -38,7 +38,7 @@ def extract_licenses(filename): def is_work_shared_spdx(d): - return bb.data.inherits_class("kernel", d) or ("work-shared" in d.getVar("WORKDIR")) + return 'work-shared' in d.getVar('S') def load_spdx_license_data(d): @@ -74,11 +74,6 @@ def process_sources(d): if d.getVar("PN") == "shadow-sysroot": return False - # We just archive gcc-source for all the gcc related recipes - if d.getVar("BPN") in ["gcc", "libgcc"]: - bb.debug(1, "spdx: There is bug in scan of %s is, do nothing" % pn) - return False - return True @@ -190,34 +185,28 @@ def get_patched_src(d): bb.utils.mkdirhier(d.getVar("B")) bb.build.exec_func("do_unpack", d) - # Copy source of kernel to spdx_workdir + + if d.getVar("SRC_URI") != "": + bb.build.exec_func("do_patch", d) + + # Copy source from work-share to spdx_workdir if is_work_shared_spdx(d): - share_src = d.getVar("WORKDIR") + share_src = d.getVar('S') d.setVar("WORKDIR", spdx_workdir) d.setVar("STAGING_DIR_NATIVE", spdx_sysroot_native) + # Copy source to ${SPDXWORK}, same basename dir of ${S}; src_dir = ( spdx_workdir + "/" - + d.getVar("PN") - + "-" - + d.getVar("PV") - + "-" - + d.getVar("PR") + + os.path.basename(share_src) ) - bb.utils.mkdirhier(src_dir) + # For kernel souce, rename suffix dir 'kernel-source' + # to ${BP} (${BPN}-${PV}) if bb.data.inherits_class("kernel", d): - share_src = d.getVar("STAGING_KERNEL_DIR") - cmd_copy_share = "cp -rf " + share_src + "/* " + src_dir + "/" - cmd_copy_shared_res = os.popen(cmd_copy_share).read() - bb.note("cmd_copy_shared_result = " + cmd_copy_shared_res) - - git_path = src_dir + "/.git" - if os.path.exists(git_path): - shutils.rmtree(git_path) + src_dir = spdx_workdir + "/" + d.getVar('BP') - # Make sure gcc and kernel sources are patched only once - if not (d.getVar("SRC_URI") == "" or is_work_shared_spdx(d)): - bb.build.exec_func("do_patch", d) + bb.note(f"copyhardlinktree {share_src} to {src_dir}") + oe.path.copyhardlinktree(share_src, src_dir) # Some userland has no source. if not os.path.exists(spdx_workdir): @@ -225,6 +214,8 @@ def get_patched_src(d): finally: d.setVar("WORKDIR", workdir) +def has_task(d, task): + return bool(d.getVarFlag(task, "task", False)) and not bool(d.getVarFlag(task, "noexec", False)) def fetch_data_to_uri(fd, name): """ From patchwork Sat Oct 26 03:25:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongxu Jia X-Patchwork-Id: 51341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FF10D116E8 for ; Sat, 26 Oct 2024 03:25:32 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web11.4381.1729913124930974082 for ; Fri, 25 Oct 2024 20:25:24 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=002900eeaf=hongxu.jia@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49Q3OaKL000751; Fri, 25 Oct 2024 20:25:23 -0700 Received: from ala-exchng02.corp.ad.wrs.com (ala-exchng02.wrs.com [147.11.82.254]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 42c8240wrm-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 25 Oct 2024 20:25:23 -0700 (PDT) Received: from ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 25 Oct 2024 20:25:20 -0700 Received: from ala-lpggp7.wrs.com (147.11.136.210) by ALA-EXCHNG02.corp.ad.wrs.com (147.11.82.254) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 25 Oct 2024 20:25:20 -0700 From: Hongxu Jia To: CC: Subject: [oe-core][PATCH 6/6] Add SPDX 3.0 include source case for work-share Date: Fri, 25 Oct 2024 20:25:19 -0700 Message-ID: <20241026032519.1968518-7-hongxu.jia@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241026032519.1968518-1-hongxu.jia@windriver.com> References: <20241026032519.1968518-1-hongxu.jia@windriver.com> MIME-Version: 1.0 X-Authority-Analysis: v=2.4 cv=UrgxNPwB c=1 sm=1 tr=0 ts=671c6123 cx=c_pps a=K4BcnWQioVPsTJd46EJO2w==:117 a=K4BcnWQioVPsTJd46EJO2w==:17 a=DAUX931o1VcA:10 a=24AZYWMyAAAA:8 a=t7CeM3EgAAAA:8 a=2V6DY6BgAAAA:8 a=YKlqnSHKsa4dvJv7hPcA:9 a=bG88sKzkDEFeXWNnvthB:22 a=FdTzh2GWekK77mhwV6Dw:22 a=ldqKKs2zR4t-S6fqr-1n:22 X-Proofpoint-ORIG-GUID: RJTnqqDQKNSa5F-9thGXL5R_t6xVrvxd X-Proofpoint-GUID: RJTnqqDQKNSa5F-9thGXL5R_t6xVrvxd X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-25_14,2024-10-25_02,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 bulkscore=0 impostorscore=0 spamscore=0 clxscore=1015 adultscore=0 mlxlogscore=999 priorityscore=1501 lowpriorityscore=0 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam authscore=0 adjust=0 reason=mlx scancount=1 engine=8.21.0-2409260000 definitions=main-2410260026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Oct 2024 03:25:32 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206384 Build gcc and check gcc-14.2.0/README in objset is available $ oe-selftest -r spdx.SPDX30Check.test_gcc_include_source ... 2024-10-26 01:24:57,063 - oe-selftest - INFO - test_gcc_include_source (spdx.SPDX30Check.test_gcc_include_source) 2024-10-26 01:28:24,204 - oe-selftest - INFO - The spdxId of gcc-14.2.0/README in gcc.spdx.json is http://spdx.org/spdxdocs/gcc-f2eaeb0d-b54b-53ba-899a-8c36c21139bf/88d5068ffd41e5ea6b4e0dd390b23bf499bb2b6674a41e09eaf2a887eced16c8/sourcefile/42 2024-10-26 01:28:26,369 - oe-selftest - INFO - ... ok 2024-10-26 01:28:33,315 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-10-26 01:28:33,316 - oe-selftest - INFO - Ran 1 test in 216.457s 2024-10-26 01:28:33,316 - oe-selftest - INFO - OK 2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS: 2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_gcc_include_source: PASSED (209.31s) 2024-10-26 01:28:45,260 - oe-selftest - INFO - SUMMARY: 2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 216.457s 2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) Signed-off-by: Hongxu Jia --- meta/lib/oeqa/selftest/cases/spdx.py | 29 ++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index be595babb3..8384070219 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -110,6 +110,7 @@ class SPDX3CheckBase(object): "SDKMACHINE", "SDK_DEPLOY", "SPDX_VERSION", + "SSTATE_PKGARCH", "TOOLCHAIN_OUTPUTNAME", ], target_name, @@ -136,6 +137,34 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/base-files.spdx.json", ) + + def test_gcc_include_source(self): + import oe.spdx30 + + objset = self.check_recipe_spdx( + "gcc", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/gcc.spdx.json", + extraconf=textwrap.dedent( + """\ + SPDX_INCLUDE_SOURCES = "1" + """ + ), + ) + + gcc_pv = get_bb_var("PV", "gcc") + filename = f'gcc-{gcc_pv}/README' + found = False + for software_file in objset.foreach_type(oe.spdx30.software_File): + if software_file.name == filename: + found = True + self.logger.info(f"The spdxId of {filename} in gcc.spdx.json is {software_file.spdxId}") + break + + self.assertTrue( + found, + f"Not found source file {filename} in gcc.spdx.json\n" + ) + def test_core_image_minimal(self): objset = self.check_recipe_spdx( "core-image-minimal",