From patchwork Thu Jun 19 08:47:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Turull X-Patchwork-Id: 65288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B80DC7115E for ; Thu, 19 Jun 2025 08:48:03 +0000 (UTC) Received: from MRWPR03CU001.outbound.protection.outlook.com (MRWPR03CU001.outbound.protection.outlook.com [40.107.130.41]) by mx.groups.io with SMTP id smtpd.web11.9932.1750322876199724832 for ; Thu, 19 Jun 2025 01:47:56 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@ericsson.com header.s=selector1 header.b=LsFPJUsQ; spf=pass (domain: ericsson.com, ip: 40.107.130.41, mailfrom: edaturu@ericsson.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=II5MMY0WNOc/UEkVdGiidl81GNUjxbp4rSwcsEsE8nRuJMReVs+OVLTW92+4Slf/h7t7KAeqdq/2PDEN9kuIwM9781hjeMSKkJaqNSxFirO/oAjsPV2Qv5ei4TNHLOwe5ePSss1Xkei96JZMSLNYrdINs6e3FKmD+co63WjAn+2owrYNS7J+eOeDjnN6JVOTwGMG639qi4DXo/lbavVyknb82CmZhYSCiUWtp4/g0ZyzF94MLOtY/6MxnryK5N78I/2le+p5MQ1EzMVc9R1uwt7SfM+s/f9X8d02TxL60nJEzaTqlL159NSux3pJcrvsy/fIBP7w0wqV+s93a8jkDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E31gYZ2YdpuJsfLjXd2jBO0GVBdrdKeuLcfxNC1yeUk=; b=hsNtPWx50MYdTg1p7do7mL3ssi44xkMer+oyCIXnXMtEg/auCsDeEmvXU/KyiKwXisIYwynSpGztw4ktLsHoZqMlauoodMgKqClQZbBGtg1eWBoNTYRp2rVWpRpTTMZadjKJ0EFb5NVKYlypQG3MjAc9TFUb48xEJe9ZsCHzVlAaL/MJmLz30KsJqHir8skQxbHEX30NlkLezQJQniBl9ewdaxy3DMiNX02odb8tDr3C++QrgtU2R/cMF3b8LYPE1H3FeNoMRU//o5MgToVq+XoK4kIBiC6OkVa2ClZKvmZUMe/QGfbKD6m/mzriu3qvxDz5lSXxGUMKl2drHxvV5w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=bootlin.com smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E31gYZ2YdpuJsfLjXd2jBO0GVBdrdKeuLcfxNC1yeUk=; b=LsFPJUsQ1NY+FCLJG1BNtFlsSP/VyLxVIgD1ICcBcmtq+QkgZpNohSqIJ2P3XSnv191222mnEPxoHTwdIkyOTSjpN6PKxLTBLqEsgc1HkAtlYHWqj8AjJ33wqNC+aJNLT1rORRSPhhQlhQzHwq+4YbhWBNfvxddIbnABXOpsqtCtkdLFjm8NTSHxTgbRwMrd/6MoS2bbd10gzEYfnurv+Nzh6zWMpbvKcgtwNvfPtaPozruZtTJU+4s4JvZx6/jiglE8hk0RNQsBgEWTQZ0YStjjab7Td+gATttbTI57/uIvhistEA9OAW4qJUa4n8jFgsHO34UQIhMi8/Mm/c4ARQ== Received: from DUZPR01CA0230.eurprd01.prod.exchangelabs.com (2603:10a6:10:4b4::7) by PAWPR07MB9393.eurprd07.prod.outlook.com (2603:10a6:102:2e8::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.22; Thu, 19 Jun 2025 08:47:50 +0000 Received: from DU6PEPF00009523.eurprd02.prod.outlook.com (2603:10a6:10:4b4:cafe::b0) by DUZPR01CA0230.outlook.office365.com (2603:10a6:10:4b4::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8835.36 via Frontend Transport; Thu, 19 Jun 2025 08:47:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by DU6PEPF00009523.mail.protection.outlook.com (10.167.8.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.21 via Frontend Transport; Thu, 19 Jun 2025 08:47:50 +0000 Received: from seroius18813.sero.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Thu, 19 Jun 2025 10:47:48 +0200 Received: from seroius08462.sero.gic.ericsson.se (seroius08462.sero.gic.ericsson.se [10.63.237.245]) by seroius18813.sero.gic.ericsson.se (Postfix) with ESMTP id BD01F9582F; Thu, 19 Jun 2025 10:47:47 +0200 (CEST) Received: by seroius08462.sero.gic.ericsson.se (Postfix, from userid 160155) id A61797083F54; Thu, 19 Jun 2025 10:47:47 +0200 (CEST) From: To: CC: Daniel Turull , Mathieu Dubois-Briand , Richard Purdie Subject: [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as json Date: Thu, 19 Jun 2025 10:47:35 +0200 Message-ID: <20250619084736.1940747-1-daniel.turull@ericsson.com> X-Mailer: git-send-email 2.44.1 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DU6PEPF00009523:EE_|PAWPR07MB9393:EE_ X-MS-Office365-Filtering-Correlation-Id: c6074597-252a-4e0f-6be6-08ddaf0df936 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|376014|36860700013|7053199007; X-Microsoft-Antispam-Message-Info: CRRNj+pDNv/hMUk0ph/bOh16UvzAP6F3k5J1Heo6UkqLQM2vSM6O+mWJZXUj7pHV51AnqVficaHFzf7Vy0w+9sikvsxuJWW4QovrdiLr0njNASrF8AO2XtobitrnO4EmVPpIImTlJ8/jQI25asdpCvNv5qMZ4ZdFmXvC+smkGtBXFFvchWs2DCawoUYiZYRGm2KBu6hjHXqiaODR1ZrfjdlwfpVz30MZ2EficHdod0PmNxmc8rE0mOZSwjhWQynqAPqEwLumGGa2uZ7/plGADuaYEh8lBV79Ngw45jWUrxmcXYULMTdFofqU9Z/pHNftFyS/f/ZkgvDWkPT2jGs9H+GBw9OXxFTxFiV3gN3sYnRz8LHpEht86RQv7D4j01TSPcsM5fWxMd3DFfw90TdmMqxhPGDCii17kKSjJ8fKc6NFGBjIms3JYHu+oK5Izxr2tTYTpXB1YhtmyRVEh533zTflM0J3FP5kMzFxqloY+woEOVCz6ZBtTUCXkxr+xfr8Z4WV9xpratfxOIi70X1M7Iwcl07yDeppZ+QIQZfHZSFJ8jqMmr2JOlg19IK5gPNm7AggANXreIw5JxtCVu1eSmqcGZFOAqOlMRAArXg/5DKWOjQWFGZDYcknUVmWomj0bKqMHL/UEL1d6lHpCptznVBvN1Bx4VPcG1O2wvUSNfRZdblumTCQJmVcldn3pe2uEHmfwcgTrx1Tut0JRr232p+h9WQFkLzCbWNrnck2eY+dQ9wpXnc49zhrLCSaprMR6+PU2+Vlo1o4i5mPwDvABMsrfz6ikWYUuIE65CdlcBMbFWhY1BfATT28Qu7qq/jmrewBStAV2vsnFAoH43OiUH9a0XhZJ6u4kUSGUHZs3g54UDOKwZZh2Wzc68YgD+XF47Eny+tjAvQgd3sNdbxqauABvzeTT9J9avZ30B8/3Se34jSXrTJAibN4Y7wuuolfQQULGKeCSxKczU3hQqLP4jbdXwfmhdF6SZyi5gsMGS5tM7PU1Sh4fBRcmlzxrKkqts7hZ2IoOU85uQOxEBjg6IFH/a6Yb1ct7Uh3ypvViH0lZ0KkblEbljrAdhK/1fuUFsCjx2OWToyrfCT/9ce2Ssu59asw8nPuLk7XqhLkUN4mg9Z5/FFnl8Jyy+9kvHkYmSx9Sb17hi57U2G+YBKtqyU+O7vwnZeheUoGY1ik9qxcDVw+WqgemxBV0u6jyNQ31Q9zS2zjrBh1fhSRWzLTF1I+4Fk600D/vvoKRWWTxZQzF7RZROj4dI4BxknojRTu1F1CwtgnUUerivvEbCjBNkovvL95dHzEhEls0LR/lTq9PUfO36/lLy4Q1YMsrmI0v38Z+7uB111m/vzwb/n5922ZAbChS3UqetUCZPE6mvPmcUdqNNcHRGNjtJYOI7ytvAKeua+SmVX/F2K4gEFHBGfbu6SGuwilC9u+Oxpy//UGzbCPIuGigO/0hU+AT5jYnUVUTGYbEqcjABBGrkKO2se9ZbSt0jS4OOHAKRxSejEHyY9CuB4ALIyJTqYxwsyg X-Forefront-Antispam-Report: CIP:192.176.1.74;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:oa.msg.ericsson.com;PTR:office365.se.ericsson.net;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(376014)(36860700013)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jun 2025 08:47:50.6206 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c6074597-252a-4e0f-6be6-08ddaf0df936 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f;Ip=[192.176.1.74];Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: DU6PEPF00009523.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR07MB9393 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 19 Jun 2025 08:48:03 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219060 From: Daniel Turull The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy. Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd Format: { binary1: [src1, src2, ...], binary2: [src1, src2, ...] } I checked the sstate size, and it slightly increases using core-image-full-cmdline: without patch: 2456792 KB sstate-cache/ with patch: 2460028 KB sstate-cache/ (4236 KB or 0.17%) (From OE-Core rev: c507dcb8a8780a42bfe68b1ebaff0909b4236e6b) Adaptations to match spdx in scarthgap: change BP to PF CC: Mathieu Dubois-Briand CC: Richard Purdie Signed-off-by: Daniel Turull --- meta/conf/bitbake.conf | 2 ++ meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 78f15b76ae..acf4e2d153 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -989,5 +989,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES" oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS" diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index af0923a63f..ba0d326781 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -1038,6 +1038,49 @@ def copydebugsources(debugsrcdir, sources, d): if os.path.exists(p) and not os.listdir(p): os.rmdir(p) +def save_debugsources_info(debugsrcdir, sources_raw, d): + import json + import bb.compress.zstd + if debugsrcdir and sources_raw: + debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd") + debugsources_dir = os.path.dirname(debugsources_file) + if not os.path.isdir(debugsources_dir): + bb.utils.mkdirhier(debugsources_dir) + bb.utils.remove(debugsources_file) + + workdir = d.getVar("WORKDIR") + pn = d.getVar('PN') + + # Kernel sources are in a different directory and are special case + # we format the sources as expected by spdx by replacing /usr/src/kernel/ + # into BP/ + kernel_src = d.getVar('KERNEL_SRC_PATH') + pf = d.getVar('PF') + sources_dict = {} + for file, src_files in sources_raw: + file_clean = file.replace(f"{workdir}/package/","") + sources_clean = [ + src.replace(f"{debugsrcdir}/{pn}/", "") + if not kernel_src else src.replace(f"{kernel_src}/", f"{pf}/") + for src in src_files + if not any(keyword in src for keyword in ("", "")) and not src.endswith("/") + ] + sources_dict[file_clean] = sorted(sources_clean) + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(debugsources_file, "wt", encoding="utf-8", num_threads=num_threads) as f: + json.dump(sources_dict, f, sort_keys=True) + +def read_debugsources_info(d): + import json + import bb.compress.zstd + try: + fn = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd") + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(fn, "rt", encoding="utf-8", num_threads=num_threads) as f: + return json.load(f) + except FileNotFoundError: + bb.debug(1, f"File not found: {fn}") + return None def process_split_and_strip_files(d): cpath = oe.cachedpath.CachedPath() @@ -1269,6 +1312,9 @@ def process_split_and_strip_files(d): # Process the dv["srcdir"] if requested... # This copies and places the referenced sources for later debugging... copydebugsources(dv["srcdir"], sources, d) + + # Save source info to be accessible to other tasks + save_debugsources_info(dv["srcdir"], results, d) # # End of debug splitting # From patchwork Thu Jun 19 08:47:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Turull X-Patchwork-Id: 65287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AA14C7115B for ; Thu, 19 Jun 2025 08:48:03 +0000 (UTC) Received: from MRWPR03CU001.outbound.protection.outlook.com (MRWPR03CU001.outbound.protection.outlook.com [40.107.130.0]) by mx.groups.io with SMTP id smtpd.web10.10047.1750322875941739108 for ; Thu, 19 Jun 2025 01:47:56 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@ericsson.com header.s=selector1 header.b=RxUFcioe; spf=pass (domain: ericsson.com, ip: 40.107.130.0, mailfrom: edaturu@ericsson.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PZecj2oslq8CBWYc9U2d9hHzpqBPBHqyT/DFYmzhElH0+tSBx948b0Ub9qYiEBqDWXxxTyBoIRFruxs9wrEUf4sXFx81w9XKxNehcG5dVklyKiTkx/DxsAiCx2Mz9xEtycptfpweLro7CFI1THh62gWQbgUKAIlb4LTp8GWKBfcrlfT4pqAPDEuoQpCXcXf1pEdg6cuy568UDn+YOJhCORH2t+AbXMQJHwBaD0R6HNSrdmk8UeukA2BkPd8UCt4HowHBr9TnVvpNLHIzjKPHD69MF1oITUOyiVtNEB5uyqL7FbIK/ZcazUENY5hBGK/IM5Zum7P7UcK6+q2ozPoNxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=j8DlDBCMbw4BPWdXEDBP7Y6YY/junut6TKL0qg+bQ4I=; b=DzZetwLcLUkrh7mUnVc6P/SiJLjg+7LCBD+Kb2hqddhP4zTdKE04udZvrOtoNM5fDzgJdKWMYkSd5tAqfugN2eMlTgxldOSPQ+amxePDNmSQLS38Iu5fmy6C5CDmiZgApsHD7HXqyeSeu4l7UV6+lvwantjxP65vWY4U9jhpSWvAuB2U/cDm72fjhjgm1lPjdT3lDhMh4mEcXsDGPDS7cjn0v3hpRU+VqJUmwRWFjrdXwyz87YKgCrmwv0eZBtm6LVEKemXPIz+YKZwHyFWg8GrRp8zlMXNYu0zau4J7Knz3+9tZpI7WCn6/JnncGwhYOSEgSmu+02CRK30S25e0Ig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=bootlin.com smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j8DlDBCMbw4BPWdXEDBP7Y6YY/junut6TKL0qg+bQ4I=; b=RxUFcioeg9l3IcOBbP54VNYSwbS8Hhv8L3hV445oFMptcRgiZzXDsxtxDU8VjpTaHsAHdBmZMXHoVP1Su1sW8zxynSd6yKtr5fHswYcik1gYeYm+TnkfQw2PpsR/6c/qxZsepnMCOiJMxmjEhfG4mcaUneTXvmVHjExKT1sQxOHnM/3XjYaW7Md0Osc+T4zlNXhNbymUgtoW1yhDUHqcxU5NPtd9WIuLzUaS/MTPyLamU3+QVXRwTCTIYvAd7vuEaRFDBZmfHdaTY+3o4uFB1VrpwuoOMIqx2tX80Z31GUB4uLRbVZJvrD8gvQgACMvOWuBGrWJSZZCVdl6l/T9k/w== Received: from PAZP264CA0158.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:1f9::18) by VI1PR0701MB6959.eurprd07.prod.outlook.com (2603:10a6:800:194::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.20; Thu, 19 Jun 2025 08:47:52 +0000 Received: from AM4PEPF00025F95.EURPRD83.prod.outlook.com (2603:10a6:102:1f9:cafe::4) by PAZP264CA0158.outlook.office365.com (2603:10a6:102:1f9::18) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8835.34 via Frontend Transport; Thu, 19 Jun 2025 08:47:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by AM4PEPF00025F95.mail.protection.outlook.com (10.167.16.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8880.0 via Frontend Transport; Thu, 19 Jun 2025 08:47:52 +0000 Received: from seroius18815.sero.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Thu, 19 Jun 2025 10:47:51 +0200 Received: from seroius08462.sero.gic.ericsson.se (seroius08462.sero.gic.ericsson.se [10.63.237.245]) by seroius18815.sero.gic.ericsson.se (Postfix) with ESMTP id 90D694020A8A; Thu, 19 Jun 2025 10:47:51 +0200 (CEST) Received: by seroius08462.sero.gic.ericsson.se (Postfix, from userid 160155) id 78DE37083F54; Thu, 19 Jun 2025 10:47:51 +0200 (CEST) From: To: CC: Daniel Turull , Mathieu Dubois-Briand , Joshua Watt Subject: [scarthgap][PATCH 2/2] spdx: add option to include only compiled sources Date: Thu, 19 Jun 2025 10:47:36 +0200 Message-ID: <20250619084736.1940747-2-daniel.turull@ericsson.com> X-Mailer: git-send-email 2.44.1 In-Reply-To: <20250619084736.1940747-1-daniel.turull@ericsson.com> References: <20250619084736.1940747-1-daniel.turull@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM4PEPF00025F95:EE_|VI1PR0701MB6959:EE_ X-MS-Office365-Filtering-Correlation-Id: 757b7455-c601-432f-2f41-08ddaf0dfa38 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: Aou6R3vg9Ap+B0F/G0fYD03Zwp3aTuKndLipThNGmmFEUzyycm8Ac1PDUY0zSEv4jpEGQh80wOE/sbNe8UDg0HcNYKFURZpQdMWKhGbvJ0QLZK/MV0GZEZrRJrmSwDEoEhqMRBRKeHGdpXPMXfL7/ImzLUCxQnUKCYZb6ryTSeYn9j2gGW+Yk7rE/ZwmAcAX93omCcwfvD5W9PteoWt9DW+xPtKvbYtRMaEnjL6+ZczZFFLnvopKToOH0PD5rIEbM1EIi9XuHC4E8EtI1vcpC8AT8Ib2RFdzTEGRjrRgAULyuOPMklH0GQuPOXXTp3cgTIvlYEXy01ShLejHq6ToRf+Qv3lvPoxeEFLEcF7Jy+D/I+RU3zg8DH0q3vpUgggkZsRZfVRI8n2I0EhSEEz5mcKjcbrtci+aG26TZkrOdx/vfoQCMrn4qZw/ZJzuaXkO2JC69FfT7L+7xJ0sU71YUDHMSvbiu5buAAQkn1gXKFjAtAknzCcOZ823sj4AO50FJa16QBJCNUF2SStC9Zod4/yOQ0bPTT/NDkFO8rr5KFDEIXhDQBSDtPuU/QBvIzkPRR5zGGRnQm6ijMu+EXyljBkzd/S0cv+H1/MrqSsKjRL+YsItaZE1laHY6+ANHmfxMAwab+Orghr5J30zbHFqw0QoVqnqXUMfV1ASLkLZgNNsGaLUTAmvKytUptQy46dKe1R/RN2JiN+zKbGDnd/y+UPrOWBfoxkjkd/Y5YYls2edpBeGYWR3BVnu5hr+Du8e1yMTk3eAR1TkKZW4mO1cE6P+q6OodnBd65RUmx+Siq/zxD9xsDhLojI5nzAJEthxTnnFD55jjDPU16LmTSZX3p7Y8AFYWwYPG3EPD8rnOrtBaRk9WZ+8x4FsmbUI5PgAZ7AKz+T85TsaTZC2lZndxDu40UPUZ0AWovkJWz2IMTc6INYialDd9ZCek8KCRyxOonWKxmaTA5rjWOdU5Wvd0rSBcxIYrZyhpwoUe5jxEzWOUjTLO4qfH1XXH5zX+wZqD1wv/d4qf3rdO/ZDGHmOX43wBe4hcy0UKBloG14Twb3dYNvrLMuML0zPFac/am8atm807KYGz9TCFjMv0R9OaFpNtosFtUvqiQ5kIKlSuaTeD7SGDeKPC5bwFNMMlycwihOL1BeSV5n+jv84kQRglGsr1KtyenmVTz+2ZkV3KxiU14oGT8UBMfhIpDhYO4hbKWIELQaIX8za5duipIuhuhlCysyP7iQ1IGU5C8knr8Z0LcbuEx15cLRj5ztLOqz2OgZuAxcrueVsciy2B0jUjxXrRT9Qo7yJhWALxc61RbHnwhb0VlCXachzzfLamBwcl8QASRhfga2NifgzWd04l8FezEEbIZkl9WS59tjHKwgNaOdlF6QFxHGB4Co4l7qG3eQilsSLmJ1WaQxY7ZILY9bQFKodYzHb/nrkgk+JUlAO5OEQvS2a+Ywk4/6kdySewmlHOY4xUlATIzvNB2+T4lamonH/gHF+j8IWTtXY79g+kwj15wB5b2+E/wu20r+Z X-Forefront-Antispam-Report: CIP:192.176.1.74;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:oa.msg.ericsson.com;PTR:office365.se.ericsson.net;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jun 2025 08:47:52.3316 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 757b7455-c601-432f-2f41-08ddaf0dfa38 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f;Ip=[192.176.1.74];Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: AM4PEPF00025F95.EURPRD83.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0701MB6959 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 19 Jun 2025 08:48:03 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219059 From: Daniel Turull When SPDX_INCLUDE_COMPILED_SOURCES is enabled, only include the source code files that are used during compilation. It uses debugsource information generated during do_package. This enables an external tool to use the SPDX information to disregard vulnerabilities that are not compiled. As example, when used with the default config with linux-yocto, the spdx size is reduced from 156MB to 61MB. Tested with bitbake world on oe-core. (From OE-Core rev: c6a2f1fca76fae4c3ea471a0c63d0b453beea968) Adapted to existing files for create-spdx-2.2 CC: Mathieu Dubois-Briand CC: Joshua Watt Signed-off-by: Daniel Turull --- meta/classes/create-spdx-2.2.bbclass | 12 ++++++++ meta/lib/oe/spdx.py | 42 ++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index ade1a04be3..1fc11ad7ac 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -100,6 +100,9 @@ python() { # Transform the license array to a dictionary data["licenses"] = {l["licenseId"]: l for l in data["licenses"]} d.setVar("SPDX_LICENSE_DATA", data) + + if d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1": + d.setVar("SPDX_INCLUDE_SOURCES", "1") } def convert_license_to_spdx(lic, document, d, existing={}): @@ -215,6 +218,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv spdx_files = [] file_counter = 1 + + check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1" + if check_compiled_sources: + compiled_sources, types = oe.spdx.get_compiled_sources(d) + bb.debug(1, f"Total compiled files: {len(compiled_sources)}") for subdir, dirs, files in os.walk(topdir): dirs[:] = [d for d in dirs if d not in ignore_dirs] if subdir == str(topdir): @@ -225,6 +233,10 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv filename = str(filepath.relative_to(topdir)) if not filepath.is_symlink() and filepath.is_file(): + # Check if file is compiled + if check_compiled_sources: + if not oe.spdx.is_compiled_source(filename, compiled_sources, types): + continue spdx_file = oe.spdx.SPDXFile() spdx_file.SPDXID = get_spdxid(file_counter) for t in get_types(filepath): diff --git a/meta/lib/oe/spdx.py b/meta/lib/oe/spdx.py index 7aaf2af5ed..92dcd2da05 100644 --- a/meta/lib/oe/spdx.py +++ b/meta/lib/oe/spdx.py @@ -355,3 +355,45 @@ class SPDXDocument(SPDXObject): if r.spdxDocument == namespace: return r return None + +def is_compiled_source (filename, compiled_sources, types): + """ + Check if the file is a compiled file + """ + import os + # If we don't have compiled source, we assume all are compiled. + if not compiled_sources: + return True + + # We return always true if the file type is not in the list of compiled files. + # Some files in the source directory are not compiled, for example, Makefiles, + # but also python .py file. We need to include them in the SPDX. + basename = os.path.basename(filename) + ext = basename.partition(".")[2] + if ext not in types: + return True + # Check that the file is in the list + return filename in compiled_sources + +def get_compiled_sources(d): + """ + Get list of compiled sources from debug information and normalize the paths + """ + import itertools + import oe.package + source_info = oe.package.read_debugsources_info(d) + if not source_info: + bb.debug(1, "Do not have debugsources.list. Skipping") + return [], [] + + # Sources are not split now in SPDX, so we aggregate them + sources = set(itertools.chain.from_iterable(source_info.values())) + # Check extensions of files + types = set() + for src in sources: + basename = os.path.basename(src) + ext = basename.partition(".")[2] + if ext not in types and ext: + types.add(ext) + bb.debug(1, f"Num of sources: {len(sources)} and types: {len(types)} {str(types)}") + return sources, types