From patchwork Tue Jun 10 15:24:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Turull X-Patchwork-Id: 64711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A092DC5B543 for ; Tue, 10 Jun 2025 15:25:07 +0000 (UTC) Received: from AS8PR04CU009.outbound.protection.outlook.com (AS8PR04CU009.outbound.protection.outlook.com [52.101.70.43]) by mx.groups.io with SMTP id smtpd.web10.90071.1749569101566887176 for ; Tue, 10 Jun 2025 08:25:02 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@ericsson.com header.s=selector1 header.b=GbEb43aA; spf=pass (domain: ericsson.com, ip: 52.101.70.43, mailfrom: edaturu@ericsson.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fDIUOVqkRekYknEJVGDfoVBr9TWD3g1IyfPR/k+0PdLoJ6oRWHLq5eYrUvLCsl/SyT4tB2pXMr2u57qVjmy9cvZe9ltiXiHPn5fetGpVza2ndk8KFGHvlAIcuCWC8c6H9K7+nmQIAem1XH9Aac+F3zmHPWJHGCptwfJQe1y20xr75dbiqS5lOCg/HYONFqXyru04aF5npFFhtjsZvIfqof0AauYvMti2wMfVllTHAHqO5sh/88iNl+644h9E7froApNUszRM1HqgH8k6b6kDQqyRwueOVBYY8rdO6RsllYxdnMmRT2GT06f+QQzxzfRQW2mLPjGklI+rD7gWe8frHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=I+zoS5voh8VglqTZUBbHWHFTT1YpUTgu6da4JaPhqao=; b=IwQ20S/cgxqbK/L3WISIuTwam0v/w4g0NJg7Pk23lymvj4SSJvuoK4n+ZR6O4woP6ymzCbERSRVMkZijSC4oOySxql7LOSspSeNOPqfysBWH0SpT/dL7pf7HZABSjS2a0pOf422GkualCtoOFqpfrylcMTOgfNs/K42+BOrOMigYcRtyn8Gy5tP8lkNAoybUF3EtuzpufRBJgOaKb5HcSszt9HKFo3vBm3zrAjzzND5JY75En7evxW1sTcjQdElmtsR6hW+rYY1I2yCSgj5RwO+7uyxIDRzMLeW7g1NxKTacIsJxX1BBA7Z+rl2FYCKjBwbRaEnDnwAaygZ2f1JOxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=linuxfoundation.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=I+zoS5voh8VglqTZUBbHWHFTT1YpUTgu6da4JaPhqao=; b=GbEb43aAkreFj7QNH6UwiAKrbeGAAa0IPdnJkVc87UoDwPPHc8IOVBR4oNGZBou+DdQqrEawkMcLpxleANNxZ3Mqt6Xdq2769Gj+gfMBY/ePk5J6/cxFzUrYzDmoXPyNAJYYKM18O2lAfmtz6xeev64PScyBS168BTq22yjghtg4NmU1deKGChN790vQfsLnIlUaSINqrB+FCXuAc4w9fMxBBKkt9xivtQG8VPtHFc3hFjJ62v1p3wvzlyf7Td0enzYzle5BFYUvcRag9++jVwG4BHTX8YHqPspUqyXdR1eqqIVOkZaGS7paXy7TcpBx7N77hfr6i4FeqEOF82kqnQ== Received: from AS4P189CA0016.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:5db::8) by AM9PR07MB7203.eurprd07.prod.outlook.com (2603:10a6:20b:2ce::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8813.30; Tue, 10 Jun 2025 15:24:57 +0000 Received: from AMS1EPF00000042.eurprd04.prod.outlook.com (2603:10a6:20b:5db:cafe::6f) by AS4P189CA0016.outlook.office365.com (2603:10a6:20b:5db::8) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8835.17 via Frontend Transport; Tue, 10 Jun 2025 15:24:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by AMS1EPF00000042.mail.protection.outlook.com (10.167.16.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8835.15 via Frontend Transport; Tue, 10 Jun 2025 15:24:56 +0000 Received: from seroius18815.sero.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Tue, 10 Jun 2025 17:24:56 +0200 Received: from seroius08462.sero.gic.ericsson.se (seroius08462.sero.gic.ericsson.se [10.63.237.245]) by seroius18815.sero.gic.ericsson.se (Postfix) with ESMTP id 127964020B6A; Tue, 10 Jun 2025 17:24:56 +0200 (CEST) Received: by seroius08462.sero.gic.ericsson.se (Postfix, from userid 160155) id E89717083F55; Tue, 10 Jun 2025 17:24:55 +0200 (CEST) From: To: CC: Daniel Turull , Richard Purdie Subject: [PATCH v8 1/3] package: export debugsources in PKGDESTWORK as json Date: Tue, 10 Jun 2025 17:24:41 +0200 Message-ID: <20250610152443.2162164-2-daniel.turull@ericsson.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250610152443.2162164-1-daniel.turull@ericsson.com> References: <20250610152443.2162164-1-daniel.turull@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AMS1EPF00000042:EE_|AM9PR07MB7203:EE_ X-MS-Office365-Filtering-Correlation-Id: 0fc504f1-8aa9-4740-4e1e-08dda832f4f4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: 3IJdPGMKgMksBCgbtOYx0U4AuwoKwL8IHEY4oJWWkEH38VaR74K4Q/xYHQ7DVV7EAnvPcrrOuCXBC+llTepbRuygqeJFFoiYVFgTsAbvAmqugR75Ih8mij2we6msFQ+uXVm+hs7MD//vRMvBtg+Gjb+rFKaQssiGz4Qp+w/KSRN7LyLWgmq8WA5hlWLWo1fu/W/YAyXT8xctTWJJ8E0I26M5IjrlvpK9PG6UOC0pKNcuv0pSRJxbrIB+ce4VM8XO/ghoPv8+n1aAlK/zD/jWYkcd3M29SZQM7itlQKamw0p0oSp0IzryZh4TJ2lqg7TaBps//0kbGjKH+egklwKp062k7dTpA4ZAihya+Y28X6ium068y4D0PuFvDPp3bBXEpqkXLYzeNUfc6/rcleuKEumYWi5gNNqjMF3dECRis8p0AsDQ1fD21RlLwv/Fu9a3AxCWIv56EzXAHtDed2rrhZlJ3oofgLKM68mvFfTcPZ4G81v1+DjR4/USELXLWN9tMZ75tDJZMDltCzup6UxPwmZDmVeXfGqK6j5V6xueaQXkZEFUB9+iDYPxpIoPY5D0n8BPVN4zb/MkJaKEABV+wyE/sffQcZlOcY5U1/zi1ZfqAQBmfqZg0eZ8MfxplNbhb4lWTGp+CxijqL8W4u8vc98YIpfXdscF7CoDrz6Isuhucz9Xg8tIn498RIb1MmJGVCVgjtjQ4tX82tWAd/hgDQBPgdlxHDJ6K5Zh30MtAks50wyk9UFtNyrTlSQ2mtjOPXY69QeU8BFh7cp9hqG0bHuxOEg2VVkFYlfFLIUBnl18rSKHgCcpvFaj9tZIMX44CdrOdJAnICoBpjRC0QD/OvRrbYq5dpaQH/yypMHxfTo5pc0SoMLuYiU/Egp5lfyiU7QVVcDQ2SbXX+HPqGXNnTeSP2UhM03xyqICqyLFs1/K6wCg8yBSmp82liQfGuyirbAMYnkH2I0hj2Jgwqb+QT6iZ8A96tJ9tpbSQmQBAc/U3JV0C1xexKOic6+0+PMrP90htZkT5ffLYum7mH1l/C8y/FKsdTdYBJuj8pRo3YwrjEBSWVh8n9Abgq+dje8nbZb9z01UUc7i818x2k9SWQRffVEYDGxDNKZTeazVN1uYsOqHyldyaS0THG0rOAcm62Q0egKkS5kQ08hLNTYgiWYWdmLyVTW0JSrxcyObuzojw5cOSJhZsdttfPNtirqm6Nua5Bhg4a0A5GQqa2PGAR/ihl1F8CvCLglGMHPLFfcpU7zHTdGmcW37G67P7Z4IvowZgzM5fDZrRVJmn2lF4vmUvCsquLJlG7ub3OjH1rENORg3w8VFONURKt0RFC5FWD0mH9zegjSvXPhFn1lXcciY5xBej7piMRgetFu1J9PKmqswcfCYwQYfDYS0Q5zDVi2QOLP2iYvJQ4nQffBIFESoYfWzXril/u5e+257cUIvbfW5fyIUQofaBGFzd9PHaInnMWVjpvCXfZC23kGpCi+KqleHFDfOL/S32/A+LNDBjnFWY8JkicrEZdA+aKcw X-Forefront-Antispam-Report: CIP:192.176.1.74;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:oa.msg.ericsson.com;PTR:office365.se.ericsson.net;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jun 2025 15:24:56.7359 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0fc504f1-8aa9-4740-4e1e-08dda832f4f4 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f;Ip=[192.176.1.74];Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: AMS1EPF00000042.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR07MB7203 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 10 Jun 2025 15:25:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/218361 From: Daniel Turull The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy. Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd Format: { binary1: [src1, src2, ...], binary2: [src1, src2, ...] } I checked the sstate size, and it slightly increases using core-image-full-cmdline: without patch: 2456792 KB sstate-cache/ with patch: 2460028 KB sstate-cache/ (4236 KB or 0.17%) CC: Richard Purdie Signed-off-by: Daniel Turull --- meta/conf/bitbake.conf | 2 ++ meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 52e74a6879..3ec0f31e8b 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -991,5 +991,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES" oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS" diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index 0bcc04ea54..60392cbced 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -1049,6 +1049,49 @@ def copydebugsources(debugsrcdir, sources, d): if os.path.exists(p) and not os.listdir(p): os.rmdir(p) +def save_debugsources_info(debugsrcdir, sources_raw, d): + import json + import bb.compress.zstd + if debugsrcdir and sources_raw: + debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd") + debugsources_dir = os.path.dirname(debugsources_file) + if not os.path.isdir(debugsources_dir): + bb.utils.mkdirhier(debugsources_dir) + bb.utils.remove(debugsources_file) + + workdir = d.getVar("WORKDIR") + pn = d.getVar('PN') + + # Kernel sources are in a different directory and are special case + # we format the sources as expected by spdx by replacing /usr/src/kernel/ + # into BP/ + kernel_src = d.getVar('KERNEL_SRC_PATH') + bp = d.getVar('BP') + sources_dict = {} + for file, src_files in sources_raw: + file_clean = file.replace(f"{workdir}/package/","") + sources_clean = [ + src.replace(f"{debugsrcdir}/{pn}/", "") + if not kernel_src else src.replace(f"{kernel_src}/", f"{bp}/") + for src in src_files + if not any(keyword in src for keyword in ("", "")) and not src.endswith("/") + ] + sources_dict[file_clean] = sorted(sources_clean) + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(debugsources_file, "wt", encoding="utf-8", num_threads=num_threads) as f: + json.dump(sources_dict, f, sort_keys=True) + +def read_debugsources_info(d): + import json + import bb.compress.zstd + try: + fn = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd") + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(fn, "rt", encoding="utf-8", num_threads=num_threads) as f: + return json.load(f) + except FileNotFoundError: + bb.debug(1, f"File not found: {fn}") + return None def process_split_and_strip_files(d): cpath = oe.cachedpath.CachedPath() @@ -1280,6 +1323,9 @@ def process_split_and_strip_files(d): # Process the dv["srcdir"] if requested... # This copies and places the referenced sources for later debugging... copydebugsources(dv["srcdir"], sources, d) + + # Save source info to be accessible to other tasks + save_debugsources_info(dv["srcdir"], results, d) # # End of debug splitting # From patchwork Tue Jun 10 15:24:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Turull X-Patchwork-Id: 64712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6916C678DA for ; Tue, 10 Jun 2025 15:25:07 +0000 (UTC) Received: from OSPPR02CU001.outbound.protection.outlook.com (OSPPR02CU001.outbound.protection.outlook.com [40.107.159.23]) by mx.groups.io with SMTP id smtpd.web10.90072.1749569103511828373 for ; Tue, 10 Jun 2025 08:25:03 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@ericsson.com header.s=selector1 header.b=mvK0Csom; spf=pass (domain: ericsson.com, ip: 40.107.159.23, mailfrom: edaturu@ericsson.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=pTFINRWxtQD+iUWdqG+PfapF+7y8TYQ+a+I0TWhhFhg62UK7BsrPe3kb/GAs+dBmpV9XPJh5ji60ypcmAcpGmo3VsUYm9Jxig5BGUCVF1y3RvH1HyzrfeqRJdd3huhgMhcudvMC4Ah5u319BAHuotuaFW2jkjK1pWbUqzn3/95lHr5y5yHZqe+Dy1tOBNC3vAu+SttZhUIZWOdHuXPc8c+J8TdXxywfSD9Lln29crMak6K5tk6MiqasjIuZWVf9vgJ9zEzY5pV465pmP3ja9I1vmQb1mq9U1qOTgBI9dsVohnma0EtTsFpAvTFRYoWrISZ9e6ydtpO+6/mdAvyrqLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QUJMWecbuoWo0cyvVht33qdnLpuNRnAqY0gLvA1/+Mg=; b=GaU0clfKGEAJf0yXzzZks55sZPUP6cq3kLks5xX4mh8NcPF0LQ7GO50xv3b75k2to/GlzK+hMHejXdVBC9+9A3KQSTZIePuUGLmO7E3Xgndf1hEsgc6D8bVco7dxcZZqZX3o0xIh5nWSjJzaZY8eDFzJe8Uwqo506xDI6LBaoN4EMSTNPmEnNWDGwI/+BZ7FKI8YsmAhULj0PLf1YVHk3TKMQW0AP+J566C/+9IL0eZa8UZxPRoSmBIYL3TVUFmVrjOCBpcV+5AUK7if9+IdzKtAqtLBFNFzTYuhLhH0JF8eJEs/gwDv5ZSerPN9waqBLmfBltTx6Arf1AKBThl7vQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=cherry.de smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QUJMWecbuoWo0cyvVht33qdnLpuNRnAqY0gLvA1/+Mg=; b=mvK0CsomB6wxUVfdQYo0JRvr8wmN9A8cWk0tPA9KCr4WM83FQRUW2pw1yStt6LDeyYXF/1RSGyl4yhHa82xvO3D06vLHbfD/JzVKVMv0eUtvlBvfbsGG0o3HzdsDKnBN/ejhsKXTC9Kw4g4aqrjEjWWmYMArQPrj77xAz4Icz8KoG6QzcaCDlp/dum3CgNbjpw5NbEP2UVI1oRNYOW7UpsZx5PwuQ3x56dXTgzozvR41DN8+cpR+GQKenpyPhHi6z0KuEVg26qWjclsl3seJg+Pe7g8jEesFVA1GOCGjUoLCUJomnyE8ANT04Gbj+QRTr24rpy9H7co6v1a0hiND8g== Received: from AS4P189CA0027.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:5db::16) by VI1PR0701MB7024.eurprd07.prod.outlook.com (2603:10a6:800:192::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8813.26; Tue, 10 Jun 2025 15:24:57 +0000 Received: from AMS1EPF00000042.eurprd04.prod.outlook.com (2603:10a6:20b:5db:cafe::54) by AS4P189CA0027.outlook.office365.com (2603:10a6:20b:5db::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8835.19 via Frontend Transport; Tue, 10 Jun 2025 15:24:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by AMS1EPF00000042.mail.protection.outlook.com (10.167.16.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8835.15 via Frontend Transport; Tue, 10 Jun 2025 15:24:57 +0000 Received: from seroius18813.sero.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Tue, 10 Jun 2025 17:24:56 +0200 Received: from seroius08462.sero.gic.ericsson.se (seroius08462.sero.gic.ericsson.se [10.63.237.245]) by seroius18813.sero.gic.ericsson.se (Postfix) with ESMTP id 11E1C9582F; Tue, 10 Jun 2025 17:24:56 +0200 (CEST) Received: by seroius08462.sero.gic.ericsson.se (Postfix, from userid 160155) id ECAE87083F56; Tue, 10 Jun 2025 17:24:55 +0200 (CEST) From: To: CC: Daniel Turull , Quentin Schulz , Joshua Watt , Peter Marko Subject: [PATCH v8 2/3] spdx: add option to include only compiled sources Date: Tue, 10 Jun 2025 17:24:42 +0200 Message-ID: <20250610152443.2162164-3-daniel.turull@ericsson.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250610152443.2162164-1-daniel.turull@ericsson.com> References: <20250610152443.2162164-1-daniel.turull@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AMS1EPF00000042:EE_|VI1PR0701MB7024:EE_ X-MS-Office365-Filtering-Correlation-Id: 75a8b419-2e19-4c3d-373b-08dda832f539 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|376014|13003099007|7053199007; X-Microsoft-Antispam-Message-Info: +I+XBhg7PWOLAaBlzMONycR/vmLE3xyI+iI/TnzWzc49e8ddAddNHczlWqoXyImcMwh49ZQS87/O2gUU/sBDlamr82VHySOYFUAeC+YFJqmOsyXGUnB//ezN2ZpDIqSOjn350QLEhFiB19yRhvk0O9Z1tD6ujD6F56ykqhhpUHPt+QTanX78YxtBdsHr8EnHvTkbpmFCWLQCFzA2IYfxH9HL+WUecMcQzSKoDnktnebtvp6oK+ml1GJJSL7zZfwTkfx4gwEmCMXpE5q/Fe/8XFsNOTqwUu3UvCovDPtOODvgdoa2tskmIYg/1ELliGW8iYf+kqBNjM7Wq+0scOw6yuln6O9MNsOlv2BHKYN0f772MLtP16gt9jsfyEgJWp2gViAAM97oilWtDXDcwmfQ7DT/iqf7X0oeq5rhrxEnj3Ii8UlTdvXNnhVtH94xYwzna844ChiZ/lQVKk6J8P2lviJJ+BUtTcRIVslhEVYbAh8GEpZ1zlabRfpIpEJT3ch2ex3c3OiV0prFN0YRU54CvAC105H4xnpVBqy1YV8IWeL1EwR2TAP8eVQ1/l8SwtAWZpWl8nvEE7H0nG85H3lbaAuKawn6JM+8GuwPGmnQW+yfxQAfQCoR1lM1hEQbMj7jf5LyRXLS8OAE/alddoDpn94U4NZ6vbgvXtL+lFfM/WbFBFMSuWIgHGoCvL43150yGocoEPkC9LRWB8rIevU3Wl2YCYdnH80N15CSuSw8jkX8yTCDc43MkVYDX/rZzgM7HXPV2vgURXzAGNCsBdNBqoyY8RIMRdrabFqNazmvIFUTxFpA6rkc0TbquqBlUHXItfoWkFHZC5afaMfsGkWtrkMPRfJBnMFXLYAfXACdJBQ5BZpMOKmFpeoYIvPBSIvEPcLE3ovlVCFhzXTwtEIOAEKc1YaL72DPu7DU++hjse0Pbw7gz+MSJfg5tA9wN5ACgMkmDqc5ree/e8ZKlhQf0BCgGvQLEKWXkYomtJuOfs2sJ/ebn8xn9Kbb0VTfLDGE0FH4JTfRX7MywrLGAv78q8ZIROato4LzKfQsLDVVwnp8HLAb1eI6vO+vXBzVw/ySaYb6KsZXRLTiXyDElnKSGNkW7k+mmfokjOYjpFkCEyk/a/JT7yu+Xt3T1mVALBmWFGotEza8MQSq6DagctS/7a9hh0JonbIVmQG3gSuxmCD0wKw9qi7EcHPpt8d9h3yJntWPO2Ax+2a//blonZNuGVS+QyTn0QVt4j+Sy60Q8QTz85xbFeZaK701cJbVvtITkj+/clv8rIKc3RKegByH10fEM0quQbDcXBYTlCeovzDbtRS0amHh/5EnY4T+M1h4z+rnFuE+KiFdG7IATbo2UDzXZU1SqfgDJ2VYH57QTAwIODR1t3cFBD/k9Nt7cK5dbrWcSg//HfG04B3MSV8HDW+lRvlqbc7zKb9UJT8MJ3C35v+4meiBQrgD/CmgFjs9aXyeAjiWkXHkn+814lItNQ== X-Forefront-Antispam-Report: CIP:192.176.1.74;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:oa.msg.ericsson.com;PTR:office365.se.ericsson.net;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(376014)(13003099007)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jun 2025 15:24:57.1882 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 75a8b419-2e19-4c3d-373b-08dda832f539 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f;Ip=[192.176.1.74];Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: AMS1EPF00000042.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0701MB7024 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 10 Jun 2025 15:25:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/218362 From: Daniel Turull When SPDX_INCLUDE_COMPILED_SOURCES is enabled, only include the source code files that are used during compilation. It uses debugsource information generated during do_package. This enables an external tool to use the SPDX information to disregard vulnerabilities that are not compiled. As example, when used with the default config with linux-yocto, the spdx size is reduced from 156MB to 61MB. Tested with bitbake world on oe-core. CC: Quentin Schulz CC: Joshua Watt CC: Peter Marko Signed-off-by: Daniel Turull --- meta/classes/create-spdx-2.2.bbclass | 9 ++++++ meta/classes/spdx-common.bbclass | 3 ++ meta/lib/oe/spdx30_tasks.py | 10 +++++++ meta/lib/oe/spdx_common.py | 41 ++++++++++++++++++++++++++++ 4 files changed, 63 insertions(+) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 7e8f8b9ff5..6fc60a1d97 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -137,6 +137,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv spdx_files = [] file_counter = 1 + + check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1" + if check_compiled_sources: + compiled_sources, types = oe.spdx_common.get_compiled_sources(d) + bb.debug(1, f"Total compiled files: {len(compiled_sources)}") for subdir, dirs, files in os.walk(topdir): dirs[:] = [d for d in dirs if d not in ignore_dirs] if subdir == str(topdir): @@ -147,6 +152,10 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv filename = str(filepath.relative_to(topdir)) if not filepath.is_symlink() and filepath.is_file(): + # Check if file is compiled + if check_compiled_sources: + if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + continue spdx_file = oe.spdx.SPDXFile() spdx_file.SPDXID = get_spdxid(file_counter) for t in get_types(filepath): diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 713a7fc651..ca0416d1c7 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -26,6 +26,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0" +SPDX_INCLUDE_COMPILED_SOURCES ??= "0" SPDX_UUID_NAMESPACE ??= "sbom.openembedded.org" SPDX_NAMESPACE_PREFIX ??= "http://spdx.org/spdxdocs" @@ -40,6 +41,8 @@ SPDX_MULTILIB_SSTATE_ARCHS ??= "${SSTATE_ARCHS}" python () { from oe.cve_check import extend_cve_status extend_cve_status(d) + if d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1": + d.setVar("SPDX_INCLUDE_SOURCES", "1") } def create_spdx_source_deps(d): diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 61d7ba45e3..beeafc2bb7 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -156,6 +156,11 @@ def add_package_files( bb.note(f"Skip {topdir}") return spdx_files + check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1" + if check_compiled_sources: + compiled_sources, types = oe.spdx_common.get_compiled_sources(d) + bb.debug(1, f"Total compiled files: {len(compiled_sources)}") + for subdir, dirs, files in os.walk(topdir, onerror=walk_error): dirs[:] = [d for d in dirs if d not in ignore_dirs] if subdir == str(topdir): @@ -171,6 +176,11 @@ def add_package_files( filename = str(filepath.relative_to(topdir)) file_purposes = get_purposes(filepath) + # Check if file is compiled + if check_compiled_sources: + if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + continue + spdx_file = objset.new_file( get_spdxid(file_counter), filename, diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 4caefc7673..c2dec65563 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -242,3 +242,44 @@ def fetch_data_to_uri(fd, name): uri = uri + "@" + fd.revision return uri + +def is_compiled_source (filename, compiled_sources, types): + """ + Check if the file is a compiled file + """ + import os + # If we don't have compiled source, we assume all are compiled. + if not compiled_sources: + return True + + # We return always true if the file type is not in the list of compiled files. + # Some files in the source directory are not compiled, for example, Makefiles, + # but also python .py file. We need to include them in the SPDX. + basename = os.path.basename(filename) + ext = basename.partition(".")[2] + if ext not in types: + return True + # Check that the file is in the list + return filename in compiled_sources + +def get_compiled_sources(d): + """ + Get list of compiled sources from debug information and normalize the paths + """ + import itertools + source_info = oe.package.read_debugsources_info(d) + if not source_info: + bb.debug(1, "Do not have debugsources.list. Skipping") + return [], [] + + # Sources are not split now in SPDX, so we aggregate them + sources = set(itertools.chain.from_iterable(source_info.values())) + # Check extensions of files + types = set() + for src in sources: + basename = os.path.basename(src) + ext = basename.partition(".")[2] + if ext not in types and ext: + types.add(ext) + bb.debug(1, f"Num of sources: {len(sources)} and types: {len(types)} {str(types)}") + return sources, types From patchwork Tue Jun 10 15:24:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Turull X-Patchwork-Id: 64713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFED4C5B552 for ; Tue, 10 Jun 2025 15:25:07 +0000 (UTC) Received: from DUZPR83CU001.outbound.protection.outlook.com (DUZPR83CU001.outbound.protection.outlook.com [52.101.66.67]) by mx.groups.io with SMTP id smtpd.web11.89529.1749569104967939760 for ; Tue, 10 Jun 2025 08:25:05 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@ericsson.com header.s=selector1 header.b=KBZc1x+j; spf=pass (domain: ericsson.com, ip: 52.101.66.67, mailfrom: edaturu@ericsson.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DCXJV+7fuiVaoD/In+lPICoc5MRTgQy0jmcqqbLLIc3bryGUBkTdD/Rf3VHHXAabu6G12lGfPAuv7YzNUTzuHm3mhOEUoaMEj6gx8XwdDYc63oZ000BuJVbOI22FT7UszYq+PJs2GiR/EYpIu8DepS87dFTJk5irg1FYiGiHveMTOpZGuK2DW46OTpt2vG/MiivLn0JqRnMW+KTPmtHy283kpEXVR2DVtxNuqwflSUlRaEu78/R5942DbFrNhBMHJedAUDdCIYHYZOH4jOZeMQk0NAW1tffgZ49Fb+tRnELA6O2GaY8sP2Y0hQfTcFBXO/ltR6GKpLEiPD2VmUk2fQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=60bZ4rj/t8mQXWNl4Bx5T1hSRtaFFSCRT/Dc2YF3B0M=; b=DhjXYPm7j3PE+CFmh8ZG5S3kAG2u8A+fBCx7jlBniiD0BWt4df68SDzF4xOvr5gtlsf6jP9aIYOckNIY1qjsT+vdRhnD2K5nOubeXix6b6K8ZWVnSqeiuEk3mvBUZNt/YtV79otJmyGzNdtJybkgMu9e87oizweFC7bFQPTv2F+YC/8CCoQyxe1bpVCsSF26IZTXFWnNAvuTLHP9C0m/aVZwnYuLVeI7NdrLIPqJ2VyU4LQsEsacEDtHcEIReatPP3wd3Y/dfsX5IPb4KnIslJlcwk4b771/wSdV6f1DJLI0K9BsabnZ2ywxmPqnPih7SGDKqUGwNAGlX7o/3Rfz7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=gmail.com smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=60bZ4rj/t8mQXWNl4Bx5T1hSRtaFFSCRT/Dc2YF3B0M=; b=KBZc1x+jXqgggok4/GkolvEzoUCqc1RCh34he2M+oB8E6Ow/ypg6ebVM0z7NQqGhghGzuhns2y3sSSoTAAoZpvTpRlZzC5Xzceim8+JGFvRIbW4BGQHhPtayprSLfzQrKONO6OvEqDSu+RkpL0JXQa9FmcqGobVh5TWZV2OygMngvtEpSQs4u0lmt6M6yi4YgJTgXRtSdqjB9mqrXUfukAbarbsGqX4S+dcVE3o/0237+b3gZRL+v93/639JYfa8DfkGRDPJGYt51PduTg3TENFc3satF8ndgjDuilP0frXmmNG8s26UNu7H1W5ASzBWH0pumiaKjYs8ePGCwMOHBw== Received: from PR2P264CA0044.FRAP264.PROD.OUTLOOK.COM (2603:10a6:101:1::32) by GV4PR07MB10507.eurprd07.prod.outlook.com (2603:10a6:150:29a::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8835.18; Tue, 10 Jun 2025 15:24:57 +0000 Received: from AMS0EPF00000193.eurprd05.prod.outlook.com (2603:10a6:101:1:cafe::f8) by PR2P264CA0044.outlook.office365.com (2603:10a6:101:1::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8792.35 via Frontend Transport; Tue, 10 Jun 2025 15:24:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by AMS0EPF00000193.mail.protection.outlook.com (10.167.16.212) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8835.15 via Frontend Transport; Tue, 10 Jun 2025 15:24:57 +0000 Received: from seroius18814.sero.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.62) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Tue, 10 Jun 2025 17:24:56 +0200 Received: from seroius08462.sero.gic.ericsson.se (seroius08462.sero.gic.ericsson.se [10.63.237.245]) by seroius18814.sero.gic.ericsson.se (Postfix) with ESMTP id 11E6C4020AA6; Tue, 10 Jun 2025 17:24:56 +0200 (CEST) Received: by seroius08462.sero.gic.ericsson.se (Postfix, from userid 160155) id F10F97083F57; Tue, 10 Jun 2025 17:24:55 +0200 (CEST) From: To: CC: Daniel Turull , Peter Marko , Marta Rybczynska Subject: [PATCH v8 3/3] improve_kernel_cve_report: add script for postprocesing of kernel CVE data Date: Tue, 10 Jun 2025 17:24:43 +0200 Message-ID: <20250610152443.2162164-4-daniel.turull@ericsson.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250610152443.2162164-1-daniel.turull@ericsson.com> References: <20250610152443.2162164-1-daniel.turull@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AMS0EPF00000193:EE_|GV4PR07MB10507:EE_ X-MS-Office365-Filtering-Correlation-Id: 13588f52-239e-4ec0-9c56-08dda832f588 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|82310400026|13003099007; X-Microsoft-Antispam-Message-Info: d5w8a2eu5mn/9nEp6i/C0jtt9fT4IER8Qx/uavkJt+yxlHcklyfaHNIOc1Sn8dX9A5VJ3ugZGIz9l2ZqNJTGbq9z9MwvOt4XkiCFkJ2A58M52nNi0hCSBbX18I996UOXQlZiJkqthR7ULc3vmWKlHCRow6kHnStteQVc4FjUg9YePaGdN1iStC/SZvb5ln4boArhsBc4INa7Ha7NWdHC+1yoi502O/myjX/vl8rg90lY132zVPmPxFPbQ8QtG6fi6f/03G49g+SVcqOpzNXyMEW7GjqMxpvbomsuBb/N2mzz1KTgqa+rHYKQ8LV/O6mD5NWvv54GklNFtd4Shy9BiYy8SJ8MXfK+Q9ByHVzApgjjpeQUIdvbhkW1BO857pahPmjKhPkR11hiISAtJswEmUoR5tG7GL/DoIe11DgmkGxItwxJ7HuoJSKSPn7d9ioDTe6zSynCaXnZ8kPCCdSjzKve1vJ8FSiB2bD+Bz8C6Wr06lpeLz8UwX6qjgoO9qPAMUzozAVcnX9MVWbn2MO8MVOuw2zwbs5m1kSB6Vtj/PbpFdt7CtiFrTkiPvA+ojie2C1GRW/S9CI53xSTGBE5UXNObJk69my6733fPGgy4uWnITTOqeJrHTi6yYodVWlQ52a5sO444v17EyK+uhCRTjKfKGmuph48jOdN/s4Q4m2J1mxLNRbB+oSsAoNhKSSfHbGPwGhjZBPkTnl7k41KiVnJK6uTVX5lCcH+06B82MAn01CRpTWbAqQXfkiyQdXejC9WKtzaV6OhiR+7iw8HzyYdXFtUYCyI0rzhFRSKDQyJJ+IN/rzi5+Ilg3N5aePX7Qs5mBeBjOYsx6nipSaK9kWWq2zS6n8bKwMWPNb84wEypcandKdgNJiau4H7gXC+ZlGh+UzuLlJ3GNpp6JcqlREucF8KIRMnsb9N2jEa+IIIBJYJJ2xxDK/1KZf6St6hE7Aqa8Jp2UaA1aN7FLB+2S6gcu2DM+iORdN4c3UW0AFjCyzlF+hY7em38ZiL3/P6Lfgkb6YOhZyUGVKgZ5hgBvbvYhzatqilOx0wo2gUMOWpxSjFYx5ZY04MehS2v/kOu4ytEt8IiRQcV5bbpCSP/0rCBa8bRcSgmo2XdYfz05XIwj9FeZhkt3zrFW6msJ0/AqSaj6k21SA7skCBNohh/Nnh5wpI2C9ECRBnOmm8UpUcEMQkQQsZmHP9af4jdvzskt5AC+SUsEZRNxFrtVTqq1akWWhNuHXUAyzAiiJ4TIbnMLTDkElxaSMfwLaV0kDUH7Rr5UvaFECkxKUacfAyET0VRXOEV+aJpFhZf3W3v2qzBUWeYKpTEIfFmo1RVqG0FzSvNqT4cnpRyuekSQPXo1UhG4iFBAZyspcHDFl8Eq3c63euM3tHodaDxBFWSI/DL9wz6BR/0Z0BhqR1s7XTGY6uHTm7kCAJFGaseH/C2kCF1ZHaGIR7fG+D6mySo/ztYoH/NJZNs1U74ZmjiKGU+uf+nzAYnYN4/u7xl5mTwmwVkz3elIpracHTRrOWV+EYrN7JJiCg3w2dmlNLH8rHaw== X-Forefront-Antispam-Report: CIP:192.176.1.74;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:oa.msg.ericsson.com;PTR:office365.se.ericsson.net;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026)(13003099007);DIR:OUT;SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jun 2025 15:24:57.7062 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 13588f52-239e-4ec0-9c56-08dda832f588 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f;Ip=[192.176.1.74];Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: AMS0EPF00000193.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV4PR07MB10507 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 10 Jun 2025 15:25:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/218363 From: Daniel Turull Adding postprocessing script to process data from linux CNA that includes more accurate metadata and it is updated directly by the source. Example of enhanced CVE from a report from cve-check: { "id": "CVE-2024-26710", "status": "Ignored", "link": "https://nvd.nist.gov/vuln/detail/CVE-2024-26710", "summary": "In the Linux kernel, the following vulnerability [...]", "scorev2": "0.0", "scorev3": "5.5", "scorev4": "0.0", "modified": "2025-03-17T15:36:11.620", "vector": "LOCAL", "vectorString": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H", "detail": "not-applicable-config", "description": "Source code not compiled by config. ['arch/powerpc/include/asm/thread_info.h']" }, And same from a report generated with vex: { "id": "CVE-2024-26710", "status": "Ignored", "link": "https://nvd.nist.gov/vuln/detail/CVE-2024-26710", "detail": "not-applicable-config", "description": "Source code not compiled by config. ['arch/powerpc/include/asm/thread_info.h']" }, For unpatched CVEs, provide more context in the description: Tested with 6.12.22 kernel { "id": "CVE-2025-39728", "status": "Unpatched", "link": "https://nvd.nist.gov/vuln/detail/CVE-2025-39728", "summary": "In the Linux kernel, the following vulnerability has been [...], "scorev2": "0.0", "scorev3": "0.0", "scorev4": "0.0", "modified": "2025-04-21T14:23:45.950", "vector": "UNKNOWN", "vectorString": "UNKNOWN", "detail": "version-in-range", "description": "Needs backporting (fixed from 6.12.23)" }, CC: Peter Marko CC: Marta Rybczynska Signed-off-by: Daniel Turull --- scripts/contrib/improve_kernel_cve_report.py | 467 +++++++++++++++++++ 1 file changed, 467 insertions(+) create mode 100755 scripts/contrib/improve_kernel_cve_report.py diff --git a/scripts/contrib/improve_kernel_cve_report.py b/scripts/contrib/improve_kernel_cve_report.py new file mode 100755 index 0000000000..829cc4cd30 --- /dev/null +++ b/scripts/contrib/improve_kernel_cve_report.py @@ -0,0 +1,467 @@ +#! /usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# The script uses another source of CVE information from linux-vulns +# to enrich the cve-summary from cve-check or vex. +# It can also use the list of compiled files from the kernel spdx to ignore CVEs +# that are not affected since the files are not compiled. +# +# It creates a new json file with updated CVE information +# +# Compiled files can be extracted adding the following in local.conf +# SPDX_INCLUDE_COMPILED_SOURCES:pn-linux-yocto = "1" +# +# Tested with the following CVE sources: +# - https://git.kernel.org/pub/scm/linux/security/vulns.git +# - https://github.com/CVEProject/cvelistV5 +# +# Example: +# python3 ./openembedded-core/scripts/contrib/improve_kernel_cve_report.py --spdx tmp/deploy/spdx/3.0.1/qemux86_64/recipes/recipe-linux-yocto.spdx.json --kernel-version 6.12.27 --datadir ./vulns +# python3 ./openembedded-core/scripts/contrib/improve_kernel_cve_report.py --spdx tmp/deploy/spdx/3.0.1/qemux86_64/recipes/recipe-linux-yocto.spdx.json --datadir ./vulns --old-cve-report build/tmp/log/cve/cve-summary.json +# +# SPDX-License-Identifier: GPLv2 + +import argparse +import json +import sys +import logging +import glob +import os +import pathlib +from packaging.version import Version + +def is_linux_cve(cve_info): + '''Return true is the CVE belongs to Linux''' + if not "affected" in cve_info["containers"]["cna"]: + return False + for affected in cve_info["containers"]["cna"]["affected"]: + if not "product" in affected: + return False + if affected["product"] == "Linux" and affected["vendor"] == "Linux": + return True + return False + +def get_kernel_cves(datadir, compiled_files, version): + """ + Get CVEs for the kernel + """ + cves = {} + + check_config = len(compiled_files) > 0 + + base_version = Version(f"{version.major}.{version.minor}") + + # Check all CVES from kernel vulns + pattern = os.path.join(datadir, '**', "CVE-*.json") + cve_files = glob.glob(pattern, recursive=True) + not_applicable_config = 0 + fixed_as_later_backport = 0 + vulnerable = 0 + not_vulnerable = 0 + for cve_file in sorted(cve_files): + cve_info = {} + with open(cve_file, "r", encoding='ISO-8859-1') as f: + cve_info = json.load(f) + + if len(cve_info) == 0: + logging.error("Not valid data in %s. Aborting", cve_file) + break + + if not is_linux_cve(cve_info): + continue + cve_id = os.path.basename(cve_file)[:-5] + description = cve_info["containers"]["cna"]["descriptions"][0]["value"] + if cve_file.find("rejected") >= 0: + logging.debug("%s is rejected by the CNA", cve_id) + cves[cve_id] = { + "id": cve_id, + "status": "Ignored", + "detail": "rejected", + "summary": description, + "description": f"Rejected by CNA" + } + continue + if any(elem in cve_file for elem in ["review", "reverved", "testing"]): + continue + + is_vulnerable, first_affected, last_affected, better_match_first, better_match_last, affected_versions = get_cpe_applicability(cve_info, version) + + logging.debug("%s: %s (%s - %s) (%s - %s)", cve_id, is_vulnerable, better_match_first, better_match_last, first_affected, last_affected) + + if is_vulnerable is None: + logging.warning("%s doesn't have good metadata", cve_id) + if is_vulnerable: + is_affected = True + affected_files = [] + if check_config: + is_affected, affected_files = check_kernel_compiled_files(compiled_files, cve_info) + + if not is_affected and len(affected_files) > 0: + logging.debug( + "%s - not applicable configuration since affected files not compiled: %s", + cve_id, affected_files) + cves[cve_id] = { + "id": cve_id, + "status": "Ignored", + "detail": "not-applicable-config", + "summary": description, + "description": f"Source code not compiled by config. {affected_files}" + } + not_applicable_config +=1 + # Check if we have backport + else: + if not better_match_last: + fixed_in = last_affected + else: + fixed_in = better_match_last + logging.debug("%s needs backporting (fixed from %s)", cve_id, fixed_in) + cves[cve_id] = { + "id": cve_id, + "status": "Unpatched", + "detail": "version-in-range", + "summary": description, + "description": f"Needs backporting (fixed from {fixed_in})" + } + vulnerable += 1 + if (better_match_last and + Version(f"{better_match_last.major}.{better_match_last.minor}") == base_version): + fixed_as_later_backport += 1 + # Not vulnerable + else: + if not first_affected: + logging.debug("%s - not known affected %s", + cve_id, + better_match_last) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "version-not-in-range", + "summary": description, + "description": "No CPE match" + } + not_vulnerable += 1 + continue + backport_base = Version(f"{better_match_last.major}.{better_match_last.minor}") + if version < first_affected: + logging.debug('%s - fixed-version: only affects %s onwards', + cve_id, + first_affected) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "fixed-version", + "summary": description, + "description": f"only affects {first_affected} onwards" + } + not_vulnerable += 1 + elif last_affected <= version: + logging.debug("%s - fixed-version: Fixed from version %s", + cve_id, + last_affected) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "fixed-version", + "summary": description, + "description": f"fixed-version: Fixed from version {last_affected}" + } + not_vulnerable += 1 + elif backport_base == base_version: + logging.debug("%s - cpe-stable-backport: Backported in %s", + cve_id, + better_match_last) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "cpe-stable-backport", + "summary": description, + "description": f"Backported in {better_match_last}" + } + not_vulnerable += 1 + else: + logging.debug("%s - version not affected %s", cve_id, str(affected_versions)) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "version-not-in-range", + "summary": description, + "description": f"Range {affected_versions}" + } + not_vulnerable += 1 + + logging.info("Total CVEs ignored due to not applicable config: %d", not_applicable_config) + logging.info("Total CVEs not vulnerable due version-not-in-range: %d", not_vulnerable) + logging.info("Total vulnerable CVEs: %d", vulnerable) + + logging.info("Total CVEs already backported in %s: %s", base_version, + fixed_as_later_backport) + return cves + +def read_spdx(spdx_file): + '''Open SPDX file and extract compiled files''' + with open(spdx_file, 'r', encoding='ISO-8859-1') as f: + spdx = json.load(f) + if "spdxVersion" in spdx: + if spdx["spdxVersion"] == "SPDX-2.2": + return read_spdx2(spdx) + if "@graph" in spdx: + return read_spdx3(spdx) + return [] + +def read_spdx2(spdx): + ''' + Read spdx2 compiled files from spdx + ''' + cfiles = set() + if 'files' not in spdx: + return cfiles + for item in spdx['files']: + for ftype in item['fileTypes']: + if ftype == "SOURCE": + filename = item["fileName"][item["fileName"].find("/")+1:] + cfiles.add(filename) + return cfiles + +def read_spdx3(spdx): + ''' + Read spdx3 compiled files from spdx + ''' + cfiles = set() + for item in spdx["@graph"]: + if "software_primaryPurpose" not in item: + continue + if item["software_primaryPurpose"] == "source": + filename = item['name'][item['name'].find("/")+1:] + cfiles.add(filename) + return cfiles + +def check_kernel_compiled_files(compiled_files, cve_info): + """ + Return if a CVE affected us depending on compiled files + """ + files_affected = set() + is_affected = False + + for item in cve_info['containers']['cna']['affected']: + if "programFiles" in item: + for f in item['programFiles']: + if f not in files_affected: + files_affected.add(f) + + if len(files_affected) > 0: + for f in files_affected: + if f in compiled_files: + logging.debug("File match: %s", f) + is_affected = True + return is_affected, files_affected + +def get_cpe_applicability(cve_info, v): + ''' + Check if version is affected and return affected versions + ''' + base_branch = Version(f"{v.major}.{v.minor}") + affected = [] + if not 'cpeApplicability' in cve_info["containers"]["cna"]: + return None, None, None, None, None, None + + for nodes in cve_info["containers"]["cna"]["cpeApplicability"]: + for node in nodes.values(): + vulnerable = False + matched_branch = False + first_affected = Version("5000") + last_affected = Version("0") + better_match_first = Version("0") + better_match_last = Version("5000") + + if len(node[0]['cpeMatch']) == 0: + first_affected = None + last_affected = None + better_match_first = None + better_match_last = None + + for cpe_match in node[0]['cpeMatch']: + version_start_including = Version("0") + version_end_excluding = Version("0") + if 'versionStartIncluding' in cpe_match: + version_start_including = Version(cpe_match['versionStartIncluding']) + else: + version_start_including = Version("0") + # if versionEndExcluding is missing we are in a branch, which is not fixed. + if "versionEndExcluding" in cpe_match: + version_end_excluding = Version(cpe_match["versionEndExcluding"]) + else: + # if versionEndExcluding is missing we are in a branch, which is not fixed. + version_end_excluding = Version( + f"{version_start_including.major}.{version_start_including.minor}.5000" + ) + affected.append(f" {version_start_including}-{version_end_excluding}") + # Detect if versionEnd is in fixed in base branch. It has precedence over the rest + branch_end = Version(f"{version_end_excluding.major}.{version_end_excluding.minor}") + if branch_end == base_branch: + if version_start_including <= v < version_end_excluding: + vulnerable = cpe_match['vulnerable'] + # If we don't match in our branch, we are not vulnerable, + # since we have a backport + matched_branch = True + better_match_first = version_start_including + better_match_last = version_end_excluding + if version_start_including <= v < version_end_excluding and not matched_branch: + if version_end_excluding < better_match_last: + better_match_first = max(version_start_including, better_match_first) + better_match_last = min(better_match_last, version_end_excluding) + vulnerable = cpe_match['vulnerable'] + matched_branch = True + + first_affected = min(version_start_including, first_affected) + last_affected = max(version_end_excluding, last_affected) + # Not a better match, we use the first and last affected instead of the fake .5000 + if vulnerable and better_match_last == Version(f"{base_branch}.5000"): + better_match_last = last_affected + better_match_first = first_affected + return vulnerable, first_affected, last_affected, better_match_first, better_match_last, affected + +def copy_data(old, new): + '''Update dictionary with new entries, while keeping the old ones''' + for k in new.keys(): + old[k] = new[k] + return old + +# Function taken from cve_check.bbclass. Adapted to cve fields +def cve_update(cve_data, cve, entry): + # If no entry, just add it + if cve not in cve_data: + cve_data[cve] = entry + return + # If we are updating, there might be change in the status + if cve_data[cve]['status'] == "Unknown": + cve_data[cve] = copy_data(cve_data[cve], entry) + return + if cve_data[cve]['status'] == entry['status']: + return + if entry['status'] == "Unpatched" and cve_data[cve]['status'] == "Patched": + logging.warning("CVE entry %s update from Patched to Unpatched from the scan result", cve) + cve_data[cve] = copy_data(cve_data[cve], entry) + return + if entry['status'] == "Patched" and cve_data[cve]['status'] == "Unpatched": + logging.warning("CVE entry %s update from Unpatched to Patched from the scan result", cve) + cve_data[cve] = copy_data(cve_data[cve], entry) + return + # If we have an "Ignored", it has a priority + if cve_data[cve]['status'] == "Ignored": + logging.debug("CVE %s not updating because Ignored", cve) + return + # If we have an "Ignored", it has a priority + if entry['status'] == "Ignored": + cve_data[cve] = copy_data(cve_data[cve], entry) + logging.debug("CVE entry %s updated from Unpatched to Ignored", cve) + return + logging.warning("Unhandled CVE entry update for %s %s from %s %s to %s", + cve, cve_data[cve]['status'], cve_data[cve]['detail'], entry['status'], entry['detail']) + +def main(): + parser = argparse.ArgumentParser( + description="Update cve-summary with kernel compiled files and kernel CVE information" + ) + parser.add_argument( + "-s", + "--spdx", + help="SPDX2/3 for the kernel. Needs to include compiled sources", + ) + parser.add_argument( + "--datadir", + type=pathlib.Path, + help="Directory where CVE data is", + required=True + ) + parser.add_argument( + "--old-cve-report", + help="CVE report to update. (Optional)", + ) + parser.add_argument( + "--kernel-version", + help="Kernel version. Needed if old cve_report is not provided (Optional)", + type=Version + ) + parser.add_argument( + "--new-cve-report", + help="Output file", + default="cve-summary-enhance.json" + ) + parser.add_argument( + "-D", + "--debug", + help='Enable debug ', + action="store_true") + + args = parser.parse_args() + + if args.debug: + log_level=logging.DEBUG + else: + log_level=logging.INFO + logging.basicConfig(format='[%(filename)s:%(lineno)d] %(message)s', level=log_level) + + if not args.kernel_version and not args.old_cve_report: + parser.error("either --kernel-version or --old-cve-report are needed") + return -1 + + # by default we don't check the compiled files, unless provided + compiled_files = [] + if args.spdx: + compiled_files = read_spdx(args.spdx) + logging.info("Total compiled files %d", len(compiled_files)) + + if args.old_cve_report: + with open(args.old_cve_report, encoding='ISO-8859-1') as f: + cve_report = json.load(f) + else: + #If summary not provided, we create one + cve_report = { + "version": "1", + "package": [ + { + "name": "linux-yocto", + "version": str(args.kernel_version), + "products": [ + { + "product": "linux_kernel", + "cvesInRecord": "Yes" + } + ], + "issue": [] + } + ] + } + + for pkg in cve_report['package']: + is_kernel = False + for product in pkg['products']: + if product['product'] == "linux_kernel": + is_kernel=True + if not is_kernel: + continue + + kernel_cves = get_kernel_cves(args.datadir, + compiled_files, + Version(pkg["version"])) + logging.info("Total kernel cves from kernel CNA: %s", len(kernel_cves)) + cves = {issue["id"]: issue for issue in pkg["issue"]} + logging.info("Total kernel before processing cves: %s", len(cves)) + + for cve in kernel_cves: + cve_update(cves, cve, kernel_cves[cve]) + + pkg["issue"] = [] + for cve in sorted(cves): + pkg["issue"].extend([cves[cve]]) + logging.info("Total kernel cves after processing: %s", len(pkg['issue'])) + + with open(args.new_cve_report, "w", encoding='ISO-8859-1') as f: + json.dump(cve_report, f, indent=2) + + return 0 + +if __name__ == "__main__": + sys.exit(main()) +