From patchwork Wed Jun 4 11:21:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Turull X-Patchwork-Id: 64251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBC16C5B549 for ; Wed, 4 Jun 2025 11:22:01 +0000 (UTC) Received: from DUZPR83CU001.outbound.protection.outlook.com (DUZPR83CU001.outbound.protection.outlook.com [52.101.66.26]) by mx.groups.io with SMTP id smtpd.web11.14200.1749036112002160466 for ; Wed, 04 Jun 2025 04:21:52 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@ericsson.com header.s=selector1 header.b=ctU6pPLu; spf=pass (domain: ericsson.com, ip: 52.101.66.26, mailfrom: edaturu@ericsson.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=WJvwYxblZ1bxhp7YI86LxE/Cuk+QN6K9NFBSBlP4FKj7ZMb0HuD5tUgZYrJVi2GvmjqpZ6rEeCzyTwjSLTl441dl++xQL7S/+N6usYpa3IrzSVDN6DlIG4+wJHa1ixi40cQYX4TZRDFoRu4ne6vuc3cqe+sWr5g3saXn7U8d2ZQpPv+frvW2PckE8wkSEVoAyNf3FD4LdjtcpHPsiGfhIuu+kaGWSr4YcQOycN+7OJJFX1kwfrtn6A8x1cyg2++5DB1zKL/4PanTyi0je87onZelt+H0TCPbLngT3fDiGUxplxiL7pFKFd78ijST+W5nO9DL2IK1DDTwckKk4fpZjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bnSW74qMcBydtWb7+i5K+VeqYjIm2JbO3YMLAQgAbdA=; b=snQzB4lgmuYEmriJTun22RCTjpTSrpbbHk6QkxcSGJ1tewLVGcOe0iGLDWMngisfczzi8L8jI0yCYvEi65i+OM9Idvcw+IdN89TBc91jgSyYush0FApcXOpZHWFPAuQKrEQ5nIMV2mUw7eoxTvOIc1Cy885fkzAuhxWRXmwkYxgzwxueYGzv5AtNHDpavZvTWE2rctm2Z48/604zLNbaIXD0HorQ33UcUckd/tYArq+osVUbsZwJbOpPkh6yxBPILH2APedjjmtvLAQa1Jue8bSsgAiuc5yF9PPEHCzXWCKv13TLzAFPxmzUIDCHDwMgnKPjhB+uqjq8HtyZUFZ7rQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=linuxfoundation.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bnSW74qMcBydtWb7+i5K+VeqYjIm2JbO3YMLAQgAbdA=; b=ctU6pPLuml4Vbcdn8Gyz8Z08tSIo4N3SZfcmS5r/P8n5AMJUhSigJWhVkBe3fzLRnKVJiGikbQcT6PDIvEFkxsCR7wAvkiL5QEGvrl2FuAkiA4t6Y18qCIZE39UbSBM0NCYkWq4YgwnYlZzt99arYdtu7wekiRo+WmMaREK27JcOve4pbmfrI/X5xPMh8vNdokVQPWFpNTgFH/DUnduSfHnLPgsFm/vXgortNe/O2en5f9ax5RSb7+22gQ1rk7QQiPCQmdebn9JotoTzA4zYP5kYVTTyIblNMcXVoudHQiaj7mN4CNsG1Fc5qh3+JWQbputAcwFAKtGphrP7Dg0nEw== Received: from DUZPR01CA0276.eurprd01.prod.exchangelabs.com (2603:10a6:10:4b9::25) by PAWPR07MB10045.eurprd07.prod.outlook.com (2603:10a6:102:38e::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8813.20; Wed, 4 Jun 2025 11:21:48 +0000 Received: from DU6PEPF00009526.eurprd02.prod.outlook.com (2603:10a6:10:4b9:cafe::14) by DUZPR01CA0276.outlook.office365.com (2603:10a6:10:4b9::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8792.32 via Frontend Transport; Wed, 4 Jun 2025 11:21:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by DU6PEPF00009526.mail.protection.outlook.com (10.167.8.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8792.29 via Frontend Transport; Wed, 4 Jun 2025 11:21:48 +0000 Received: from seroius18813.sero.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.63) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 4 Jun 2025 13:21:47 +0200 Received: from seroius08462.sero.gic.ericsson.se (seroius08462.sero.gic.ericsson.se [10.63.237.245]) by seroius18813.sero.gic.ericsson.se (Postfix) with ESMTP id 3AD5695145; Wed, 4 Jun 2025 13:21:47 +0200 (CEST) Received: by seroius08462.sero.gic.ericsson.se (Postfix, from userid 160155) id 209B17083F54; Wed, 4 Jun 2025 13:21:47 +0200 (CEST) From: To: CC: Daniel Turull , Richard Purdie Subject: [PATCH v6 1/3] package: export debugsources in PKGDESTWORK as json Date: Wed, 4 Jun 2025 13:21:31 +0200 Message-ID: <20250604112133.2581063-2-daniel.turull@ericsson.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250604112133.2581063-1-daniel.turull@ericsson.com> References: <20250604112133.2581063-1-daniel.turull@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DU6PEPF00009526:EE_|PAWPR07MB10045:EE_ X-MS-Office365-Filtering-Correlation-Id: 6e1eefa4-26fb-4c7a-0c0e-08dda359fef1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|376014|7053199007; X-Microsoft-Antispam-Message-Info: zgDeN4Hq6YUcokNY74pUb3w5Hn/3GB8OXfLajDfp3oaekikpe/uXvw9N8hsifkFwrcT+3psYyRJXZR8W/5+1YcXAHeBrosLeL8rO0ie+QYWW5il56YHMOuBpqNj/6hb9UC78vC9G3JfQdOhe8VwFligdORiFAjV9KnHBux1ZOc5SOs3zG51jkRTkIk4TH9RQol1+cGGKomVh8UCl/TzBrE/c87tQR+aYPAr6wRb0NLqs/lMFhhUgrBqWQmTzjIXxDtlr62kWpgCjNnzJfPOnVTTrU/GoXaZddkfR1SQ+wH4v3Pby6kMR286FQdVf0tm944x/RsxzzUFXdJNVMMb7h3MnDUrMhQT8xDjZUKdWhZxCxIB+N3CoS7QSl/ahGbP0JcRjempeF0Av06Fck33O+EIq4natF5SfEwa2XEa6dshY8lS2fxSv2v0V1AvRuQFdZxdjw+1uv1sohP9ZDgJgBtNUw7/sn8IBYnNjPM/b1fiFQFVG3CiF//dcqEKMjMQ1DPD3kDwk753dr/MiVtWh1UTBgzdnRRDKHVH7rbO2Lq0DOgIlEQq10J0kbAYmOWHXNH7lHmB0s/3WP4tmg2EiVFcjaLVmP80Hcr205KQ6+8cbAYoQXwAKKL52HnKxeV06Ztkx4iouPF/HdU7Elam9CP/KqLB4Klqr4SiDX57V7NXaUndA1IS0Pq1mJ4UdVUILsucSg3WBzc+s+nG5FJ3PQ4Pf2p8jpNTweFhhNNVaZDQF+vkev1bKZNGau1aHTvEADNq+R/AAWUc3OKzTmRboS5oKLxXfr/HwKtZqdmFXwfXQArbrKxAeLbpfklvMgy1nd0NM3o+CMw1Rlwx3014d1orSmWmQ9mq5N6oz8ZezZx2A4HEcrOXkR+oSKfJaTIzRPEG/1o903qusBCNn20ceab8ovjvNxCf2uPt0nhC0TC458eQ/oGgOKwC9Mc9tYCNpBkMOKxP8NPwpTdDo1t1HmaIE2SB2T4B9Mqm6hFPifibDZXHlWoFAzBdF11p1V5LJRRQc5aoxIUhpOlS0qZwqRUTYuzhapPm7gjlh6clt0w+9TeMBIGntXbV1ktcPDhzrFcvdxGDFchhwQxKUyTlO5GZkhLT1irXFCXU/dmVbUARS55RDyf5ZReRcIh7O1u0jDFEvZirFsDKV84m63MgapZ5P0gSbVkvJDF2ZatPn0ZOoMYfvpt1FYXQVH6NBFBJJ/BlRg5RziMBYBxYC/HFOA7oSLYKfiyAnxQ84G5V7MUYcEhFsVMLsDF+yYJfpPlM9Vd9kRADsViHRjDtSw0Mg2V8WyeVil2LCnzanELCD+NXai9MxudRS4WFMgcs9fmCgq28PIyXSKg0cMP0JkigisDIwFrJqQ3X4Tt/XMiQCHfcwe5J+pVoqiWBhTBtjx5UyHfgwOb4/qTSBqm/UXUtmSEjSoTqUXZ3mWyd4y4jrm/tbMMbMdnNifem21kR92fkarhHOUsA/yVU7syZfh8VWKYUqcJzWD0CHGIKGGwmrimU+NgBFZVu7D79cOOu+lkxW X-Forefront-Antispam-Report: CIP:192.176.1.74;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:oa.msg.ericsson.com;PTR:office365.se.ericsson.net;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(376014)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2025 11:21:48.0221 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6e1eefa4-26fb-4c7a-0c0e-08dda359fef1 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f;Ip=[192.176.1.74];Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: DU6PEPF00009526.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR07MB10045 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jun 2025 11:22:01 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/217908 From: Daniel Turull The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy. Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd Format: { binary1: [src1, src2, ...], binary2: [src1, src2, ...] } I checked the sstate size, and it slightly increase using core-image-full-cmdline: Before: 2454884 B After: 2456860 B (+1976 B or 0,08%) CC: Richard Purdie Signed-off-by: Daniel Turull --- meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index 0bcc04ea54..c5acb7d205 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -1049,6 +1049,49 @@ def copydebugsources(debugsrcdir, sources, d): if os.path.exists(p) and not os.listdir(p): os.rmdir(p) +def save_debugsources_info(debugsrcdir, sources_raw, d): + import json + import bb.compress.zstd + if debugsrcdir and sources_raw: + debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd") + debugsources_dir = os.path.dirname(debugsources_file) + if not os.path.isdir(debugsources_dir): + bb.utils.mkdirhier(debugsources_dir) + bb.utils.remove(debugsources_file) + + workdir = d.getVar("WORKDIR") + sdir = d.getVar("S") + pn = d.getVar('PN') + pv = d.getVar('PV') + sources_dict = {} + for file, src_files in sources_raw: + file_clean = file.replace(f"{workdir}/package/","") + sources_clean = [ + src.replace(f"/usr/src/debug/{pn}/", "") + .replace(f"{sdir}/", "") + .replace(f"/usr/src/kernel/", "") + .replace(f"/usr/src/{pn}/", "") + .replace(f"{pn}-{pv}/", "") + .replace(f"{pv}/", "") + for src in src_files + if not any(keyword in src for keyword in ("", "")) and not src.endswith("/") + ] + sources_dict[file_clean] = sorted(sources_clean) + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(debugsources_file, "wt", encoding="utf-8", num_threads=num_threads) as f: + json.dump(sources_dict, f, sort_keys=True) + +def read_debugsources_file(d): + import json + import bb.compress.zstd + try: + fn = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd") + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(fn, "rt", encoding="utf-8", num_threads=num_threads) as f: + return json.load(f) + except FileNotFoundError: + bb.debug(1, f"File not found: {fn}") + return None def process_split_and_strip_files(d): cpath = oe.cachedpath.CachedPath() @@ -1280,6 +1323,9 @@ def process_split_and_strip_files(d): # Process the dv["srcdir"] if requested... # This copies and places the referenced sources for later debugging... copydebugsources(dv["srcdir"], sources, d) + + # Save source info to be accessible to other tasks + save_debugsources_info(dv["srcdir"], results, d) # # End of debug splitting # From patchwork Wed Jun 4 11:21:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Turull X-Patchwork-Id: 64252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0601C5B543 for ; Wed, 4 Jun 2025 11:22:01 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (EUR05-DB8-obe.outbound.protection.outlook.com [40.107.20.64]) by mx.groups.io with SMTP id smtpd.web10.14157.1749036113797800544 for ; Wed, 04 Jun 2025 04:21:54 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@ericsson.com header.s=selector1 header.b=Cz0Wme5z; spf=pass (domain: ericsson.com, ip: 40.107.20.64, mailfrom: edaturu@ericsson.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uD3wxPibHIKk5SuAhXYfUw6lAtIYaaQE1amSODAbLSJ/E4rB1ADmjH5sxjB9aHpFYB0OKJ+9LfU9fBrUlLGwhsvVVGvoX67LwvCZosK+kpyiwlxUa5iZek6RaJpar4NRldKCQ2ARtN5nUnc4NiSa8DnCFg1jx4Zi3cM7c9xIYvHMhi6t3YmLxcZKlj9TO67PXBQ/zLsoq+tbSn/zb3jHU0HXhye6tmfQdsCD7ZWbliJseCUu4Q1JwKDixRH6IP+jnbyleCMA6KKTRxbk6RuNyVEgMBNnSlnPlWPN1Gq8VbfD1wxVVOFhoEeD7fqxhBcnkHFD2pq4wmb16WemtRLl6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=snDIhAMaYpV80q7YuGsKJkdxefOcGynPOcY1wAYyppE=; b=bXrwy80e6E5WKFlpxzD48jayr4eDu0JLXsHBugmQt9XUv9fJnQGOKRQxm8ITCjZqAir3iyELI9IV6r4fwopT9RUIIXjrftSXc+Dtui4yHW6GdFTS++kYh9okVXdHwtTGLoSWmd7CWjWfrcPL+rgKlqLDRy8Bmn364z76iweLyCvbMG4akx/8ZeUCJ+ULHnlPMoUKAWbpeOfoxf5bmlidIBMUCXWYYvoiSmzM5gfT3F6yDKD3gHmk7d+Qz5ryuxO2YorddPayoHDA8rx8E2WzCwNX7itXCy2iI3d+b/LxwA7CHcsnn2Om2lvl2K03Z9WNZnzOuEmN5EGWHyG3GUMj9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=cherry.de smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=snDIhAMaYpV80q7YuGsKJkdxefOcGynPOcY1wAYyppE=; b=Cz0Wme5ziMmKRpLDSmL11lWC4jzz/tZ76W4/WLjV0n/9OWuMbJKcStm0eVILfT55prkpdOCw1lIFvaXOEOCzddXD/XUFJRcbM5WAmdABiS0m6bvGiiDlUMIv1+S95CPQNFTNs+YtczWuJPvxveOCUdf9a+xRjlMDu8aLLPSoyF+wIkVhErS5YIhkacYOBScP59X5buKr3KCTHTxwKo2KKI1fX6745Wm3U3YG0RVOpm/XhmSplKy8hjK5NLE1gIDE8/ibFOroJf5b/nOqlKBTI7B+HyhTZYJEzGtsVJgEtlPkKBklmnVHHcKu7hnpH9TSeOlhGQA1Xrte22jpv/d+Tg== Received: from DU2PR04CA0337.eurprd04.prod.outlook.com (2603:10a6:10:2b4::35) by DBBPR07MB7626.eurprd07.prod.outlook.com (2603:10a6:10:1e2::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8792.33; Wed, 4 Jun 2025 11:21:50 +0000 Received: from DB1PEPF0003922F.eurprd03.prod.outlook.com (2603:10a6:10:2b4:cafe::65) by DU2PR04CA0337.outlook.office365.com (2603:10a6:10:2b4::35) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8792.19 via Frontend Transport; Wed, 4 Jun 2025 11:21:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by DB1PEPF0003922F.mail.protection.outlook.com (10.167.8.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8792.29 via Frontend Transport; Wed, 4 Jun 2025 11:21:49 +0000 Received: from seroius18814.sero.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 4 Jun 2025 13:21:49 +0200 Received: from seroius08462.sero.gic.ericsson.se (seroius08462.sero.gic.ericsson.se [10.63.237.245]) by seroius18814.sero.gic.ericsson.se (Postfix) with ESMTP id CF9B94020B54; Wed, 4 Jun 2025 13:21:48 +0200 (CEST) Received: by seroius08462.sero.gic.ericsson.se (Postfix, from userid 160155) id BA8327083F54; Wed, 4 Jun 2025 13:21:48 +0200 (CEST) From: To: CC: Daniel Turull , Quentin Schulz , Joshua Watt , Peter Marko Subject: [PATCH v6 2/3] spdx: add option to include only compiled sources Date: Wed, 4 Jun 2025 13:21:32 +0200 Message-ID: <20250604112133.2581063-3-daniel.turull@ericsson.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250604112133.2581063-1-daniel.turull@ericsson.com> References: <20250604112133.2581063-1-daniel.turull@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB1PEPF0003922F:EE_|DBBPR07MB7626:EE_ X-MS-Office365-Filtering-Correlation-Id: 969d1b42-c6af-4e7c-8025-08dda359fff9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|1800799024|82310400026|7053199007; X-Microsoft-Antispam-Message-Info: Y3U0BCOM6HNO6AtKxcJ8cocWjRMU5MQaKGLtIp2qivOZmKUZ5Lm6bRjeov7gMgEdMq8CqpuE17Qz+AyfzIWAc2Dz7kqXXtJNvind56Bv9BUKRuiKjmtrGRBYS2qUzpSPyqCPkopjIdAbCGKpwEmhw2xMqeDcwiExbasDaUXeHUrwnGU8X5Y3ISghW5b2D29BK6EMC3y27AV81Hfx7PMunpO98xRkvai3+LOC9HZAMEiptsm6ziACsv59NWNFyz/lXTrPXIU0PwSmTmaeb4qzJMK1hXyzKVzO7NYePCr9hcghJxmm5DkAHluSQQpehBgnKnwUywPItH/Hk3RknW2qhERJeDpCy4b1uOecy2aqchWB58u+c0eDOslkctTkeZ2+s9UtGLAWAY5P2Lo1iyIAVq8c1gkYDoapJOeX70hQRk8ntMcEVK+VnevvrFUdeXV40dCijL3E6y3oHNCxqEdkHY0gS1VecBlE5ApXKF5KWGsWpKMNYP4A5zutgAEzPtSa4LrCrqWgtvoHdoVXhNTTGHelUD6S7NSiu4cKzkPvZxGLpS3FeldPzTZ9t8weGjGpZEMPWAij5qOCdz/cpXyDQ+p/vp3heC/ekkFt3PlqLb2sGES7rOBMmIvUUxezcnGXxqOfjqLM5tK5Myccof4xzBMykJ1W3LBkOj7M3KDb0kdmlTlrjJ2g8p7cG63NLIB8m83Hlna52Dh2ESXN6wHeeIegf/Ecy1qiq+wZcIuTILzheH48KdKW750/9b4nNSpYCHDWbaS9DbqCsV1lwxY3iMjERFACEyS1fYa8OM+lAvx/qQyPC8elsefBog7Fxqr0qNEfMB5/yUQqlhy5emwfv4S+2vBqfDFs8+c5hU08hlTwCo4ZvOFDinzL1H+FDxKH6wdaEw+1Z2vJm1eMSgVpO9E2nMu8ca7c8h4JSaiMPwSx2FR1cHBJ/8FENlnAzJ+qEfTtTzVOZ8NOEWaus+dNigLCUF9hfZ30yqwfhHBut51O/Gm0f+X1AFTu8DydH5QLVYaft5gG2rmBm2CHq/+bTR8hLehUGeq7CDsCI7Fadp16J0/tka38LqrwAP4Qcaxjo6c6gR1+nQy7rxM/7WvkjNN+YliYsTO90QCet+VYfeNx8MMKHNyhcWHi0Bc4k58LfS1opf9Bc1hP2V9k5QFUkvkA21fJ+UFy4RrxaS7TyPoDHbLl4XX0KAVTPR0z5KKZpTyGyWd/cPK4utVQfu8geKGpj7MaXMlRWUFBDMHvdl9awojrXwIcuASY1P00m0bVNKThgF1vDVC0WEopP8IQgSmkjCVJ+kIpvDZgGlRDjypT7MIo31LigDuEgAvzoAqnlYYnoEZK1nz+qSoTfLhZ91QPeiSmR1HWAJ+aT2hq51V+fExckjwL+mhRCons1p9Au6l0ayOFM5j66+9DZRtghf1JiMMn9DANDibehP51jXvpT6Pg/bFP6ftboCaIS3Y5WmQ0kmsLj5D+xQmnB+39AA== X-Forefront-Antispam-Report: CIP:192.176.1.74;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:oa.msg.ericsson.com;PTR:office365.se.ericsson.net;CAT:NONE;SFS:(13230040)(376014)(36860700013)(1800799024)(82310400026)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2025 11:21:49.7565 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 969d1b42-c6af-4e7c-8025-08dda359fff9 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f;Ip=[192.176.1.74];Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: DB1PEPF0003922F.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR07MB7626 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jun 2025 11:22:01 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/217909 From: Daniel Turull When SPDX_INCLUDE_COMPILED_SOURCES is enabled, only include the source code files that are used during compilation. It uses debugsource information generated during do_package. This enables an external tool to use the SPDX information to disregard vulnerabilities that are not compiled. As example, when used with the default config with linux-yocto, the spdx size is reduced from 156MB to 61MB. Tested with bitbake world on oe-core. CC: Quentin Schulz CC: Joshua Watt CC: Peter Marko Signed-off-by: Daniel Turull --- meta/classes/create-spdx-2.2.bbclass | 9 ++++++ meta/classes/spdx-common.bbclass | 3 ++ meta/lib/oe/spdx30_tasks.py | 10 +++++++ meta/lib/oe/spdx_common.py | 41 ++++++++++++++++++++++++++++ 4 files changed, 63 insertions(+) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 7e8f8b9ff5..6fc60a1d97 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -137,6 +137,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv spdx_files = [] file_counter = 1 + + check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1" + if check_compiled_sources: + compiled_sources, types = oe.spdx_common.get_compiled_sources(d) + bb.debug(1, f"Total compiled files: {len(compiled_sources)}") for subdir, dirs, files in os.walk(topdir): dirs[:] = [d for d in dirs if d not in ignore_dirs] if subdir == str(topdir): @@ -147,6 +152,10 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv filename = str(filepath.relative_to(topdir)) if not filepath.is_symlink() and filepath.is_file(): + # Check if file is compiled + if check_compiled_sources: + if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + continue spdx_file = oe.spdx.SPDXFile() spdx_file.SPDXID = get_spdxid(file_counter) for t in get_types(filepath): diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 713a7fc651..ca0416d1c7 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -26,6 +26,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0" +SPDX_INCLUDE_COMPILED_SOURCES ??= "0" SPDX_UUID_NAMESPACE ??= "sbom.openembedded.org" SPDX_NAMESPACE_PREFIX ??= "http://spdx.org/spdxdocs" @@ -40,6 +41,8 @@ SPDX_MULTILIB_SSTATE_ARCHS ??= "${SSTATE_ARCHS}" python () { from oe.cve_check import extend_cve_status extend_cve_status(d) + if d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1": + d.setVar("SPDX_INCLUDE_SOURCES", "1") } def create_spdx_source_deps(d): diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 61d7ba45e3..beeafc2bb7 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -156,6 +156,11 @@ def add_package_files( bb.note(f"Skip {topdir}") return spdx_files + check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1" + if check_compiled_sources: + compiled_sources, types = oe.spdx_common.get_compiled_sources(d) + bb.debug(1, f"Total compiled files: {len(compiled_sources)}") + for subdir, dirs, files in os.walk(topdir, onerror=walk_error): dirs[:] = [d for d in dirs if d not in ignore_dirs] if subdir == str(topdir): @@ -171,6 +176,11 @@ def add_package_files( filename = str(filepath.relative_to(topdir)) file_purposes = get_purposes(filepath) + # Check if file is compiled + if check_compiled_sources: + if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + continue + spdx_file = objset.new_file( get_spdxid(file_counter), filename, diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 4caefc7673..daf43bce56 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -242,3 +242,44 @@ def fetch_data_to_uri(fd, name): uri = uri + "@" + fd.revision return uri + +def is_compiled_source (filename, compiled_sources, types): + """ + Check if the file is a compiled file + """ + import os + # If we don't have compiled source, we assume all are compiled. + if len(compiled_sources) == 0: + return True + # We remove the top directory, to match the format in compiled sources + relative = filename[filename.find("/")+1:] + basename = os.path.basename(filename) + # We return always true if the file type is not in the list of compiled files + if basename[basename.find("."):] not in types: + return True + # Check that the file is in the list + return relative in compiled_sources + +def get_compiled_sources(d): + """ + Get list of compiled sources from debug information and normalize the paths + """ + import itertools + source_info = oe.package.read_debugsources_file(d) + if not source_info: + bb.debug(1, "Do not have debugsources.list. Skipping") + return [], [] + + # Sources are not split now in SPDX, so we aggregate them + sources = list(set(itertools.chain.from_iterable(source_info.values()))) + # Check extensions of files + types = [] + for src in sources: + basename = os.path.basename(src) + # We check that the basename has an extension + if basename.find(".") > 0: + ext = basename[basename.find("."):] + if ext not in types and len(ext)>0: + types.append(ext) + bb.debug(1, f"Num of sources: {len(sources)} and types: {len(types)} {str(types)}") + return sources, types From patchwork Wed Jun 4 11:21:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Turull X-Patchwork-Id: 64253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D14BDC61CE7 for ; Wed, 4 Jun 2025 11:22:01 +0000 (UTC) Received: from DU2PR03CU002.outbound.protection.outlook.com (DU2PR03CU002.outbound.protection.outlook.com [52.101.65.11]) by mx.groups.io with SMTP id smtpd.web11.14201.1749036117843588580 for ; Wed, 04 Jun 2025 04:21:58 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@ericsson.com header.s=selector1 header.b=Dci2QlFm; spf=pass (domain: ericsson.com, ip: 52.101.65.11, mailfrom: edaturu@ericsson.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nckidG+AdaltzFuqDnlglYAzcE/s7YUeBpj/7BpJV4bGbwoZrs+H3YOalP+ZyJGUgRFPrXTG6BXJJTmgElp9PXEi926rg+n+Odr60sbnN8yh89thVFW2WmN0Rz+X3UlfYDR/8GUo2Ry/g0lIp1tzOltrcL+jeju7nmXP9hXExqPT3QHkvj4sE4s875NXJ1E0aEF9DTRnuIdhnb1OlhJmRJnC5r9sUAJeXNyN54RuV8xlw+KkqrlCBmsNfcu9YyWo9MKNzXSkzoiTkX+N+5Pm+87zXvjzQys8Iuhq24OI9MyX0JWeMo69sUEA/1epFpBoIDYmAIJUCsTZu4aEb+52sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=G5X9QZc11nQWAqvLmsG0r1jxwdzW7MbuUQhXgBH1/S4=; b=jc0r97G4so5U1KYT41NiLb1zaP1JB0HP9eZ/tcynnbPg8zT4r3bBNkHgcMFJggFTTm22KBn49/+YlFVPB43yJN8cwYr4jQfg2cdLB5hoHouGBb3Qv7jwatHK7E4n2iIw4A66GPNJgxhU2+StqiDqAqnp55EtyGafXl1rQHfz0uwGba7LzKHrleIopX911kptJrpPrWhu/QRXS0oBRr+4Z9tTqQHOMEFMR5ZtvzQNxhZKJ21xmGBbT7ZDauZTJl39k5t2fOXS5JmXtlVutzW6SiBGyWJsdA//G4JZpdoQk8oaWMM16ObbwMrKgy851gFi68GVgZec/zrQ8x1x76n8fQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=gmail.com smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G5X9QZc11nQWAqvLmsG0r1jxwdzW7MbuUQhXgBH1/S4=; b=Dci2QlFmGl0zdk6x5OJDET0cR82DhaFUb1MDrFUx4NCoiJ2l0EV8vyJpcsXx14u4FmmzMJDgV4C1q3nL8SJibCm/Hxvrpz9vlceaX6+xOTZogCGrodoBikemXbL7UmGApYP4ISLqOdmHfyYf8MQ4gFLbc0AKfXnu1QAEVYLWpUb1NZk7jp5ZDsbjGveDSdTz9iNLt+29xJ6vvPvdK5RjWR/nzep9kUPtKv0lLijBABoWBJO8vfqxHmV02lMj90SBSmDbF8xGuzvect4l6L52wr5NwLYP4sOs7mF0AFXZLM6xhj69V/shhWD9iUW+hAeG3xdSseCsw7NsR4kyJk6e/A== Received: from DB3PR06CA0015.eurprd06.prod.outlook.com (2603:10a6:8:1::28) by GV1PR07MB10216.eurprd07.prod.outlook.com (2603:10a6:150:211::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8769.37; Wed, 4 Jun 2025 11:21:51 +0000 Received: from DB1PEPF00039231.eurprd03.prod.outlook.com (2603:10a6:8:1:cafe::3d) by DB3PR06CA0015.outlook.office365.com (2603:10a6:8:1::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8792.29 via Frontend Transport; Wed, 4 Jun 2025 11:21:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by DB1PEPF00039231.mail.protection.outlook.com (10.167.8.104) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8792.29 via Frontend Transport; Wed, 4 Jun 2025 11:21:51 +0000 Received: from seroius18813.sero.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Wed, 4 Jun 2025 13:21:50 +0200 Received: from seroius08462.sero.gic.ericsson.se (seroius08462.sero.gic.ericsson.se [10.63.237.245]) by seroius18813.sero.gic.ericsson.se (Postfix) with ESMTP id D8F8695145; Wed, 4 Jun 2025 13:21:49 +0200 (CEST) Received: by seroius08462.sero.gic.ericsson.se (Postfix, from userid 160155) id C2D767083F54; Wed, 4 Jun 2025 13:21:49 +0200 (CEST) From: To: CC: Daniel Turull , Peter Marko , Marta Rybczynska Subject: [PATCH v6 3/3] improve_kernel_cve_report: add script for postprocesing of kernel CVE data Date: Wed, 4 Jun 2025 13:21:33 +0200 Message-ID: <20250604112133.2581063-4-daniel.turull@ericsson.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250604112133.2581063-1-daniel.turull@ericsson.com> References: <20250604112133.2581063-1-daniel.turull@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB1PEPF00039231:EE_|GV1PR07MB10216:EE_ X-MS-Office365-Filtering-Correlation-Id: 8e1106f2-24c7-479b-be8e-08dda35a012b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|36860700013|13003099007; X-Microsoft-Antispam-Message-Info: U+svhIUm/FZAC607GWLBgUzcRejpiJR3fJwr93eC9OmfuqJ0a5yk7aw+1+QoC5N4PUzLoSKbWawBt+t151VCZGMKAyUTw3ASeWyFuMXLTCKUfnHSTevltN425HvQWRj+IBrfcz8LeGSd7+zn6Ic6DIsrM2wRp0k3R7OHCL6WMkAWjzy1TKndkp5vnaB2uUPhMb7ickVJsue4h8uTW3q2SwxttdObJDplQF0j771zJ45EVflWAYhvLTugGuXXbMqB66Dv4S5gYOUupOuxj3QI40Zgw/+T6q+TRUN0v2hgpQICmYYingPP8+HM6Ts+MIXeeXr0a3ZL6MZptsb+0L2qKhAifWZ6GxHe7CS8scO5AghbbH/zOMVUvfbxTti5fk/fFlytqZjuwInvgGohulQbl0H9e6ruOkdGFKf50wz50+cY7qPc1UDhZIhHH2ffl6QFauwJ6x/RcDLmwAuM1FzhJT00fd18AuCWBiq8sNPM/AhHxWrpRka/sgt4aFkio7H76Fv1kLJW9rooH+tTu93W8yOOGGgg/nRzhtXPzlr0YDnzM3Rhh6caj/r0xqPYE1UT4GBfutlQ2og//iyoi6zD7TeWh55Ms4/6SytXd604lJV+6T7zUc0O+/r1U/dyZtm5hRObJ/G/hGIkGiuPjljwxU56iWRkY/Fs3Oau8OTWtSY2/jLrsphzwqeVqS+oKh9+vK4rsuLjtjsBQJzgrQrvePvBf3Bjz462Ru138pxm0zuOmQLhExF7btwA+4l5cl7GBFuCtNzwjXouapmU0MJX7rqI4QlEBfStOI2WKQ4mQaCp2G6i3YqP24x/A4uX+p0V+bfYIVqfB+H9jzgJoQn87OhZoEPV1iEowEcoYy5PdkqNQP3pTiTl8GiVsz+767J7I1p7JNdXoHyPvDmpsv/AoXRIg6m3/8Ggl1WyTJhBf7E2zGWpr55GhfszT/heOanrpKf0zk03/ZTmDoEV9JsHwuYgrPFrWyNPBfA14+x7P+H+cafMKfD5lb4u4/mj2PTBy/SVSybaKNHD8SqB/7iZBtwe1WZ4YyioVIS/3/5cV/PTx//kvvtR3N04wJFzfuAHMZ49FOxHR3aSWLsQrZHdKpD1oownEEZXC5cxNuB4im+ZcJ9hGyMVhsESK8HYnghFSb50DAUjjfSxz7ZESP/5RBo/hQQTBLT3gcEdUDa1lniZJdFFGrLmAds/q5ayyM8QWxa1abd2jeCpr/UsRfl3uz8dJEnd/mJYxfZPm8xTw18T52R4MTkSXvE5SBA2V4FxvkUhj99FbgxE+KF6rYd+UbUBOxINtDlw1BQRtwxfQxOEGWOhw0Vkp/UwgCKTDFsNAn6Yl94Y6riqMZPcW48lgp1CB7iTSLqBDTBN/yVREfdmT8uE4AAB1CYskBCQT7/4/hl9qytf7D5MBwJHF3MvtnBqKtsecsPyvVAAeRc2nuCrI7xLIgTwE9Nd/WpCVqPo9KOwAZDywDysXp972r2ZUPJHokIY9KtYJaTI8f6IadxggkDONaGfjenBp9BuGkOxQHoKLx8gfaQzuu1wm54hsw== X-Forefront-Antispam-Report: CIP:192.176.1.74;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:oa.msg.ericsson.com;PTR:office365.se.ericsson.net;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(36860700013)(13003099007);DIR:OUT;SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2025 11:21:51.7624 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8e1106f2-24c7-479b-be8e-08dda35a012b X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f;Ip=[192.176.1.74];Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: DB1PEPF00039231.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR07MB10216 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 04 Jun 2025 11:22:01 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/217910 From: Daniel Turull Adding postprocessing script to process data from linux CNA that includes more accurate metadata and it is updated directly by the source. Example of enhanced CVE from a report from cve-check: { "id": "CVE-2024-26710", "status": "Ignored", "link": "https://nvd.nist.gov/vuln/detail/CVE-2024-26710", "summary": "In the Linux kernel, the following vulnerability [...]", "scorev2": "0.0", "scorev3": "5.5", "scorev4": "0.0", "modified": "2025-03-17T15:36:11.620", "vector": "LOCAL", "vectorString": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H", "detail": "not-applicable-config", "description": "Source code not compiled by config. ['arch/powerpc/include/asm/thread_info.h']" }, And same from a report generated with vex: { "id": "CVE-2024-26710", "status": "Ignored", "link": "https://nvd.nist.gov/vuln/detail/CVE-2024-26710", "detail": "not-applicable-config", "description": "Source code not compiled by config. ['arch/powerpc/include/asm/thread_info.h']" }, For unpatched CVEs, provide more context in the description: Tested with 6.12.22 kernel { "id": "CVE-2025-39728", "status": "Unpatched", "link": "https://nvd.nist.gov/vuln/detail/CVE-2025-39728", "summary": "In the Linux kernel, the following vulnerability has been [...], "scorev2": "0.0", "scorev3": "0.0", "scorev4": "0.0", "modified": "2025-04-21T14:23:45.950", "vector": "UNKNOWN", "vectorString": "UNKNOWN", "detail": "version-in-range", "description": "Needs backporting (fixed from 6.12.23)" }, CC: Peter Marko CC: Marta Rybczynska Signed-off-by: Daniel Turull --- scripts/contrib/improve_kernel_cve_report.py | 467 +++++++++++++++++++ 1 file changed, 467 insertions(+) create mode 100755 scripts/contrib/improve_kernel_cve_report.py diff --git a/scripts/contrib/improve_kernel_cve_report.py b/scripts/contrib/improve_kernel_cve_report.py new file mode 100755 index 0000000000..d729f8ed64 --- /dev/null +++ b/scripts/contrib/improve_kernel_cve_report.py @@ -0,0 +1,467 @@ +#! /usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# The script uses another source of CVE information from linux-vulns +# to enrich the cve-summary from cve-check or vex. +# It can also use the list of compiled files from the kernel spdx to ignore CVEs +# that are not affected since the files are not compiled. +# +# It creates a new json file with updated CVE information +# +# Compiled files can be extracted adding the following in local.conf +# SPDX_INCLUDE_COMPILED_SOURCES:pn-linux-yocto = "1" +# +# Tested with the following CVE sources: +# - https://git.kernel.org/pub/scm/linux/security/vulns.git +# - https://github.com/CVEProject/cvelistV5 +# +# Example: +# python3 ./openembedded-core/scripts/contrib/improve_kernel_cve_report.py --spdx tmp/deploy/spdx/3.0.1/qemux86_64/recipes/recipe-linux-yocto.spdx.json --kernel-version 6.12.27 --datadir ./vulns +# python3 ./openembedded-core/scripts/contrib/improve_kernel_cve_report.py --spdx tmp/deploy/spdx/3.0.1/qemux86_64/recipes/recipe-linux-yocto.spdx.json --datadir ./vulns --old-cve-report build/tmp/log/cve/cve-summary.json +# +# SPDX-License-Identifier: GPLv2 + +import argparse +import json +import sys +import logging +import glob +import os +import pathlib +from packaging.version import Version + +def is_linux_cve(cve_info): + '''Return true is the CVE belongs to Linux''' + if not "affected" in cve_info["containers"]["cna"]: + return False + for affected in cve_info["containers"]["cna"]["affected"]: + if not "product" in affected: + return False + if affected["product"] == "Linux" and affected["vendor"] == "Linux": + return True + return False + +def get_kernel_cves(datadir, compiled_files, version): + """ + Get CVEs for the kernel + """ + cves = {} + + check_config = len(compiled_files) > 0 + + base_version = Version(f"{version.major}.{version.minor}") + + # Check all CVES from kernel vulns + pattern = os.path.join(datadir, '**', "CVE-*.json") + cve_files = glob.glob(pattern, recursive=True) + not_applicable_config = 0 + fixed_as_later_backport = 0 + vulnerable = 0 + not_vulnerable = 0 + for cve_file in sorted(cve_files): + cve_info = {} + with open(cve_file, "r", encoding='ISO-8859-1') as f: + cve_info = json.load(f) + + if len(cve_info) == 0: + logging.error("Not valid data in %s. Aborting", cve_file) + break + + if not is_linux_cve(cve_info): + continue + cve_id = os.path.basename(cve_file)[:-5] + description = cve_info["containers"]["cna"]["descriptions"][0]["value"] + if cve_file.find("rejected") >= 0: + logging.debug("%s is rejected by the CNA", cve_id) + cves[cve_id] = { + "id": cve_id, + "status": "Ignored", + "detail": "rejected", + "summary": description, + "description": f"Rejected by CNA" + } + continue + if any(elem in cve_file for elem in ["review", "reverved", "testing"]): + continue + + is_vulnerable, first_affected, last_affected, better_match_first, better_match_last, affected_versions = get_cpe_applicability(cve_info, version) + + logging.debug("%s: %s (%s - %s) (%s - %s)", cve_id, is_vulnerable, better_match_first, better_match_last, first_affected, last_affected) + + if is_vulnerable is None: + logging.warning("%s doesn't have good metadata", cve_id) + if is_vulnerable: + is_affected = True + affected_files = [] + if check_config: + is_affected, affected_files = check_kernel_compiled_files(compiled_files, cve_info) + + if not is_affected and len(affected_files) > 0: + logging.debug( + "%s - not applicable configuration since affected files not compiled: %s", + cve_id, affected_files) + cves[cve_id] = { + "id": cve_id, + "status": "Ignored", + "detail": "not-applicable-config", + "summary": description, + "description": f"Source code not compiled by config. {affected_files}" + } + not_applicable_config +=1 + # Check if we have backport + else: + if not better_match_last: + fixed_in = last_affected + else: + fixed_in = better_match_last + logging.debug("%s needs backporting (fixed from %s)", cve_id, fixed_in) + cves[cve_id] = { + "id": cve_id, + "status": "Unpatched", + "detail": "version-in-range", + "summary": description, + "description": f"Needs backporting (fixed from {fixed_in})" + } + vulnerable += 1 + if (better_match_last and + Version(f"{better_match_last.major}.{better_match_last.minor}") == base_version): + fixed_as_later_backport += 1 + # Not vulnerable + else: + if not first_affected: + logging.debug("%s - not known affected %s", + cve_id, + better_match_last) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "version-not-in-range", + "summary": description, + "description": "No CPE match" + } + not_vulnerable += 1 + continue + backport_base = Version(f"{better_match_last.major}.{better_match_last.minor}") + if version < first_affected: + logging.debug('%s - fixed-version: only affects %s onwards', + cve_id, + first_affected) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "fixed-version", + "summary": description, + "description": f"only affects {first_affected} onwards" + } + not_vulnerable += 1 + elif last_affected <= version: + logging.debug("%s - fixed-version: Fixed from version %s", + cve_id, + last_affected) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "fixed-version", + "summary": description, + "description": f"fixed-version: Fixed from version {last_affected}" + } + not_vulnerable += 1 + elif backport_base == base_version: + logging.debug("%s - cpe-stable-backport: Backported in %s", + cve_id, + better_match_last) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "cpe-stable-backport", + "summary": description, + "description": f"Backported in {better_match_last}" + } + not_vulnerable += 1 + else: + logging.debug("%s - version not affected %s", cve_id, str(affected_versions)) + cves[cve_id] = { + "id": cve_id, + "status": "Patched", + "detail": "version-not-in-range", + "summary": description, + "description": f"Range {affected_versions}" + } + not_vulnerable += 1 + + logging.info("Total CVEs ignored due to not applicable config: %d", not_applicable_config) + logging.info("Total CVEs not vulnerable due version-not-in-range: %d", not_vulnerable) + logging.info("Total vulnerable CVEs: %d", vulnerable) + + logging.info("Total CVEs already backported in %s: %s", base_version, + fixed_as_later_backport) + return cves + +def read_spdx(spdx_file): + '''Open SPDX file and extract compiled files''' + with open(spdx_file, 'r', encoding='ISO-8859-1') as f: + spdx = json.load(f) + if "spdxVersion" in spdx: + if spdx["spdxVersion"] == "SPDX-2.2": + return read_spdx2(spdx) + if "@graph" in spdx: + return read_spdx3(spdx) + return [] + +def read_spdx2(spdx): + ''' + Read spdx2 compiled files from spdx + ''' + cfiles = [] + if 'files' not in spdx: + return cfiles + for item in spdx['files']: + for ftype in item['fileTypes']: + if ftype == "SOURCE": + filename = item["fileName"][item["fileName"].find("/")+1:] + cfiles.append(filename) + return cfiles + +def read_spdx3(spdx): + ''' + Read spdx3 compiled files from spdx + ''' + cfiles = [] + for item in spdx["@graph"]: + if "software_primaryPurpose" not in item: + continue + if item["software_primaryPurpose"] == "source": + filename = item['name'][item['name'].find("/")+1:] + cfiles.append(filename) + return cfiles + +def check_kernel_compiled_files(compiled_files, cve_info): + """ + Return if a CVE affected us depending on compiled files + """ + files_affected = [] + is_affected = False + + for item in cve_info['containers']['cna']['affected']: + if "programFiles" in item: + for f in item['programFiles']: + if f not in files_affected: + files_affected.append(f) + + if len(files_affected) > 0: + for f in files_affected: + if f in compiled_files: + logging.debug("File match: %s", f) + is_affected = True + return is_affected, files_affected + +def get_cpe_applicability(cve_info, v): + ''' + Check if version is affected and return affected versions + ''' + base_branch = Version(f"{v.major}.{v.minor}") + affected = [] + if not 'cpeApplicability' in cve_info["containers"]["cna"]: + return None, None, None, None, None, None + + for nodes in cve_info["containers"]["cna"]["cpeApplicability"]: + for node in nodes.values(): + vulnerable = False + matched_branch = False + first_affected = Version("5000") + last_affected = Version("0") + better_match_first = Version("0") + better_match_last = Version("5000") + + if len(node[0]['cpeMatch']) == 0: + first_affected = None + last_affected = None + better_match_first = None + better_match_last = None + + for cpe_match in node[0]['cpeMatch']: + version_start_including = Version("0") + version_end_excluding = Version("0") + if 'versionStartIncluding' in cpe_match: + version_start_including = Version(cpe_match['versionStartIncluding']) + else: + version_start_including = Version("0") + # if versionEndExcluding is missing we are in a branch, which is not fixed. + if "versionEndExcluding" in cpe_match: + version_end_excluding = Version(cpe_match["versionEndExcluding"]) + else: + # if versionEndExcluding is missing we are in a branch, which is not fixed. + version_end_excluding = Version( + f"{version_start_including.major}.{version_start_including.minor}.5000" + ) + affected.append(f" {version_start_including}-{version_end_excluding}") + # Detect if versionEnd is in fixed in base branch. It has precedence over the rest + branch_end = Version(f"{version_end_excluding.major}.{version_end_excluding.minor}") + if branch_end == base_branch: + if version_start_including <= v < version_end_excluding: + vulnerable = cpe_match['vulnerable'] + # If we don't match in our branch, we are not vulnerable, + # since we have a backport + matched_branch = True + better_match_first = version_start_including + better_match_last = version_end_excluding + if version_start_including <= v < version_end_excluding and not matched_branch: + if version_end_excluding < better_match_last: + better_match_first = max(version_start_including, better_match_first) + better_match_last = min(better_match_last, version_end_excluding) + vulnerable = cpe_match['vulnerable'] + matched_branch = True + + first_affected = min(version_start_including, first_affected) + last_affected = max(version_end_excluding, last_affected) + # Not a better match, we use the first and last affected instead of the fake .5000 + if vulnerable and better_match_last == Version(f"{base_branch}.5000"): + better_match_last = last_affected + better_match_first = first_affected + return vulnerable, first_affected, last_affected, better_match_first, better_match_last, affected + +def copy_data(old, new): + '''Update dictionary with new entries, while keeping the old ones''' + for k in new.keys(): + old[k] = new[k] + return old + +# Function taken from cve_check.bbclass. Adapted to cve fields +def cve_update(cve_data, cve, entry): + # If no entry, just add it + if cve not in cve_data: + cve_data[cve] = entry + return + # If we are updating, there might be change in the status + if cve_data[cve]['status'] == "Unknown": + cve_data[cve] = copy_data(cve_data[cve], entry) + return + if cve_data[cve]['status'] == entry['status']: + return + if entry['status'] == "Unpatched" and cve_data[cve]['status'] == "Patched": + logging.warning("CVE entry %s update from Patched to Unpatched from the scan result", cve) + cve_data[cve] = copy_data(cve_data[cve], entry) + return + if entry['status'] == "Patched" and cve_data[cve]['status'] == "Unpatched": + logging.warning("CVE entry %s update from Unpatched to Patched from the scan result", cve) + cve_data[cve] = copy_data(cve_data[cve], entry) + return + # If we have an "Ignored", it has a priority + if cve_data[cve]['status'] == "Ignored": + logging.debug("CVE %s not updating because Ignored", cve) + return + # If we have an "Ignored", it has a priority + if entry['status'] == "Ignored": + cve_data[cve] = copy_data(cve_data[cve], entry) + logging.debug("CVE entry %s updated from Unpatched to Ignored", cve) + return + logging.warning("Unhandled CVE entry update for %s %s from %s %s to %s", + cve, cve_data[cve]['status'], cve_data[cve]['detail'], entry['status'], entry['detail']) + +def main(): + parser = argparse.ArgumentParser( + description="Update cve-summary with kernel compiled files and kernel CVE information" + ) + parser.add_argument( + "-s", + "--spdx", + help="SPDX2/3 for the kernel. Needs to include compiled sources", + ) + parser.add_argument( + "--datadir", + type=pathlib.Path, + help="Directory where CVE data is", + required=True + ) + parser.add_argument( + "--old-cve-report", + help="CVE report to update. (Optional)", + ) + parser.add_argument( + "--kernel-version", + help="Kernel version. Needed if old cve_report is not provided (Optional)", + type=Version + ) + parser.add_argument( + "--new-cve-report", + help="Output file", + default="cve-summary-enhance.json" + ) + parser.add_argument( + "-D", + "--debug", + help='Enable debug ', + action="store_true") + + args = parser.parse_args() + + if args.debug: + log_level=logging.DEBUG + else: + log_level=logging.INFO + logging.basicConfig(format='[%(filename)s:%(lineno)d] %(message)s', level=log_level) + + if not args.kernel_version and not args.old_cve_report: + parser.error("either --kernel-version or --old-cve-report are needed") + return -1 + + # by default we don't check the compiled files, unless provided + compiled_files = [] + if args.spdx: + compiled_files = read_spdx(args.spdx) + logging.info("Total compiled files %d", len(compiled_files)) + + if args.old_cve_report: + with open(args.old_cve_report, encoding='ISO-8859-1') as f: + cve_report = json.load(f) + else: + #If summary not provided, we create one + cve_report = { + "version": "1", + "package": [ + { + "name": "linux-yocto", + "version": str(args.kernel_version), + "products": [ + { + "product": "linux_kernel", + "cvesInRecord": "Yes" + } + ], + "issue": [] + } + ] + } + + for pkg in cve_report['package']: + is_kernel = False + for product in pkg['products']: + if product['product'] == "linux_kernel": + is_kernel=True + if not is_kernel: + continue + + kernel_cves = get_kernel_cves(args.datadir, + compiled_files, + Version(pkg["version"])) + logging.info("Total kernel cves from kernel CNA: %s", len(kernel_cves)) + cves = {issue["id"]: issue for issue in pkg["issue"]} + logging.info("Total kernel before processing cves: %s", len(cves)) + + for cve in kernel_cves: + cve_update(cves, cve, kernel_cves[cve]) + + pkg["issue"] = [] + for cve in sorted(cves): + pkg["issue"].extend([cves[cve]]) + logging.info("Total kernel cves after processing: %s", len(pkg['issue'])) + + with open(args.new_cve_report, "w", encoding='ISO-8859-1') as f: + json.dump(cve_report, f, indent=2) + + return 0 + +if __name__ == "__main__": + sys.exit(main()) +