From patchwork Thu Jun 18 15:57:46 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anders Heimer X-Patchwork-Id: 90443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD49CCD98F2 for ; Thu, 18 Jun 2026 15:58:04 +0000 (UTC) Received: from DUZPR83CU001.outbound.protection.outlook.com (DUZPR83CU001.outbound.protection.outlook.com [52.101.66.67]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.23044.1781798274316705747 for ; Thu, 18 Jun 2026 08:57:55 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@est.tech header.s=selector1 header.b=TNYRMYho; spf=pass (domain: est.tech, ip: 52.101.66.67, mailfrom: anders.heimer@est.tech) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=X5WXOJrddbmRMycvYOBjvKuMGF1cLycqRygxZaRRLB8zUxF4Bz50cNpEKTGMj0IyRbELLmpqQkA3hvwPav7t7UcJA3SdBlmhD6YqycrdJ4G4grb6+7JAyXMWi9ZHb8t86HCFWDyH9PhBub5Yua0kBPqhPDgUdEVBolNDsbESYZoonZ+CJY+SnAl63jaDTzCLvgRD91SuQZD4rOvyxM9mifJBrs/HJMl7+h4k1I4mj1L5hUeIHBZBj9JzS7FURDsxnCEuEzxeNwr2eeCgygmhiZ7xbHSABcv6bDfXKKZbXzyCu2zkE3C2qoAoABQwYRcU5d8kez/P7tsJURnToMLJnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=F9MKU9BS73MTJfpUZ9TLFhWzR4q/2L6fNjAyh+7EenI=; b=kJP8Mji2bJEiJ/b3O7VQCojQ3mdNdwjHE46xKvCoTdvr5JvFZmvEBBM5UmSvLbAB+ZQo2TPHchg+G37o30ifo21+4+yAUzxw+OCsXNIzOtQHM0NVtmwvlTTuj19NFhq2hxCFOQI14LwK4OphpMkwdwYRuzDDpDCXO1MzElDq+c+OQ8AytlMeDg3wXY8WT7V0E+cw1LM7FP72XI3aLguC6jTUssk2M+r67j9tp51kQEXrmFRQkShzow8suRwhR5BYn9P0Q2w8fi9cGQaS63ufG4TTrVfZv5JUG26VBLgNLrwN8sMatJV1NvoFgO0Hc+3D2Y5b5f04102SnGTZiGuxgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=est.tech; dmarc=pass action=none header.from=est.tech; dkim=pass header.d=est.tech; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=est.tech; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F9MKU9BS73MTJfpUZ9TLFhWzR4q/2L6fNjAyh+7EenI=; b=TNYRMYhoJos/fnXEOwRhYj18/gUzgRU8FU1ZWk97mt4d2NcsXr5ulnoathyjD7sbtnkn3NT7JZd4LrGpQWwh3+YmrOtEyDqpzejPBJKdWu00Umgkhpnh651KhOkt1Z6c3Te7FpsNeJyYKypMdWANxuUoNdUVKVIAz4BfK+WMMaFE6ZlAm2TNI9T+Th1rcCw/tBTcKjv+zL65Z2yHBikdCGrE3Cp6I79rNdETNT0ppLo/PI16dhkIOZ5yGuXKcEE14WDOZ3MSiASdVLQX5I7vgT1hZvwlzLkD+TuJY8j/0t/SV1Xb6o0lqpyBmxJDkHWadKOpcy7693/kruNbWfn8Gg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=est.tech; Received: from DB9P189MB1641.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:2ac::9) by AMBP189MB3277.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:6ad::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.139.11; Thu, 18 Jun 2026 15:57:52 +0000 Received: from DB9P189MB1641.EURP189.PROD.OUTLOOK.COM ([fe80::90da:b700:f102:5c82]) by DB9P189MB1641.EURP189.PROD.OUTLOOK.COM ([fe80::90da:b700:f102:5c82%6]) with mapi id 15.21.0139.011; Thu, 18 Jun 2026 15:57:52 +0000 From: Anders Heimer To: openembedded-core@lists.openembedded.org CC: Anders Heimer Subject: [PATCH v3 1/2] package: replace copydebugsources shell pipelines Date: Thu, 18 Jun 2026 17:57:46 +0200 Message-ID: <558e19ee3aa1a0dcccf5d46de7ac87ce21c80808.1781796358.git.anders.heimer@est.tech> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: X-ClientProxiedBy: DU2PR04CA0031.eurprd04.prod.outlook.com (2603:10a6:10:234::6) To DB9P189MB1641.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:2ac::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB9P189MB1641:EE_|AMBP189MB3277:EE_ X-MS-Office365-Filtering-Correlation-Id: 2b8ed477-6dc2-4c65-57d2-08decd525a5f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|23010399003|366016|56012099006|11063799006|22082099003|3023799007|18002099003; X-Microsoft-Antispam-Message-Info: p1sy0jG5bFIvUIMeWxPekyzxrdUzJGv7Vrz3OZHh6Dqk81i6gsdXdShmS8HeFLswaSFy87Mh8Q1ChfmoLHYFmajiq2y55bJ9YOEluGr/CQvAeEkigP8TT/akTbOILMQ/fWnUlQo3zainJSQOU/wc3dJp8GxWmFFG//OcFVsk9bxH1Z2nEduPCxKaGD0dguCLEIUx34112OYSZXeFkDnMFqLExAlsmyXU9xZIOsUPsfH7IR/8HZmidByR1EKDmUKzIhFSeOO9q1RzgFx/FizhmNdkX3V/1CMiDuHUTXM7MwvZzl7QecBCMAVv+Q1YZIJ3YvvslO979vyWMJA/NxU1VT6d2mTp1+o3eGFrso86Nii+ofAbJh+7ajfEP9MB16swdOaKZQBJJ5Nqj0rjKtJT1W37RUxSfoDlp48DVqG7zJ1QjKRTSS/WtsflmQCHL8kdq4N44kIgxJUeX4XIbVrJ6SFnyJklV+KEkbiSAxhNUoQvto/RgFIpuqF3Z3Je9mj+pfY8LMGbNgXsgizu0NRtkPgbOIWXecMiBjW22c4PxgvsmhKfMo673tAflCdfTJYxahUw8iI6kwuZsohVZrW77rAllZo9tkf+opYfpJLSaBMUxGUpSjahKdTZT6BiHiAqQa880crwfRK7BNL3NMi37ct70j/aQ0Ed9gjA25N+yxoeYZoYiOas8KOWfl3mxp9q X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DB9P189MB1641.EURP189.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(23010399003)(366016)(56012099006)(11063799006)(22082099003)(3023799007)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IV/gKGVXbLJUWzpyf/IcbZuuKBJD+gDkSlL3BTrM6KTe9vir3PZ19b3xtsKgaiSjwoBCstx0Q8KzvC4X4uV9a4b1SpUtl+gFagmkbYnLHOdaTZP0mU9+IuM3EQlIZJsqglLAQEyR0Ne5wCQVoR7cEDLcdAo8xlzTJ6mruyKSeDmZ5sZB8HUIVjurPN4AtOJdoVfUpv/F+P2v+AlVVKrZL11CF+oVJZGmNWla5v72RT1Hpc18RgC49D11ag4ncK4zmNeisKosCRD/NTQVdboia0nQVWIQV6EKJs4dFC+d0LPu1aMv5r9O501VhnsKiYHONKRZfb6dwP5dOO0rxJ1Q7tjJ0Nz6GpmGVH62HUoSjd97bAp3xD2w+T3mzsQ3uMfjhzblblgigRhybXDpIMRC8cUOXI+VNQvEiLvwsuXVIThrCA5L80Q9SG4IG53NqGrSRHOh7OJWe3dr/9Rj+iF28Mvy5zCYdYk9l/CjBCYQ9ZVNBFtlRP+WM88mq3S1VQxG9WmiTIQajUBdNnGVkn2Bou+QFcCj+YmA2RfMh5dIMoNw3p0KCfmsUZSZkzDCnZ57KZQpEiYoH5dR8QYrYt07GKAnLeslHUoWXPdgMvi2Jx1zuuyTrShAXGXaFB4Fb67/ktZOg32qL3TJ9SH5KCqy0dCP4hFZvHIzjjMM8I2V44w+hBW5JcthIBjc/dXOFAU4BBepENfk59M5ok5vNkp/6cDZDkCAzV329qKnqWUeY6I28dczcVw7Xpoio43KMuAmYyPrbQpCVf6t/Wykm7LywQWZ6pbz1IzPpLhUZmz4cRmOqOXHZHz0AbcyzhYZ53h9/XANHwL5xTEvveYvxkFQQSlrKM8SRseQjsTUiMy3weJXiLOYzaXElmx1PBuuNcOG2XNq+T2TWFnCzMmdweURDj7OMYa3gRRI3hvgfwohR1FEocnBEk2uZm3WlbdToqbNYRZwHiedIUiApEr7wqW02SsSYX08VFq6q+VbmpxbyQ4PPdlU27OCwnXb+JgmqCzs0eFI+5MYcBNrLcqT9CAQvC9DrnX6XxKssjhFjLAMPcepMnFfT1/6lKHz6tyhKdluFZ5opCtpNpu0E5KzamUFIBQEursb9o4XdYtffvezQptInOw9K3SFQq5eT0oQYu7oZLPFm8QBUfYJ71qmYbGPHyQcTIQeXJkuLcdK9GG7x24JkDx4ioyhX5IAVqgsK2mRYy3K8hlEFIe5MYO6KxoVaq6i/XuPknTtDJY5ZwZO4HlqJSNehRNWqanB0v6zi4eEoy8NBrEwSfznIYFdgzToJWeQ2r6PZu1nKAxrHH/RV/oEtEVGonnrcke9LPW4+RXCpC4vV6HbYIB9Ac1wRDStGGVGoiF/NGV/R6jWteH23Xvgjr7TQ99VWLuEx/Wi8EoPdIp8FMMZX5TyrR8FACS4gfFTZOwx+UAHDuOPIrHvumaKclJCbnjt84ey0mGnqe1qgmO4ftB8mTaAIoaxNBmOv16B14mVQjfIuxaNom5Txiya97LhzkcOHQZoZDc4FuS6ou/eKt2gA0mpCfBw70oHdqybO/odMWZ3tszD89y0KDoiOUsU+HbWPLTT3T4S7vDe444EJC3FtQaTVCL993ZAJqM3wyrBlr5c4Vh1j8GNRYtblUZSxoWw0xNHP7YUNWtFMOV0dJZAvIHa+BzUYIM2+UwTJHJ+7L24alq7kGctzIsszSetpUbGLGmd9J8qs1072KoYgnMWr421FoYao9Z3HQ== X-OriginatorOrg: est.tech X-MS-Exchange-CrossTenant-Network-Message-Id: 2b8ed477-6dc2-4c65-57d2-08decd525a5f X-MS-Exchange-CrossTenant-AuthSource: DB9P189MB1641.EURP189.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jun 2026 15:57:52.0995 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d2585e63-66b9-44b6-a76e-4f4b217d97fd X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JZnd5q9jhamNPWaP5ENmTBduf30/1etj6MGIFmzaVRoC6hR/15cG/fVTvN2YAWm2rCDcC87T2drVQFtKG+RxEw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMBP189MB3277 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 18 Jun 2026 15:58:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/239088 Replace the sort/grep/sed parts of copydebugsources with Python filtering on the NUL-separated source list. Keep cpio since it is faster than doing it in python. Use an explicit prefix + "/" match before stripping the prefix so source selection is limited to files in the mapped debug source directory. Replace the find/sed symlink fixup pipeline with os.walk() plus cpio, use an argv-list mv for the externalsrc relocation, and pass the empty-directory find command as an argv list. The first cpio copy pass continues to ignore failures as before since some inputs are expected to fail. The symlink fixup copy still reports cpio failures. Signed-off-by: Anders Heimer --- meta/lib/oe/package.py | 72 +++++++++++++++++++++++++++++------------- 1 file changed, 50 insertions(+), 22 deletions(-) diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index c375acc124..d047e41a78 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -1017,26 +1017,50 @@ def copydebugsources(debugsrcdir, sources, d): bb.utils.mkdirhier(basepath) cpath.updatecache(basepath) - for pmap in prefixmap: - # Ignore files from the recipe sysroots (target and native) - cmd = "LC_ALL=C ; sort -z -u '%s' | egrep -v -z '((|)$|/.*recipe-sysroot.*/)' | " % sourcefile - # We need to ignore files that are not actually ours - # we do this by only paying attention to items from this package - cmd += "fgrep -zw '%s' | " % prefixmap[pmap] - # Remove prefix in the source paths - cmd += "sed 's#%s/##g' | " % (prefixmap[pmap]) - cmd += "(cd '%s' ; cpio -pd0mlLu --no-preserve-owner '%s%s' 2>/dev/null)" % (pmap, dvar, prefixmap[pmap]) + with open(sourcefile, "rb") as f: + rawpaths = f.read().split(b"\0") + + # Ignore files from the recipe sysroots (target and native), and + # compiler internal entries. Use a set comprehension to prevent + # duplicate entries. + sourcepaths = {path for path in rawpaths + if path + and not path.endswith((b"", b"")) + and b"recipe-sysroot" not in os.path.dirname(path)} + + for pmap, prefix in prefixmap.items(): + dstroot = dvar + prefix + prefix_slash = os.fsencode(prefix) + b"/" + relpaths = [path.removeprefix(prefix_slash) for path in sourcepaths + if path.startswith(prefix_slash)] + + if relpaths: + subprocess.run(["cpio", "-pd0mlLu", "--no-preserve-owner", dstroot], + input=b"\0".join(sorted(relpaths)) + b"\0", + cwd=pmap, stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, check=False) - try: - subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) - except subprocess.CalledProcessError: - # Can "fail" if internal headers/transient sources are attempted - pass # cpio seems to have a bug with -lL together and symbolic links are just copied, not dereferenced. # Work around this by manually finding and copying any symbolic links that made it through. - cmd = "find %s%s -type l -print0 -delete | sed s#%s%s/##g | (cd '%s' ; cpio -pd0mL --no-preserve-owner '%s%s')" % \ - (dvar, prefixmap[pmap], dvar, prefixmap[pmap], pmap, dvar, prefixmap[pmap]) - subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) + symlinks = [] + for root, dirs, files in os.walk(dstroot, topdown=True, followlinks=False): + for name in dirs[:]: + path = os.path.join(root, name) + if os.path.islink(path): + symlinks.append(os.fsencode(os.path.relpath(path, dstroot))) + os.unlink(path) + dirs.remove(name) + + for name in files: + path = os.path.join(root, name) + if os.path.islink(path): + symlinks.append(os.fsencode(os.path.relpath(path, dstroot))) + os.unlink(path) + + if symlinks: + subprocess.check_output(["cpio", "-pd0mL", "--no-preserve-owner", dstroot], + input=b"\0".join(sorted(symlinks)) + b"\0", + cwd=pmap, stderr=subprocess.STDOUT) # debugsources.list may be polluted from the host if we used externalsrc, # cpio uses copy-pass and may have just created a directory structure @@ -1046,13 +1070,17 @@ def copydebugsources(debugsrcdir, sources, d): # Same check as above for externalsrc if workdir not in sdir: - if os.path.exists(dvar + debugsrcdir + sdir): - cmd = "mv %s%s%s/* %s%s" % (dvar, debugsrcdir, sdir, dvar,debugsrcdir) - subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) + srcdir = dvar + debugsrcdir + sdir + dstdir = dvar + debugsrcdir + if os.path.exists(srcdir): + entries = sorted(glob.glob(os.path.join(glob.escape(srcdir), "*"))) + if entries: + subprocess.check_output(["mv", "--"] + entries + [dstdir], + stderr=subprocess.STDOUT) # The copy by cpio may have resulted in some empty directories! Remove these - cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir) - subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) + cmd = ["find", dvar + debugsrcdir, "-empty", "-type", "d", "-delete"] + subprocess.check_output(cmd, stderr=subprocess.STDOUT) # Also remove debugsrcdir if its empty for p in nosuchdir[::-1]: From patchwork Thu Jun 18 15:57:47 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anders Heimer X-Patchwork-Id: 90444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE48BCD98F8 for ; Thu, 18 Jun 2026 15:58:04 +0000 (UTC) Received: from DUZPR83CU001.outbound.protection.outlook.com (DUZPR83CU001.outbound.protection.outlook.com [52.101.66.67]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.23044.1781798274316705747 for ; Thu, 18 Jun 2026 08:57:56 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@est.tech header.s=selector1 header.b=MbPAdWJ7; spf=pass (domain: est.tech, ip: 52.101.66.67, mailfrom: anders.heimer@est.tech) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hUFWIWXTOzBWXz/uXiFC+syf/93kbfP7fW3AZJEo7pt3ITASFe+dbj6n4lh0oR5rSgWRjFYtsnGoWJtYhZZTwLvA9bFi4LaHlh4NppVG96Y9rP8z6QMOgCpjE/kdRbqJ7f2JDEqYB27phEoj0NqOV1AgYnnWEWh4rlFSCTZIV5rRiRKJE6aRsit/gHcAugrprBivTH30rlPR7E8zNbDYGdKCzqUv6Z2yvn+SfrhW7W1Yby3lc0HxYsvO3AXZLGRXXb1DRj6wbtoZWmKWg0CQgbdwZRuhn5+dJNFeQM3UpNG8sZVQXVNmtQJVfFPbfnh7mo73MvMm0Id+Quk6coRjhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ziTmeqtNbOWDoJN6xGRsqIDQRwUmFWu2GFYpmpdhjkI=; b=gJBxomdGDia9m5RjdTcGRlXVdrLfHAQ4en0qlX2uaFppnpSycShoyePdRNYo7wcq0Yjk3wl9wre55i+UBGXPyo2ixlMJgRa7mu5RlVh04H9CMXQFvW6+1PA9CKOyEbU1tLZqH1lqA6EbhViY9MgtELPXNP0p7ci/Jj2JrISroYfIgI6PuRgMb2tJpo9j1pztNCO0vLnnHAKhaWADEj5V1mU4XTDtsu9NRFEA3t65CwcBuHXf2p4RaqApAIsqMkxl8uNih8cyLts8gWxvGZjl1Y2rqAjPbk12taWimiFdzNCE+FSgQT2DfDpLV84iwdcw4TTVJn2PAn1EBoU6QDMNHw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=est.tech; dmarc=pass action=none header.from=est.tech; dkim=pass header.d=est.tech; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=est.tech; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ziTmeqtNbOWDoJN6xGRsqIDQRwUmFWu2GFYpmpdhjkI=; b=MbPAdWJ7Yjew4v6B6d/0kBBBNo8a8SFipIwlj+r+fwGZX8fVLIhUTttC9CuNEnjh4eOm+8VXcwAnN9W5NjLYTyxcIp86frAhMNNGVkvndqGJSCDPQ/OhtBREp/JsgBWexWZgglnJWGaCfBVwx7Bv1xIj1BOBJWSFTwJsYTzi7RlY/o29Hg6oso4lYF/qB0guiWo9SEcid9NKUmkIlH+ySbq/OhAKqxZicsQqtF/ZUoW6s4EL91qeBMtst7F0H2/B9DDeeVpsN9qWte1M82gvvB/R2G76+BmtaGSkXrMLRCfWtsvsTcg58K+sySml0CEGkm4cUWHhMk4SjQR8DGBdfg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=est.tech; Received: from DB9P189MB1641.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:2ac::9) by AMBP189MB3277.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:6ad::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.139.11; Thu, 18 Jun 2026 15:57:54 +0000 Received: from DB9P189MB1641.EURP189.PROD.OUTLOOK.COM ([fe80::90da:b700:f102:5c82]) by DB9P189MB1641.EURP189.PROD.OUTLOOK.COM ([fe80::90da:b700:f102:5c82%6]) with mapi id 15.21.0139.011; Thu, 18 Jun 2026 15:57:54 +0000 From: Anders Heimer To: openembedded-core@lists.openembedded.org CC: Anders Heimer Subject: [PATCH v3 2/2] oeqa/selftest: add copydebugsources tests Date: Thu, 18 Jun 2026 17:57:47 +0200 Message-ID: <8aab102dc05984d47067c2f50c834b1ac71a1554.1781796358.git.anders.heimer@est.tech> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: X-ClientProxiedBy: DU6P191CA0017.EURP191.PROD.OUTLOOK.COM (2603:10a6:10:540::28) To DB9P189MB1641.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:2ac::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB9P189MB1641:EE_|AMBP189MB3277:EE_ X-MS-Office365-Filtering-Correlation-Id: 806c1281-0e40-4d48-35db-08decd525bf0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|23010399003|366016|56012099006|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: m39XcYoe6yS0Wy7n8OOHINkvsdHLM0PUje4YyWZvEf5g9nGNSIr7ZRDdXt/f8F1gqgE/OAiV4zuCS20f+Cz+FcF19HxRvyzUoLtTTWe25tl3TK2TmDR4wNg2LmvuTRg2OvbeX5Qf7sYkGRcjQXGqZyj8CCt0e8M51MkudHfIfv53Bfme3Xlm3JPHhvuQTPGe1uLzakMhV8fyokkkE1Z4itjZjW3sZluyzkmzePhv6uE0dmb853x3nyI51CMRv4+TnoZ5SClIMupSMGjUVns2BMPzXMNW3D7xxVhH3WaqwZwGnMzuySskIJxBppaE4HQGZKzxo38l2hAHtTQQHjgyeRCztckQDS+3+qZaWCCCrPn4D51v/2K+P+r4jt9uG9hKf1MBaI449FgLNck4stp6y12e3XywgFaZaSYYh8zauDQzAT3u9nZ5vHjbtl6zaGwtFKSMd7vFHI0M//MPVN43gSRYqNLf/WzXSqgHdzKcix91ADdX5+QKG+KDpib6/kdcFCUXMExEe/ydU0yjHEHlaRvrFY/yE4v/u2FNslYo0KXw0PMZZ+AZ1+0yn89RPG9+EqwasZOFUEZ9PB2AllWhnPFFhX0GG9MO4QBU70K91tFjgLZRiZw0Jj1oWPNZWp0/4c7etlFVhXzTytlPvVQ0RNml+qxRC46bKr1saYHIbWy9gwG8daJ3i/0JUbdwt6MR X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DB9P189MB1641.EURP189.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(23010399003)(366016)(56012099006)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EUI8MDC/BWAXix/d7ExEMbcP/bN/mylX9iKzbDABp+yo2GGhBdfQJzF8dBxPeflWzYpJJ3mM3McXJ7ZtSp/0SyJm7c+1csMnE3uYCrlTL2djXkgE1Nz/gJnwBBPIBpD0t+8mN4ImYJmYu+LIxeo1izbKF3J9AdvBREP0o40STG8jCY4kp5Xlrz/ZBKlNLk2eyQYdML6kSecqcHBGWVxrn+zG3qtg5Wd/uKX9AmdtKKr0tGzbvE1DBg1BUJrz21mvB5WKG5kjjATOgslW2ZUDn8X6GdUvNMIdoPxME0XLpViTRp43s0XVWz/uKiIqZX9JHeBFs4opF2XXZ23TiMWRg6/EttuE8HvRJpOPCFW53aj1fGDOb93PX0YdpRtXUlTe8r+Zlh4nCS2Dn5mCLNd7UXEyW1TxIozgXUEdqFv63/wRhs78s+PEqVufz2Ys2o4JsSLJPQdBdDY7eiQjcMrnbAm4vrD00CzeZtcVmF5vAne0K+jhhQs/EMOPBSvkKxC2d/f58OHcWSzMmOC3ZEzk4VSrCxZ6Hhby4zsSgM0WqHu0CwUfin/FcsTTPKTVTRrJrqUZq1zUAe0JrwOpWv6KupPXw8BSvLaheVeBCrzbmvnhDGHBuVEturqCyJQG2yGM37ACsjGd8GjwzmM6JB7iijKLFx0Y/Q7C+ZtTw5zjCMf/yHzO60pLUS9GbOu4sWhzt1x7BM3wU2o91kzrVRTi1E+qNgUmck+013VcgbfuuinDoOG7RKTcWWFrTj75YaWTyAbUp/+Ugnv4ElVE1050h3uYIE8aS6CBgk2nxz8W6bshVOHCahteDzIyMYN3w3nytkjGbfbs5RAkLJLPEwulOojggqABI1UFq+2+UplL1H8LtPj8xxajorcHxufK3yRx3tmsvhaa3UphCK2ZzYPnsXoaM4QFz7ImtcWJJ8RYMU713+9NjC0YfXV14E1kmCFsFckwCEZkF7zBvZ2HqWEsbyY+3pw3hRqdE4J9TPSdWM3XJI/Ufk1OIyXnbt7y/2Hk+qS6xfVUao47bJSpFdft6m9YZx7uguGeHaT8iCCigFoaidZUR1dZ0R6pquGGstVvajvPwStNy47H1m5U3ZKpY31Aw5D60sFV9o0cwpTBjz01qPPs/MLtV5tgw91LzgEgBa1JWfb3K7km+iDHvNVoGJxmnjyPbFwLKUxFEEgHJaJo1EaPnNCMeuk8Rz1dm/n7xgzwGxg5b7xLg3ctIEarihpHZe5Wp7z2NSlhnp/gwX5S753I34ZxkVBQgCIq1aSMucL+KbVRKlc34CTgw9bdhx5y1zy+tQ4ZQumB4YzpGl3+psn72EMPi1zdlnj8kuWKJAbX+/2hxTjcFHGQsUnYi3aUXyHe9xmEinzptyKNoGqVUpT02xF2ZEEc8+j2+PveSH7Z6tplA1Py+xkuEAgY7F99AVX0NH1JSNWwvvU6DyPftO5mWbBD9Ef5v93vrmM9XlJTLbpUUCUjFaxNMJTdSLTD+EoryqA35uJb1DanQRnRmVRDISC2cjiS+KATsoDa4HKINdHSYsiuikkxABY9rQH4kEBQTyzi5iZhj282MvrxD+XpUq6Go7G4kBIklTNWrfUKcTgtUuzQp6Jb3diaNIcAW89U4Qub5jDhoph9oF1U9G28Lya49uN0ZC6sBVJ+qc3jPJPJnypyLuS15su3V9/dUL20djSsD68e3dC6big2tdY2OS4UPfJAY2hJoG1eIQo0ddsxxfzxn4nTUAC8AQ== X-OriginatorOrg: est.tech X-MS-Exchange-CrossTenant-Network-Message-Id: 806c1281-0e40-4d48-35db-08decd525bf0 X-MS-Exchange-CrossTenant-AuthSource: DB9P189MB1641.EURP189.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jun 2026 15:57:54.7365 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d2585e63-66b9-44b6-a76e-4f4b217d97fd X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FGNvLepPeYj7WBbsZs/h1uryI1BYhc9cdVFUIF2tmbgMd8BSwTExy5ANoVMLh2/Rs/wFGOEX9TMVNIH/JS2wIw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMBP189MB3277 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 18 Jun 2026 15:58:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/239089 Cover multiple source-prefix mappings, recipe-sysroot and compiler entry filtering, filenames requiring safe argument handling, symlink dereferencing, tolerated cpio copy failures, empty-directory cleanup, and externalsrc relocation. Exercise paths containing spaces and glob metacharacters, leading-dash filenames, mv overwrite behavior, an empty relocation directory, and a source copy failure which leaves the prefix-specific destination uncreated. AI-Generated: Claude Opus 4.6 Signed-off-by: Anders Heimer --- meta/lib/oeqa/selftest/cases/oelib/package.py | 265 ++++++++++++++++++ 1 file changed, 265 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/oelib/package.py b/meta/lib/oeqa/selftest/cases/oelib/package.py index 5751057a8d..d1b0e2ed4f 100644 --- a/meta/lib/oeqa/selftest/cases/oelib/package.py +++ b/meta/lib/oeqa/selftest/cases/oelib/package.py @@ -6,9 +6,12 @@ import mmap import os +import shutil import tempfile from unittest.case import TestCase +from oe.package import copydebugsources + class TestKernelModuleFilenameFilter(TestCase): """ @@ -106,3 +109,265 @@ class TestIsKernelModuleSigned(TestCase): def test_rejects_unsigned(self): self._tmpfile = self._make_tmp(b"\x7fELF\x00" * 20) self.assertFalse(self.is_kernel_module_signed(self._tmpfile)) + + +class FakeDataStore: + def __init__(self, values): + self.values = values + + def getVar(self, name): + return self.values.get(name) + + def expand(self, value): + for name, replacement in self.values.items(): + value = value.replace("${%s}" % name, replacement) + return value + + +class TestCopyDebugSources(TestCase): + def setUp(self): + for tool in ("cpio", "find", "mv"): + if shutil.which(tool) is None: + self.skipTest("Required tool %s not found" % tool) + + def test_copydebugsources_copies_files_and_dereferences_links(self): + with tempfile.TemporaryDirectory(prefix="oe-test-package-") as tmpdir: + source_root = os.path.join(tmpdir, "source") + second_source_root = os.path.join(tmpdir, "second-source") + workdir = os.path.join(tmpdir, "work") + pkgd = os.path.join(tmpdir, "pkgd") + debugsrcdir = "/usr/src/debug/testpkg/1.0" + second_debugsrcdir = "/usr/src/debug/secondpkg/1.0" + + os.makedirs(os.path.join(source_root, "nested")) + os.makedirs(second_source_root) + os.makedirs(workdir) + os.makedirs(pkgd) + + normal = os.path.join(source_root, "nested", "normal.c") + special = os.path.join(source_root, "nested", "name with ; spaces.c") + leading_dash = os.path.join(source_root, "nested", "-leading-dash.c") + recipe_sysroot_basename = os.path.join(source_root, "nested", + "recipe-sysroot-file.c") + internal = os.path.join(source_root, "nested", "compiler") + built_in = os.path.join(source_root, "nested", "compiler") + target = os.path.join(source_root, "nested", "target.c") + link = os.path.join(source_root, "nested", "link.c") + ignored_src = os.path.join(source_root, "recipe-sysroot", "ignored.c") + ignored_native_src = os.path.join(source_root, + "foo-recipe-sysroot-native", + "ignored.c") + second = os.path.join(second_source_root, "second.c") + + with open(normal, "w") as f: + f.write("normal\n") + with open(special, "w") as f: + f.write("special\n") + with open(leading_dash, "w") as f: + f.write("leading dash\n") + with open(recipe_sysroot_basename, "w") as f: + f.write("recipe sysroot basename\n") + with open(internal, "w") as f: + f.write("internal\n") + with open(built_in, "w") as f: + f.write("built in\n") + with open(target, "w") as f: + f.write("target\n") + os.symlink("target.c", link) + os.makedirs(os.path.dirname(ignored_src)) + with open(ignored_src, "w") as f: + f.write("ignored\n") + os.makedirs(os.path.dirname(ignored_native_src)) + with open(ignored_native_src, "w") as f: + f.write("ignored native\n") + with open(second, "w") as f: + f.write("second\n") + + empty_dir = os.path.join(pkgd, debugsrcdir.lstrip("/"), + "empty", "dir") + os.makedirs(empty_dir) + + sources = [ + os.path.join(debugsrcdir, "nested", "normal.c"), + os.path.join(debugsrcdir, "nested", "name with ; spaces.c"), + os.path.join(debugsrcdir, "nested", "-leading-dash.c"), + os.path.join(debugsrcdir, "nested", "recipe-sysroot-file.c"), + os.path.join(debugsrcdir, "nested", "compiler"), + os.path.join(debugsrcdir, "nested", "compiler"), + os.path.join(debugsrcdir, "nested", "link.c"), + os.path.join(debugsrcdir, "recipe-sysroot", "ignored.c"), + os.path.join(debugsrcdir, "foo-recipe-sysroot-native", + "ignored.c"), + os.path.join(second_debugsrcdir, "second.c"), + ] + d = FakeDataStore({ + "WORKDIR": workdir, + "PKGD": pkgd, + "STRIP": "strip", + "OBJCOPY": "objcopy", + "S": os.path.join(workdir, "source"), + "CFLAGS": ( + "-ffile-prefix-map=%s=%s " + "-ffile-prefix-map=%s=%s" + ) % ( + source_root, + debugsrcdir, + second_source_root, + second_debugsrcdir, + ), + }) + + copydebugsources(debugsrcdir, sources, d) + + copied_normal = os.path.join(pkgd, debugsrcdir.lstrip("/"), + "nested", "normal.c") + copied_special = os.path.join(pkgd, debugsrcdir.lstrip("/"), + "nested", "name with ; spaces.c") + copied_leading_dash = os.path.join(pkgd, debugsrcdir.lstrip("/"), + "nested", "-leading-dash.c") + copied_recipe_sysroot_basename = os.path.join( + pkgd, debugsrcdir.lstrip("/"), "nested", "recipe-sysroot-file.c") + copied_internal = os.path.join(pkgd, debugsrcdir.lstrip("/"), + "nested", "compiler") + copied_built_in = os.path.join(pkgd, debugsrcdir.lstrip("/"), + "nested", "compiler") + copied_link = os.path.join(pkgd, debugsrcdir.lstrip("/"), + "nested", "link.c") + copied_second = os.path.join(pkgd, second_debugsrcdir.lstrip("/"), + "second.c") + ignored = os.path.join(pkgd, debugsrcdir.lstrip("/"), + "recipe-sysroot", "ignored.c") + ignored_native = os.path.join(pkgd, debugsrcdir.lstrip("/"), + "foo-recipe-sysroot-native", + "ignored.c") + + with open(copied_normal) as f: + self.assertEqual(f.read(), "normal\n") + with open(copied_special) as f: + self.assertEqual(f.read(), "special\n") + with open(copied_leading_dash) as f: + self.assertEqual(f.read(), "leading dash\n") + with open(copied_recipe_sysroot_basename) as f: + self.assertEqual(f.read(), "recipe sysroot basename\n") + with open(copied_link) as f: + self.assertEqual(f.read(), "target\n") + with open(copied_second) as f: + self.assertEqual(f.read(), "second\n") + self.assertFalse(os.path.islink(copied_link)) + self.assertFalse(os.path.exists(copied_internal)) + self.assertFalse(os.path.exists(copied_built_in)) + self.assertFalse(os.path.exists(ignored)) + self.assertFalse(os.path.exists(ignored_native)) + self.assertFalse(os.path.exists(empty_dir)) + + def test_copydebugsources_ignores_copy_failure(self): + with tempfile.TemporaryDirectory(prefix="oe-test-package-") as tmpdir: + source_root = os.path.join(tmpdir, "source") + workdir = os.path.join(tmpdir, "work") + pkgd = os.path.join(tmpdir, "pkgd") + debugsrcdir = "/usr/src/debug/testpkg/1.0" + mapped_debugsrcdir = os.path.join(debugsrcdir, "mapped") + + os.makedirs(source_root) + os.makedirs(workdir) + os.makedirs(pkgd) + + sources = [ + os.path.join(mapped_debugsrcdir, "missing.c"), + ] + d = FakeDataStore({ + "WORKDIR": workdir, + "PKGD": pkgd, + "STRIP": "strip", + "OBJCOPY": "objcopy", + "S": os.path.join(workdir, "source"), + "CFLAGS": "-ffile-prefix-map=%s=%s" % ( + source_root, + mapped_debugsrcdir, + ), + }) + + copydebugsources(debugsrcdir, sources, d) + + self.assertFalse(os.path.exists(pkgd + mapped_debugsrcdir)) + self.assertFalse(os.path.exists(pkgd + debugsrcdir)) + + def test_copydebugsources_moves_externalsrc_relocation(self): + with tempfile.TemporaryDirectory(prefix="oe-test-package-") as tmpdir: + source_root = os.path.join(tmpdir, "external-[source]") + workdir = os.path.join(tmpdir, "work") + pkgd = os.path.join(tmpdir, "pkgd") + debugsrcdir = "/usr/src/debug/testpkg/1.0" + + os.makedirs(source_root) + os.makedirs(workdir) + os.makedirs(pkgd) + + source = os.path.join(source_root, "real.c") + with open(source, "w") as f: + f.write("real\n") + + relocation = pkgd + debugsrcdir + source_root + relocated_name = "-relocated with ; spaces.c" + os.makedirs(relocation) + with open(os.path.join(relocation, relocated_name), "w") as f: + f.write("relocated\n") + with open(os.path.join(pkgd + debugsrcdir, relocated_name), "w") as f: + f.write("old\n") + + sources = [os.path.join(debugsrcdir, "real.c")] + d = FakeDataStore({ + "WORKDIR": workdir, + "PKGD": pkgd, + "STRIP": "strip", + "OBJCOPY": "objcopy", + "S": source_root, + "CFLAGS": "-ffile-prefix-map=%s=%s" % (source_root, debugsrcdir), + }) + + copydebugsources(debugsrcdir, sources, d) + + copied_source = os.path.join(pkgd, debugsrcdir.lstrip("/"), "real.c") + moved_source = os.path.join(pkgd, debugsrcdir.lstrip("/"), relocated_name) + + with open(copied_source) as f: + self.assertEqual(f.read(), "real\n") + with open(moved_source) as f: + self.assertEqual(f.read(), "relocated\n") + self.assertFalse(os.path.exists(relocation)) + + def test_copydebugsources_ignores_empty_externalsrc_relocation(self): + with tempfile.TemporaryDirectory(prefix="oe-test-package-") as tmpdir: + source_root = os.path.join(tmpdir, "external-source") + workdir = os.path.join(tmpdir, "work") + pkgd = os.path.join(tmpdir, "pkgd") + debugsrcdir = "/usr/src/debug/testpkg/1.0" + + os.makedirs(source_root) + os.makedirs(workdir) + os.makedirs(pkgd) + + source = os.path.join(source_root, "real.c") + with open(source, "w") as f: + f.write("real\n") + + relocation = pkgd + debugsrcdir + source_root + os.makedirs(relocation) + + sources = [os.path.join(debugsrcdir, "real.c")] + d = FakeDataStore({ + "WORKDIR": workdir, + "PKGD": pkgd, + "STRIP": "strip", + "OBJCOPY": "objcopy", + "S": source_root, + "CFLAGS": "-ffile-prefix-map=%s=%s" % (source_root, debugsrcdir), + }) + + copydebugsources(debugsrcdir, sources, d) + + copied_source = os.path.join(pkgd, debugsrcdir.lstrip("/"), "real.c") + + with open(copied_source) as f: + self.assertEqual(f.read(), "real\n") + self.assertFalse(os.path.exists(relocation))