From patchwork Tue Feb 11 15:00:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 57092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FA61C0219E for ; Tue, 11 Feb 2025 15:01:03 +0000 (UTC) Received: from DU2PR03CU002.outbound.protection.outlook.com (DU2PR03CU002.outbound.protection.outlook.com [52.101.66.76]) by mx.groups.io with SMTP id smtpd.web11.1132.1739286057297610302 for ; Tue, 11 Feb 2025 07:01:01 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=jmPagI7W; spf=pass (domain: weidmueller.com, ip: 52.101.66.76, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=WifE9ZQjEKlWuBWrEpQJF2IuOc8ts4ok9aFrWIxIAs28ScBK7K9NNe/7cscld4jPe8NJ/qBWUcbuylaLI5HI7PLxNwKRQlFIhqVSB7u+WorsDNGwH10A9MiHQb2BxOr/dKJcOicYM1hLbgylRe5VUCj+FGMpaLiHMIeTgsNig5hiWEpEqFJRmyR/1DekgZeeLOe+e0HbB41bWPNahAXOld/YUV0O8DOGUbcL3L+pdWGa+IcE0CWU2gavQ2+ZMsDnXlyoSzoGZF67LBoxEkP1VaRYbTnJboXO9ywJW+19Z0NSPjbRyW1oa0QMyCZHL5iFzPNU2KbXekyEcesxPTsAng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iS64VFU91IZ7bMi6ReKYIFCU9H5fDjvr0X5kRqdz/Xo=; b=tZitswMxPCYfRxxd7KdduwViI08FvaNTe8ENwMBolliIjEp2xVPNNPVfpxfuZlCBbc3YufzgjPw2Ha+tCtnpgKuhC1tH7RJ5/bpK7wEZOsrJqf1IjcWTmtwoCFf/hcpPqwxER7LTSaGLba1Lx2hztZ/WwPPdmiv5rC4UMJHGoQ7hst/4j6WKdau41o5vcwQlINW0Iq38KgqJRps8ieRaK9jU0y5L/dHcv92+FB+mFd7doQP1ZxQckpeVYEdhBppPDNzyHbHUMuxBPXl0W2j3bL850a9uhQVEzpGZzAc/Cu5XgjZaGAXwodkW7kZaoWGqvU9P6lOYrUw9eV8r2+R9DA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iS64VFU91IZ7bMi6ReKYIFCU9H5fDjvr0X5kRqdz/Xo=; b=jmPagI7W4WnJZT1Y/AL+Pbtq2dN1SPxdLVNBuBl4dNYaCVQmif3UcPcSxCug2v3eDHGHA68olqjvhdOFHF5aXLl0RuYJkQbOoN8sNN+Y0YZEdAXW0S+HnBDFooT5SCbpgUXccf2oAxnjdPZljmoNv3BfE24B83BvEu6G3tpzbjcAXgMJWCzo/g0AJJJ6NlUPFyL0JlYC9BmO4rlQGArt6sxomP7HQxri8tmCWJxoKrH+66/l8+pWn2QYvlDLIcMYOLmGNOP+xdX8Ks3JMQjLiSsdU6wUd2/QB52VF659abjd0b+ioYvAGDS9COFs5VrSB4ogmV37dO8+B9LxSboRAA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from GV1PR08MB8426.eurprd08.prod.outlook.com (2603:10a6:150:8a::17) by DU0PR08MB9396.eurprd08.prod.outlook.com (2603:10a6:10:423::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.20; Tue, 11 Feb 2025 15:00:57 +0000 Received: from GV1PR08MB8426.eurprd08.prod.outlook.com ([fe80::f9f5:b4bd:9e01:9013]) by GV1PR08MB8426.eurprd08.prod.outlook.com ([fe80::f9f5:b4bd:9e01:9013%7]) with mapi id 15.20.8422.015; Tue, 11 Feb 2025 15:00:57 +0000 From: Stefan Herbrechtsmeier To: openembedded-core@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 14/30] classes: base: add get_src_uris and unpack_src_uris functions Date: Tue, 11 Feb 2025 16:00:18 +0100 Message-ID: <20250211150034.18696-15-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250211150034.18696-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20250211150034.18696-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0103.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a9::16) To GV1PR08MB8426.eurprd08.prod.outlook.com (2603:10a6:150:8a::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: GV1PR08MB8426:EE_|DU0PR08MB9396:EE_ X-MS-Office365-Filtering-Correlation-Id: 618a9279-f656-4274-e260-08dd4aace116 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|52116014|366016|38350700014; X-Microsoft-Antispam-Message-Info: NvfKKcsY2fY1lCFJho4przcxYEJv2hPFzDzKDFCJkdPY/2qKkFa4DO4NNkbWjMK+4xyVkYDjP6mVBTMzVBERGsnGe/BsFg3JK73n+HtBlou7MkBxkr5zxvuOI5UBy63BvlmtMe4LNqDm4BfmZcshtqAAqqPNgZT4+yt5xh+y0dCxqoPI1lWClHwYRNiddhGrk9egXkAEYmh/5uaYrov3efohlJbBNQS60R0jCFkmA4GR2ql29D0m/zlzupO7+YJ/7T9+AJ0M7vQ+WT82pNx6yVVPrxSOMW491/EbgKpzLPmFJGyi7Fb8Ipx/sRgyVNUWRBxnY8kCGdhFv2FF0pARJE/forNNejocpVEEJxtdORn/qYA5IYVjsWINChP64gdmfZ0ffUyFmAXwiEfsfIxzPuqv5XSegHYJjQa2ov+qUywKPXGIgo7uVCQNv+7Fp+hDbfx1DFpLhxGLCpI3EfkEo46OXl0SVsYNVyZazgzqDUBb/vQ6Jrf2VUo/9gcTP/u5IloIv1+LFBCQFLJOi+z58suhcoZINl/gcgodlFpWVQpBmsAYskzJSJkqjLYTyv+t2C8/vb/DUJKx6klwvH4O3NnCbFlxEIx+hvAtaaB2c32QK/bgEinIMj+RSRIhx25cgdQz0tuADPlly764fGwIO9FTWu7xExYTHwC6OZKgBpvvPPX+W8OaD/3rV84G/p6TXfF5CkeNhjsNrQEXHmewNNnmHTMIudg54tjprpsHbwG+2HWmXeSDxQbhwtyh6Uyu+Cc3YiOvEfWo0/MySdeXdMSd/Z5/NfBKAZ6sMobJx4AbMkJvANHKL5nuujSbZQowdAfm4fflHVdwMOKJLugO1Njze9paGYIvAMFu/2HlEZIdhZyQzDm+zvgb99prspXy7b8qK1X8jdUw2i52YhpZ2WsIETWz2qUexJCn4NuW2WhRyaOqxqzCrnmx2sLPFSVeIJdeE92lBqOtMQtoEjkZP8v3HreP3aE8sSRY22pt+DyZrBM/o77mrROOVKX7a51DtW7G5IzCO4ZN7ajUInB7fmoQytmCOh9uQvPtsEY+Mm7vrifV8NZR88UI9yhGttHNaR3CQMOwxH/FIv43zoVP++Jmd3C6boSPoHgAVAnumCxaPZNsWjvOjnEs89LRWuFaHViciMJxs1d7VR/lMlJZMrUvt3euDbT/xe3JB06fElQNcbRbpkAIPtKopfrRTeG3rkJ/gAFbazYeX/0Dpaiia4RM3+c0Mu8f2CiwHLcem8iph9JHJwuDnmGcD7jNBS9yZGlFLrXCRZ/otG0BJCHYUBG1qAlXZLt3iQT5Ucw7nlfYQ4W3XAhKpI4kfvMnr9ciAI+3YS8+PjXwpNvpkCTceia/cxbg+B0yVVbqmhumYZV7OB6mpJLxvFBoE8BcCR2fSuRqQ1eG9KsfIYMJ+hBcB3033RCJgIfh3H++Y3/cMzHlo3THyVqcYWB7JLSvuTUm X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:GV1PR08MB8426.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(52116014)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2Cnnyg0e+jZNkg2lGfgl2SS/s6XIdWOJKyTDrtFHsoU7d15J7j290l+ak+ce+M5St9s6yhE+C75J4MzMwHtBuNbKf4t2TGhIONSdlJ1nq9oY8x8cLm/Xdbw1cCo/debQopEb2NBIEaYJgdQTqQ3bmMlvHVCnoDfR4xeycMm2j1PMwGae9J1OGzpdCDLUXIt1V8kKQC5W30qpu4iSO+g19shs9ccAId+O0JvxLJAJM2bRAAM6tbbGxXDwQ2uhsCc/6J2j+UalHHUWnhj5CocmWrB7MSacXjG/XqWJT58mLGpyof9HYVXjOSI/pUs5YvxW/Ka5FU7E3+dquOodBh0Q8gyZdJCTSZv77QAxdRfPgsFkGXFKHb6KPDfaa5Z5wrq2bRPTEeZ4WOJIeQkdxXmRb1MDObrAKRDnnlQWm4LladjyrxIiRCmjbLGilZnmYTV+PevaiZNOdDLNcbjCTYzTVkXuOWf4V1OTCm2GiUeNvgJlR389IkGyH+dnApt/SngYp+vgrXzcAgZmD+Zj2TXSDXI+VUyKiXS8WB6sQPCMHCWgiQuykv2bsXg4ETcvRMqt1Hf6R4OJF8gYX231ciH/67WqBTtTcht4+RbMocD7pc2NEhPoPKTgV0tXaY5MMKdxbw4fUKdWarJrOR1TEgsNRSUvBavzmGQLAZ8MJZR2BlRAeASxdViLVayGA6vaUs/KvRorE0/ESgRVEXwC8C9bFr+3e1ktKphMSfiFRKLgXWGoX8RIEbw7nmNWoKViR251hpVCjq+mF7OT5+dOHPsTVBOmLqyM7J9DDpEzdbai78UtZFf8MqTqbvRx9A0vhKTvfhy4NENNxEeHbGQVANTq2t/fqeMPcC/MAe0uq+RpXgt3J1+bSQMAuXZr2ZU02kmKqJDQ543USpxmsLzeLaO4EsQ3kEHRUsvfPTXajgszHX2jn4X0PUjPlBhn4jtmmlMeTPdV/VnmfLvY0r9SAqJJlp1VcUo4NIyMu3GTzeYqNhjmtaYy2A0/s4vL+LLdhTaGCPfefiRxcLGHPFzWUhNhKkOn+Xo3TnlhYdFK7cEsnyW/eN3E9pf9AmAm/oadyXzsM0oYtKzxA86rHmT52fEUbYrJTqJendbwcRwxRGsEJpc8X+vV7BOAXpeC8ll0CUX8HBndmvmLZAkaBigLQkZRCju+PbGlkX3w8FEnT/LvZmU0UiNIGJcB/JYs4Cln0FWzfOkbhzJoasjeEQbgkLjlmL5IKcUEF6fZCk7ijttfb5Tad1gK6voTBusg/DZvm3iKAN+/5LnB3jETAMGeUtjZalpR7wapzOwf88qzNB7noRiGrbRIx8K3+Yzic3sg71gEr7mSjehoCk2m0FU5LX7DoPE0aEEJFVPeqBQXyDqhB8VJvmOdnp5rvWZPTZvlFkOiEUrWOIVwam7ZnWU3guwyV8wNpUOWPET0TOE86S/ci5wrw0JamE16Txhv6VtUJlQQMxiMZA7dfXIH1xmLMEmnI/adrqstqLF3JfsgCBKGo//X8yJluQw1Bq57hty0X5LTq4R5KFDBtOL3kHWZiUnqlvFG/zlkJZP7KrLV2xof7UqzzZTSCT6XBte2X1cMuG994YmD1dvn7rz6y6WWLRAd9w== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 618a9279-f656-4274-e260-08dd4aace116 X-MS-Exchange-CrossTenant-AuthSource: GV1PR08MB8426.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Feb 2025 15:00:52.9332 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VLzGYsV74h7RrbE7n2aKUfX6aVxYCXTBycXjLLiYJfNvo2gnQtomfWqnw3vYUrTusyVrr4lfoztlPtMubXt/Ug== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB9396 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 11 Feb 2025 15:01:03 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/211140 From: Stefan Herbrechtsmeier Add get_src_uris and unpack_src_uris functions to share code between the early and normal tasks. The functions have an early argument to only handle the recipe SRC_URI and doesn’t parse the SRC_URI_FILES. Signed-off-by: Stefan Herbrechtsmeier --- meta/classes-global/base.bbclass | 47 +++++++++++++++++------- meta/classes/archiver.bbclass | 4 +- meta/classes/buildhistory.bbclass | 4 +- meta/classes/copyleft_compliance.bbclass | 2 +- meta/classes/create-spdx-2.2.bbclass | 2 +- meta/classes/externalsrc.bbclass | 2 +- 6 files changed, 41 insertions(+), 20 deletions(-) diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass index 924f319999..2571dab3ba 100644 --- a/meta/classes-global/base.bbclass +++ b/meta/classes-global/base.bbclass @@ -139,9 +139,24 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" do_fetch[network] = "1" -python base_do_fetch() { - src_uri = (d.getVar('SRC_URI') or "").split() +def get_src_uris(d, early=False): + import oe.vendor + + src_uris = (d.getVar("SRC_URI") or "").split() + if early: + return src_uris + + src_uri_files = (d.getVar("SRC_URI_FILES") or "").split() + for fn in src_uri_files: + with open(fn, "r") as f: + u = oe.vendor.load(f) + src_uris.extend(u) + + return src_uris + +def fetch_src_uris(d, early=False): + src_uri = get_src_uris(d, early) if not src_uri: return @@ -150,11 +165,25 @@ python base_do_fetch() { fetcher.download() except bb.fetch2.BBFetchException as e: bb.fatal("Bitbake Fetcher Error: " + repr(e)) + +python base_do_fetch() { + fetch_src_uris(d) } addtask unpack after do_fetch do_unpack[cleandirs] = "${UNPACKDIR}" +def unpack_src_uris(d, unpackdir, early=False): + src_uri = get_src_uris(d, early) + if not src_uri: + return [] + + try: + fetcher = bb.fetch2.Fetch(src_uri, d) + fetcher.unpack(unpackdir) + except bb.fetch2.BBFetchException as e: + bb.fatal("Bitbake Fetcher Error: " + repr(e)) + python base_do_unpack() { import shutil @@ -162,12 +191,8 @@ python base_do_unpack() { # Intentionally keep SOURCE_BASEDIR internal to the task just for SDE d.setVar("SOURCE_BASEDIR", sourcedir) - src_uri = (d.getVar('SRC_URI') or "").split() - if not src_uri: - return - + unpackdir = d.getVar("UNPACKDIR") basedir = None - unpackdir = d.getVar('UNPACKDIR') workdir = d.getVar('WORKDIR') if sourcedir.startswith(workdir) and not sourcedir.startswith(unpackdir): basedir = sourcedir.replace(workdir, '').strip("/").split('/')[0] @@ -175,11 +200,7 @@ python base_do_unpack() { bb.utils.remove(workdir + '/' + basedir, True) d.setVar("SOURCE_BASEDIR", workdir + '/' + basedir) - try: - fetcher = bb.fetch2.Fetch(src_uri, d) - fetcher.unpack(d.getVar('UNPACKDIR')) - except bb.fetch2.BBFetchException as e: - bb.fatal("Bitbake Fetcher Error: " + repr(e)) + unpack_src_uris(d, unpackdir) if basedir and os.path.exists(unpackdir + '/' + basedir): # Compatibility magic to ensure ${WORKDIR}/git and ${WORKDIR}/${BP} @@ -704,7 +725,7 @@ addtask cleanall after do_cleansstate do_cleansstate[nostamp] = "1" python do_cleanall() { - src_uri = (d.getVar('SRC_URI') or "").split() + src_uri = get_src_uris(d, True) if not src_uri: return diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass index df271feddd..c297f6008d 100644 --- a/meta/classes/archiver.bbclass +++ b/meta/classes/archiver.bbclass @@ -190,7 +190,7 @@ python do_ar_original() { ar_outdir = d.getVar('ARCHIVER_OUTDIR') bb.note('Archiving the original source...') - urls = d.getVar("SRC_URI").split() + urls = get_src_uris(d) # destsuffix (git fetcher) and subdir (everything else) are allowed to be # absolute paths (for example, destsuffix=${S}/foobar). # That messes with unpacking inside our tmpdir below, because the fetchers @@ -332,7 +332,7 @@ python do_ar_configured() { python do_ar_mirror() { import subprocess - src_uri = (d.getVar('SRC_URI') or '').split() + src_uri = get_src_uris(d) if len(src_uri) == 0: return diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass index d735dd5fb5..1a6bed8a84 100644 --- a/meta/classes/buildhistory.bbclass +++ b/meta/classes/buildhistory.bbclass @@ -289,7 +289,7 @@ python buildhistory_emit_pkghistory() { rcpinfo.layer = layer rcpinfo.license = license rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') or "")) - rcpinfo.src_uri = oe.utils.squashspaces(d.getVar('SRC_URI') or "") + rcpinfo.src_uri = " ".join(get_src_uris(d)) write_recipehistory(rcpinfo, d) bb.build.exec_func("read_subpackage_metadata", d) @@ -933,7 +933,7 @@ def _get_srcrev_values(d): """ scms = [] - fetcher = bb.fetch.Fetch(d.getVar('SRC_URI').split(), d) + fetcher = bb.fetch.Fetch(get_src_uris(d), d) urldata = fetcher.ud for u in urldata: if urldata[u].method.supports_srcrev(): diff --git a/meta/classes/copyleft_compliance.bbclass b/meta/classes/copyleft_compliance.bbclass index 9ff9956fe9..e16427e9f4 100644 --- a/meta/classes/copyleft_compliance.bbclass +++ b/meta/classes/copyleft_compliance.bbclass @@ -29,7 +29,7 @@ python do_prepare_copyleft_sources () { sources_dir = d.getVar('COPYLEFT_SOURCES_DIR') dl_dir = d.getVar('DL_DIR') - src_uri = d.getVar('SRC_URI').split() + src_uri = get_src_uris(d) fetch = bb.fetch2.Fetch(src_uri, d) ud = fetch.ud diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index aa195f5aa7..43db4f6e3b 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -349,7 +349,7 @@ def add_download_packages(d, doc, recipe): import oe.spdx import oe.sbom - urls = d.getVar("SRC_URI").split() + urls = get_src_uris(d) fetcher = bb.fetch2.Fetch(urls, d) for download_idx, f in enumerate(fetcher.expanded_urldata()): diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass index 70e27a8d35..17a0159bc0 100644 --- a/meta/classes/externalsrc.bbclass +++ b/meta/classes/externalsrc.bbclass @@ -65,7 +65,7 @@ python () { bb.fetch.get_hashvalue(d) local_srcuri = [] - fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d) + fetch = bb.fetch2.Fetch(get_src_uris(d), d) for url in fetch.urls: url_data = fetch.ud[url] parm = url_data.parm