From patchwork Fri Dec 20 11:25:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 483BEE7718B for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (EUR05-DB8-obe.outbound.protection.outlook.com [40.107.20.118]) by mx.groups.io with SMTP id smtpd.web11.149744.1734693993154756588 for ; Fri, 20 Dec 2024 03:26:35 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=rvUaJ610; spf=pass (domain: weidmueller.com, ip: 40.107.20.118, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HSMd+QmbxzHR1cUPQk7XlTQz9rWS68ASE3ZhJaftQNrsWGEA7LEQoMchQ6fmU/OSL2Z96LUf2xt7gkWx4IQ3tFVhv7dbFpnwBT5JbhqyP/SZq3IhHhLNKdAf8LaqGYwSVXVKInPwM46NvIDirjTadbEVAFdntGk3YYZ/0bcLf9Stu9R8CWGoJ4Bd36+X9i22EM0OOi2IswPA6xuy+SdWR+/lcUF4utrRPzOYH935N5YGFsifiPZQGOcEeSfUPw3y+XCpwJPh6/GPjofGMroouFoFrW8+F2p7qvYSZDW5Ya/kNz25VUo9n7IPP1v0yEC7Ww/Kg8FQIMvJZjfhCmwHww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eTPvKRjIRugd0DfnF+zHlcgXQw+aAS5AGA5VFuhyZQI=; b=u1j27tXk80XQR2BBUG7IwWACHzvJsYupHr+MO5wUL9sMGcu+JOymLqM/IoH8IWL4o3Z9Ma/jSE6fyL2pZifOcPX64XC2D7UhFb6da+YdwfQM9krPfdoCOxccA+TyO3VhHFB3PTl69eRJqFVFFvZs+JQA1Yw++6lCPGDttPUuuRBL6dJlebuI461e0MbGwzDbfrrs4XMagIDu79jkbw6HcswrmYUFtsPybPCJQ1c9PVJmrbU34t/Z2OKUQTPJmTEHk6h6O35OL0GXrg5mcb+fXXdVvfd0V3bNuf+jBNTj30//RIDDHH/RSv/PwPrW5xYfRu3N0dlS8E775EFOC3pYeA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eTPvKRjIRugd0DfnF+zHlcgXQw+aAS5AGA5VFuhyZQI=; b=rvUaJ6102i8PJSx0+iAlPhSm5yF+OViYAW1vnte5KjlMDJ9FgqEVjgIhUeZ5ER6nZPCdr2uKovY+ltPtBFaDDiMNLleDuEEt2C3R+H8XRbNRn3053QSWzojKEnZ7xnBNQ7E5Me/OuKEtqFYBa4dTHxTHeOgL6QnauuSUhsFmOT3eUxRwUcQO/YsBmeJyLvFdppxGii7RjR1+vVd6Ibb1CU7pH4BjKw0NCDk0lfMwnSIRFjae7Q/3jV8ODY8BLh5mdC+xQu/SpJAqHnKoJ7tuUM8HVHpjBmkguxyFDk/v+lWMQM3rVTcBGWPN49ekgBZ33vFLlhvgu6fS3VAx0TMBuw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:25 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:25 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 01/21] tests: fetch: update npmsw tests to new lockfile format Date: Fri, 20 Dec 2024 12:25:52 +0100 Message-ID: <20241220112613.22647-2-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: 412751f3-9efc-4887-3d59-08dd20e92312 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: CDv0kHn4W5CgaDnmynLTTBxz9yz1YHMHJHkk8JzFwvue7wv8FM6cSN/YCHpA3sz47YBgxeSypfn512Sm884ogd0uR0m7HPjA87eE5t0tc4kSBNVmqGkkyF0+ydQm3MPIPzX/K1Jvz8aURemceMJDJ7UccIYpzqBlKmjuuRsUh/slcZG9AaBJ1Z0AQYVoFFCBqoZBLH3NhfoQOV5KJjog/CvEDa/CQheGaW9+EHsN0A62JfNZo84xBUuX9tCxtPzS0LxP4ZbEA6j/7F+OC5UsM+KRA1PnANIoqqfm1WhxjTbNw0uEbw1iw5eG8bVcyZVbFjG/6wWePG9nadvxBON8dnPPd9QbksQOjLvhCsw6mb6jm0I/EVUQi3IciTUamQb5lupckGoCFREVZ7xa9y88QY9NeCGHd134OMtvItkG0iT9jI91uJz9G8tJEV03fsNDgwLyqH7JGtlRQCy1euMy27+KGdnrSbiR3dX1QyFXnuxGcs10aaKdjTW6wzkmIuPJgriX5WHBae/HtvljYSrm1HZPkIAko1mZm5swuazHHx6qg1Ww1ahke+sVDpc9wJ5vBzvNRT5JacQxZp+MIeuU6AQBhQ40cHMMwH9u1xuF89mNUIhWAYs7vVMcC/p6OjcB0r3SBE+Da5UCLXszJ33jkAPEMpGDlBof2BqtAhckbi5I5EQ9poZvAgahzt3XyaHr4CDYXGrTPyArUMNjc2gSz+D/CYuSYxOIMFr4Jfullw0NrIM80LQ+SzfUKDL9LNtJbQZOzO5IX1tBdPwq4bM6SkxK6FEh5LvQTgiioZRHt5H2bj8KPiHGB0mQJhdSIIyBnvlnS3MCiRKoJsQroBYf6G1pb0ziIE68UvmXrJx91ClAUexZKYmoissNGL0tkz0DgUjvuRULhcdaUHfSEmolUzVpJBq+bORgfeLYr8wuENbDftn3YNbZNE5v9FjQLz7SxptzlPFm6Di+gUUsXPCAbwrn8e9ZuiCwgk50Dj1Re/Pda2cvY+wCZkmAZM+mk1PDQK9eOGLtEfW8OGxetS+wkPXRwN3PD4Vud22M6ptZtfM6Varz4nqmOLrfmMJRLrgD6+/a1/OSUQouZzodxoO/4g5mY0DDzIRba++pOWk9eSVqt+e84KkM9nelLM9Usc7si8Zb+L+HGZhgKm1LIKxjBf4XCrc8dS5g6Fc7pIGWUFFqQ19XzFFnypNCdFjgSEWG8XiinyEEBdd6JS29Fp+tx4zeX0705QqDvb2PQB/UuIyBxB16FyxW+f/Ap23Qe8zLxh9l3izUGy/RcSyLHWKx+Yav0xBWWsuhAkTEV95nqGK73KsizJrStmkDI9k9JZ9IqNo+jbzs1i+y/dlUoNpCz/nDoygl0nxlTgrKUi/7aYt1oHOKDSSmHg5Fq+A7JEqYGVzsvTUW9vgr/NoohVBrcg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: zKXgLSFdqAd4DXgqm2co+jDFq5GFX5L4pH5HpC5GFt48mYAE836vudmGDNh/RqiXYYf5q/XtwLk2JXHNNPL7kHxlEF/bYnXcjef8lwOxIzJUKCfsE2hCuK03isArAu3jgf8dSMXRHpfnqpL3eMw3EAo8SL3zjDSxXdH6lKZuYGbbjjZCNhc1uRtlmslonjNbRK/JhvhUULtGDLkfkvFQNTIHu/vDERAkQkVqQiPqlVUewIwQsCkygSbFnWplypVyqXxZpKwV1CY6GeEZyoaJusHakNpmAemQ3dPaaOhPTdewSac2ReOgSKoUt//DPJmTPADVlX9vuTCyXMhd5UXU8A1g/yswSzhYjNAJ3LMZsWsoMUjezHr/6o6+vpNyAPoW5O4VU7paErwenx9wcecjHwaNpWBW/u7xDIYuxXnWAMiHBfaSXu6KICL6asr5sJ7eXxV/nn5OoSAEiXRJ4hoP7w/f/J0tQ59gn16x9KEEmusgCHCQqjemjSTVHyO5xk1NDovM7VTb73X+TH5B42rontQ69VDuqB0yKo8IrCDtheGyQsunHc1Nv1+/6uuCTNT+0EhphAEPWadK+jIdYRkhL2eHOTweV9+1c06xdVsWLQmCKTPUtw5L1Uxik9huNmmEbgj/AKQi5NUQxtAUe5LNQsmfamzrGcrFF8ODF70DYvlevZZhVealncGw6t1mBQv5XrTtpi8jDav2mfi6cB50zgpeKNs1ZbKDiLlWEfevqrG5YvCsqywreWipNffkkzP4LC+9SlNKXnpxqOnTgyPt6B5/HKO0WzYr0uxkTL/JSutqnEblgOCCQFD1V8+gCKn/cTcWUNcepZ1qH0xdCD1FWvVmSecZ0q1qTCAJS4Bo7v1gozmEKzQltgh+Wba4X4TBGvdAUZlykk7Cwb90UbNfy2y0kbMEG4ZCJv9MFovaD5E2x4mgxCL8eoG69Y2CfaJnPheSzjGC4+W5w4/pu1FVpUHWKaUh2jX0SzM8rxnV1FwFavJy11BavRIfDjJu5M32IXvjIoHv8tqAloKnNko6pPi7SL4G2t+P4PEJvARPuKi8hl/tfoQsCITgUDg82rQHqAhlRdOfTS+9BoucmlBynPVn8hPKh5LST4l6Fl0q4wyFO/ZPGdUhhXoSehySV2lR7TckBs1gvB2PZu3kh3sjBbAhXDGydnXCjGJEerJ6++6v0TLmgkmEHyopCEG5QY4SkKtmU7s/cZ3aGhDRXZ9WpQ1jaPiKbuFiwUnATvGTi2ra2RJUb8Ljh9OI1p4ThjnqpzQlCf4veXI4j9lc+L7sEcxMWkKnOkpOC9A/y2be5ynn7FgUMU8EYrThuk9vlSph96VQz9XdtHQFpR2+sWYOF+1Rl0xl+fubjZFJ65pBxYTYY8pMhNmVRLBhq39+N6lFKLwDcJvtvZpqd3Azzgkj0J/SDQMhxocoER+tUfMRtp3xZrhgZAVZSh3VP81q0AxfsMm/70NXH0/OBIOHk4q7wUXkCCx68aCGCcf6BtQB71JEKdmkBE20r5TVuZ2bplTqVB5PrajUcv2cDAXtzGq+0m1uyeRLg3JJAg8g+ibMh/76kRnlPNBg7J5BQAUWr8PVlhjMOHZ6+5ClzW2fZt87VA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 412751f3-9efc-4887-3d59-08dd20e92312 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:24.5661 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KUtyVsNzbMkNt8wmTwHYqu6Ymixq+CJIxIyd+5NFbAve/+zxjLzPU0G5Sx5sdk/Y2N9qiPNhK5FLregm4N4Z/A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16930 From: Stefan Herbrechtsmeier Update npmsw test cases to new lockfile format. The old lockfile format is required by npm 6 / Node.js 14 which is out of maintenance [2]. [1] https://docs.npmjs.com/cli/v6/configuring-npm/package-lock-json [2] https://nodejs.org/en/about/previous-releases Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/tests/fetch.py | 135 +++++++++++++++++------------------------- 1 file changed, 54 insertions(+), 81 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 6dda0d381..b89348236 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2843,23 +2843,25 @@ class NPMTest(FetcherTest): @skipIfNoNetwork() def test_npmsw(self): swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=', 'dependencies': { - 'content-type': { - 'version': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', - 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', - 'dependencies': { - 'cookie': { - 'version': 'git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09', - 'from': 'git+https://github.com/jshttp/cookie.git' - } - } - } + 'content-type': "1.0.4" } + }, + 'node_modules/array-flatten/node_modules/content-type': { + 'version': '1.0.4', + 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', + 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', + 'dependencies': { + 'cookie': 'git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09' + } + }, + 'node_modules/array-flatten/node_modules/content-type/node_modules/cookie': { + 'resolved': 'git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09' } } }) @@ -2877,10 +2879,9 @@ class NPMTest(FetcherTest): @skipIfNoNetwork() def test_npmsw_git(self): swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'cookie': { - 'version': 'github:jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09', - 'from': 'github:jshttp/cookie.git' + 'packages': { + 'node_modules/cookie': { + 'resolved': 'git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09' } } }) @@ -2888,40 +2889,16 @@ class NPMTest(FetcherTest): fetcher.download() self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git'))) - swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'cookie': { - 'version': 'jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09', - 'from': 'jshttp/cookie.git' - } - } - }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) - fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git'))) - - swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'nodejs': { - 'version': 'gitlab:gitlab-examples/nodejs.git#892a1f16725e56cc3a2cb0d677be42935c8fc262', - 'from': 'gitlab:gitlab-examples/nodejs' - } - } - }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) - fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'gitlab.com.gitlab-examples.nodejs.git'))) - @skipIfNoNetwork() def test_npmsw_dev(self): swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' }, - 'content-type': { + 'node_modules/content-type': { 'version': '1.0.4', 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', @@ -2943,8 +2920,8 @@ class NPMTest(FetcherTest): @skipIfNoNetwork() def test_npmsw_destsuffix(self): swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -2958,8 +2935,8 @@ class NPMTest(FetcherTest): def test_npmsw_no_network_no_tarball(self): swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -2981,8 +2958,8 @@ class NPMTest(FetcherTest): self.d.setVar('BB_NO_NETWORK', '1') # Fetch again swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -2998,8 +2975,8 @@ class NPMTest(FetcherTest): def test_npmsw_npm_reusability(self): # Fetch once with npmsw swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -3020,8 +2997,8 @@ class NPMTest(FetcherTest): def test_npmsw_bad_checksum(self): # Try to fetch with bad checksum swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-gfNEp2hqgLTFKT6P3AsBYMgsBqg=' @@ -3033,8 +3010,8 @@ class NPMTest(FetcherTest): fetcher.download() # Fetch correctly to get a tarball swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -3072,8 +3049,8 @@ class NPMTest(FetcherTest): # Fetch again self.assertFalse(os.path.exists(ud.localpath)) swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -3100,8 +3077,8 @@ class NPMTest(FetcherTest): # Fetch again with invalid url self.assertFalse(os.path.exists(ud.localpath)) swfile = self.create_shrinkwrap_file({ - 'dependencies': { - 'array-flatten': { + 'packages': { + 'node_modules/array-flatten': { 'version': '1.1.1', 'resolved': 'https://invalid', 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' @@ -3114,29 +3091,25 @@ class NPMTest(FetcherTest): @skipIfNoNetwork() def test_npmsw_bundled(self): - for packages_key, package_prefix, bundled_key in [ - ('dependencies', '', 'bundled'), - ('packages', 'node_modules/', 'inBundle') - ]: - swfile = self.create_shrinkwrap_file({ - packages_key: { - package_prefix + 'array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' - }, - package_prefix + 'content-type': { - 'version': '1.0.4', - 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', - 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', - bundled_key: True - } + swfile = self.create_shrinkwrap_file({ + 'packages': { + 'node_modules/array-flatten': { + 'version': '1.1.1', + 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', + 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + }, + 'node_modules/content-type': { + 'version': '1.0.4', + 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', + 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', + 'inBundle': True } - }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) - fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz'))) - self.assertFalse(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz'))) + } + }) + fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz'))) + self.assertFalse(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz'))) class GitSharedTest(FetcherTest): def setUp(self): From patchwork Fri Dec 20 11:25:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3416DE7718A for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.106]) by mx.groups.io with SMTP id smtpd.web11.149743.1734693991344736713 for ; Fri, 20 Dec 2024 03:26:31 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=HX4gYcZq; spf=pass (domain: weidmueller.com, ip: 40.107.22.106, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LZq9YMdg3uzRbEE/+J8TX9dmYP6fcViP3eaowbYndjlMW6g2Kvyf1z5oGWE+FaLzT1iiCOmFLzD4Zp8LNROzweb/Z+BdWeOVbf7MRVl5O1dl7ZqFlBPByzYQ+/I+kB1b0HZIxoVxGbw8/OAv7PL8QZt3ZDWicpuVchuyt+o4h84HXdMdL5DPeONv1JuDwZhXxXx5dm0jEEyl666cZmypAu+3YK/W+T/EhkVDb0x09HAFpeAqY9DwCzcahXwbc8bELaurFqtCGxQ27lA/neQ2NBV/fFHGssXIO6R7OZ6LuywB+HHnjndChg2CIC9QjWue5IomSytCDlYzxt6YkqdlfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aTPZuyQ3aOrnIzP3GZtpG3WFcmfRMw3Q23kRVyZCQTM=; b=eYE2rRb1QbKcUKo4umiK/RLCcwn1QUXIi1Vq3DwXJjgeZctaFG297Mv+rDjJIGywX3U7IBauTaJlzEsE9cPM5nr+uRs3yPMSkh5d63q5K/o9cq4ukX7GhXh8yLKxb7YG/h4gCA3YoTjMSuhU/qwcmSqpBm0m3WpUgIoRrqFIRdUkaZ8wkZnNff5n7/6ML33n9SEM1jkL8LJ1xOPeBDx/q/hIDL5W22/ccSQU9FuaX/RRg6b2vVnnGS+jf9gHpMij7uf+G/MqqgSVq859wtoPzt9c9k8u6lUqn1JrbRVa2HzC8PcFerkEa2zOptCAltBJi0F4LfMTqy6VhApomKQ6Ew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aTPZuyQ3aOrnIzP3GZtpG3WFcmfRMw3Q23kRVyZCQTM=; b=HX4gYcZqGP28y95EoYe4Gzov4Me9ASNxlBTFtTGH1ts94lJWPNL11kapuamkHOAvcVtDnz7lVuz/SXYDZJe9J1R9BdeMKP2Dbn+DVw3+DZn02ce/scUTZiuMGSLRzWKyKSL7bcABHm93C+rgRCLr9cPg6wEPiTuAjtbn80zfhgGhq3n8NNR0+S8cRXjAS2KWhXzM7vDMBGIdta9BmzRJTsT9mNJQMiQ2DdIxj2Hm7C8v+iyZQuZR4UkQN/BGmvvNOkvPFPoDYVXajN2YgJAkR9n0pTBUkjXK/3mmSHbBszB7vDRLctPrf2ybCGW/mWO39R+ZWPxPzf8RyVo3ZJ0Ucw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:25 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:25 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 02/21] fetch2: npmsw: remove old lockfile format support Date: Fri, 20 Dec 2024 12:25:53 +0100 Message-ID: <20241220112613.22647-3-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: 29d552c8-9f2a-4138-42e0-08dd20e92345 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: EKsS0jh0Xn0RTsP2UfeLHfD+mU42TZPRhoyD4hjAWpK+DIP0eR/A/9+M8C0Wdtm3c7TfFeZHH4cWW93krFwFEMqXTzT8zem6WEpbpykfRC+4HBhpgbvMm4GlKOoU03oMKU8s2Nb7r6UK42/f2I/w6lOYTixgsdFUTbOweUyhOzBh8Vq0EKRY9BrNEuSA35zw3qE+4StN5UVTvrfC5dnHXEAD0DuvzajzEnUhg01075pIhrr5WGiQmrfES5fznVRTL0ITsQw1a/760zcZ7yXAnIMusKY/ws50veNTv4auiYgycSUFM5AxMrhVpnKcghNXiaFRNFQW2MMrHkzaOcv3LrnyN4onoMTuAYHZP2DewlTJSH5dq4brwUGfeplnuSpbDiNfRW2gBeWjTYpL5sDvf/Yb0C21aH4JC16WqNFeziwYQU1C0NNp/JtoRpslgBIAiKB3v7JWNIFhNTP38q2ZkWyIvDJz4WVDOkDuKEUUnUskUGIabt6j50SKOa557fldwrIfM0Jp55kiLSmAi/h1dHnszLXXv9OAdBh7j+asX/OyknUSZjWap42wNIQYPIIv1GfNpMzZt8O8qKbje45NzHnAe2DIzJkrgNQQsP6lj25xDtIDjDno8HTXDYnTzuv9wJ7L8qtqN6Kv9luM3EKZa/PfIIVkKte5O+sdw4+rSLzhWhtgoKU7Ke4E77o18fDyRDw33PT5KbhNQ6VdPI++oM2+T0CsTi//MCtCSPG4KKmO2DuHmZ3RY2sibyzatPTnPOSQgte1XtwZLpNZCdlE4TrThrelQAaSm/zoGimPnMD8NbUfM0DGqihb0UV6nmOviVIEn0/gFBNJGuSTxEfxo2CVmAxrHfLs+XGDsTl7V/VLdIQVSim1J+FYdkTeqojJ33HdMLG8/VrYf+wUG1NAZgIT6Ru7e88QiKX2rdysF+gFsyWQvpEqtoSd5FY81gfWJf3P9H1vaSWegwPdKQqqEemJEo6Chsx9K5EcEVsH1/3Tnamy2BoKAH+if0dlypS3mOos51vZLm+uA7fD969srG9ybzISHIPv6SApOdh8Yey7nqOTrb+B/VvYdndDy9IeZ651ZIHP3fN+GghTz63oAYK1n4M2WfilSC125H8Z1cq433TAJQDNWIXYtsaHmB4CfhjFs2Fo4UPlGKPyTOL0oJSy7uqhhQehngGjxJXa4AvaFCpQTkIjxas+OCJPnZsKwdPrwCRgDbyFubbnEnQF5dUP92bV7PSCSak7OZghRoUCEBc9wYkRltSb+kafusbzANdbSMTVowHE2F5anDNZIl1lJgYeGm7vQG8zUdPO3xH4alV1rxpJzbYqgwTzUhR4Y+FwT59lT2ZTZ197VEuSb42KEQyxw8oCJoYsXA54IHeHybe7F4fQMyCcNhnCEOEC X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: SmxsuJ63qlx6dX6qCb6myMeI7n+hf9tmd+aZ/ds5paawAetDR2S0AWUkJm5DLm4stHk0AKfAj701+O+lQbZF/gknLKylxYNF+rJHF9Je7LssZxXZhhPFneO4eQDXX9UdLSpTfmIaTsKISNkamXTVq47AC5sPAP7ICl5LZ4JvfcjsC++94d86kwbLMDZwWdH5lRaYgf4poMGXxp9e+WeIt2ovQIkiprF3m+SJ+GTJr6Ydqnvw9MMgUiYrYapyFq5YduUpUIiqyfc+cWLtTwy93ZFweLhwXWR4mqIrlxwH/1UiPbj1f7G47GXb24XT0wfDWrPz9eM9rk6tYGndiz+rM0LmOFmbJJ4uhGIJ9Pb6kOp2Af5UR/eCYdlaEv06YnEfQ2FLxyQ5Bz+1/p+S++WRbAxs803b1yEzG59U39jYwMlWP/wLOAgQLJDXbATus1Nkw3BBgSfNAdt9OcS2jurEoJ3WlSLbN6xSKb1Swc9RW1ZwhoC9loZ0BHp8wR5Eni5PBNMPcuLqAxOyzm2aHJ76W8ipuEqHAUeAWGHnbQHe2Tc/tY2F4/lhIwx4oipiKAPyEn2B3TzRitoM8YQXgRHnhl4pCBKqbCuiauHWTGTaxYgl2MRSX3xlaW0k+pw5j7JBm+TDbp7R2JuHJ8hvYAfs3LK29xKeUkI0K8hm4MIvOzIQF+liJrL+rciA1v0AhRC3M/WpacRt87piajz3v+MJMuGPn5M3s9f0eMQ3Gwggk1jYihYC+0vbJEuFDkV9Dlrd6Q79Zdpbm0NuJ94v1dwFsxS4jUeNLoOBa3J7BtV0tEMH0vUnBHWKkx7aT/fdaFyDwqnPKBAE5lZ6043sT1jRtwPkqhLQtP+/mtcCfogRWwqc4d75neN+b6v+S+K+48mMA0PaJTHtS5sScPXbZC9iRlGyUGd4NMI2GF9hynJB6dDDdXw8/OVRsM6zMnikMAIshbltOH3penn/VdtU4N0sGUB6fiomBZL/ijk8Tgr1cBWpdw990SokrgxHOFrYjdP5xuvBXaREv+tfAA6AYSONCHfO5g2HrcsqDYIjPFinoyhCCeXVBsY0NJQwKXh/QD+Rwoig5fi8s8l6tXrXxW6SW8nlNb1u6H0H8jcSL6A+uiff8iYh1IkHGXkHzv9jz8/s6bZL8pkFZrO9w2FOHMNp1tBssMs2ZlKYVVFU7ZIwa7bQhBCjkDM3znZivL+bFsiDrjzeMEC/RFmzPAfARiXPu9tMVuW6om5fvQLk6jxMNj55mDpJZCCWPZVj+3KCaU112T35NanwYxnOlC7wLg24YWUaHVALHbDqUEXCu9/n7hUnDyzoSgp4iaeVnMCqzswgBUpIyOERaox2ScutMJd/Do8AsAh8yEvbosKEPnXAqAeHTvi1B6tQ4v3pfUXxIKhMYiAcoGOhj/TUGpikUPlBP08gLv8hGQcJTFCPkFtgJVZ6Tw3AAwgmoVaN0kaixqyzgRL/O0CLntCBmTepY5FLcTfWjFKhdGGWuUTn3PxBNuhIUcwvCmnIT91Fm+8d6hevz3L7QAYOypi8+MUU87RGhK+TVCtjMbZboELsJ8yjZRWmLTKrIJ9HXkRMs7LojWeTdlcRfgzBD8GHNQm2YR+rLw== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 29d552c8-9f2a-4138-42e0-08dd20e92345 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:24.8670 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: OusTP67ncE99U3Wr6Gvbwhp0TfBcQHlViR2eaUVr8GImHEGm7kMpHCUQFVI1P0yv52QuPcUfeQ2YB4f3vuTc+w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16921 From: Stefan Herbrechtsmeier Remove support for the old lockfile format. The old lockfile format is required by npm 6 / Node.js 14 which is out of maintenance [2]. [1] https://docs.npmjs.com/cli/v6/configuring-npm/package-lock-json [2] https://nodejs.org/en/about/previous-releases Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/npmsw.py | 97 ++++++++++++++++-------------------------- 1 file changed, 36 insertions(+), 61 deletions(-) diff --git a/lib/bb/fetch2/npmsw.py b/lib/bb/fetch2/npmsw.py index 558c9a2b0..2f9599ee9 100644 --- a/lib/bb/fetch2/npmsw.py +++ b/lib/bb/fetch2/npmsw.py @@ -37,40 +37,26 @@ def foreach_dependencies(shrinkwrap, callback=None, dev=False): """ Run a callback for each dependencies of a shrinkwrap file. The callback is using the format: - callback(name, params, deptree) + callback(name, data, location) with: name = the package name (string) - params = the package parameters (dictionary) - destdir = the destination of the package (string) + data = the package data (dictionary) + location = the location of the package (string) """ - # For handling old style dependencies entries in shinkwrap files - def _walk_deps(deps, deptree): - for name in deps: - subtree = [*deptree, name] - _walk_deps(deps[name].get("dependencies", {}), subtree) - if callback is not None: - if deps[name].get("dev", False) and not dev: - continue - elif deps[name].get("bundled", False): - continue - destsubdirs = [os.path.join("node_modules", dep) for dep in subtree] - destsuffix = os.path.join(*destsubdirs) - callback(name, deps[name], destsuffix) - - # packages entry means new style shrinkwrap file, else use dependencies - packages = shrinkwrap.get("packages", None) - if packages is not None: - for package in packages: - if package != "": - name = package.split('node_modules/')[-1] - package_infos = packages.get(package, {}) - if dev == False and package_infos.get("dev", False): - continue - elif package_infos.get("inBundle", False): - continue - callback(name, package_infos, package) - else: - _walk_deps(shrinkwrap.get("dependencies", {}), []) + packages = shrinkwrap.get("packages") + if not packages: + raise FetchError("Invalid shrinkwrap file format") + + for location, data in packages.items(): + # Skip empty main and local link target packages + if not location.startswith('node_modules/'): + continue + elif not dev and data.get("dev", False): + continue + elif data.get("inBundle", False): + continue + name = location.split('node_modules/')[-1] + callback(name, data, location) class NpmShrinkWrap(FetchMethod): """Class to fetch all package from a shrinkwrap file""" @@ -97,12 +83,18 @@ class NpmShrinkWrap(FetchMethod): extrapaths = [] unpack = True - integrity = params.get("integrity", None) - resolved = params.get("resolved", None) - version = params.get("version", resolved) + integrity = params.get("integrity") + resolved = params.get("resolved") + version = params.get("version") + link = params.get("link", False) + + # Handle link sources + if link: + localpath = resolved + unpack = False # Handle registry sources - if is_semver(version) and integrity: + elif version and is_semver(version) and integrity: # Handle duplicate dependencies without url if not resolved: return @@ -130,10 +122,10 @@ class NpmShrinkWrap(FetchMethod): extrapaths.append(resolvefile) # Handle http tarball sources - elif version.startswith("http") and integrity: - localfile = npm_localfile(os.path.basename(version)) + elif resolved.startswith("http") and integrity: + localfile = npm_localfile(os.path.basename(resolved)) - uri = URI(version) + uri = URI(resolved) uri.params["downloadfilename"] = localfile checksum_name, checksum_expected = npm_integrity(integrity) @@ -143,28 +135,12 @@ class NpmShrinkWrap(FetchMethod): localpath = os.path.join(d.getVar("DL_DIR"), localfile) - # Handle local tarball and link sources - elif version.startswith("file"): - localpath = version[5:] - if not version.endswith(".tgz"): - unpack = False + # Handle local tarball sources + elif resolved.startswith("file"): + localpath = resolved[5:] # Handle git sources - elif version.startswith(("git", "bitbucket","gist")) or ( - not version.endswith((".tgz", ".tar", ".tar.gz")) - and not version.startswith((".", "@", "/")) - and "/" in version - ): - if version.startswith("github:"): - version = "git+https://github.com/" + version[len("github:"):] - elif version.startswith("gist:"): - version = "git+https://gist.github.com/" + version[len("gist:"):] - elif version.startswith("bitbucket:"): - version = "git+https://bitbucket.org/" + version[len("bitbucket:"):] - elif version.startswith("gitlab:"): - version = "git+https://gitlab.com/" + version[len("gitlab:"):] - elif not version.startswith(("git+","git:")): - version = "git+https://github.com/" + version + elif resolved.startswith("git"): regex = re.compile(r""" ^ git\+ @@ -176,10 +152,9 @@ class NpmShrinkWrap(FetchMethod): $ """, re.VERBOSE) - match = regex.match(version) - + match = regex.match(resolved) if not match: - raise ParameterError("Invalid git url: %s" % version, ud.url) + raise ParameterError("Invalid git url: %s" % resolved, ud.url) groups = match.groupdict() From patchwork Fri Dec 20 11:25:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B416E77192 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (EUR05-DB8-obe.outbound.protection.outlook.com [40.107.20.118]) by mx.groups.io with SMTP id smtpd.web11.149744.1734693993154756588 for ; Fri, 20 Dec 2024 03:26:33 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=bXZJQXfT; spf=pass (domain: weidmueller.com, ip: 40.107.20.118, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=k7gT83QPELUNx/jFHit73ks08XqB09WO8xqAeKN94oVp5a0BVVgidWkp76UOOTbml0u2NeyzUduuUbAplmokmdf8XAxxWh8GjsaAShVcNkCB6gTAa/IQCCHX424kK1SXIOMSJuVVr9Lpno1C9++Ck5n9gHjm/MicYLA7y1Kv7CcObk6INrjTbxaCsRWv8Kd9tQ55fdmBREPPVLfkBVH/+vlkYt8Ho34XvrNDfqjds5qU1cJUIvw535jMtpxGrrM7s2bBHgc3eFlTncOmwEXQe8HkVhtF0FUYehVdhDVn0Ar4O9wWUENIzIeTATTUdVBzsQOwVpqQu/X5RNUknSJzPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tECYzinVbDQxMsSSm81qc/TD7t7q7X2mvyztaVAu010=; b=Om8y1Md/q+l3Z152K5wULO47awSn6wAVAPSF5jTh+hSFalx+/yM9e3lMK4+ih3LbKlPVda8dTn6ZQ9o87AfGmJpiN6vmHM1/rpUPN90/y8q8QZOfmThjuaw0TKwmZf+I1oRdHa9FQsbdGIsU6+F70XR82T6MXf+JkgPpdg9hiKNPtRWj5wMPCBs4ovBZh6qFNnN6LOWXep08Aw9jRl/VqEAUETrr7pIfoGUe5oqRihMShQwJp6OpoZ7Nyaf7J5afm9YYbTHuqMucGKfDcphdQNemOnieCVRwTxN7jHJA0uFvHvZ2QVY+kKrHNGF/+j7TAsZKH2tpSklKWU9JQ68kFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tECYzinVbDQxMsSSm81qc/TD7t7q7X2mvyztaVAu010=; b=bXZJQXfTaJRPq3dzbOJxrfha5xezB62n6dTZYqEeW6iaTyC3L+sMOPmXER9S/eq47uGeWBbiIbhhQDXB9swAMYlV7NGcaD8IGbqVVmvNnY/3zXssEZK+bhZgwlkK/KF+2dqHH/bDyeJT4WsZ5ghSRLHgvhMztIljGCnKk1XLQPcqRgpFwCnWdMhuP1OD74T9Bu+MjfNXEXy5pu+An+R2t/I3qmGI0PuypATIdkkXrNL9irU2MvMC+xZ9F9WKvlEb47bWwgZmsoHCU3Z7B0AJcgZ/eQr6gYrpWGNij023Z8ir2niIdU8Mj3uEdXQPS05/DI7fT+/DrPZuwy75Ld1S8g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:25 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:25 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 03/21] tests: fetch: replace [url] with urls for npm Date: Fri, 20 Dec 2024 12:25:54 +0100 Message-ID: <20241220112613.22647-4-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: 2ad32487-4c16-456a-8537-08dd20e92373 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: s3sQEqh8SEv4A8HRNdW5RBsHt5EZ0RgPRPiXkuuzYXoIncyvy2SjJwz7/2TBq1x+i/UoYiQUQrvEBRJkX6OQi8hOugftrhnBMA5HYdVYFjWqDlwV4qLbq8+98y6JnTSf5rGuE6jMrw/iEfSedoEYLVRXeFTJGokNs8eTDU194ZOecUjxePSwtJSzWAqfe33WyzGVrMlvBxWXoocWdXM3DoxlZT/mJ1d3WLgkgIKqBHQ5tjBhmYXeXWUcFPcXgiQn4n6iqPIP0IHMnmh0FNL39HRhDeAxKbrua8nMwkqWHTrHA3XarorvH6Sk2wcEZiORfznJrYfhXzdjTArDEkW96+f+ndAN4zNMyG3yvoCHQJo14O7+HsIi9xIb59SbIPWLiMS005BzSCQ3FVTdgfv4pS7uFiZipEUUsGPtpJ/sjrJMzZyvsec+j2JysV3o+p9PvVKBnC/71xyjt/y9W1z+b76cxshUClkbGFpsIGvVbH2y8FpHgPTfnOOP58tAm+e/LsvlK1u4AugygZxnRqVHSsMWKq/QnsnX4Un7f6MugjQN0gu937tqvxatNmZpTb0k2Eq6y3FtXMU7lDxo/uZSPnK3lQqEvWkzbVeteGHvY1hLLdfbkeKOkc8rDYfqS9NMwiJ42FlsKswZZdZEtcyB4pjzbzaXulvLYSNJK6ReMoW26Ul06haCwdQjiKdN8UCjH42tv5Opki8D1hrubwe8BIV6/gHcfE5ktM7u+NeukOYuZrZW2QzRVa1d9LeapunVR0Wk3vdrrqXWRnPpWVb7NGs2MPqaFq2pWKIuHCqm8Bp3SR7ENKCGCm90UJZAlc0aiLdRcwswN9OUBPDQELGhNmZ2ijNSp9mqoexKYFoYF/Nf6kXdbbRTkD0mCl5s7EKLJfE5LcMpXdE6/wHi6wN+2iaWz3rQOOtD7JhEjT7n5RI26dPiI0dsHMHkFsCJ/RbFYaZTNUuqaVreE6iCiK16UhW/MKLcSn1bv677+VC3oxlyyXQ1UMsEwRD6ulJFuMTLUEZwzStHU+29AgW30+C1AOYSnbrCi8RT0sDkyCMEqCaZB4sWG9puZpK7nQ/xqgFD5cQBXgTis4mtZERs7TjlyDCILHO22ilJE7pt/JQRMeE9+JluoYknJJfSX/CLaWjcdYvcUeXJrx1gp5C+JqVzQtUewC+Y7P2Ew1JwYjQPNcVv/YBkqNxfzqi/cblKLmHVdFmJvIdQx8FcZzVVnupzGGg4Rl3bPbeevmx77Z7tedq5BOgJEBfzHOv+owrd/p4rBFYfa8zJD9d6g/B7IY9bt/4HfYak48d3iMrh4eEK2bu7h6ashdCUc/Q9hWfpbjnv5Yuhna1CZV/VFRwkIhsFGfL8YuOZZb/kC5hZYS4HMECORkZeR0IfYSi+HbSHRYRxPfkbSC7I2gI3hhHREsMhm73G6Bo2sxrJcWkw3UPAReD9ZNiAz+Pse512nnmRCtr9 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: E1sUap5oPuFP+LfqJy71/NVxeAdP0pHHLLPFpqwbHQUOgL3J4aKHZoHEfgqDSkylJb8daf62azJzqmqJ07WqmWui6isahjetFmzfV1L0ezx0wuNwEhaNg053fi9Fx7eUYMYRn2SMlCKu9oxKa2lTEXfnYLMYOL0zEjnRv3hirBsUlMgHE5JlGEOi/vbxPIqEe+eGmsDyzflEGUVRIOHRjiMFY9GH8cHVllop24urrpu35eagcTmHEr/Mlj9VVQXfCSf2xE0v5YHZ2lY1Elqj0oNYS04mjZU6x77h7MXVojeaXPT3Iy3Cg/cYmWNKZ/oCqozk60xDBcRM2jlkYOoZq9VSEuwlpI/5bPuxtMTHDebaN8YiHUFFLW/Rgmamg4IDSqkhPGL7RafcI/GricTaxt98jk/nKn2Ym5pcmTW2+oNXOX7uJTp6aVbjmVN1gByw3qlVMIQJp8ycGXHS3v63dJbeG9qn7OKAInuwU4GPtmB8y7TLa+L+rGogRgjNFflqoFcPWPBB6d31Lwc2gsKQN2UoKeK5LQxGSf3mQuZJlQ0863g9SvlwalbB84vJbPESGrmFEfF1+51FRGxlDIiyqWH13SCspNSDI1pXkfb2LulXgcjj0BjayZsiJutqWinJnmpgucVOLqSoVA7SUePvv8YmO+VpDq+HtFBnuT7phslKzZF6iIgoq7hMyfPS3OLGsEOMKsJcmlnzT7ZhI6RGgGyInQxbArxTXE1ocyLdqoOuSsD7mxV/NPUCaAxkU13fLEgTwCmlPGLybA8MOexa6ld/LjXp4k70fWs5Hu6anT/Drn3eNf025M9ZB5/V3OFimg0/XGY9bVQYLaaEcbeCQTaAvWr6VpLR3xtcXPRU8agVMHW1Zy8NkhfGr9pcf8ny3WFDBCy4EyeZ4H1NQjWwZOlGpO3hBMgsvuTjipi2sr2YUCCNJtJm6BdkboJYeB5d8MMi0Jc/u39M8xulIbun4k7YjgMGbMXeEaLC/1bioGFAoDenz21WrSy98I7wQAivN3lxzQKpgw1y6rFyuREj+/5yArm3XoHHX0uV3CWsFZL2bzfxNKGTblQdqxi3suhqaOZWRmyXqXODg4RMjYqJq8plGmiQpLkLTotOmy7LiXbdhDhOH8KujSLkisSfGB4p661Gkd+srU7rMOOrODng/EdmM1uSJe1tdUI9438SxNTu8bNZd3JBxnrE9xtO/zZHP4xvxcs+kxYL4Hsx5/SwriUtkEHj4pNSknTLgou0rO4sEtrKoM7f6AQZslKJhrFnp4gicfkxp+dE1I91Rgk7Sv6LajnP2LFDQPxF3t7+5nZ1JpVjKm8h21SKiV6Wk56DSTU22uRcfd7yZMXstiwFoUAXyeM5NJi51qq7Nv2IZ3L484g3fuxL12zk4+HDKcWbiOIIO8srh9pDDKxjAdtQrv3c8Gxkr2h3r3bAligp+wZOOO+iQzoT/Imb2nCmvN1VL7Mj5CVRHBi1H+dbs0Pso3SjcPPwKnM6N9WzvqT6C0e/vifXjF1qGtKOx/i72biKvvvxwhYB7py/l1+CxyF+DNHoDu+P8gvwI8wWydFhC9t1BqeXfUMl5TlP3RauQuIArJ7MupDLuVaIJV2mliilTA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2ad32487-4c16-456a-8537-08dd20e92373 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:25.1848 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jID07k0939LSwM/W3bgsLddUAnh7IHzb1ntanmjaJaAntZGzWtp4+LvzaDANR2TwAkhizCroEijV/iIM9P7C1Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16923 From: Stefan Herbrechtsmeier Replace [url] with urls to simplify future modifications. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/tests/fetch.py | 66 +++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index b89348236..c5ec84dc5 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2627,8 +2627,8 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2641,9 +2641,9 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_bad_checksum(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] # Fetch once to get a tarball - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2660,9 +2660,9 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_premirrors(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] # Fetch once to get a tarball - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2682,7 +2682,7 @@ class NPMTest(FetcherTest): # while the fetcher object exists, which it does when we rename the # download directory to "mirror" above. Thus we need a new fetcher to go # with the now empty download directory. - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2690,9 +2690,9 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_premirrors_with_specified_filename(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] # Fetch once to get a tarball - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2712,8 +2712,8 @@ class NPMTest(FetcherTest): @skipIfNoNetwork() def test_npm_mirrors(self): # Fetch once to get a tarball - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -2737,8 +2737,8 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_destsuffix_downloadfilename(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0;destsuffix=foo/bar;downloadfilename=foo-bar.tgz' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0;destsuffix=foo/bar;downloadfilename=foo-bar.tgz'] + fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'foo-bar.tgz'))) fetcher.unpack(self.unpackdir) @@ -2746,18 +2746,18 @@ class NPMTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json'))) def test_npm_no_network_no_tarball(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] self.d.setVar('BB_NO_NETWORK', '1') - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) with self.assertRaises(bb.fetch2.NetworkAccess): fetcher.download() @skipIfNoNpm() @skipIfNoNetwork() def test_npm_no_network_with_tarball(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] # Fetch once to get a tarball - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() # Disable network access self.d.setVar('BB_NO_NETWORK', '1') @@ -2770,8 +2770,8 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_alternate(self): - url = 'npm://skimdb.npmjs.com;package=@savoirfairelinux/node-server-example;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://skimdb.npmjs.com;package=@savoirfairelinux/node-server-example;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() fetcher.unpack(self.unpackdir) unpackdir = os.path.join(self.unpackdir, 'npm') @@ -2780,8 +2780,8 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_latest(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest' - fetcher = bb.fetch.Fetch([url], self.d) + url = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest'] + fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() fetcher.unpack(self.unpackdir) unpackdir = os.path.join(self.unpackdir, 'npm') @@ -2790,46 +2790,46 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_invalid(self): - url = 'npm://registry.invalid.org;package=@savoirfairelinux/node-server-example;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.invalid.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) with self.assertRaises(bb.fetch2.FetchError): fetcher.download() @skipIfNoNpm() @skipIfNoNetwork() def test_npm_package_invalid(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/invalid;version=1.0.0' - fetcher = bb.fetch.Fetch([url], self.d) + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/invalid;version=1.0.0'] + fetcher = bb.fetch.Fetch(urls, self.d) with self.assertRaises(bb.fetch2.FetchError): fetcher.download() @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_invalid(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=invalid' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=invalid'] with self.assertRaises(bb.fetch2.ParameterError): - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_none(self): - url = 'npm://;package=@savoirfairelinux/node-server-example;version=1.0.0' + urls = ['npm://;package=@savoirfairelinux/node-server-example;version=1.0.0'] with self.assertRaises(bb.fetch2.MalformedUrl): - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_package_none(self): - url = 'npm://registry.npmjs.org;version=1.0.0' + urls = ['npm://registry.npmjs.org;version=1.0.0'] with self.assertRaises(bb.fetch2.MissingParameterError): - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_none(self): - url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example' + urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example'] with self.assertRaises(bb.fetch2.MissingParameterError): - fetcher = bb.fetch.Fetch([url], self.d) + fetcher = bb.fetch.Fetch(urls, self.d) def create_shrinkwrap_file(self, data): import json From patchwork Fri Dec 20 11:25:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 553B3E77191 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.106]) by mx.groups.io with SMTP id smtpd.web11.149743.1734693991344736713 for ; Fri, 20 Dec 2024 03:26:32 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=rXAIepYX; spf=pass (domain: weidmueller.com, ip: 40.107.22.106, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=sN1j4gTgt0gFVnnNclEDv63U1B1MrcQBA/47qrfjwkoR1ab5cdEscyRffLyqdi73l1fdjB6jZkUA1DjL8iLjZ3zvF2VCvZiIIkg/Etls+tfWvA/vWqmUjoiLlwmaCUgpE7KQWEpewDbdk3tyqbQmSrd50kybYp+7V2gi++zDYS4WvBc5bO03P9+PL36IFOu0cpJZquJLmk6ANn3nan924paNm7LpKZmep/997JjW48k6pQGnHwEt6mu8q3IsmRV++HtIJHvRusp6KnKOwe2JGC8/9UybQJ3WzvKxAO56GfnWNyNydxckwTvkybzpeQwz7Hfh4np6NnZtFn4YACaRdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YAH8aLxGBZMJvoHCM9F/eJsCXT6U6VjnD0ZS3FSViBI=; b=LDddvS/Pi8BBSGtT80LaN+YD4IHRoKt1/1xev0G3k92kRvq++LbdVNSZYMgr0uLLSBWj+VCoeLhV3d1AjyMW/kd8t3fkhJn70oHesJ3ibi2JBGuRPv4JVggjFJ4aoIv1zb0jJU/BfEqGcwShB8YCGKD+c+F0BLg+NA5789xLCdQrEBGYJin/Anb0SrJZ68yTbA0RsJD+uKnm+jDcwSAEbJDUzoy9ppzqr2c3M6ebbVYFie9yFFdHnVrlclUEz5LKltiYITwgoh4UwFM8gOmZ5pheU2EzJ7VSfCf3V1qEAmwaZCtMVjexsCQoXpFhuu5jpa6Y+QlpwfAfTqbiM0tsJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YAH8aLxGBZMJvoHCM9F/eJsCXT6U6VjnD0ZS3FSViBI=; b=rXAIepYXQvzYzsujaq+7IovggxRBZ7IvcD/ghM7aOQzO54lS6PLrjjQtGSomZmuwqINr7XezJMiC0JYlGfArCQVz8uRRg0N3HJ3hWhz6Hm2pPb++OgSR5w+dlhiSGQKRH8ZnzUxiec3+IOIOJ8Ye7ve9jW9IOpbQ317qP5v2muTk8+BqoGdtWEjYUiOOOpefROJW0HcMcQwv208i7Z/g8NBuR3ulfqczqNu5/YucvftAddkc8EutEsa2RXEfYKKUvYdO04uOHABGBpnyx4muaasrNXi2mdCGMJMzzk+NYdE+bmFOWC8ncpeGdQ4A1watFgVmdusGmkkq8gxHG+4qLg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:25 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:25 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 04/21] fetch2: do not prefix embedded checksums Date: Fri, 20 Dec 2024 12:25:55 +0100 Message-ID: <20241220112613.22647-5-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: 8148a115-8c84-4564-40ac-08dd20e923a3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: bSu05CVs8wNRnvtQ0mOWz3WEsoN9ciOm9MGazlaox4qfFuNBL9VnJOAfRjDS1KqeDgyWsmlZURKhleCgpTt1Ri3t/gSCG2FG02huXbiEatP/LBghmcUyU6zlNZSkW9V8+Hx9PKF7JS08GKfoUeD4UhH0idHz5BI6h2fE8DG1a7a169nXw3hmdYHRey2Aj6+QpJEWqnKb2K8lcr6H5AcodHKOZAYt7n4WAuxhBTU8/Y6RE3mNQFn3Va+fKQU3q/wQgQiYMBzNc+39kpgBP5K1u4jyswLAfRl7xswihi5eGo1yhpM71Bg4S2B8bgu3Q5puUq6hh9DVKYi+EIcF8c0v1JHTfKFeFG6FN+sj1Bd7+VMsBRVmKka3MGElsZuiUfyAH2eMJ49VM5Kq28VyHYX4YwetIaJRgw1/JFKJ62sZv3I1Vd55TgOe+e+IcuZAVcm4Np7VP0zfXwH+aebyTudHQWjjx2a9kTvAL72w1IAHnXUHl5WAqv/mpfjHNRC0WdDtX5GkvLwsqUX5DtZ65NJlkOl349EXmUcvYwJv5Q01g4MGpWPX80ndTIDn/NUvkhXlepy6q1ITDql7A4s5/WaFc9BotZr+t+5Y5Q7NPOvYk33AxDGhIyojJfzOnljbLXqblo8EQXIhkCJM99Ov2IGRb4HaEvuNqc1EwcZWlLSet2r8ds3ttqWcGDqww7ffZsNVNQ+3oH0X3OtNUvkwIG8D6dslJg6vwaTZCZk3ugBroxhdCr0T2tzC4v0WS6wbBrBDszt/1uhS6i6PRKJzRDuvtU4nnxlO74ZqvPcVMhdgfuwpwR4+YHeKxXQ5MDHXPvKNM/E/mJdMUGnkRDkEr4sc1OF68IcWAEKz0Y9swAbDK+XlYcgdh56VDyatL8tFo3wmoKTplH7KSv7CRAMKIya1loegANrnCWx685cCPP1eDM3vl19mjjaVEoUvfHcizsNopM/ryC5KfpARegKDMvj4DS5kgR0l7maLHlm3DJuyxA5pmlaph/n5gu1kEj0GTkoaRFyfiPmytXiwDAOHsOTJAzgLlV593my9xlnLJzGg4F+PaR5OoQAuIqqXDDr4pGd9OI42Jmnp43rCa9cp5sui15ljwLbrBSaWeQg4XkGy3lic7B1dBD9SWrjqFYyUgIt90hpdFTCM9zUXU2i87K50JyfSyMExf2own+hRrWbGpNnbBuxiglCIMLYuukZFQ4TvuvPcAU2Sa8tWoHkr5U+HaixIO0T3hAn8t841xN7ma+JqAF4gNUG4A08njz4rh2niRqKtW5KiPA8HonFGDHUPOyr8SyiCd4sI/Wo7BRGRki8q+60ZOnKCwxc5N1tu+SeWS3UXIVYeNcB7xe8AWXCqJ8wHc2z5GyMx8bTq06b1XTSaIFFJSgxGX4kM2Spc0H6KHWbbLhdJfVRGbZfkh5xZ1Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: FwI/MLDzYp2XoU/q/TGR00IOyCx8tuxiKY5RlX1ZfXmhUUuEKF6hLbFMCaRNmrI1RUIw8r+N+lBo1GU0W2yn5zRC9J57H3ea379z03fcsd5mpU60qmjVcnhm52Ln1p/EP+MzqXKiV3Xy7pRlPD+AZVfVu1NJ8k/iz7H14+AgA8qyMomzIUm2bR39lZwm54vLZy9rgJKDqfDP+s8a7ffDCxv6BIxrIOJipVxgYrrdiuIWcb7IFFP3guEQZ/IYw1V27DEMdv0P2qUS6LZrxu+KYv0nyZ55lcU4sqMm5CTO/WelMDJ1p0S/jrNfyfeHHqdfX1wDuggRh7XD8hOO/gM7FfFI2JkuNHH1R/KUjRKMXG+W987UDh3gd4BFOqLrJgazf4ahZnPXMC68yEvqLrthXSpg9uK9LVpYR4A9jSrVwDps8WAS8zlVgXPwvOVMa1/kbOLEg+hThZxPVvDVbcEHxSCaLdSBnoexpGjE+vfEGebhMrpwlxqtH//Q8KitOtGjPdjRxqT7gHoLjxX8EI9JvwBTMjUYnB2YhUEBhI1RucE/pQTQ8UfXjYDdzT+zyWfvjFVkXHN81+GcwcTFoX2InjZKQ0e+SE1ve87P2d8ZPWG6p5ickZwOloB1HqsvYQzXnlkWPcbmdhWJXAwNWn7j6x3D9ILr8C6xLjmyhgBu8gn67MMIDoMoJLMNYPH7myzioy2UrkFGjO5DsXz8F+SB1SXUmHWlf1Fi6W7yEnWY5kOOSnlpFYShjZjlvLZ+ZFKRSw2bhivZjWd5OW2q83zh06njNDiX8R6zayKzIY+ulmhzf3z6td4zrnUuIDGiiK+Vyh91S66tdawROZ+KraS0hLXZpQCZo82bS0KFYVzqgRLXT8bwhQnBy+2YmGVdWL1xs7tfPLVqXl/aTS6/vuvvWqd9pa1dUgXOpFdPiQtGXb4h5b5OOrKMVQZIMBYmMyl7BrzMH4Wrvi6RAllpuoh00I9wF6+4dlpX/sRvrKEWmUzo/qkfbiEBiYJ7z8l4QRVZWdTqcqNDnqxHHv8I2oyv307IdfvrLEHXBq73eOXsvku2vpsI31Kmh+cEKHIn16zoN8PFrxeNUtZjXOzerrL+xynpJsDn4YBllucvcE5wkTP2pISq+SjPwkmlG8K8H10ONjbhXCkkuZGMjey873MU1A5wRV4wKYHkudCZ4cE2EzUeKPc+xOIsPE9RXGEeX91Swv4qxYizJHaN2dPs0VPnfQyRT73rohoajsITo5mzEr8mxmV0HkC+FIx5SvYIznukhqtEsqVSOSCQ+PofxcBOhTj38M1/F/EBeh48A0l47mxuYOoJktjcU3VvTjty6izw0ql8FDv7qVB6u6sQzq9HMu+QfqIEgrSRqLec/Xmww/REAECcmZU8Yrzwk/Dqc5B8y/OGjRO/hX1TdFqX01va8bSX/tCYMELdHXPlOrKmZU2xOWI42D3MSYDUaQPUyaJ42R7WHgn8tUJvKrZ/2B1t8kzRhmFdeLlYEcEt2Ur8xp5KLLzB7YkV+yy7TvLiA6M2RyMI7fnEgP9XS7uK4a0Tg3AmaBg9oJMj0hcnUAAbzOkvn1ZtCa0VxgkoMncxsLfqs1SLPKTz3IPQ7xTw3n1rKg== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8148a115-8c84-4564-40ac-08dd20e923a3 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:25.4978 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: D9c9EtLnWN362Kt/XKffG+0fqV9K9+rhInft9vti1iFlch5tWB4mI1i4Rgq0NvL8PguWDhaZm2uF/7DdaaU+XQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16922 From: Stefan Herbrechtsmeier The fetcher support entries with an embedded checksum like 'sha256sum' in the SRC_URI. It adds the parameter 'name' as prefix to the checksums if the parameter is set. This behavior is unexpected and leads to hacks in fetchers. Fallback to the checksum without the useless prefix and set the parameter 'name' in the gomod fetcher unconditional. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 9 ++++++--- lib/bb/fetch2/gomod.py | 5 +---- lib/bb/tests/fetch.py | 4 ---- 3 files changed, 7 insertions(+), 11 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index f79e278b1..3d31a8d4a 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1316,20 +1316,23 @@ class FetchData(object): self.setup = False def configure_checksum(checksum_id): + checksum_plain_name = "%ssum" % checksum_id if "name" in self.parm: checksum_name = "%s.%ssum" % (self.parm["name"], checksum_id) else: - checksum_name = "%ssum" % checksum_id - - setattr(self, "%s_name" % checksum_id, checksum_name) + checksum_name = checksum_plain_name if checksum_name in self.parm: checksum_expected = self.parm[checksum_name] + elif checksum_plain_name in self.parm: + checksum_expected = self.parm[checksum_plain_name] + checksum_name = checksum_plain_name elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs", "gomod"]: checksum_expected = None else: checksum_expected = d.getVarFlag("SRC_URI", checksum_name) + setattr(self, "%s_name" % checksum_id, checksum_name) setattr(self, "%s_expected" % checksum_id, checksum_expected) self.names = self.parm.get("name",'default').split(',') diff --git a/lib/bb/fetch2/gomod.py b/lib/bb/fetch2/gomod.py index 21fbe80f5..6c999e8ba 100644 --- a/lib/bb/fetch2/gomod.py +++ b/lib/bb/fetch2/gomod.py @@ -119,10 +119,7 @@ class GoMod(Wget): ('https', proxy, '/' + path, None, None, None)) ud.parm['downloadfilename'] = path - # Set name parameter if sha256sum is set in recipe - name = f"{module}@{ud.parm['version']}" - if d.getVarFlag('SRC_URI', name + '.sha256sum'): - ud.parm['name'] = name + ud.parm['name'] = f"{module}@{ud.parm['version']}" # Set subdir for unpack ud.parm['subdir'] = os.path.join(moddir, 'cache/download', diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index c5ec84dc5..6b8e3e060 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -3391,7 +3391,6 @@ class GoModTest(FetcherTest): fetcher = bb.fetch2.Fetch(urls, self.d) ud = fetcher.ud[urls[0]] self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.zip') - self.assertNotIn('name', ud.parm) fetcher.download() fetcher.unpack(self.unpackdir) @@ -3409,7 +3408,6 @@ class GoModTest(FetcherTest): fetcher = bb.fetch2.Fetch(urls, self.d) ud = fetcher.ud[urls[0]] self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.mod') - self.assertNotIn('name', ud.parm) fetcher.download() fetcher.unpack(self.unpackdir) @@ -3442,7 +3440,6 @@ class GoModTest(FetcherTest): fetcher = bb.fetch2.Fetch(urls, self.d) ud = fetcher.ud[urls[0]] self.assertEqual(ud.url, 'https://proxy.golang.org/gopkg.in/ini.v1/%40v/v1.67.0.zip') - self.assertNotIn('name', ud.parm) fetcher.download() fetcher.unpack(self.unpackdir) @@ -3460,7 +3457,6 @@ class GoModTest(FetcherTest): fetcher = bb.fetch2.Fetch(urls, self.d) ud = fetcher.ud[urls[0]] self.assertEqual(ud.url, 'https://proxy.golang.org/go.opencensus.io/%40v/v0.24.0.zip') - self.assertNotIn('name', ud.parm) fetcher.download() fetcher.unpack(self.unpackdir) From patchwork Fri Dec 20 11:25:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A7D2E77196 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.106]) by mx.groups.io with SMTP id smtpd.web11.149743.1734693991344736713 for ; Fri, 20 Dec 2024 03:26:33 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=pw0toZgX; spf=pass (domain: weidmueller.com, ip: 40.107.22.106, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gGeDRp6AklxvtAnt2Q6V1+ZLrEZphLtIizv9DlaXhX/A1L+dqOEMmQU72KqwuJPOQaspCFNotfS7RlkQLPJt8j/slfnhs2GUT7p1vYyGtyUbupJELBPgxVxoaO4ipdM6fuMiVtVAt8rIAB8H5ZefwcOyd3qXQRpJ89ojNm5XFjFPKQ3/e6v413XfyntNqspNltoBAgxONHmda1LwPrHGsAn4edptUG8ijOVDnD78dP5AplZkYMPnS+YvpFlHHXrQLS1Bhvp1rJWr3yqDC0tvbAMlwpBH1cGxg7XcEG6C7Px5JAfQ7A3ppT2A9nnEMNC3rWdHxoa3yHwInPI1ZqEYCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Tit6rmzEZRai1FkdRp8QgPPUeB/0AyLmW9ayX20M9nk=; b=IdGsPMXrntCojUhQ7yAg1DJj1cuhcK+YBKElbi7RTW0yRCDB3whWVpvZ20FHJRh1eqbTvpBwzZXQaNVh8ZjDG7IkGRRAsdJVQtB6DpD1XsR1NpIqw8rDp82D4LR12gYcRY/zR0IEv5EpkSv5xIarB3dsXYUWLqP6/g63kKGLgqiSdbFgYvwB74rUFk9v+eIAcVhW77wkS1nC7uP7d7aAimmFVTbhqJejTBvJRHvXpBlgv6uVteNO9bcBzJ9skZUBHTBzpK8myz5KQhOi0UVIgoa27wT9MTzs/Fh5Ky5cLPQmvksbuA2Y6yn7wXxsl4WZoy7SP8FLzh+Ra28l2svfjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Tit6rmzEZRai1FkdRp8QgPPUeB/0AyLmW9ayX20M9nk=; b=pw0toZgXoh7L2KzaY8IGER2lfJ70qFM+TSwoyyhWn/rrT6pWlz54h0znJwSDedO9gpvPR7LWJcRzrvZVLdeo0pEJQkIifqSoq10QPwu6ZvnzUowxDIzIzqqR8PxT7snRyHMF+KA3ng6Zb6g+cg1HV8U7+yAikvHevjD71WwjnHpY7V1vCDp6DxUi3Tr/0ObE+g7YcJ/lKtSO0re8LTQ76nmPMxsOJzHN6LuI/u8smhFbf+95VtpyIJV+4AQS3IZ6iD55jh1XDQqYgpXzgVLs2c/pOE6N2ds1O0xOx0PdofKdV/b84Xj0RvNCpqqAFtkImf+Gzb3rRNzdow8alLdxyw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:26 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:26 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 05/21] fetch2: read checksum from SRC_URI flag for npm Date: Fri, 20 Dec 2024 12:25:56 +0100 Message-ID: <20241220112613.22647-6-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: a88e4f9f-b850-41cb-fc6b-08dd20e923d3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: xtRXQRA9SuKeo5QOsJl44ynR5AkwSR0T33IY9mNNwnyUsOvKec/qjwLngUrspfOi2lZZzN/P2XBRleKUC9r1BL2qFxj/stJdc26w7DGjJnNx756t7G+d8qZAzPQSORuWQKmfvGMpDvzO57YZrNbHrfOOAGypWV9GDpr2Ysle3MgNDb2dEAwQ+j3ZwjI+8ptaUTsY6nnwp/IxJQerKvrYcCX0cKfu47w/3F6lEfmDyjOHdIkNM6I3UnTSOyjC68+qVn2QuQ7hJMOoFfXrB0bQsHQ5cwNxoGaHtNym9meqD5mr7P+D1KYuR+6o52GIX6PaxY0ogX998L6TcCYhmZGX06fX+TRZB3Qi/hFWuAMzsZBWQK+UkdVgK8iLfXi3sQMQlniIDHd0z+6A4fRrZKkEHLTPLGXBbH3Rz2CxG5gb2yqM8ApoqMD9RyC379WAvI/NXx4xKj5X7VrNVB7BYgR00odXZmvfNk31PmnOEX7kHEPznC1VljC5uywPz2XMnxwkXGF2iWVppvz7oCPYbJuRLrG6yp4lOorlAHoAYLL9lqhiT/RYDMOLHlXx2gX3M9+UFDk9Nn7r/7mjlE/mfvo46qdbsyrJSVOp8Tr11QPlBWLTFuxjpL5cMQAP804em8OiRsZre3impApEkSct5692qOL+jkbsC4xjTmMYUDAGupz2yHP9uuaPqWz3z0eaJiAVnzDcCb4iU7FrYqhdy6Y4bEr2LXQu5Q0Jx//AJauXUbHkT3TuZSaAxDsa6Rq+8iIJ9brUR0HDd56Yzrv72/As7LW/RF2h1wMi8rhEmgpWq97YvnofXxhSoW0f3m+745RRMLqiR1ZQwLdK8aiOmJoqh+8ZcOy7MUUlpjVRlSq87s5qJlbTHYCsY5zG9E3QnBQ+e5fCuMfHaaXHOJzi9szHIgBGwy41JcT3yKtZ4LknRabfql3NaKuN2NEiWnXcPpmIgN1djxwvmrc10pexKcu2O1kp7Jwr4eH/mCf8q/Ws4eJgG8LZSnVqN3E7OUQzNaT1jAZbLLDJ7m6ozmpRVR3u7Gn2Q10cEYfSQhhPzcb/D/VWsQam5l0JtnpUELQesImmw/OidshQh75iYRLcli/KFlialVPhNENdTi2ZDuFXsRoGVAe3apB7PYXLo8NuaOGUVdQtorw9oyARvgBcAAK0AwrA2cfJoY1UWyU0b6FGsNppfwRJpsp3omRU6171I/x9FxjTmQGdyNI1itMEwkopAzrdNLgB55BZBejjzdkj0nM8gs0hTuNiaX3Ky2a5JckSKTQIVH7J6VTc7n6UcVUo+4Bd/uFpYgCowCeT8uLogZmnv5T3y7IrHr3YSX1fhtI9H5FRp3ffmn3E8YqnJ1bVImLd7tR3kyvWi6B6knLbuZsebHBTn4lX+W3GFT4kXYjuNNw7xE8fIA5/eIxbU2GmPmBIaXP5mj+y5qD3uapiKUS/3d+XI8F5KscCKfqe7Msn X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: w1N9wEgI795VNm7xZPny2627cx4IQmao+f9sPGUzEfGkt4fRZr70AFEI6pWoBNCr0ASviYm0Kzte6+ey8IBfqJQnL6842TNNmyldEM7+s1myvKpT3yDRR15qPQnWaKPF3cSz+I5Yz8U8Ty2TyZqujAaITX2GM0mi2DI+8TMdATSTczLoEiJSppXzs4E9ywqnztWKutWG/BdNHRpTizmbqZpPJOmOisXnTBkXEruedyko2x1SgKxvOuOG5UQsF2O+HXAgucXRGhIk5Tv4reKMBHd8dk8Xb9/Ik5BOfXLRgl+zC3sx2rrgLSI2POCI3cDTq6MLthVNVKuoniTbJZXh4nVpFwuQb+hxFQqrus86yQ3cRB9kkRpJBWCeTKW6mMsJ3rvkFKve+n7YqLdiN42Hc3cH1LAmZGrVudYR/8YvNhkk7KdMDJxaHQm2apu2bzwaWld74dMw2JobHk05zjNguub6iizlURME2QUuRj45EWcV3t/m8BXp0PKQRoZpSV887FjBKpeauagj0/6a5p9YXDYcifkyqoeBC+zUddhvMWD8XyfJ6CrRE48HYvPYeFSAa2gH6kkF1P9nI13aM+HDm202Uf35N1zBfWbVRMO83zcVYjWVvg+L4qzI3oNA/Uv6FttEqdqknH8wKGSYuRyto3m5CTBLBvFwpAiGCsfvLnBswFT/mqT4kzuUHM3JlnI3wGUQe1KM3jq6FvpRtHQ5aIBd7buJRYxsoAmR3Bxqe8EvbcvlVyUrgVZ4oI5QAlWuJ9MldRjdP+iBh/QAKCIf4TxT77xUL1KIyyxdg7VRv7lmZJhBuc+1NuA1IXF7Z4Ir2JiRgekKaZy27mKE0FV74SWJJDKD8fLsx8VxKgtx1lVev3hWf6pQFOp0NbJUBH+ZvHwcXhXa4AHRgldN9tf2I+GuWEDMMFm7U9Fr+deX6YslzNI+3IK8kkdaZd2mjQQyuF5FIItIhshxAbA5FoI7yfI9HT94598TAYOBT/pmw3clqHNwm0GyA/8OBuNVyI+JkXQqEbdmOYeeinsu+SxIqoLtceiX3Pqjv28Sp0njLD45GKlZljUUTzWZqH4le9WnqxBc4uKUip2b8L8VaTWskXSJ4m/3NHabpamoJ9Ktg3V3EaJT9YNpVzZfzBy14TbgoVDWCfQqS/3yVf9OKXK78VhiXJoNpyAftnPr/d5VrXnCBSsy3Hf1n3k9uTdqRZwNcwgKhhEMiMcWXM8lgjGfkfiYFnkelx0zWVwvBmGg57q1e7uR7C3Qnv/SPwxfbxAtP22fK9VWvVp0EYj8hf1U6RdIFuj5xalAHxouvGxJGz4R6kLubTSuUvBnoUiV5f/NPOEzywAApQlCd2x4iD/jvsHmGbWxphwhMKAXxNKAz1pbmx/GIgWzmnx1L+TqelQlrKPXNU+XPenSgJQ3S9Hx3z2f6WQgSI2WUOLVfMit+qN0uFAcR93lXKOVb+a0nk7VpUBga68HrwlgpqU/3qSAm/dE2FLRAl/gYEdm0PPrquuI9yHANitRVY7/aulq3s7NWY1ZaBLpBhpdsFSlxlmle4TcyJxS6O+KvAbq4VKf75VFBdah6tUveQbDJ0iu0PPpT+9ARLd5xoE+0yJK06MYTw== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: a88e4f9f-b850-41cb-fc6b-08dd20e923d3 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:25.8084 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Be1s3TXD8DHsN1WwDpXUGwlxXypVEnU0wxJnx0Pq/Hc6knkdVTAJeei+ka1YpnmeCmiNBBN6zAfl5/ZNoi2B9w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16924 From: Stefan Herbrechtsmeier Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 3d31a8d4a..d2a30c18f 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1327,7 +1327,7 @@ class FetchData(object): elif checksum_plain_name in self.parm: checksum_expected = self.parm[checksum_plain_name] checksum_name = checksum_plain_name - elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs", "gomod"]: + elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs", "gomod", "npm"]: checksum_expected = None else: checksum_expected = d.getVarFlag("SRC_URI", checksum_name) From patchwork Fri Dec 20 11:25:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D9C4E77194 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (EUR05-DB8-obe.outbound.protection.outlook.com [40.107.20.118]) by mx.groups.io with SMTP id smtpd.web11.149744.1734693993154756588 for ; Fri, 20 Dec 2024 03:26:34 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=e4ZAy5mD; spf=pass (domain: weidmueller.com, ip: 40.107.20.118, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hfzrwhVg3kPznq0QHVe40PF35dUPLpTWtTqvM2QUat51zX5qsJ7531AqK4j1TxsqonwSmAJPPLy+bhaf7zfkkgLNjxdFzchPqWGfPubJUo6mc1puwwnpmMyEF7+bckmP6YNqvMJZc2ERZoB9XMDvoxXdQliKKGMzYvj87d9kMfY1EmX/hL3/AkIdkHihOmbpDHwM9eAUgQ3zNCO96Qjrje0ljnyEIxAG1OR77h3fAcbgs4a9I7bMvxtIuXHBAByphXYQroG7jkAZmNZ1quURgae7NXYX67nHhqC6NgcKAMX6mo2eIGj4u0W8E/2hro8lGv19LIkRSoMCCtX854pHow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Mh/WNWuuyisgUp+5E56nzUtPFSpNFcDOw4PpKhURkpY=; b=Wwe35Y2OaxIBOLjkYbJ2AB+V7feXEYIZXyOa1eYrYM7PGZe3KTV6Z9zw1RQbZlnM3Rd59GGzU2av5FovwJM6CJ3mGR4lDPFErilts/nCn/jL80ajqoP3q7eMujyCBvrFqJLCmeKL4DLSs67z4/dWgpEnF+qrm+w9qYqDffpyjCA1XJl/ZkZhvzu1rbQ0Azd5UXWJkAG/jMrzqtXOx1hGY7drG58bl/UT/18/1dnCbPUUtH+BaaZGDULM4um8LLoQTVcO0YxR6wcAPDGBUqBktt8U6NivHpMI+kgt6QZ46+NVWXCu0jtm5+Ldh8yjA8Lr2i/zSlFcX55uzcLzenzJQA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mh/WNWuuyisgUp+5E56nzUtPFSpNFcDOw4PpKhURkpY=; b=e4ZAy5mDyrqa0GzLtp+Eq12tgS8B9e3MCQLOLQxz7fjIaS8JqZ51JSz7NV7bMpoQXPybptSu3CI28jRKy8ooRutNthY5d3n1/iMAXrFuT9Aow9tYXpbKLQIAY8jfFn1i43kvl6mUbOFH6uD99VWeVRqtxeo6/k6nrAFpOAQVJhAOVUtE3yuEdi992d+sg84Sg+Ikl93gGL/lxyvSPxM5ZlGmBWJFpFlJmBwrGHtzCtoU+vY6p3qwNSHDty9eP6oyiFESCAhp6LNvQsFtwZK1oKZclaQM5mFyBJPgDsH3ffxe9aboRpdpiyLXLspguYV07fUL2pgh2aFQRtDIYS282Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:26 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:26 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 06/21] fetch2: introduce common package manager metadata Date: Fri, 20 Dec 2024 12:25:57 +0100 Message-ID: <20241220112613.22647-7-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: bd213d8c-1309-482a-0af5-08dd20e92403 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: LUNC65/KAWfYZhDMuBlokxxGgjvh3N2lFnc+6WTBTIf5N941PtqaOAxJHSFhyKYE6RcilMyg5UHt0XsqttxWnhy6OiCyK1D3TiRF+kXS5KGERnmD9nOwy/JNwNFV/dn9vFlzYGSzkVAIA6tnssjFrmRgBHMSXHIxkCKYTkitdQG+1I8lKBqNKf1MLEPRTkF7agwniBNCQ8HxFBEr9RhxZr+rNVUAVJpQKdWUGivIXyB5eUnmKIE5Em++tQsY1U5I5UPF37tT792XeaXfj8+m6iO5a8GDdmdlzSc23f5wZU8t5W2AQ8Ioj+BeN6oywNynoeQBdj/adE+YLSxUvE1OK/el7bPJlaztVfAI0B7kKN+C/bG79kuit8wgIN23E9rcMBFegqZzBVa/h8Wd+cpx45FDCoSzNflir6WAlKsJ20B0ifMpIL/7Awi0kL30NRdxrzbIQ1XiUhupeMObDuRp+EfjkBrxPwWxoWN4u8SXjvtcINwqLxur5cXI/Cql8MWV2vdymvWGs2LgGG6zZap1og3/wpcGQxX3pyCGGdvteBADgQj83/r1189pS5RL+eufUt35C+5r001FDMdMVJBktZQzgF/bw3ydjJlSppqjtq1co/kfIkYlP5oZxukyUZLKNR2RgHrRDdH2VxwlWAiG0urmJZe3pUb/9fhSclT5uUZCLxLyjbTO676H0lw30O9tYBu3P7rA3s+AWqXTlh3AVBJjPWX7BFLDCzUq3dAKtKQbuL5yXBBE59zXiRnIM1ejZ87I1VK2D3jPigScOvXvEoeLXi7nRB0N/9O/6iFApfs/FShPVQx59VPs5yI3QZ6dtvmXppTWNa/lorqQDrXqxKzPJ5626YcxgwtgFUAbIpL0muf6/QPIJETOyXElnKfYWv52HvzwtON9/DSIijqSj8XECtV7b/mNxJ/wutPVRVm3zTNBF0jIpGJHGR65GlopWEzn7lWpVgMDAd/4J2k9lb2WXuH+tmjj8190Sw25ibZYMAHA0bAPghUcVoiaw8rSXLyxIdjQf3OJ6cfeiXct6HZnErOpbfN+OkJx93+H8rEQv6Xd2zj9X7wIst1ubwtukxBjapgSkF01QgtPTLJaAMISsw1eR7Sg+fW8l3XfkUA+Vy/onm2FC+R95ff68iYiVCFFAVA4yBeVhPnoIds6RXOySjKp0FoGfMPibeUKqMCZKqERpX4IZH54Xocd/mLo3j0iXaOiLzBiipJylywWMNJ0o1DTdjyCy6InaxJ+npNPmbDPr38C8SAaaZDBIRobXgDwQu5r2z1cHFe8yY6dIqajr8LwleFY2szZ0K3e/Ru73k5EQYtiwoQZ9CLk3SgG9gD9XLSEqbCVDcQA339jsk+XKTzGA15PvvtADPJ8ze36ShT7k7E39kh1OAsIhXKmYW7mdXanml30Q29eUoMvoujOFjlMaNxHk9ueebHTExau9UHVqMRRb9zOjYuDhBym X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: rj3OcZS8vE4ntN9fk7srU8c6YXIZzVMM6VvYl5J2LcF/LBr3tLmdbJzh/+lDopqR3P5QMy2FohwCrdbw78wZLtWKN1RUmGnkAXOeyi5hm4SLv3OiY6tvoK5L9f/MFED92Hy2z7vko4FZv1L66iPDaab+wt3O89VmQJehsUP+AqWZgsp/2XK95hGIDklCNtUmGzkMP3quiauCSKrovuP94soIEEqHGOygm3hNs7s/WtI0FQ09juPGj9zLuUc8+SjeFxT8u5iINBpKHOIQwo30DRWf8m1X3uVgzUbJbSR/5s91ThcKthbkH+tJhbZEf/wkZE2gVeRUcl9ovVpF6A7lkYs4Mudyqnrr0ve+yzOa52Ur/Xe8gCdugdM7eAYguSMKYKWyBi69yT6NzbHDHM9xvXCSBCurYggSjeZbY4H9FMsdzhuqq2DuOXpxPVphxU+H3pDMez9aqXi27GEi7iCZfM4DI25SeCLrM+VfSO7hSCL8ju62IB64KIYSJCOCeQ3RJNYzd6wgdd6kMJcUBZ/mYlmXq2/Uu6UkX8qsr5Lci2eRRrsIVdukDVEaZ23RevQ6eFmnr7wVK3RWt8Gf5d+RcHoncdVJpdTta/jYmH8SUA6Nc11i5+zYdd1ajW0ee/XDrVuuR3k3c/qGa0LYYYjs1aJhka+FxbfirPB04dArma/JkIRvn6E52w7RXPUNdl9QyKPsYZ5uW50nAUfZuU23XjjfYA/L2WHg+fMRVHIQ0O+D5rcmy+H3zWS5JLOueGqfvq03NJ8Ny/cyMShWbM1btE0F1gT6fnrNzeAKotTPp/nhSQ8m5FR+pPhZVmpcDvElVaEbshMgo454fqxQxAiDCJ3CyA6C+6CeJLu+sOBpTuLHVpHgxsdnmd2VEzOm+Uu84kaKrP5PDfzt4NSFYQfVI63R8gSTblCHfj1oeiTVG6CrxGdTIuWrA2Vu/jXSOXOqmbcCb3JobczVBXCnXY4gdxAFGM4l3fsBINJU/PNAcncUTvJvB6Oa9vhl9jhujwhpywtQf4X5BcvraUQJrVFM5bAv7q57gJmrqrb8BDR0Vl5SyBCCzGWG8VEgpM0ip689Qh2mPkKnACsJchPNqJTn1hxCnWzthYAGN/Cl33Due3PFCOAPOy55W/N8Jt6dEwtHbWYi0s8trXAbzBTdaTxT1XeJOhMQz9CVLyfiUvsf0IAGLpMa67EwHJ4ib/V67+7t3GeIHtPez/U9Bep2aGin4xWzWIyLjEgwRoean2D+t9I9IClASh9eLOOmQ/toALsNjRQGilSMeUAR7RXFt7tMZoWqbK5mAcT9IbHgpd+01kUgypzWCE7/RvE9b1csPHzuSVslekvX2eIPKxX1e3PZKnADi1KRCcrZ+QyNND3Ngys5j8pROdkdPSnEAgjuV4+oWEFdzmo2aKiZwUTDadgzcSI+eKYpLB2j4a0LKU1am3o19fFkLIQ6ARkCdLkP8ZvxQ9IdrC7hnvZ1kgRhSQJS/ZGlU6fhmFput1GFn/B3w/lk6bKH9G+h3ILJAKgeDfsL+kcJPZV5Po75MU/gSjR5fJTehPYt7x7UjbHRqxKwVNGW7xTeFyiekXCMnookIvVS6hyx1xsY/fOAW4tco6u8UA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: bd213d8c-1309-482a-0af5-08dd20e92403 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:26.1271 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: N5HoyF+y396nHoWh8bZv6kfJdraYR2yZ+Ci7pM6qry1+u4nPkH59yhur6Mlp6ueLHiLSEpam5UFTPRPwdvLlcQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16925 From: Stefan Herbrechtsmeier Downloads from package manager repositories are identified via registry, name, and version. The fetchers use individual styles to define the download metadata: npm://;package=;version= crate://// GO_MOD_PROXY = “” gomod://;version= gomodgit://;version=;repo= The name and version are important for the SBOM to add usable name, version, and CPE to the SBOM entries for the downloaded dependencies. Introduce a common style and check the existence of the parameters: ://;dn=;dv= The style clearly separates the metadata and supports slashes and @ in the name. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index d2a30c18f..4b7c01d6a 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1356,6 +1356,12 @@ class FetchData(object): if hasattr(self.method, "urldata_init"): self.method.urldata_init(self, d) + if self.method.require_download_metadata(): + if "dn" not in self.parm: + raise MissingParameterError("dn", self.url) + if "dv" not in self.parm: + raise MissingParameterError("dv", self.url) + for checksum_id in CHECKSUM_LIST: configure_checksum(checksum_id) @@ -1711,6 +1717,12 @@ class FetchMethod(object): """ return [] + def require_download_metadata(self): + """ + The fetcher requires download name (dn) und version (dv) parameter. + """ + return False + class DummyUnpackTracer(object): """ From patchwork Fri Dec 20 11:25:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FCCAE7719B for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.106]) by mx.groups.io with SMTP id smtpd.web11.149743.1734693991344736713 for ; Fri, 20 Dec 2024 03:26:34 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=i8++gokS; spf=pass (domain: weidmueller.com, ip: 40.107.22.106, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ZbUYSy3SGP2rkL97IuKesJk4ibvNnDQwQbjRDH4XvLZMVACh9a4Uqsi6W1rlPFxgzNv0qcvESxlR+hgCRJs+hZWbKvX8pu9mxBIeTllvPiiAogHJJ+CPPjwXLe3izQZMm0VdfrhDbxbpqv3ShgCTpzd2MdWAhVLKo3clwSTPPFVDmoeNIc+hCIu7qZ+2YItA/DhiXoTqpoi1VViwwHu7sodaGwozxzXlrGRQGQmnb21j0+TJSqx/ehoMhhVInsXOsgrIZBO4Zci/hjKO+qjuT9KMW+kX0Won/LJx/HWNwFGdJa9VlP7+llOl7GO2VUCijhKGZbY6YYYvxwOnA3TxKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dWGmKD9aRvjtx7U0RGt1riNrMstd+3AfVtEM6WR3xqc=; b=QII875k3ZE9Gx1DC3ni2vVgx31bOhEiiWTAMEdmDeFhxPQvRbZ5JieMyMR5wk1MXiaG4ors9TrrUR4V2QWADouTXA8mT99a1iZaloNpGIndn/U4nUNf7RiiNNyaoUnvyOiK8cjFPsU3L0degmcHT6ICMuXPM6YQ6xsCbc0ABGOU1LnOUEQlxhyQSFJtQ7rBw49wFTkgaqfM++V79NQSJz/g6A0ZoBIhDwl7ix+R27srRjC17Y/tSR6b2JTuHsYjTDQiuO7uyHCkIXN04K8TJtMGed/OKHy+dVQ1B63d34IRGx2B2aWMGT9MRMcRkpQmUWpjdHIoiJMlofqTZsBSZKw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dWGmKD9aRvjtx7U0RGt1riNrMstd+3AfVtEM6WR3xqc=; b=i8++gokSTJR4A5kb7f2X5mTPGAZDUrmqs6K3oZ+txWcr1ZnV0YsgPiEVQr0aOIzuodcoCR1y1nAwcdtjVpGPtBP4RRgQXTf3VYLQFG79Y32M5k5lw7jzEv1kuWLsv+SvDH4Fn9v31XNrEJYL+9XihcN4Flb5un4lO7rm4HsXXnH/gKmBBxYayzYmDGxa167rSkNEEX4I1o1yNxxiOBv8KVimSeUKAUtXny5D9FiJlmaqHv6EQRE5U8lGw8eQ/55pXmWt1xCWLTC/h+ALUze+wprf9N3dFJDdAuj+6syEuQKletuAaCDqdZEXNby3Lsibtr3ESD9PetHDDHWMcXNBNQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:26 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:26 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 07/21] fetch2: add unpack support for npm archives Date: Fri, 20 Dec 2024 12:25:58 +0100 Message-ID: <20241220112613.22647-8-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: ea2a2c58-e7b7-45da-a7f7-08dd20e92432 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: DgNLjK/nHkP3652lmCOOczhqciHVF9OJxpQ8zi13/GoVuWyxi+EMkRrlHS0aMBFjpvymHZ4xhBOmcW3iGgsjcoK+AhqmzP9nrsLiU1dELEUutWj1RxUUzUenVgexREfDXck33YlRvpDUCEfwRFUOP14+K8sDs9V6AVMOvn5gAqDQh89FR+7W4+CivUYtH0y2/GX2zZ/vwr1iVQMcdKTDJ6oeMOmLIAJTIY0RWl/lVgYmRs2Ub6NGNogyGxLLY/kqZN4ZASiXR4whb3Wp7bJXb4EhOsCI4T8dqPM7HZH7sdRQ0JIra0NqEFyiZk0OS2vkXMGTH80vUiZ2nhe2gkHluiF7NtvzDTpE0pk8MizKChBy/IkjZoBtvAOKFy4mZJkop9AYsEoPxrOGn3/YXMK7WMd3jjQIiGrMMiYZMpCnHlr9DtEPWM4f6oQc22DPKxgLo7NtXaKx04GWAIxXu6ezYbsxZXqZ7rUWmQJ76FqIUTD8rOeCcRU7Jk93XZvIksKs4wkXAINkuf60447qLZJDDuQmUPTlf9LdRdMC8HHDxReggswDpp9dAlMAZisx09Ls+VoeV1VsRk0gSeGSN3n4cjOaOsBc/X/EZ3pzYp+d911IDm4r7/yExvei1JSIwt9Whv9Gk58bl3KvdyimAI03a5fqg5fE9evnd3Q0wbJRdxeHSPMW2Gyj2L06/+Z/E3VWNQb9jQmWriQFjpo/g/fYgj79V8pJ+21/4MCCs5Eu05Bp5B8d5fY7RENSpDUwuZX85zCNcLjfwxqePb577EdK9GTnhPKEP8iJWTLhZS2ks2OzB3qGfxHprUbWAHhQlRd9Php6Pwia8OTRPen5RG1JCtu1+6Te7xzvi8Z8S8DKU7wsiHmoICZykg4lveir0s+ArBT0uGL5dnWdwILpmn66anSAE17FA6kBEzzebHMh+N8SI4VU+Ntl6DgtdbwnCk7ZP6o26zHOEpwY1OcY1B7nCChxJ/t5N5W9ge6L2gbaLNJjVBFTf4XJavS03jTcBxYk5SEfe1/RjTDfn3Q1BSfFmcJFEYXo7bsCzw3yupOtW2gLR+ENUOLFpbaawyeXRKJeMQvOmfcHHvbhTyI+u7Y9va4z1X4ZcVPVlsvDsDFDOFAyMwBxtaN4fo1//tdXwlTDk8S6t9cxUO44e1kSPVgRQHyVWPUme03Cdy+o/LpJ4xkBxi0ffDDPm+pum5MfYrsY+lXBUEgkcWFHQPcX3Dp+0hVdxd0AiD+ep5Yr32R9+uMxqog8ZFJUwnyjZNhFqc3fbvKY2Nr03iVjvMsxAia6sUDeSErDm7rskQONxyzjPYyFRSQQM5eaa3IjDTLXnbUfxjuxwNhzrC1AOmX5UefSZfAI969a1z2kUu7/0cyR/zbrSAAHoWDpP2iDHMfMR1Y7oa1usz1vQJRWGSZuzI941eikrIxgQlS6OTBX/iT6OA8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ApdbZLG/3a8JdqJmPEUbvLt47jH34yjs2HzT5sOIHCt49rrm42tXK06Ns8uaSesIfuRJDbrMrI7tkmjRFw8pXTF9Xc1JZp6Fc4Db8s0HcO2kMsV9gagL7RiXkRqiKD0iPXXZlcwxGdotwoD9dZUdR+URxoXKzVtQdgp5+S+HBZ0GBmsATZUZFBSy3pbwPJCl51B+WhxQZcsWo9uwjzXzyawhM7lmnTJGS08UlZAe7/MWjYX4hyI3oW/Fej51Gc8e8ds+3Kn61tVxNTxezmPjCZN7vYjq/Hb+vrA5IzwsR7R3yKcJWg9uBhWJ6HAgBWKJnVTTyC+JGgqYgiieGY6y9qappkpQaDD32JwKpX+Waz6j7wmmBPvp5ABjbiDMzzjU8IZDSUejOENUyVv5Hukr95VQrouZS9Ix6yYoYs83vadPHGWaifsgZ9ntAxUFIi+/1FI2xs72OlTGrolKrk24k18uMCxOwnZwUHc9ei75iuciCyg98uIQcXpw2DIavCIwCcKDJzw+g00dglANpCdbw6FV6IF9i/nTm0vAfnZQlMgaN+RozN8myD1piqMXWo4Q5NQ1mngBTcSDQAYQJX84SwegGZf47O5WCs8S8Hve1hcXOXikHmMwrLfityAmwghKK3PskKSlzXH8GC2Dn5AOIDcoydwn3smVrltrQhZi5rObLbIOCKC3tAaTpYLu99gR1vRKr1El45N89Os6aCWyyzrJFFf4LI7whKNmuv3Ie8j/q4vFgbQd5nygGl1FilF0JOwjxi9EUQdeFemS0vxN27FF9QrWBO99x2n5e0iMo2m+PkMy8vHUxE/Qp93wOt6EFaZ/IgZou7GO/n78crWFrrdqsRj4DKr7x7QZLwr8a6T80dO2UrrYXkJTcXNhiPGLuL+3wBFpQR7Cc+cmXFS/VXZV9FT2Aluql/KTlwQI83roOmgYAx3PibEcdxlSaf26RNSdcKLK/eD8T4p6QKzo3kLd3EdFkGYJ8n9eIW22gzo3yy8QLH5LQIL6cwdTUUU3f5tCyoAmL4cNsr9qYqSbagYJmXYdLsAzhzoj9nfSq5jc8xu1pl4254YY1txFOGHl3YOMzkYWGss7XLDq4VY4WcFicRquADFO5BHGDNQ00IR7+6nBaUNawbGzwXZtKmVkLY6VStCkkWUjOegzXO73O6vpjesidenw8rKz5czHyEGuths13b6fJQur0J8yNtz94xEKemS59U4epE44jH4LGBbiUMmgKGS0Y0P4NwPOoNjkQj57p4ApLrqVOId9LwGA6o3ziRzirMUhkPCZSUx2nNet0ExC3Y0tNVyLWzUY1YhR39m8E4o4X7gUOwxapbQYbqSXQC1eiNRoHP9E/M386Qp8IseRcQ3llGKc/nGgwKS7vBxj8cWehbRlV90k5cgi4tOsxBxXf3cfrk6e5VUywAplyXvre1HtEV4VIAQqPq4inJ/WppG3mNQywykmEx8E5Xhj+XrZxAfuHI8Z1igXQOlBY+5bOCLzHWk4Kpt3AHcVUmKLrQ+MZ9A23vJJm2zb22ZEDFE4NpoEdetYV7dOPh1UVipzl8/qWaWswHoycd/vyq64UmYvuEgQWBxsdh4m7Nh3w2k9GhzEcFeabexAJg== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: ea2a2c58-e7b7-45da-a7f7-08dd20e92432 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:26.4386 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hYlMpGezDOQ1zRvyat3zWaqZFJSJUAvJj4K0ODrK4ASBs2i0uTq8ub1KbmEFdYv4OcMhmVYC8I7224jz6I8ndQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16927 From: Stefan Herbrechtsmeier Add unpack support for npm archives with unusual member ordering and disable warnings for unknown extended header keywords. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 4b7c01d6a..7d8f71b20 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1535,6 +1535,7 @@ class FetchMethod(object): if unpack: tar_cmd = 'tar --extract --no-same-owner' + tar_cmd += ' --delay-directory-restore --warning=no-unknown-keyword' if 'striplevel' in urldata.parm: tar_cmd += ' --strip-components=%s' % urldata.parm['striplevel'] if file.endswith('.tar'): From patchwork Fri Dec 20 11:25:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3096E7719A for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (EUR05-DB8-obe.outbound.protection.outlook.com [40.107.20.118]) by mx.groups.io with SMTP id smtpd.web11.149744.1734693993154756588 for ; Fri, 20 Dec 2024 03:26:34 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=CpvTifc3; spf=pass (domain: weidmueller.com, ip: 40.107.20.118, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fXkMpLdg8sYEVgSYUY3vXQbBnTIDMuw/DytaEbXWGcFY8Xe3ep5I7jkViprovS0V0dDn2gFn3qnwtcFefg3gw24TCAgsadebAronVWjuj/ZMbUrj2+x1SfrEbhBMl4tTKBUG20hsWjVLvZps8H4o8udAR28P7TdTKd/e9tFgeoqI9gpbDv8w6jlB/JRr7AQKdT4Wqxn56kCokgXjKbaKde79V47w6dedUQByCa7hfX2eSzjbAJyFXm/HE0uQgnCeeMrNIjJrM/rKPVUFxXIucisz0qT0Xafklt+w30F0IIaa6RvilGKfH42kL4CaLL23g/tJjOh4dtPPshx0vKeulA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eBuSYNeLoQQrMRmg2THzhSs0jVpnuplbbs/jpD8Dqec=; b=mjEYMeD4TotwVIWAD6JA/cDAAAJMf91i7C6SEPO5zQCm6/xtCDzgN0Zvbfs0JGzCIYV8RGXhYb7f03V74FmCMqE6AxlSj3iZ6mb5Qu7gxqOYjghauDl9gVLZyDR0WDMN8uevWScyg2gHwaT+uGLqBXXMnyJbNyHeOmnLuJqkmQpE1YKNAqZm+lriz9onnUd4r8JekqUKE6856zkt5MfsmGQxq694AXpdeYKhicYCGaPy8BH7ljbJLpgDg9OqiJmB+3pRu84b19hhpXT3uicUo8E5clP3/ra8SYxxi5jOUJM1zCQWjFCfD9Ke0dMsF5IVSO5H4DYQWiF3NmKZ7EcMww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eBuSYNeLoQQrMRmg2THzhSs0jVpnuplbbs/jpD8Dqec=; b=CpvTifc3kK+E3kyeW3LHpGm+gFF7CmPXoE+njJv2I3636Oq1446ugICdjT/MT3xT1j9n67CJ3ZSRcJEgD6YrlF/LJ7wwY+FCJJUGteD/Drkwl2SzmFzN3BIvS77VptZUZG+Ty3NTc/8zd7qaRZ8gkm8B2SbkJlrXawdXQZHFRkCJWII+ktbmeuf2uk+lmNpcEAeJhhUQ+VXpunYVxV6J82vV/6FrG5toTMDMAUpnCSGWchvv1faFnYiWrU9TsUJEsGwcLlwbpwEj9mGOcVT01fWFTwzDAYALEqJ38uembhVLdz3CdBCXvdnx52/eokgu3ErjAcptprbm1/yM/r/n0w== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:27 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:27 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 08/21] utils: add Go mod h1 checksum support Date: Fri, 20 Dec 2024 12:25:59 +0100 Message-ID: <20241220112613.22647-9-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: a039e0fa-2f3e-4292-75db-08dd20e92462 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: Do62e+HtasyjBSg6X2jQN4zk4TdqCDae99QmxOHSuWXDwV9KIG79qHIqT5ckrMFeZ4RmkPYy6cuieK50rsognzosbSJeAD57Q1fHhItydRERSuEfKYgZnmBUJ5nWTtf1AvDjbJVpwhWCaSxuEzgf/NCP9LyrbMA2NCotz06wBmri7PtjpSJ9qu7g989X7HglWQsEUMD6OXaKzXJCMKlOGnDLiPZ+I7IzY3LJk9fEtvCqDHfv3HpmXhQU4HeqWbxiKB6bp3MVvkjkWdfcQ3BM0gG2k87kmZbWihKraPZhPGQUKjeHFwlwY+JqhQdngbNKfxyRIDb/oJFMPafWZKjEr9I/fH6cIi/N9A5EOSNgsZLf9tMkt2OnZ97eRFUNI2S417iWehHLLx/AbW+yne7Pn3ajvy4elzmV0BnxHQGffRGpKBOfsrAT4MgrAQ2xK0ohAKZ7i8j8b63mEUrtoJbxolubeqZHYbY8G8CZOd5zFBFUd8ep1FgtVl7eMN9BKHwwQfML8/6+ADkFdqo3rr78ha0uEZH58utEQMIWmuwAVVrpcta1KdblIq+wzKkIb5Sbmnmhl4tXs47g8JFlCX0Q1UcaXn/zmKqjxNDGDtAKioLKulOBjbNpuKVtQi/o2E7r1rXUY+cwOy1X9a8U5FhPxpci48kroyLUZDc3BOpyDP/NA+i/BL8t5sjM1GouCxWOrXl58OmOgkL2ulHX6RMSe+byOh8a5y4F3Gz0Ko47mGEqZyCVrQvuV18O3y3CkbxRv8NqUz+ipCeMbKcYg4vg0kP1I7nPNK5bbzoQkKR305DDXOkxoJAS7ZPwr6SwIe79/Ml7KM+R8R/T4YybJyYLn0DJkepq8SuOKdahDuO0sQpu+DYvYkzf35KlFwwkwG7dqn24t2FgmNnV+VJaPueNOy0nqKh2eAnle2FbJRKAqDv3AuKdVOOF/Y1FfnPGgU2IwVhYBjkcGqdc/jR5E7+W5F8aZ2/dTVQNKFHA+k2FOSxQAOFIxVIZLgMUT66QtmQbXCgfYQ/NH3R3kvSTuiwPLwo5fBoeoTg46xiZysd7pYgJ1O8C3j4caC5A0aPZHvQrDLo06JmbLwJ5VhFbuptQJ16OMHo0kLksImrdUL8SiQFZvRHOHRkcydqS1A6+TPwu/9ZeyO21mbyNa66gSPcKWuuhYBVepBoEWGBQML6xlSH9DCMPKxjZuAhNm/lQWr3XGpstyF+nLBKmqOrD6JDr4et9zjX2O6Un+8Z3IjTsYywHsuuvXAZ9B18YyKy8591oTLt1blealg2PE0SVlYPsa9umV2hpFs++Tyfr9cJCerMRNCpjYfnzFlAI8JCETrdk1LE8ZgEqYAGuEHmQfDAy6accnem0QaZ8sl0ZVYUDP5IK9JVhkYCTQGDAdtrCSTiTj0IJeEMKzF3R4sAGfkB6CkEwSxen6MFe0hUL0ZWiytSmGMRTSpunUL4SpUKAJEIe X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: S1zkspHIswR7MYs1eEWWGpVcUDYdR3aChUTzjLZvajE4gTRvuKOTwrYwWSuiShd7/NkaAy6hj6RKmAMsrbBNft8xILmsNSeXZT9ojho2tR4IxGjdaA/tLQyFE8MYmylAt3dZO9mV7pWNpSmTjFqn7nYXl2Cn3jV9qL5XjbOh1DiMVUUVsJkUZ2y4o8voVMVqd2/UyTCKQK8nm5JCOGDzvNVkuf2i0IPMwTeOaBZ8qcmZSosL8Ks9ewpwIVW7C0kGYPvkqQyB4FiwLqwuaKmdPGdc1QbIOlbpvHqxJxMR8Q1KUGjHlahnDs0YII4tQfFR9nUdTHXedCPwRGE6h/2yULBc/hcPLkSQJTEyBg4EYszyzMUYicWRsxaGZKAJiG9Cl0J0losK2GMg1PjowHw8bs25NJCZtgV/3urce/fhd/NXPunWRM3C0qcF0m3Dm75o/oZ3cIpuX8Fmz58uhQ9pZFQ2p3/3VAwDLB6BaexxPpBZHIjytNybU2+Yx3jWdDCKH3AICP92XDMS3gksVvarHekYY0v5OMZeBGLAOFtDDq+FbekH5NTZUoOUDxkYFHUu1ACKDWQ1TO/ZvLNyowjWonDdb5tOVsoKkEi7RstQDpl7Sr8KOdMoGTlWDk71XZr0uQjaHI2s8+YukfMRiU92549adAiHV4y3An/8qxphMTtWsIWHBbfEn/4yfz/qQdmF6Oo+hKovMWs8GlibnoEq3/3qDf5M+2K/Smr949jeAZJp2Xzrv746pXjCLB11lLw2WRgISfir3wvOcbtaaGF5g2Kxok1n+mvAl/ZIscsoN5aygO61bMT5JqZPym6FnAbk6gLOX+fJkfScP1R44UToQZxSyoQdxz5T9aDPl0s6/sszngsLtREzt3L+JLHAJd8lNVFiA1P9D7B916EP+47VBSyL01gJaIO+/yiDnTVFalrTWESdlAsOjmeCbuW2Nfsd05B7lOwfYAgMLj7/V1wItNX5Hf/A47wtt7BOtnuMZeihMFFETR76xMFE0ZivEu3r8FumY5A5/n7NlXGwyzf2zCHvGR8u4+GDcSiZVFreO2G2IICkzyTU7HpgzKlD2rNX/IW1VYWBvYiK0EwZnjS6i3sDid/hZXejArW0WYO9oTJ16Hk0q6mjEfqAOn3DNDSCNrK+gDZXNVcKqLOxJeVRO3F9HAeX/oni1FtMmlcBbQA6uQkppj28jC3qJkl6KKtTyYBzEOAaJI2QecgImkksxLuN8iWDWzoqUH3Gz5WXeKIbxPDES/VNcA0LFeGeKBMTDHevhRWNQXTCR9mQ6VDh97YmavdakzrFIk9KCQXkMIaovvmorShr8hQqZBzQ5xx3o80JMu+/kBml2Ft/owy0kUm9b9mb7DgyDyL3YwpKQ8WxQsZDXHhIPXJAWTShDuxHScZOm+LSMU4do15MRPPttEYDjyukz1S+i1nj9C7FZttGZQHH064rwDPU/a7PznqqtmmwiKhig6pMQdqSUoUXxdu8lr/4HKSKWLujeVYSsJcJ7GaXhy2CV4JzBiQ/4FiEoTvc18MU9K+UTy21bwHqSKS+fWU4IsVh23jPKDjP8csxgNzdCe2vfzAyOmKmwBeJux+DwXjVf1tz+PQoQwHHsA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: a039e0fa-2f3e-4292-75db-08dd20e92462 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:26.7501 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JgNXPaF8llY0iuYvdQuACmEMSuMXD5fxLEQI9g33QR6qvr5d6fH9j3TUkWXh/jN9vKXyERelBGNhGL9vRMuk3A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16928 From: Stefan Herbrechtsmeier Add support for the Go mod h1 hash. The hash is based on the Go dirhash package. The package defines hashes over directory trees and is uses for Go mod files and zip archives. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 2 +- lib/bb/utils.py | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 7d8f71b20..0c2d6d73e 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -34,7 +34,7 @@ _revisions_cache = bb.checksum.RevisionsCache() logger = logging.getLogger("BitBake.Fetcher") -CHECKSUM_LIST = [ "md5", "sha256", "sha1", "sha384", "sha512" ] +CHECKSUM_LIST = [ "h1", "md5", "sha256", "sha1", "sha384", "sha512" ] SHOWN_CHECKSUM_LIST = ["sha256"] class BBFetchException(Exception): diff --git a/lib/bb/utils.py b/lib/bb/utils.py index e722f9113..131766e33 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -585,6 +585,31 @@ def sha512_file(filename): import hashlib return _hasher(hashlib.sha512(), filename) +def h1_file(filename): + """ + Return the hex string representation of the Go mod h1 checksum of the + filename. The Go mod h1 checksum uses the Go dirhash package. The package + defines hashes over directory trees and is used by go mod for mod files and + zip archives. + """ + import hashlib + import zipfile + + lines = [] + if zipfile.is_zipfile(filename): + with zipfile.ZipFile(filename) as archive: + for fn in sorted(archive.namelist()): + method = hashlib.sha256() + method.update(archive.read(fn)) + hash = method.hexdigest() + lines.append("%s %s\n" % (hash, fn)) + else: + hash = _hasher(hashlib.sha256(), filename) + lines.append("%s go.mod\n" % hash) + method = hashlib.sha256() + method.update("".join(lines).encode('utf-8')) + return method.hexdigest() + def preserved_envvars_exported(): """Variables which are taken from the environment and placed in and exported from the metadata""" From patchwork Fri Dec 20 11:26:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 552BAE7718F for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.106]) by mx.groups.io with SMTP id smtpd.web11.149743.1734693991344736713 for ; Fri, 20 Dec 2024 03:26:35 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=Yg8YWHwT; spf=pass (domain: weidmueller.com, ip: 40.107.22.106, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=jX/BGaZI3emQ1O30/5z9eAoR5JuKmzVS2F9WdYC8AKX+cfavTFs36U6YPaGbME4gw2kkSdOiFehq9ZWnDeWGQqc9r0SY1slqnsCoQkI2mZqzjoJGX+LxCc3v/CeFH2lVNKqTedo+S70AAeJzMDlGjmHURu3FMemS9GzUxKQP0lM1HQbFmmQFDwg1tLZvhSFYcH3nbe4GFUKzjI3gySqOSFLNI8ECwvYy9CZpC42Z3/GVPqvz5K9muIIFOkXxvSwWiWXLYezY6nhMp1Icz88SYMWaqjZ9PBpr3HwFPsCFplD1tMSqK7uM1guX6VJ/r4I3rj3B6riAXX7DUF8wo+WzHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lLkihzEIyZkYbf8tBr7koBGCxXvyngDqbEgVCGtUo80=; b=Hb7KsFFaaOK1Jyj821Ef8g65RPoVMpm1bd7WxYlcyY6ET7gzPEdoRBPkDssj2LiyXDpsvZjV2PWkjYQz57poyf8RKYSFsae2Q1aKxtC1x5qwtUmjY+Ke6Oy6JjWKactOF7lkmBZMOJTuU395swxrbo8dUe5UYwtGA9algc64h4IVrfz3F3MjKFpVc8dTa3SOjm7oXXDtx+DcHHGDwNBLuhKcR74lFHqfqqGN4SupP/zN5wdY5kHzpYOF8mIohgCmoYYdQfN8QmZvowRKrGR+ikuJnwHX2PTnwh/wQYrFNP0yGbkhb3euLFOBHGKb3cSqcObm4C9jFLY6Ms5IJnADrQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lLkihzEIyZkYbf8tBr7koBGCxXvyngDqbEgVCGtUo80=; b=Yg8YWHwT705Fkybtk7Eaoge0Crdfo1NhA8KZtmTOcn8sHHctvDEOxdC4XkMNCwyH8PjfHVH0WNKHqzwotZGRxbK/LWvuLlkR8Cph7g71zvcgWaIg5Rfi4HwQv92bfrK2+hHc07w9Z9sSAsOWFSAYI1SBLJRD4W/hXX3FZDDaH7n7D6Q90zWHzLCu8Hw5EeMUgNoCCvtVtpqfkg7ggfP8FtUVyYsA6kiUnjxuo4D6oOVe1yxOYlD2ISj9osYC+F7Rk/qYQnlHpdnfRGegrJ8C2x/aNoDxw/LSvgCOIKNB2UkG3Cx7QNSaZflcmXfAnsmWIRCz9zCIlEUA79mjjLyyYQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:27 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:27 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 09/21] fetch2: add destdir to FetchData Date: Fri, 20 Dec 2024 12:26:00 +0100 Message-ID: <20241220112613.22647-10-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: ae3f3697-4e20-47d8-be7e-08dd20e92492 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: E1DwSfG1kl9aXqnnTtj5dppDIEo5JQU2snpq2vLV1z/qS2i0Of2aOkCIJGWR/sAX+N/Sf27wuKm8yjgUdL59HwdoIsQ6Iq5I3JDns1AFiAIOrXXqhxElA3Sn5EtitcRWrFfyWJkhTPtpCDQRa2GgRVkD7Vjbw8jLgML/e45hkuMZwbcSEmzemXW0hOUkaAWZD+NF07b6bKd++Dk7Ln95ucJDJar06G+hXDbLqNRu5PjF2MMdc9tFPW8Cedqg4/FLpd62LFlAl3Td9lEUULKOpi/fKZZaHhMbsVjtEhUxNORLUBMVQIdwO4mfl0RLxD4BM82Svoz3AM7zSduA+5T45eJ51BbbtC2Tnwb0XyQredpQcvNM4MUDSSvyBVxk16/llJyWzPA31g1nZ1VWGpFSPS0bq3krZSLcgF4jFivBSutaTyS9heB/O60CPFkCsebIy/sdBY1pgh3wAdD/S6iLVVTYxe38Hq4Hg4UIm0a4oVARZTEcjcIKDlHy/K86Aca3PL2FeixNPUzw0B2lc6BloDGgsfkmpaFgzYUCZT4wOvYpb1aZeV/IKDdZBQCHr52eaRwywkISfRH3qjIIX+dhBkiIyP9083/ljtWv8Yji5XQmVGiqTKbWBhDOgYc3rAD+SulhRh7e7obyCLz11jR5K944OrTPKmf6Skk7b07EYPb+GmU9M/eN3qRBlyiUAQruydnf++qd/RDzuYQCOSl9g30tOlaswRINMbBAdnp+cPng//c24m42NmWSMk9rZ8TWPkb3sjvJxOENwcs32YqaqxmtykjgAEM8BQ91/obJ+XRhq/magdm+z3EgfYeiwldqJ42sqbnk6kfqdCudVs7ww70Y8gPndsRlqQR5FChUri+dshOHVbCgjoKG37cMbQpNijqx//8y7a5o5rg8GJooJosOJyCI0vYcjviQzW+Ks+nr6V0Ix9LDJ4xHl6r7UvWcdouyLZadipYASOTrHcvbeho7yGGBKpLZquW6s+yw/rgfRZJEj1XoB3gyE3iBm9LRcu7BHvB1OFeFV5y42/a98ME/+k3HL7ktgGBC75X9F/vTlOsOMVVi6cHOuZeGdm+wiekj34PJl869Yi+7yG92AO6gjJSPs6Jj7+cMn1swGx43DrFaeF5W6EDxPOzIcatcjG91sw76fZc1s/ePpp3yjdLFcYB+tj2UyCmS032SdSO0IYi+6JMNu9keA5pc4Woe3SQJFPUCwt6l0DfZ/jQ21OQC6XM1EoEINutDtW7ggEX8t+A4Bbzhs3uRN0RMcRx6lff65jDPqt+3bBruIEpjrj2x4sp9TCKfX0wIGeShMr8pbj+dLcOUETIEuBQWYzGV4LH5SPuRiBYMRJhzmxOZ90YSlm7NbA6nYcXo4Cq4Onoab3WtX+KoKlcII4XoHS8lD9UhG/UFa9JqvJK1e0bvQJcghvZOdPJEd25r6N+wXyuPF6EVqPUBYZr2+KmmOByy X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: nQZpKVDYs/FAWebug8BIZtMPB8AO9pR3GKDsn4TCwJqbFlQ3KgmGzDLasj/PMyip/zeDNun6PHc0/9XKz7JSWQKAxeYlMXf+uRpftPDqbNEIIqvvNNmk0gbHUCX3y3ODeFEHWdeE8DO5wkfwfdDiFT//Nb24khKyq+JccZocUw2uSOjzz/wTd7QrbcbsNDLmJMLePN33xEfDkjFQtt8oxDkuo9bV23wgOo3LgdXOJCYY2QUHLxW3+l+i1nbZ5jx3zAGmuC6PcMqY71FZUzB57XtbuwHm/05Re9/RivogWiJUhfNmrV3Cqd5bHtvqEz74Pj7FUPC0PY6rLNcVynF+ogILOhiKFol1aL0BHz4G7ytRh7pGs/p4BYPp7repMLPPjck+Midci0gTo/njn5QVMZMpcbNuxqVF5OxpckjW6/PiDizYJJ9gCWge8+e39xOSq8f7L871SSAvI6bHu93sbarzhufSmcqGNKnedW05yarRKGUHC7dIvy0nUkrAZuLGItrcGc+OGH31+mw6MHQqmO65e/AKb0d0jEFXSnTrer7T+cYSsoEmv+QFDvVx1jyyzY9u9Cia0eDy7KNHZ4vToOKCcXluw1IFtErUtda6MYzgrF0/5ob6X09zEtiYlyzyr2VVLUY33ixInR8tbR3D4rbgzNay8w8tMc0hb6p8utL/En9cXawGnBng3j3HIMfyHVHA42TUsYB6MIt4gvGNq1aGnBkNbF6KOAYvzZ0dTZgXlJYe4YwKEYU6QlM2FgU0b0Nw1BSjvklKoTnrYrLOypRUhoIoh111Qeh19loZ6ymPYypAhK6CP9LfccfLy4g6Nm47H+2OgYSajHT69vzaotaryK8p5pM+dCSvgp9TEt00weTIAHQoBmW1VI1OdwyXdo1OKZfp2UYSx2J5X0xZYeZeDh47YlJIHvAG7CSAk+SIjRAjIO+EwcyVhMnzFREUpr27gJxmtjh/aLhmwt+cT5XfT7jvApxYkO+uNP4cs+wOywPqGRvymIH5lsYjVmfJqdWiRT0AVjtLZNMn5xX4eOknHB0sPFzQBAKmsFyj+pm5VMIjSlQJL3kF5TydUgourwn+oimuju/Fe6hAipN6RQ4xErwd1mz65Z7kIYXluYdU69d95UIVfEvpqoTsxkvPbmi5To6ET/04yXYuypboko9Yn+zBHVb6gx771tlp60HRFQPbC6VSNaLl5druLziEahX2knqphXaMPkRo8bge3ydNeYiO5vJJ/H1CHtTMTsNzO6dy90KrnvaBKGd6vaPLMEF4l/pxw1UDqXctTVPzamBask3R+O1gQWhoJLtJHnvYEf2B+OvKUHbdGWwzpKc68Iu/hMf1mb6UWcGQ5luQgF6BCEBut3VSRWweA1TnweuPGTZC950JCKEPY8681fnTivGjE+m2/IhQ7X1vm+oCwNKThPnyQjc9KGPgf3nbDBsPAcUIypYdZHPQ8shSUls3MHdocdp6XBUSGQ/pVczKu7qh8+YS1Ug9uJ5nk9XUUQYQvxbGzWIJ3AYFRL81MlrpfArKUgo3cL8J2fW9QcCuaKEUPran/dzJnfVlOsWjmNyPj8lasGs+TMncmUG1r/XfjQoF1D0jgegZb5sd5+MUKQ== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: ae3f3697-4e20-47d8-be7e-08dd20e92492 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:27.0621 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7hQxQR5XeNX1KhJg6/aa3iP5B9aki313tXEhn6JZoPLcaZ8iRDWSFAuYp3ryUMY8bQDg/6Xrh4fRV8sybeGFTA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16931 From: Stefan Herbrechtsmeier Add a `destdir` variable to the `FetchData` class to record destination directory of unpack method. Users of the `FetchData` class can use the directory to unpack additional content into the directory. The git fetcher class already records the destination directory in `destdir` class variable of `FetchData`. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 0c2d6d73e..3a7030bf3 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1314,6 +1314,7 @@ class FetchData(object): if not self.pswd and "pswd" in self.parm: self.pswd = self.parm["pswd"] self.setup = False + self.destdir = None def configure_checksum(checksum_id): checksum_plain_name = "%ssum" % checksum_id @@ -1609,6 +1610,8 @@ class FetchMethod(object): else: unpackdir = rootdir + urldata.destdir = unpackdir + if not unpack or not cmd: urldata.unpack_tracer.unpack("file-copy", unpackdir) # If file == dest, then avoid any copies, as we already put the file into dest! From patchwork Fri Dec 20 11:26:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 516AEE7718D for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (EUR05-DB8-obe.outbound.protection.outlook.com [40.107.20.118]) by mx.groups.io with SMTP id smtpd.web11.149744.1734693993154756588 for ; Fri, 20 Dec 2024 03:26:36 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=VO5ow6Ql; spf=pass (domain: weidmueller.com, ip: 40.107.20.118, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DOc4Uii8RJ7gwMAeVZU0jcdv83q2Pa3Ba9imBH75HKpk7zr2uSy/F9xyupEVMoih8uCSdy+9in+Q4k5HrWISsstSUBYgXuGd/3gW1XFX7IXS7wZKC9rXJDw2gG/rtgPVrMSL7hZ3PeFFjXWxx8/bhEElL9LgJR2u/auqrD0SVPZzms1ULBetyNNTLcolgb2ru8GDgru1MJlPCbMyGrVDn0aw+kmWUudO/6dHc+BFn/xFHZZoDSWKOEAP0AfAedJuZEVAUT2tnfPNmuJcKRiFyYBjWtNpyeQlf3YBcMCGcAD1p+bk2IH4s70Qn8avMXQpz/GYjrAT/dfulZASLHMzuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fUnEYLBpnFS6KFFX+hiKjOr98Rh4ZfH9Nvm05jFPbj4=; b=V/Fq/Nv77HZgipuacnOJIeF9yCE40J7khw8XzjacL0z565rGR+FvIQ24//Ev5dH4JhgU1x0pwNGccmbZpuimW4Jx3Zmelp1Y8iaDfLWUH/owkuI50kP1AKFa3ZY/WriWdg/Ru9EqfZjRhCTLzpDjcXZScDMT/uvZQrQx/IPzqdtx03yFsmEClJFh2CWuooUHQLBW0qU8ydPNCnsYj7xSpNuFAEUFqBM6TvWDqEtWX1P9hxkY+Qw5KGZ/B2ldHDf/TcogXKHzVadEHzcbSHg8oxG0B8aHJdG2Shrprhi4Xcq5XJIyZDyjDpR1Ayxv0GbSyVuxhjkUqhscGrLrozkrpg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fUnEYLBpnFS6KFFX+hiKjOr98Rh4ZfH9Nvm05jFPbj4=; b=VO5ow6QlI8FcMDXD3ZFfPSRBWr3Q8MtR0JNCUYdo2iEHRO39hnd3gTOzfWUJkO2P9zexswyivWwgV+urzqLNVJwhMhgGD5K4+X7xfCFB/pLK8zxdx2QHF+UCXVvXxEFt+Xtl/NeztV5kMsoq1R/NczEqUvwCUSHOkFioJmnpIE/mSMAE4OfVEyVZwhF3duZOKiJ39omXbyCpCxGmW6iiv6SL8rKjwmfHPKTFVp8vBw+OKlm5EhF48xiMkWCFojaFzX5q5KVji0+KSi3Cmg5T5FGfPyEAOLsU6d/by1SOtg7aMqSJ9Rj1Ujfh0AXdhjllKLoMSQSu/SMKnMjAnLp8Gg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:27 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:27 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 10/21] fetch: npm: rework Date: Fri, 20 Dec 2024 12:26:01 +0100 Message-ID: <20241220112613.22647-11-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: 71c27570-ec4a-4cfc-6699-08dd20e924c2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: Tsm+zvu1Wrq9wXmFWkb+2zmr8EqMIfc8j6Gdni6GLExrJl+Bi15BqL4eVCKRj9rHsMFQ7nfwFMviHq+yXd6fn044Ga8Sxz+d0fhmFRQengkqZrWaEVGUC1lchzRWIafSqDqEEZFFu6yQC+NCMLtmKww/Xl5Dj8J+TSN0Y5x2kE40niSIxvOuVu9XvLNzVRWS6QIZvwQrqtYN3grBEOocFKTQH3WlwkEhb2f8OrMV7y7J3peTKyE+zXZsTVTLb2TZcMj+nez3hO40CLxLWFQzBeqI/UrPjzD3bI6gmRpsDcPap3jCQpMdauGzwKWUkXyh8BuTBmVERn6d6/8oASGEfB1cm2gx+b2yN0fhQb/9Mq/6umpkfZbY6ohwPrfw1RcUFn/p8FCQonK8oronNQysMVyF7p3xb4tkeKMrL6A9CQchOdKBFb6ad8c4ehOv7bcCOj8WSoPmENf2T54BTNc4ArE8S7AwBCl75s9M+rLrjAOlW3ewEAwpJif/osYG1ZI88zI7DnStKJR6ni/FEPAkRSvIW2uzXvPwMI421F9syzTJy1LWii/yl1DlMWAb+H4RHt33d0AxIGRtTLT+l0G1bQeaF/kqYz/pwEfTq2sYkJ6fbKJv8TNUv4lTchWtWjHNkAHcaCSNw4xXWqtco5xv1xOvzXKGLA5LPofiVee+wxAsH9oa3a4MRmJOQMSeWWWN6xnXwC3QtAYBRAs1Ws/QxWLPNgCM7LD3sEiCK9wdpGs3ZWsxCeYHkWtXpK6Ywk6i5C/vDDg6uST5ffWsXIQni3QcbvmFmrHtvlmMmaPWv0tAK2uQ2vRb6VOLwWI8qpvHkJ/uoRDwnYO5t/rGjENX2flrZ8OiNttsl7mvxMx8wwzl8QumtUDz73OvtyUbgNeCjfWBVHLLSRbpMYSUNpTcpj46sf9xx1LH/jfHFA/+7izbgQdTfmkT4jgBWcln4D+JLeeVps0Coi1EBz8DeJPt+6IdPxpjelIaTdKe2UFdAjz3JwpoN7qf8bnWgsckRAn7/9rnQ+318QT3TcAlUNQqR0fLIlB2hfUv2hZAxshkhGgQdCOC5WBLqsw6+adm3qzE/JaPMbhnt2ChdBehugLhlBUqhs4/WJH4XlMJrF6vZYDEhaJk73WrGdkcrETfIRroOD4+MozCRRJ8kwCj8aePH9FQa7m28p4k8aBXdb91qptQ5NWz4tX4nu8lSB03WCLtZAaWlUjbZ2HRzZomH9gwGJV9hhGJNfbHJPd6o/nIFW8RojxA2HA/B0xfFMWtyr0HIgAxxEIPeWFQFdKZl2XpaYj/zaxmqGOWKSaIkEfDG9RkNm0pBelrSiZQ487V+7b/B0iCuaoly6RtZMGN1nnb8jI+fPOERicowX3ChqePDmRRN7FglHlRmhuBLeZuxJUpMhZ6Pi0ZLcg4P6UXR6JrZw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: I7GVUc5oZEAYKcbvAUPhpb+ipCwUiD+rxjxHjEuIo1YYPyJH/4QP/qDtv0EL/LGpSD11nH1gxZSZyj4HBYo6HC8na8rrcCW6b6citvaV9+SCF0igCl+24IQb7lpXqATry2DIBgoPOSXOGncJhKqCsET40Es53qVeQedLTOVeiFb40w6ov53A9IjHDildOs06Dq8X/DkL9tqYntglYMWIXval9lqwd21ccrFog0Nixn6kbcr7hSTvMU7L9IuoVX7Vl9me2vuyv//R0LnO6EHTGhecjfydSnxHTqbPp2hq48RrjCnqcvtBXFvBDY/sp46vubClJrMpXwpmaTT5/gTgeRslXDkNn3jnlIcDMVh8ZbdbXoCBvLD9Kf5zXZXpuSmDU4bqWdahtZ9hDwuJ3Ge7AOKI+DCY2PE+QSbRFG3qKocDR5sGmmWiIFZvydNBdFm7gejUy3qJvVdrhumoaqDefOlWlVDX0PdchioDX0vaHL1rFVqh0xoaV5sJiPM8j3cMeuIE6hrZtVH+wWo8uGGIHX2ofvqfLH73+smmsThzY3m5nBHvamYA/L7Na3oKknordrZ5RpjfggmMejwfz7ucRRKaiP4SrvtD/aVZQm4uihRbrKQSWsWacuIDvyFQSdcxkD8uVH6cAFO5e6HFZoBkuFICItVqe/GM50UVT9tOYlFQTEMwaxqHBrtRHj3PNI1FAqqfE59iiNcpsa3VKprWjabo8ctMWzy0b234nASvTD64piDdV3P0z/FEe3tKdFp8Fj+8bLis5Qmg+07Vu1Bz6btUlxyEf02abNWC1BSNxxlhmCSjXWdpQB9bNdaGZs1qAAq+svwNUQYqsc7g4/8779Lat8xSDnpsqhGBJSA/jtEOjz3b98Dae/eqNYDzu6BsnGQswo25Earf79EjWpNuhAocoE5ID3hc2ELk5ECpxpk8+bLARl9xAYzqr3PInoFIfClh3n1KNF6xuE0TgRK+98bOwpREte+QKu0SZ1MYet9bEU84EfpH/DA7Z5WovBl4h62BRKpWEyrLgpd7YZRX9pAJLwRoLE1Xjr+xqKzJPQt5GadW9GcBhDim5UcY84VyvfKGElDSz2qRZvZPHDYJNP8s5xP2nIwU3WSK2RmYjdo8e+i9hgcJLYQb36D6I/96nmxs9g3N83FKyWsFrmNXRQS0gaGeoUlPJkxgYOM7/IkymB+q0wgA8esgYUeMaQ+jcdwNNG7+Mot9RMDAE7VROGJubupaQDZIW7v89Llbquw3U9fvP3KtEFhz02P5xpH0GWyScqodJw09UDDqig2+BMiWIhNolO+Hny6VCffa16cPbDU9CFw6qLJP3cEz+rv9TDovix0DmMEEepkgx+kmNOOvOo8Pv9q8nb8hqlJ9e9NLiQd5FOZTn6yCT439fppU3bQX3SQe9oo5rwKXt3Lv02Hn6VZgQg7PlsLzBqD1RJK9EGVik/y9FkxqLVvB8jByQVOKEUGMmzSvzvhIwuDGPo2HoZGu2iHevzPE4sSc60lzXE4eF52ZvPJjOXncNuUGrNwExwh3NhfDdZtq/xzJwKIj5isaInpxo0u6zqxZiMnZHzw1wWQiaTFXP7FKkhNxkLqCWlAWRJU0PO4he8Dxag== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 71c27570-ec4a-4cfc-6699-08dd20e924c2 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:27.4102 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8W69tXbIjo2br34uvBslC5qI/yn48+WzEaUXidBjvbH07vNdq3Qd0NnQoiAGuIheursbaU4iKyozO0KjjZ51OQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16932 From: Stefan Herbrechtsmeier Rework the npm class regarding testability and integrity: * Remove dependency to npm binary. * Construct URL via a fix style and don’t resolve the URL via package registry. * Use the checksum from the recipe or URI and don’t depend on the checksum from the package registry. * Add common name and version schema. * Mark unused NpmEnvironment and npm_unpack function as deprecated. * Use Wget class as base and remove foreign done stamp handling. * Add support to compute the latest release version. * Remove support for latest version because it requires a package registry and should be rarely used. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/npm.py | 244 +++++++++++-------------------------------- 1 file changed, 63 insertions(+), 181 deletions(-) diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py index ac76d64cd..120dddbfd 100644 --- a/lib/bb/fetch2/npm.py +++ b/lib/bb/fetch2/npm.py @@ -10,20 +10,19 @@ SRC_URI = "npm://some.registry.url;OptionA=xxx;OptionB=xxx;..." Supported SRC_URI options are: -- package +- dn The npm package name. This is a mandatory parameter. -- version +- dv The npm package version. This is a mandatory parameter. - downloadfilename Specifies the filename used when storing the downloaded file. - destsuffix - Specifies the directory to use to unpack the package (default: npm). + The name of the path in which to place the package (default: npm). """ -import base64 import json import os import re @@ -40,6 +39,8 @@ from bb.fetch2 import check_network_access from bb.fetch2 import runfetchcmd from bb.utils import is_semver +from bb.fetch2.wget import Wget + def npm_package(package): """Convert the npm package name to remove unsupported character""" # For scoped package names ('@user/package') the '/' is replaced by a '-'. @@ -64,14 +65,7 @@ def npm_localfile(package, version=None): filename = package return os.path.join("npm2", filename) -def npm_integrity(integrity): - """ - Get the checksum name and expected value from the subresource integrity - https://www.w3.org/TR/SRI/ - """ - algo, value = integrity.split("-", maxsplit=1) - return "%ssum" % algo, base64.b64decode(value).hex() - +# Deprecated def npm_unpack(tarball, destdir, d): """Unpack a npm tarball""" bb.utils.mkdirhier(destdir) @@ -80,8 +74,8 @@ def npm_unpack(tarball, destdir, d): cmd += " --delay-directory-restore" cmd += " --strip-components=1" runfetchcmd(cmd, d, workdir=destdir) - runfetchcmd("chmod -R +X '%s'" % (destdir), d, quiet=True, workdir=destdir) +# Deprecated class NpmEnvironment(object): """ Using a npm config file seems more reliable than using cli arguments. @@ -130,7 +124,15 @@ class NpmEnvironment(object): return _run(cmd) -class Npm(FetchMethod): + +def construct_url_path(name, version): + return f"/{name}/-/{name.split('/')[-1]}-{version}.tgz" + +def construct_url(registry, name, version): + path = construct_url_path(name, version) + return f"https://{registry}{path}" + +class Npm(Wget): """Class to fetch a package from a npm registry""" def supports(self, ud, d): @@ -139,178 +141,58 @@ class Npm(FetchMethod): def urldata_init(self, ud, d): """Init npm specific variables within url data""" - ud.package = None - ud.version = None - ud.registry = None - # Get the 'package' parameter if "package" in ud.parm: - ud.package = ud.parm.get("package") + bb.warn(f"Parameter 'package' in '{ud.url}' is deprecated." + "Please use 'dn' parameter instead.") + ud.parm["dn"] = ud.parm["package"] + del ud.parm["package"] + if "version" in ud.parm: + bb.warn(f"Parameter 'version' in '{ud.url}' is deprecated." + "Please use 'dv' parameter instead.") + ud.parm["dv"] = ud.parm["version"] + del ud.parm["version"] + + if any(x not in ud.parm for x in ["dn", "dv"]): + return + + registry = ud.host + if ud.path != '/': + registry += ud.path + name = ud.parm["dn"] + version = ud.parm["dv"] + + if not is_semver(version): + if version == "latest": + raise ParameterError("Value 'latest' for parameter 'version' is no longer supported", ud.url) + else: + raise ParameterError("Invalid 'version' parameter", ud.url) - if not ud.package: - raise MissingParameterError("Parameter 'package' required", ud.url) + ud.url = construct_url(registry, name, version) + ud.info_url = f"https://{registry}/{name}" - # Get the 'version' parameter - if "version" in ud.parm: - ud.version = ud.parm.get("version") + if not "downloadfilename" in ud.parm: + ud.parm['downloadfilename'] = npm_localfile(name, version) - if not ud.version: - raise MissingParameterError("Parameter 'version' required", ud.url) + destsuffix = ud.parm.get("destsuffix", "npm") + subdir = ud.parm.get("subdir", "") + ud.parm["destsuffix"] = destsuffix + ud.parm["subdir"] = os.path.join(subdir, destsuffix) - if not is_semver(ud.version) and not ud.version == "latest": - raise ParameterError("Invalid 'version' parameter", ud.url) + if 'name' not in ud.parm: + ud.parm["name"] = f"{npm_package(name)}-{version}" - # Extract the 'registry' part of the url - ud.registry = re.sub(r"^npm://", "https://", ud.url.split(";")[0]) + ud.parm["striplevel"] = 1 - # Using the 'downloadfilename' parameter as local filename - # or the npm package name. - if "downloadfilename" in ud.parm: - ud.localfile = npm_localfile(d.expand(ud.parm["downloadfilename"])) - else: - ud.localfile = npm_localfile(ud.package, ud.version) - - # Get the base 'npm' command - ud.basecmd = d.getVar("FETCHCMD_npm") or "npm" - - # This fetcher resolves a URI from a npm package name and version and - # then forwards it to a proxy fetcher. A resolve file containing the - # resolved URI is created to avoid unwanted network access (if the file - # already exists). The management of the donestamp file, the lockfile - # and the checksums are forwarded to the proxy fetcher. - ud.proxy = None - ud.needdonestamp = False - ud.resolvefile = self.localpath(ud, d) + ".resolved" - - def _resolve_proxy_url(self, ud, d): - def _npm_view(): - args = [] - args.append(("json", "true")) - args.append(("registry", ud.registry)) - pkgver = shlex.quote(ud.package + "@" + ud.version) - cmd = ud.basecmd + " view %s" % pkgver - env = NpmEnvironment(d) - check_network_access(d, cmd, ud.registry) - view_string = env.run(cmd, args=args) - - if not view_string: - raise FetchError("Unavailable package %s" % pkgver, ud.url) - - try: - view = json.loads(view_string) - - error = view.get("error") - if error is not None: - raise FetchError(error.get("summary"), ud.url) - - if ud.version == "latest": - bb.warn("The npm package %s is using the latest " \ - "version available. This could lead to " \ - "non-reproducible builds." % pkgver) - elif ud.version != view.get("version"): - raise ParameterError("Invalid 'version' parameter", ud.url) - - return view - - except Exception as e: - raise FetchError("Invalid view from npm: %s" % str(e), ud.url) - - def _get_url(view): - tarball_url = view.get("dist", {}).get("tarball") - - if tarball_url is None: - raise FetchError("Invalid 'dist.tarball' in view", ud.url) - - uri = URI(tarball_url) - uri.params["downloadfilename"] = ud.localfile - - integrity = view.get("dist", {}).get("integrity") - shasum = view.get("dist", {}).get("shasum") - - if integrity is not None: - checksum_name, checksum_expected = npm_integrity(integrity) - uri.params[checksum_name] = checksum_expected - elif shasum is not None: - uri.params["sha1sum"] = shasum - else: - raise FetchError("Invalid 'dist.integrity' in view", ud.url) - - return str(uri) - - url = _get_url(_npm_view()) - - bb.utils.mkdirhier(os.path.dirname(ud.resolvefile)) - with open(ud.resolvefile, "w") as f: - f.write(url) - - def _setup_proxy(self, ud, d): - if ud.proxy is None: - if not os.path.exists(ud.resolvefile): - self._resolve_proxy_url(ud, d) - - with open(ud.resolvefile, "r") as f: - url = f.read() - - # Avoid conflicts between the environment data and: - # - the proxy url checksum - data = bb.data.createCopy(d) - data.delVarFlags("SRC_URI") - ud.proxy = Fetch([url], data) - - def _get_proxy_method(self, ud, d): - self._setup_proxy(ud, d) - proxy_url = ud.proxy.urls[0] - proxy_ud = ud.proxy.ud[proxy_url] - proxy_d = ud.proxy.d - proxy_ud.setup_localpath(proxy_d) - return proxy_ud.method, proxy_ud, proxy_d - - def verify_donestamp(self, ud, d): - """Verify the donestamp file""" - proxy_m, proxy_ud, proxy_d = self._get_proxy_method(ud, d) - return proxy_m.verify_donestamp(proxy_ud, proxy_d) - - def update_donestamp(self, ud, d): - """Update the donestamp file""" - proxy_m, proxy_ud, proxy_d = self._get_proxy_method(ud, d) - proxy_m.update_donestamp(proxy_ud, proxy_d) - - def need_update(self, ud, d): - """Force a fetch, even if localpath exists ?""" - if not os.path.exists(ud.resolvefile): - return True - if ud.version == "latest": - return True - proxy_m, proxy_ud, proxy_d = self._get_proxy_method(ud, d) - return proxy_m.need_update(proxy_ud, proxy_d) - - def try_mirrors(self, fetch, ud, d, mirrors): - """Try to use a mirror""" - proxy_m, proxy_ud, proxy_d = self._get_proxy_method(ud, d) - return proxy_m.try_mirrors(fetch, proxy_ud, proxy_d, mirrors) - - def download(self, ud, d): - """Fetch url""" - self._setup_proxy(ud, d) - ud.proxy.download() - - def unpack(self, ud, rootdir, d): - """Unpack the downloaded archive""" - destsuffix = ud.parm.get("destsuffix", "npm") - destdir = os.path.join(rootdir, destsuffix) - npm_unpack(ud.localpath, destdir, d) - ud.unpack_tracer.unpack("npm", destdir) - - def clean(self, ud, d): - """Clean any existing full or partial download""" - if os.path.exists(ud.resolvefile): - self._setup_proxy(ud, d) - ud.proxy.clean() - bb.utils.remove(ud.resolvefile) - - def done(self, ud, d): - """Is the download done ?""" - if not os.path.exists(ud.resolvefile): - return False - proxy_m, proxy_ud, proxy_d = self._get_proxy_method(ud, d) - return proxy_m.done(proxy_ud, proxy_d) + super().urldata_init(ud, d) + + def latest_versionstring(self, ud, d): + from functools import cmp_to_key + info = json.loads(self._fetch_index(ud.info_url, ud, d)) + versions = [(0, v, "") for v in info["versions"]] + versions = sorted(versions, key=cmp_to_key(bb.utils.vercmp)) + + return (versions[-1][1], "") + + def require_download_metadata(self): + return True From patchwork Fri Dec 20 11:26:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BB34E77198 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.106]) by mx.groups.io with SMTP id smtpd.web11.149743.1734693991344736713 for ; Fri, 20 Dec 2024 03:26:36 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=5a2KgGS2; spf=pass (domain: weidmueller.com, ip: 40.107.22.106, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MO9FckC4/+pjvSl85FXCvNz8zxqy9GwgNS1P7E3RRBAgva7AHCR5Jy2QJyvpFe5Oss12H3nzSTs3E3+93XoUApuYucvu3DabJstoPsQuk8Yy2rDJTsqzn/JViFbs7LoOe74Q+qZBb2EPsRsRu/s1dn6PHWJkOoZ8kZTmwraD7qNiWNjpUYmqFsB3wyUPpeCj/MME1uxxwyj0VvavZK5xjeWhNwIsFdsyKE4LzFIfaobnp+5ZYLye746OTUxziaIXSoZ/13aefrnaMCwOp8xBVMeJPBjCXAHcMAK0tYkIPe5iyWu3SXLdrz/qoRDsWDrdJLRYLQO+tCs9pXft+uWRPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CP6PRAT6tlYMnq9DSnjr6qPmKll6BATeTNlR+i/5Rnc=; b=VG5eNNUX8/dXRUEDorGOI8LzTIy6Yi4crClecA9xIyKIPbb9KpRRJz1ZMm+9w+Rcy6SFAId7b2Q2u2XrCcZIa810exPEMamaLSyUTGwL0rWFDS1oxqOWF4BsPcgCZ8ASK3qZHev0q7K3bq3074gk5pIIcOjmpveHoiwmLz12JL1e5LLxBSB0MsutKnLzHN1kr1Np8yXsoX90jdVbkpwiUFw+XHQrm0DJSC+a0g87F5mUBY20qvieG06AgpRuwjD6eYjDs0xlq0o2zqkzLODWKQhAg+DaRxug0fWIUwUPHjiZC4r/H7N+nntxGiw10URB7RyU+f0m9tRqep/PxjcuhA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CP6PRAT6tlYMnq9DSnjr6qPmKll6BATeTNlR+i/5Rnc=; b=5a2KgGS2yCIIi34IhlL2817a2MP40dPaYpHe1hNUY4/h+gP+duKnbXCK6sYE5M96J/IhUWjZ6c7M34UJ1sv2AQ9l5Qy9gBSfMRyrVqTvOr1W/gTLxdBGYVdXRFjgNAIu30WwsWbaQKLaBiQ3mm/FmhWTxQpgu90bAG7CKq8Wb+M4EEvMnZo9cXJxHLKxHIFdNkEmgSZQE3bjSL6fnOacLdzjNTviD3XcXXbanmrc+63CZm3V/XUVsDP1MSy6qI8l/EO0ovywiJiKMNdvqGYPFRuhlAcImpfche+jQBE8/uqsJ2WhcJtT+CeyFn90lH6VoatCdJQ7HeRASkuGsGXDVw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:28 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:28 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 11/21] tests: fetch: adapt style in npm(sw) class Date: Fri, 20 Dec 2024 12:26:02 +0100 Message-ID: <20241220112613.22647-12-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: e76333fe-02b8-4f37-5e14-08dd20e924f7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: BrulRto5BH+XqfiYhocXcdWli1bp9ch1lKuMRhENo/c4Yh+goYSLPKBkrfIOXvui6lGZhlygFjkg9PXjskegqUVXxKDSQeTo9SJ+Auwi1xLhJ+vGuE/l1bOtHKKUoOp+ClncG1/s4l/CSZ4eC3reH+c8YqU17BdKD67WTfXuYUSLU7IRn/XOPC0qedotWE/NXzirdPrd9vs86EutnAsz58f2qy19tZU9lAP3w+TCKqZuQx0ee9IxcfrK2KoOaOqNul1E0UkM0Q/JSEOveHECYzoUWp+ZU74vV6lAbtb8CzG4I4nZrPwhJr5vPD3WTWTu1haT+V1D01qFLc0kY0ew8wkp6Sjrte89KW6zWy7qe+Rf7RFLBMJcYuc6ypstifScU8M6F1KYQHuFg7baFkYtdXOkqOoa1L4oSBY7WH22MnA35oUh5b/4oSdRZDIwNGIOdAOoz0TMcN/OHvwDef9hmZgT/NCZutotR2YOv5bwZd6+Ke7TZnEZhSqAFCiebVGPji7SDAu+V1cX4xdJkexf3W3we94JvNglcxuskup99vJ/jofi/HZ8Kev/ajUTFlkP5VkMeip3rv9rjsxGKDbKfXeSHc1JNx4R74ufYPe6YcvrbQhdHcPJfDiwjLX+Ng6vH20FAbxw/zPBKXThnElE5WvZ0e6Jjk3QldRhq7r7sgjxD2pqECINCECbKqIpZdGOLZZoaMfOWELQVSofNUbkGKNNLZesv+brULC8wTbFh0jIvMoDwL5MBl00IlaBRR3CBgPQJOOAmAg5QT2HhyEPUCD2ZFPND8NtrzH83PJYWkm5OBqC/V0Ydr6UwvMdTaR07AAvs/AvQHfBAnSESD/MYPPORIz7y9rORSpklaEiD2g3rJQPr7y5Wpwy2RPsqbMgrXaK6SB6GSb3gLKFRbK65o+DyBqTeXPWX1LUVJ7KqbkXxrumzswZlDdP3mTVGRuQYekqtzwvu9kuMLpEYlhVIbjNxChI6r3cI/kkWBpiHkP89xTqBqvPWlVSpY2SwHvbMBqBLI+e8nTsTct5pxL9n3/jGokrR3J1KkH94v6KNVS2NC0jwkNwB/TSWcs+JxtqpizebP7Wa5sQrw6wX424BAeH7hDSxFFNDxnGz3ztfG/ywhO6RvBkwnBpbQD3szIutdMwniSyIgaqESFs58slfoCYW0ogDRn3rt5T3CN47TmKWCfDMca1z/iPY3cIcAK2hFXuB+tkScxok4RNktlXK652VqISw/5RRbJpNZtzlVgvhGzbPdkzv064L4ZdG1tD5cxHF9bqcISQxOFOiXsid0EjZ7nsA+o13RdlAyq3FdIoCmn2fGfmVTPbAMdcrx8nxbrpiv/Brav15QqWJSJKO7utZTv/onAFROIGMKOmGc36OEYrw0vd+QMdx7UY7btJtafdaertXndMNQX7DZ9jIQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: TOr71IKYxZU8n0nlkIMK+DN8XlMu8LCxqKCVIXpahF9OBjGZGXrMRgJwoftb1ovPbe9xLlQYBlrFemwbEE3YJZfDy/ktTq3qULKuMB+XwaRva+hQDBzQz/bQGLKC2/rpm9LPrqv5Cws5TZkySo8wsJXB+NUDBr35oodhohM4z6RgX03QoesOkm4EnA+kdYjK+T0eaYAHvSd6WMI/2N7SbLTcZ67JrejC8CsH/1pfB1cyOt48QCuhzbRYHEWY1j+uz0gfZae90HPSIiDucYtNuUdo+O1w/FfO46BRn4r1NuGLPANBDKbrPzx5NbLTXcwa+ZDKxHGUGVCTPgNkkKeFrapDk46FsSDGlhGeXoJkfSYtTgpFX90Q5pTVQLran5hk9pX8NBqr7zV1rtGNwgPERYzjKRZzutEUTt19z8AvN+Fgc90vUAlVH/7nr1lVUGVb96AZHc6w46TeTJGozcELRmH5XwTbTMEsDzYOOeCdiDeEsrNeaH4HHYO7ELKdxMa14+QmFFB8VJAaptJpDBpW0fg+1C/hvIu2gg/D5hRIzcXHQVvR4SSa2WVe3GXoQIMb0oTDiM05vuUqm9IDj3QigCcBYfndzsj7Rh7BhmNwimR7byVTWXAsIOLcw4p1tymEGY27IqRe3cTf/4bRtlO1SstGuJSCQLpZCDbGKFupZ84t7pu7WBUA/Pg6LuCRdFeBHS5IYRAyo3lZodBsh0jBo/nF2m8aTfKl+K8Y5H70ZT5WDeWJvkjywgEldpGIQ7AukZG5nki5e8bDuHpP91z8OXsUawLe29sovvKtD9eTLU/T+aSNeWuzmadLFwNt6YfjZlvFM9eNQdVMRysv/JWykJybID1UwJNUgUqU4JGzd3jEPlRTSRgxh7Zt/SR4dE9xjwuH7mJ+PgjoxPsmcuAkajJgd0R06PxxpXQkI8oK0S/1yw9sKFTkQeR3sfd+0goRyo08D7J5DQBxYrkXxiXQHnFxM9cOTFbM97g2h7RIm1Z/quTw8dQBxQV1OkuZdi7XRlYA9W9DGU8U3KJ5AbgliMojzi5ByfrDO3rU2D8Ojn5eTeIsFA/K87KaMbMWtQaWNIjnoeDja/q3I607QzpJRJy82HfFqBHigHsFTtsHxp+aLv+cKlIOLICGbadHOASBXS2T/2uxi59fdhDi7y8egJqLRy81taJXJl/8sFpKm4Is2HYcfpY6Gsh1d9xzJvQvk4kc6WvP7b+B13skunwLSQfetcJKuvcnmgo/fXpTgCpMrFVB4SQsjeVGtGlCFTP3EaUk0Q5pJETA2fbb5Xk1WfbLLwQ/sup7eHNaBX8m/uKiBAqRfELJByXogICMhzL0uORVVIjTWjHHau0ZKbmzIwZEqPLADaLe+n0neGFGtAs6POIUs0/qaWiO5bshTDVn6fC760aAWNb7zO8xaAPRsVZPa7JDUxR8My/K5y/xR/ahSyizl1/HpQ/isDOMwhOSnlJtMpLqjU7kHwLexeGYJMxndg01iiKmPxEkloDSHuvIB0Gw5pZ9Ssfy09wjOGX2DZTqodcczscuz77MKBEHWRauBvRvnVe/aQe+RdXD6qfePFxqamitIe+GAV6b4ydYCAGhWmoUvAgV6XE9lHrGLA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: e76333fe-02b8-4f37-5e14-08dd20e924f7 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:27.7618 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7TinhaW4g8wcAujuxZh44lt/LmiTPcBTEOHj/8M+N/hg1c8SL5o5jNH6JMrTfHXup3BhP+BGFMdrlSIrgYe3ug== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16935 From: Stefan Herbrechtsmeier Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/tests/fetch.py | 348 +++++++++++++++++++++--------------------- 1 file changed, 174 insertions(+), 174 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 6b8e3e060..934b96cac 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2620,47 +2620,47 @@ class CrateTest(FetcherTest): class NPMTest(FetcherTest): def skipIfNoNpm(): import shutil - if not shutil.which('npm'): - return unittest.skip('npm not installed') + if not shutil.which("npm"): + return unittest.skip("npm not installed") return lambda f: f @skipIfNoNpm() @skipIfNoNetwork() def test_npm(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) - self.assertTrue(os.path.exists(ud.localpath + '.done')) + self.assertTrue(os.path.exists(ud.localpath + ".done")) self.assertTrue(os.path.exists(ud.resolvefile)) fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, 'npm') - self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json'))) + unpackdir = os.path.join(self.unpackdir, "npm") + self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_bad_checksum(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] # Fetch once to get a tarball fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) # Modify the tarball - bad = b'bad checksum' - with open(ud.localpath, 'wb') as f: + bad = b"bad checksum" + with open(ud.localpath, "wb") as f: f.write(bad) # Verify that the tarball is fetched again fetcher.download() badsum = hashlib.sha512(bad).hexdigest() - self.assertTrue(os.path.exists(ud.localpath + '_bad-checksum_' + badsum)) + self.assertTrue(os.path.exists(ud.localpath + "_bad-checksum_" + badsum)) self.assertTrue(os.path.exists(ud.localpath)) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_premirrors(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] # Fetch once to get a tarball fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] @@ -2668,17 +2668,17 @@ class NPMTest(FetcherTest): self.assertTrue(os.path.exists(ud.localpath)) # Setup the mirror by renaming the download directory - mirrordir = os.path.join(self.tempdir, 'mirror') + mirrordir = os.path.join(self.tempdir, "mirror") bb.utils.rename(self.dldir, mirrordir) os.mkdir(self.dldir) # Configure the premirror to be used - self.d.setVar('PREMIRRORS', 'https?$://.*/.* file://%s/npm2' % mirrordir) - self.d.setVar('BB_FETCH_PREMIRRORONLY', '1') + self.d.setVar("PREMIRRORS", "https?$://.*/.* file://%s/npm2" % mirrordir) + self.d.setVar("BB_FETCH_PREMIRRORONLY", "1") # Fetch again self.assertFalse(os.path.exists(ud.localpath)) - # The npm fetcher doesn't handle that the .resolved file disappears + # The npm fetcher doesn"t handle that the .resolved file disappears # while the fetcher object exists, which it does when we rename the # download directory to "mirror" above. Thus we need a new fetcher to go # with the now empty download directory. @@ -2690,19 +2690,19 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_premirrors_with_specified_filename(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] # Fetch once to get a tarball fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) # Setup the mirror - mirrordir = os.path.join(self.tempdir, 'mirror') + mirrordir = os.path.join(self.tempdir, "mirror") bb.utils.mkdirhier(mirrordir) mirrorfilename = os.path.join(mirrordir, os.path.basename(ud.localpath)) os.replace(ud.localpath, mirrorfilename) - self.d.setVar('PREMIRRORS', 'https?$://.*/.* file://%s' % mirrorfilename) - self.d.setVar('BB_FETCH_PREMIRRORONLY', '1') + self.d.setVar("PREMIRRORS", "https?$://.*/.* file://%s" % mirrorfilename) + self.d.setVar("BB_FETCH_PREMIRRORONLY", "1") # Fetch again self.assertFalse(os.path.exists(ud.localpath)) fetcher.download() @@ -2712,22 +2712,22 @@ class NPMTest(FetcherTest): @skipIfNoNetwork() def test_npm_mirrors(self): # Fetch once to get a tarball - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) # Setup the mirror - mirrordir = os.path.join(self.tempdir, 'mirror') + mirrordir = os.path.join(self.tempdir, "mirror") bb.utils.mkdirhier(mirrordir) os.replace(ud.localpath, os.path.join(mirrordir, os.path.basename(ud.localpath))) - self.d.setVar('MIRRORS', 'https?$://.*/.* file://%s/' % mirrordir) + self.d.setVar("MIRRORS", "https?$://.*/.* file://%s/" % mirrordir) # Update the resolved url to an invalid url - with open(ud.resolvefile, 'r') as f: + with open(ud.resolvefile, "r") as f: url = f.read() uri = URI(url) - uri.path = '/invalid' - with open(ud.resolvefile, 'w') as f: + uri.path = "/invalid" + with open(ud.resolvefile, "w") as f: f.write(str(uri)) # Fetch again self.assertFalse(os.path.exists(ud.localpath)) @@ -2737,17 +2737,17 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_destsuffix_downloadfilename(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0;destsuffix=foo/bar;downloadfilename=foo-bar.tgz'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0;destsuffix=foo/bar;downloadfilename=foo-bar.tgz"] fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'foo-bar.tgz'))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/foo-bar.tgz"))) fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, 'foo', 'bar') - self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json'))) + unpackdir = os.path.join(self.unpackdir, "foo", "bar") + self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) def test_npm_no_network_no_tarball(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] - self.d.setVar('BB_NO_NETWORK', '1') + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] + self.d.setVar("BB_NO_NETWORK", "1") fetcher = bb.fetch.Fetch(urls, self.d) with self.assertRaises(bb.fetch2.NetworkAccess): fetcher.download() @@ -2755,42 +2755,42 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_no_network_with_tarball(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] # Fetch once to get a tarball fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() # Disable network access - self.d.setVar('BB_NO_NETWORK', '1') + self.d.setVar("BB_NO_NETWORK", "1") # Fetch again fetcher.download() fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, 'npm') - self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json'))) + unpackdir = os.path.join(self.unpackdir, "npm") + self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_alternate(self): - urls = ['npm://skimdb.npmjs.com;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ["npm://skimdb.npmjs.com;package=@savoirfairelinux/node-server-example;version=1.0.0"] fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, 'npm') - self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json'))) + unpackdir = os.path.join(self.unpackdir, "npm") + self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_latest(self): - url = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest'] + url = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest"] fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, 'npm') - self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json'))) + unpackdir = os.path.join(self.unpackdir, "npm") + self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_invalid(self): - urls = ['npm://registry.invalid.org;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ["npm://registry.invalid.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] fetcher = bb.fetch.Fetch(urls, self.d) with self.assertRaises(bb.fetch2.FetchError): fetcher.download() @@ -2798,7 +2798,7 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_package_invalid(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/invalid;version=1.0.0'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/invalid;version=1.0.0"] fetcher = bb.fetch.Fetch(urls, self.d) with self.assertRaises(bb.fetch2.FetchError): fetcher.download() @@ -2806,145 +2806,145 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_invalid(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=invalid'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=invalid"] with self.assertRaises(bb.fetch2.ParameterError): fetcher = bb.fetch.Fetch(urls, self.d) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_none(self): - urls = ['npm://;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ["npm://;package=@savoirfairelinux/node-server-example;version=1.0.0"] with self.assertRaises(bb.fetch2.MalformedUrl): fetcher = bb.fetch.Fetch(urls, self.d) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_package_none(self): - urls = ['npm://registry.npmjs.org;version=1.0.0'] + urls = ["npm://registry.npmjs.org;version=1.0.0"] with self.assertRaises(bb.fetch2.MissingParameterError): fetcher = bb.fetch.Fetch(urls, self.d) @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_none(self): - urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example'] + urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example"] with self.assertRaises(bb.fetch2.MissingParameterError): fetcher = bb.fetch.Fetch(urls, self.d) def create_shrinkwrap_file(self, data): import json - datadir = os.path.join(self.tempdir, 'data') - swfile = os.path.join(datadir, 'npm-shrinkwrap.json') + datadir = os.path.join(self.tempdir, "data") + swfile = os.path.join(datadir, "npm-shrinkwrap.json") bb.utils.mkdirhier(datadir) - with open(swfile, 'w') as f: + with open(swfile, "w") as f: json.dump(data, f) return swfile @skipIfNoNetwork() def test_npmsw(self): swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=', - 'dependencies': { - 'content-type': "1.0.4" + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=", + "dependencies": { + "content-type": "1.0.4" } }, - 'node_modules/array-flatten/node_modules/content-type': { - 'version': '1.0.4', - 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', - 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', - 'dependencies': { - 'cookie': 'git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09' + "node_modules/array-flatten/node_modules/content-type": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz", + "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==", + "dependencies": { + "cookie": "git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09" } }, - 'node_modules/array-flatten/node_modules/content-type/node_modules/cookie': { - 'resolved': 'git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09' + "node_modules/array-flatten/node_modules/content-type/node_modules/cookie": { + "resolved": "git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09" } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz'))) - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz'))) - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git'))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/github.com.jshttp.cookie.git"))) fetcher.unpack(self.unpackdir) - self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'npm-shrinkwrap.json'))) - self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'package.json'))) - self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'package.json'))) - self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'node_modules', 'cookie', 'package.json'))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "npm-shrinkwrap.json"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "node_modules/array-flatten/package.json"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "node_modules/array-flatten/node_modules/content-type/package.json"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "node_modules/array-flatten/node_modules/content-type/node_modules/cookie/package.json"))) @skipIfNoNetwork() def test_npmsw_git(self): swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/cookie': { - 'resolved': 'git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09' + "packages": { + "node_modules/cookie": { + "resolved": "git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09" } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git'))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/github.com.jshttp.cookie.git"))) @skipIfNoNetwork() def test_npmsw_dev(self): swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" }, - 'node_modules/content-type': { - 'version': '1.0.4', - 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', - 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', - 'dev': True + "node_modules/content-type": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz", + "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==", + "dev": True } } }) # Fetch with dev disabled - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz'))) - self.assertFalse(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz'))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz"))) + self.assertFalse(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz"))) # Fetch with dev enabled - fetcher = bb.fetch.Fetch(['npmsw://' + swfile + ';dev=1'], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile + ";dev=1"], self.d) fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz'))) - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz'))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz"))) @skipIfNoNetwork() def test_npmsw_destsuffix(self): swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile + ';destsuffix=foo/bar'], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile + ";destsuffix=foo/bar"], self.d) fetcher.download() fetcher.unpack(self.unpackdir) - self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'foo', 'bar', 'node_modules', 'array-flatten', 'package.json'))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "foo/bar/node_modules/array-flatten/package.json"))) def test_npmsw_no_network_no_tarball(self): swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" } } }) - self.d.setVar('BB_NO_NETWORK', '1') - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + self.d.setVar("BB_NO_NETWORK", "1") + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) with self.assertRaises(bb.fetch2.NetworkAccess): fetcher.download() @@ -2952,112 +2952,112 @@ class NPMTest(FetcherTest): @skipIfNoNetwork() def test_npmsw_no_network_with_tarball(self): # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1'], self.d) + fetcher = bb.fetch.Fetch(["npm://registry.npmjs.org;package=array-flatten;version=1.1.1"], self.d) fetcher.download() # Disable network access - self.d.setVar('BB_NO_NETWORK', '1') + self.d.setVar("BB_NO_NETWORK", "1") # Fetch again swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) fetcher.download() fetcher.unpack(self.unpackdir) - self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'package.json'))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "node_modules/array-flatten/package.json"))) @skipIfNoNetwork() def test_npmsw_npm_reusability(self): # Fetch once with npmsw swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) fetcher.download() # Disable network access - self.d.setVar('BB_NO_NETWORK', '1') + self.d.setVar("BB_NO_NETWORK", "1") # Fetch again with npm - fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1'], self.d) + fetcher = bb.fetch.Fetch(["npm://registry.npmjs.org;package=array-flatten;version=1.1.1"], self.d) fetcher.download() fetcher.unpack(self.unpackdir) - self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'npm', 'package.json'))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "npm/package.json"))) @skipIfNoNetwork() def test_npmsw_bad_checksum(self): # Try to fetch with bad checksum swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-gfNEp2hqgLTFKT6P3AsBYMgsBqg=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-gfNEp2hqgLTFKT6P3AsBYMgsBqg=" } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) with self.assertRaises(bb.fetch2.FetchError): fetcher.download() # Fetch correctly to get a tarball swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) fetcher.download() - localpath = os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz') + localpath = os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz") self.assertTrue(os.path.exists(localpath)) # Modify the tarball - bad = b'bad checksum' - with open(localpath, 'wb') as f: + bad = b"bad checksum" + with open(localpath, "wb") as f: f.write(bad) # Verify that the tarball is fetched again fetcher.download() badsum = hashlib.sha1(bad).hexdigest() - self.assertTrue(os.path.exists(localpath + '_bad-checksum_' + badsum)) + self.assertTrue(os.path.exists(localpath + "_bad-checksum_" + badsum)) self.assertTrue(os.path.exists(localpath)) @skipIfNoNpm() @skipIfNoNetwork() def test_npmsw_premirrors(self): # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1'], self.d) + fetcher = bb.fetch.Fetch(["npm://registry.npmjs.org;package=array-flatten;version=1.1.1"], self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) # Setup the mirror - mirrordir = os.path.join(self.tempdir, 'mirror') + mirrordir = os.path.join(self.tempdir, "mirror") bb.utils.mkdirhier(mirrordir) os.replace(ud.localpath, os.path.join(mirrordir, os.path.basename(ud.localpath))) - self.d.setVar('PREMIRRORS', 'https?$://.*/.* file://%s/' % mirrordir) - self.d.setVar('BB_FETCH_PREMIRRORONLY', '1') + self.d.setVar("PREMIRRORS", "https?$://.*/.* file://%s/" % mirrordir) + self.d.setVar("BB_FETCH_PREMIRRORONLY", "1") # Fetch again self.assertFalse(os.path.exists(ud.localpath)) swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @@ -3065,51 +3065,51 @@ class NPMTest(FetcherTest): @skipIfNoNetwork() def test_npmsw_mirrors(self): # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1'], self.d) + fetcher = bb.fetch.Fetch(["npm://registry.npmjs.org;package=array-flatten;version=1.1.1"], self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) # Setup the mirror - mirrordir = os.path.join(self.tempdir, 'mirror') + mirrordir = os.path.join(self.tempdir, "mirror") bb.utils.mkdirhier(mirrordir) os.replace(ud.localpath, os.path.join(mirrordir, os.path.basename(ud.localpath))) - self.d.setVar('MIRRORS', 'https?$://.*/.* file://%s/' % mirrordir) + self.d.setVar("MIRRORS", "https?$://.*/.* file://%s/" % mirrordir) # Fetch again with invalid url self.assertFalse(os.path.exists(ud.localpath)) swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://invalid', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://invalid", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) fetcher.download() self.assertTrue(os.path.exists(ud.localpath)) @skipIfNoNetwork() def test_npmsw_bundled(self): swfile = self.create_shrinkwrap_file({ - 'packages': { - 'node_modules/array-flatten': { - 'version': '1.1.1', - 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz', - 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=' + "packages": { + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" }, - 'node_modules/content-type': { - 'version': '1.0.4', - 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz', - 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==', - 'inBundle': True + "node_modules/content-type": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz", + "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==", + "inBundle": True } } }) - fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d) + fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz'))) - self.assertFalse(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz'))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz"))) + self.assertFalse(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz"))) class GitSharedTest(FetcherTest): def setUp(self): From patchwork Fri Dec 20 11:26:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67950E77193 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (EUR05-DB8-obe.outbound.protection.outlook.com [40.107.20.118]) by mx.groups.io with SMTP id smtpd.web11.149744.1734693993154756588 for ; Fri, 20 Dec 2024 03:26:36 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=LRjoLvN5; spf=pass (domain: weidmueller.com, ip: 40.107.20.118, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RrM1i3C+Lb5DZZaZW+wDGjxNhc8rEgHvvfBzKkQ4EpYTT29ThZGQQU57ByfUasfKBAQm+boX9ROIw7U/9BdRv4sxJArwmiGQp85NZ9qGnQn6YLrBJg0HH2ZG9+sudoRq5aL5Jl6qifLsqK/Wp7fI8wgR/KaTGIxEbtw82/2/jow2Dgqlxq847VV+cXM2s34jElyatfu3Rd9NiHIvuxWx5omlYbY+BGD/LAeEjQ/ql9d7PqeZokyVKWnieSpNvYord5a666JgC5JrT121CJlGCUWyq3OqVKzD8uvcIQdxxEE6IOJWAh+tcGmkolWTHgMxlfxqpXP3VnZXT1RE7uV0ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rtxu4K68iSifp91eIigTtftCNZ8EYP0hKlTQRlFBAAQ=; b=w5uWz60RF1aBw43fJuaHaXeO7A/+AQf7IOa96dvDC0y9x0Wo85qUfQiUqlpZOtEP6bFOp/X6FEyZ+I3F3+4hM+j9U5zFU8sdR+eIIvM64SstOXCtkSKIm6uzTflzRjqaSEoh7T5IoI89KLblVMOgS9NlVSToO6K4p0TPqVp9TCHEdJxcAGkV7/shPOmiCzOsgT5UvekfGiyOd1PxSv+XUaFsyfzSsDd0t30xEyQC0wlTY3Mnw9yN5agd+ly6YxESBoFWd6iZWJ6B6r93sjEf1Vy1hNYACSwNrCPRq09Pk6B/dHlJVbmixo8nC8Ey//cIW2yQabSacM32mw5atY0fnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rtxu4K68iSifp91eIigTtftCNZ8EYP0hKlTQRlFBAAQ=; b=LRjoLvN59UdSAgkNxZNd6HbwYsn1hjnB+aywgvp9vTEqQ7YPZt1iQ+roCGuKQVbnaB6DsXXKl25VKtDB9dh5JcANmeWgQzyVIB6PRExEz2q61CNQ4cYcTvcPremFhnfyJ9LqcwIyASBMzsavWz9KYJyX61+RJYca8+0SnLSzFsnxt21PHXp3uLj+LpIrh/0dj7oGlhXUb6P8HoQzAnX4X7EKYpNcbjZ2XFHOdQoQPUImUpbOkPYH+nKJd8DzCcbISFHHlKWjxp0SOx1XjvYrVrHl33xa/Br2hsu28wednsGJB7x3zoP2+uS68Ql3qj1EyTGkYoKAg/h4P63TZl5aTw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:29 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:29 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 12/21] tests: fetch: move npmsw test cases into npmsw test class Date: Fri, 20 Dec 2024 12:26:03 +0100 Message-ID: <20241220112613.22647-13-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: b2ca5c72-67b4-494b-b5c4-08dd20e9252b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: J69v0c547JXxFA/IEyXTFtHbN/siwYYyw2zPza8+y4YF0Z4FCMeYhvs1kE5ubTLS8ShtalqQZ6LsJA93j0owf0IG67Yuq1HOZQy/W2/u+kDsbJGf/xF4pbojJayD9eUHVxQ9e6Jfaft4Hwubb/yNsUoGDp/ygqNozjzkoXoq3HzKpsHcgEF0BIgoVO+JEfDCiA6hlmDy1hopI5Ml9ZkJMCdtNk0iF4p27PuhbDqUY4TXR49NqnoMMuQ5k6lR0WhiX46KiseaTWvFfobOBR7TL6JinY+na5OZ3ArTTPr0TG0QvKYNGP23+XKkVuMTjjFrFEHqdxskPe9EC6M1GBb9hSRBGafmrgD4RKT0t3eZgp/DCpE4cFmELSMuapOAMcGKIg3h1NLme9+JDB2kcueebwAT4bZRWVcui/fZqCqbq9E0HGXHxF73UuJCFn6TVoO3ATwu1lK1bXcDBnt71HGOmZQ2vx5uU+OTh6mQ4fM7E08tKzg3wpvvbLR86Ut3/Bwf0pseZW/SlXD/mQYcbRtWgYSWspe/vT6OBEmnnIReX+cKPjYwXWdqHnWwUnzV7YSTEWAn0CD+SmF8rU3kFb+Ide6Y09KlaZuhTwny92drCG+1Jydl7WJaAipfALSpBEdtUkWlf4fJmG6ErbXu5c2Y6BDNP/TLcnmieRdXVRw+N3datrC7H650TRvklRAShHPlmF9NKWjrUPusKvELZfqrs8tIeqozzcZIs1nquK9USX0woB1eHLF2eFdJ6yogxith6ZbbMpXW3ZEW6/oxLypyIEi93EM6i3a2vC4am2DSufV53N8iZCLWAH34qqjCpf5Dyjc+l4Zj/0LtNTpC/A1Hmfje8PnDX1m6iPuenxlOmtbRY/6iIXhlBhbEOB/M2si993NwObm2QcbzyjlX1yHhcf3ZiVgZQkOKAIarcOB31WZVtMqNuG82NobmH/Yz05h+3VfiHTV+AMOp+Uy7EgzVLs/oBj4ZvpNfJaPMo/aq4BKQta74Sc+plThDF1IafIChdK9aV89hhnynLsCnXxP88IX5BZL7NPcLCguxRPtqocJMvmz9olK5JX7ri5+5tVMwqDGnZ6alOOUy+Uzc+eLnm/S2KdPpUyiIBAR+C0Ty566pBFlBAH8YsHS7GVxKL3c7gnguDwqxP5dKvqTATpBzj4tAPoeYEkF5J9imbPJQhIV+Vj13bGFXOiATWNgbnAIIuyGeohqHXopena+lp054hj2myJndrBDkh+GW5NLUqPCy0xgbhweE0/WOjvb23/lrIPVuHf8T+eMPJOyV6AH+Y6abHjzYI5NGYrO+sqIB0l9D4bzUpwnjpAw10v+8oSoi6wgDNp98JTrT/S72Z7fqolnSfUC2pnZ+kkMT+oxkOa99W8CiYbaQMbQzzfpdMX3LWucrIYQdcDEMyZBbEcHas0BwaYuO/yeKev8hQRcLF9MuGyaEnggjOz+3s7y+9hzR X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: weU/ggV+xQ4Z/Jc5mrWbp/+dtnPVy+mknutV+p6thiBGoAkx72LAq7HUKOBrwVc4D1Fd710aqDg5QqtRkeDSEFvdZb7J1E0brMkUO+lvUFBEJwdHxHMAByRyFY0GAWcxkzUlUZWYQKFIspZ2LhzXNmhYw5uMM1RCRB2PkHqQnWl3S4HCF9CKGZqwOcMSgHxUfw05Ng5bz+tTIEo2N3qvml30yyLBHfdJLS5QH2/lYAUAOLyXcwTSxnuCfNlIXdcPY0+8I4Dfxxrhp5NhieCubAhw385sf+zixZRJAbxVAVoY4+cVXqbFJ2CL0V8CRI+ORpuIwtcgwxF0UxGqitBGLhtzNO9PO46IunIm2NrOuEC7FBAz3/ciZITLmaF4+kbr/DOQAXFnDUUJdJX8+COvlzJ2nhfq1VLT22De/iqMa5+qAhAy0VEvj0GDEh94eXxHalnQdZa51dRH4CLMw3zyHd+6LkROWbrJh7iLPBYSxbH9RyoiXbRRHO8YB3wnzsNXsQNEjteBxIV3LBtir8f7jIYGrbwVpxEE7grl2ZHako+OqNGGt1cCh91EEyUU2HtD/ccN8GNpF1S+vtSDucYcdz+k8RJ2FkETRD9qU4WNYApwkJEKE9T6VjjBUH2zHaBAtzrrUiECeqmF1k3NTrLj4cX9Id3NnVhRMK09Bv3+upWKsfcr2uXMatzQwHz520Fz/gHfFCpEANEFm/BgaFlIt7YhSdjqFo45OPPWAw8XiOEnzvbCmd7rugTPC0hY3+OXj6VP9vA2IqxGZmoHhJhknwRa6L9gDqygT/QcGJ5bK0BgniDYey4vx8nui+xZ7U3kD4Z7p79KQ/svCwsSk3hRkoLnldlovgGjCcvLnElyIMdzKZgUzRb530gAZu6Fy9t5R96mdigWvFNZKD0wWJu8Sf82bL5sVkkR8pmnP5zxLJUE07lVqtqf3msN5qmeq8ikUU1mMdwTlyo8FxqWkPlDgbvKt+3exZmbefx/syWtgZMlmZy+/xwRgXfQgyrFvXwBEF8DwzPx7gAgvqaj11t4Lk6bcVeX7CYE3DLt5RtkOzaHfDH95KgxXTk5VjGuzZoOJeY0TYabbwiBq9eHunxbpeRv19U8WXSyfDJSX3Qs6tjOpaTLAZARg3v44s5CiGX+4kVHGMit4sSiM8Bw4bjK/O/i3wFoxKCo4jUuT+mmZumOTbaQa/q8QOqRO84SDPJBuJaVnAMP5oV24k53hZ7VZjxcA+3sfkvJwiHMnEmue+PJFxaFKG3JrfPf9RUwvQVwj4YZmJCUW59L2MoEpbljpnXBvvAToqHLCjwrHMnUp1qN7IN5O089wmMHRblf4v1dJOpDYRP3Ud2Z0ztjXZP+e32SAjOUkxOX/w/tFfhsEGBrXsgrNYzW8UQBxZR/5rklipWS1jy+BQsccXknmiTJRcKpgqYmafU0FxW2psTTuOUrlCk0E2dDC8I/9ajm0WeerCPDm5lWIHpsE/+E28CVcIXvYpaIq7XrxRtHsuhrG+f3GwAUOrvsHuHai8Nihm0qnjVxuRA4mffVdHkrhTZQAZ11w5rYK/QHgJpZd7LosKY4PnVzUj8Fal6dq5M5u1QclSr+UYD327+WsIeLJdfbew== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: b2ca5c72-67b4-494b-b5c4-08dd20e9252b X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:28.1064 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: BmMq+UwIPZYsAVrkl4WWGeEbD0xl52vsQnRZsSOs/tPBZZEhtDpxbT0tGNT9NqN6OriCwAduy7QQTw901drltw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16934 From: Stefan Herbrechtsmeier Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/tests/fetch.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 934b96cac..3fd4c82cd 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2831,6 +2831,13 @@ class NPMTest(FetcherTest): with self.assertRaises(bb.fetch2.MissingParameterError): fetcher = bb.fetch.Fetch(urls, self.d) +class NPMSWTest(FetcherTest): + def skipIfNoNpm(): + import shutil + if not shutil.which("npm"): + return unittest.skip("npm not installed") + return lambda f: f + def create_shrinkwrap_file(self, data): import json datadir = os.path.join(self.tempdir, "data") From patchwork Fri Dec 20 11:26:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89844E77197 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (EUR05-DB8-obe.outbound.protection.outlook.com [40.107.20.118]) by mx.groups.io with SMTP id smtpd.web11.149744.1734693993154756588 for ; Fri, 20 Dec 2024 03:26:37 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=KMKEckgy; spf=pass (domain: weidmueller.com, ip: 40.107.20.118, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=j3Dnl7DmzD8vGawq0EKZ09r8xqNZ7vm5GJzoyQidkl4iqYW4txRCE4+3NBNyRRCjJEQUDPkPsfpXcoNfXYn8/UtDRwkza78wj3vmTKCRZdjohjLrie7vb6X/wFWBLGNetwe4gEhwmiDe970zgCqUXMYBZ8oZ2hFDctvzZ6MvzaFWIddXWMFpwolwekiagfuEsdiv4mJhcv/cK3dOxArgsKY3JNXHxKu51htTN7APoSE9ut6LTX9xTTiUzvr1N93xIAs2xPjD+LgweoPlq+0Lg3fi1uMdJaU/RTjIhlVxzZJc85wK1ofS1PEogMXBkBSnJ5i2NM20F0NA2jr3nh5nWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9i3bNDHqF84Ldo1B37JTzHr5cWaUGE5wPJDTGQw9/m8=; b=Rl3KFA4iLFte/pSJdIzgt+A+vFSjv2lYs9emXPRZU5ar83+cchRLgXEELs5Epb29JEtQaD9udrUxY5vHge9N1Th7IfcmGtQsrCqigI5MoXbo2s7/LMnOC3vfF+o5aad1BT2UX2YhmKvZ8OFc49lYxYSrzhR4BHJaBy8IfwOaAldbynW5CAtjfiUiUrdeei9176Rga9bKygZ2oOgGH4MHhwbiYnk0pb1/8drSPsY0SkVlsE4tXhrfBa61nkNFevDVSh6cFPKhlsUYOnIrEcC+sWnP4hCcfMA/CiaKazo7nc0jmThgyX6kF0QpnPg671eYWQISZtSXOUGZMhMQAHoPYg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9i3bNDHqF84Ldo1B37JTzHr5cWaUGE5wPJDTGQw9/m8=; b=KMKEckgyuxsxA+v4QCLE5n4oArpZ9+shb4g3szld+Yu1d+oreYWtzUojNeAM2MIirf6erxugtlCxPTz5ki2I3XP/ZBpO/DZg6sFYpYk0h7soufhLD9TxNq3EEk9NWTMmyOWHXGwIq5e3G+eBMetv94ZgUHACoc6dD6Sfk9n6yhQTPKQnx+2vF096VI+kGkN1y9Osh6BphLWncsovL65yrZI8R22KR9fpeP2SNYFd5zogfj/dq9mRlz7Pom4g84x2jZZjhBUiAL1iTgRtMBZPS0dM9pjRqOjeJnjIQZ730O+rkOMlwarK04nJ/nxNz97ZCKLnn9kxgV7IPGD7WkQnJA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:29 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:29 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 13/21] tests: fetch: adapt npm test cases Date: Fri, 20 Dec 2024 12:26:04 +0100 Message-ID: <20241220112613.22647-14-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: 9f0e3a87-07cc-4bf8-c2b6-08dd20e92561 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: ip+DCzS7HG1sxQkO3gA0IZ2fz22QtNnhW8uzzVzqbGqZc4R0k3HNi+UjFUkHW7E3V/UkmAU+zOWW4LmXVS4lvQudQTKXyh8dv1AZZblz4FDA9dohrA7CTQfzDm0nKyBbcG44nqPP84Bl/U3tIUFDsWOUV3WXJ3vB9E+RMZmd7YMoNOgxSW7fGTLiGiUvrMLuToinO3zIaSycsuv7TjYQpVM4GAi4uezItNqqK7IqCBcrQV+NNB0IYQpHf7HtfjA9+XVu3KYy0QtrGJKp47OyAQPx7HBgO9jE3uZ7lr3qHWavvdUPino+0qZ11iwt2A106GXtVSJvEUw5bPvx2Z+CrPZPL9ObkTausOBBvq3xuqAlK2MS+lWlxyh3d1fL77PYKchVNZ+wDVkrE6B8/nUB1YNPYYp1a/fCSgpQQAf7v5r0xBxOyxzm+CHGuvnBlZRVA9KshkjYIZTSpYhnOcoJCZsj9LBAR9ElstjbTUyKr0/BH3+INtd0RhOxP4CIhLcjYWpobd+guo9AAC0HNbf/ny6UhjaYWKEkD8rKioEfJEfYqFReEeenjUrVIzEgKXtov3aJcAImhrKcQgptusS08/S6nWplkulpViaVlVqvbWx7rkaRwq7rA8QAaFutVa18tvBhl/zWCOCd4+pn9xKgbGDbMgZEPX7nuaXjU2Bt1Uy8uSDbgvk+uFhV9BAQJke7r5StI5o8EKApUA/cWtuE1y4xYkn0PKhvw1Y1L6Ce3NUbWvzrzoFLsM2dJbQUbcb7sW3Sw2to7ED2bJIcWxRzAuuYzYfrtzYFS8QINqb+vE6OSSYYVRoZWgV4EUuAn2gA+NhWZLDWC2ISUv06kf5PCQwRta3Q1PjoXHzxKQxbGOQQzNWtMHJl20j4Zz9TXEfhF+ttZFuvNS0Q8pcVhmNy1A4NGoqrTj3zvRDr5vbt6HYi3ASiBeg2MspBAua6RTFEJvrXz/cUJCEdPzj7g2clKVPxZqcw8genFxLyqUN7qjqlptmsS3YQsg/npFZlgLj2yWYdgTwFs2FQIol86paRgcimsJT7jAc/Gqa8C3ZSPGjNhTlgBWV4L1Pr7xM3ttZfRruc2+Z+UmpD3r7xgJroxZQvShgNBI7isEmd48CrKVdIpWWUu8WxC38Oj7R3cN8i6y98yLr7INi4k6LtyT7r8iFx0CHpW32vqWDHM47fd+hcZo+yYGuZCLwpGV6dHRnWnbYmJNNXJgULZFx4Oo9zfJ5IEYIsZM2VPixFwHxuc5SNA+4qtaJF1bMKtLOBEX+RTOUGLQ2Z4XFq2ElgfJVWs5dbknx2i8xUaKDBOc96KaZSaJ2w7x3pEYoEkDSSCehZXqDPGuwLYmx4BexT+Bb6U3HMhu7ImzvXr/ZPLfPLkHUPdtVmM38j8sBUmaN4iVtT4G5PXCygxZxznkoI749iIA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 46CSJ1QWhVC7DfdTkWOuJpDdh6Vrll6E4fi9NYP6XIIZ5gLf0rQ7hCw1WAWOEdnj5z3QNBMUE9v9zgaa0pz3jku6Z6Cis4ziifsoVQdywJZ7gMIJGfmVatfTEvIJxnefPdjIOxI5E8I2ixgCjeWLD1GO0kK5/Jos8sPFjcs8zGhQ0IPtKAFB1ZVAaVQ8Pg0OLj7jmTYNKh6F6cwSovbR4t6M6Gmw+a8i9WlLLZne4RlJzr1aiSKxZOgRxHdJg6uR2xzJefWgyrW3JZlYoc2heSww3pWNV8NYM/MkW7Zqji/PrkAXAYDp8WLv3cVRbsuMvUkKcz/WlPNaILzng3yNyvSgLF4rudiouWsh5zWf9oYjLveE4cTze71gUVtpwwcQ+HLclijuopgUNlqBWY/tZqNeL16VXOHu1FTES6vSbZqvSN2cmujoOw7/2hv+C8dIMzsEAS7UEubUFi1MNyiX/MF+iuMkRNaGZHvAZON39AntmybbIXihblnXW7PjN64iP9dPMbX/pbhDyhTci4GuFHeSqbdx5ja7ZyFMvKyELJrbs4LVlOScYMMIVun7KyCmdPG1BIUqNijthl9Bbna14hRxYkTPMFZAbJFJesZ7RIIdhLS5kZVxlp3OAKLnQAkJGu1Z+jQ/domX9AI5gzPk0jMk+eEsDvYWc0tep4QwUPKhZwKYW1geqfu5nagwaD8kLDwC6CyDQgb0dkh2GbmCla7H6DkN9Hd0uTKYJ7cvq92urhdQTtdVhcyBcJ1XQeE4jkOiDPe/pHGAOZoQdbxb1hIeFd8sHlU4tLZEWhgxx3e6GJow6K0SW7RJ02zzYYHXvK5TizAYH1nzZD2oDQqxK0sG/I2WD4HNAPsjORGrJYGPmSNmEq6G+L8lYXIAg7GV3dJAsUtrmfstLAw9PY+8HePFv2nyDnjp7dCnD3J8hqxOO9dQe0QNHP+5eqb2E70INF97ExqcPZVB7Ng+rihaCJkW2T/Irg1mOO/wSCiVstTgJw3l5Sj3m4rfa88zJ/9SoUwPI2OcNcVGu+GpBGd93irrtdM4bBimLwH11szq5B9T5Ebkr7n0FN493UIi/0X6CY0woi9DB0zhmDGGnBj/mAP9qVrjoQ1FGfPNNdad1UAn71n2stTfhKRAhMtAdMDyJYXbp+E7jYIBaOiLngkrkZpHCtwcmlHHOVjwqwLzyJAFmcpP4q0UBMrZOYZ4nFb4GRdTC3Q5s6a1YSQatT2wdiFoKXBCA5VRFA6xmeN3TA6H5xPYLgbnufq1RV6f+4O+K7DEOsrVBsD6/fmrbgT1C2UH8mfZ7NrflYjMaEn1/2yjGO2iULtmRCl6FrwK9Jgx4ouat9/vEaQNzTko0jSz1os73fGNhVGYQdu9sJVsKFD7o/L4/YWIjrSsleNGB0N+nY1kQ9HnFFmIhZ5hC1kgAJ+2C4cDBVJdQ0qTAHfcyonokT9nbjulcFkNW7j43aYVPSCwhYbLUZPhX1MUQJPP5lhjtqKzSHdjF2UY9GC3QAE8iIhmT7g63O66HSzLBgFIGMzknKBe36Xn3uLOr8ZLq798qAAClr783VnUPfuY4A+j5NBjLU7wid+nSsyOarGyNK4oWZXMQgTtB9L2LI+tXw== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9f0e3a87-07cc-4bf8-c2b6-08dd20e92561 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:28.4098 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vRwRHHX0n6DStVWTBc5MumiFBdYfvspvR+N60X6fptIGhNZoos6BHijCdareD8Co5sGUro93oulnayJ50f1RPQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16937 From: Stefan Herbrechtsmeier Adapt the npm test cases to the reworked npm fetcher: * Add test case for latest version check * Remove decorator for npm binary check * Use common npm package for test cases * Define excepted file names * Remove test cases for (pre)mirrors, network and invalid urls because the reworked class is based on the wget fetcher. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/tests/fetch.py | 221 +++++++----------------------------------- 1 file changed, 36 insertions(+), 185 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 3fd4c82cd..09f493f8a 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -1507,6 +1507,14 @@ class FetchLatestVersionTest(FetcherTest): : "0.9.29" } + test_npm_uris = { + # basic example; version pattern "A.B.C" + ( + "types-node", + "npm://registry.npmjs.org;package=@types/node;version=16.0.0" + ) : "22.10.2" + } + @skipIfNoNetwork() def test_git_latest_versionstring(self): for k, v in self.test_git_uris.items(): @@ -1557,6 +1565,17 @@ class FetchLatestVersionTest(FetcherTest): r = bb.utils.vercmp_string(v, verstring) self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], v, verstring)) + @skipIfNoNetwork() + def test_npm_latest_versionstring(self): + for k, v in self.test_npm_uris.items(): + self.d.setVar("PN", k[0]) + ud = bb.fetch2.FetchData(k[1], self.d) + pupver = ud.method.latest_versionstring(ud, self.d) + verstring = pupver[0] + self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) + r = bb.utils.vercmp_string(v, verstring) + self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], v, verstring)) + class FetchCheckStatusTest(FetcherTest): test_wget_uris = ["https://downloads.yoctoproject.org/releases/sato/sato-engine-0.1.tar.gz", "https://downloads.yoctoproject.org/releases/sato/sato-engine-0.2.tar.gz", @@ -2618,216 +2637,48 @@ class CrateTest(FetcherTest): fetcher.download() class NPMTest(FetcherTest): - def skipIfNoNpm(): - import shutil - if not shutil.which("npm"): - return unittest.skip("npm not installed") - return lambda f: f - - @skipIfNoNpm() @skipIfNoNetwork() def test_npm(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] + urls = [ + "npm://registry.npmjs.org;package=@types/node;version=22.10.2;" + "sha256sum=9dad888e5280e9969393d7410e26d4edf726a828ee4762318c8ddf6fcfee793e" + ] fetcher = bb.fetch.Fetch(urls, self.d) ud = fetcher.ud[fetcher.urls[0]] fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - self.assertTrue(os.path.exists(ud.localpath + ".done")) - self.assertTrue(os.path.exists(ud.resolvefile)) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/@types-node-22.10.2.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/@types-node-22.10.2.tgz.done"))) fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, "npm") - self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_bad_checksum(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] - # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(urls, self.d) - ud = fetcher.ud[fetcher.urls[0]] - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - # Modify the tarball - bad = b"bad checksum" - with open(ud.localpath, "wb") as f: - f.write(bad) - # Verify that the tarball is fetched again - fetcher.download() - badsum = hashlib.sha512(bad).hexdigest() - self.assertTrue(os.path.exists(ud.localpath + "_bad-checksum_" + badsum)) - self.assertTrue(os.path.exists(ud.localpath)) - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_premirrors(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] - # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(urls, self.d) - ud = fetcher.ud[fetcher.urls[0]] - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - - # Setup the mirror by renaming the download directory - mirrordir = os.path.join(self.tempdir, "mirror") - bb.utils.rename(self.dldir, mirrordir) - os.mkdir(self.dldir) - - # Configure the premirror to be used - self.d.setVar("PREMIRRORS", "https?$://.*/.* file://%s/npm2" % mirrordir) - self.d.setVar("BB_FETCH_PREMIRRORONLY", "1") - - # Fetch again - self.assertFalse(os.path.exists(ud.localpath)) - # The npm fetcher doesn"t handle that the .resolved file disappears - # while the fetcher object exists, which it does when we rename the - # download directory to "mirror" above. Thus we need a new fetcher to go - # with the now empty download directory. - fetcher = bb.fetch.Fetch(urls, self.d) - ud = fetcher.ud[fetcher.urls[0]] - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_premirrors_with_specified_filename(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] - # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(urls, self.d) - ud = fetcher.ud[fetcher.urls[0]] - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - # Setup the mirror - mirrordir = os.path.join(self.tempdir, "mirror") - bb.utils.mkdirhier(mirrordir) - mirrorfilename = os.path.join(mirrordir, os.path.basename(ud.localpath)) - os.replace(ud.localpath, mirrorfilename) - self.d.setVar("PREMIRRORS", "https?$://.*/.* file://%s" % mirrorfilename) - self.d.setVar("BB_FETCH_PREMIRRORONLY", "1") - # Fetch again - self.assertFalse(os.path.exists(ud.localpath)) - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_mirrors(self): - # Fetch once to get a tarball - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] - fetcher = bb.fetch.Fetch(urls, self.d) - ud = fetcher.ud[fetcher.urls[0]] - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - # Setup the mirror - mirrordir = os.path.join(self.tempdir, "mirror") - bb.utils.mkdirhier(mirrordir) - os.replace(ud.localpath, os.path.join(mirrordir, os.path.basename(ud.localpath))) - self.d.setVar("MIRRORS", "https?$://.*/.* file://%s/" % mirrordir) - # Update the resolved url to an invalid url - with open(ud.resolvefile, "r") as f: - url = f.read() - uri = URI(url) - uri.path = "/invalid" - with open(ud.resolvefile, "w") as f: - f.write(str(uri)) - # Fetch again - self.assertFalse(os.path.exists(ud.localpath)) - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'npm/package.json'))) - @skipIfNoNpm() @skipIfNoNetwork() def test_npm_destsuffix_downloadfilename(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0;destsuffix=foo/bar;downloadfilename=foo-bar.tgz"] + urls = [ + "npm://registry.npmjs.org;package=@types/node;version=22.10.2;" + "destsuffix=foo/bar;downloadfilename=npm2/foo-bar.tgz;" + "sha256sum=9dad888e5280e9969393d7410e26d4edf726a828ee4762318c8ddf6fcfee793e" + ] fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/foo-bar.tgz"))) fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, "foo", "bar") - self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) - - def test_npm_no_network_no_tarball(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] - self.d.setVar("BB_NO_NETWORK", "1") - fetcher = bb.fetch.Fetch(urls, self.d) - with self.assertRaises(bb.fetch2.NetworkAccess): - fetcher.download() - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_no_network_with_tarball(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] - # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(urls, self.d) - fetcher.download() - # Disable network access - self.d.setVar("BB_NO_NETWORK", "1") - # Fetch again - fetcher.download() - fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, "npm") - self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "foo/bar/package.json"))) - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_registry_alternate(self): - urls = ["npm://skimdb.npmjs.com;package=@savoirfairelinux/node-server-example;version=1.0.0"] - fetcher = bb.fetch.Fetch(urls, self.d) - fetcher.download() - fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, "npm") - self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) - - @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_latest(self): - url = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest"] - fetcher = bb.fetch.Fetch(urls, self.d) - fetcher.download() - fetcher.unpack(self.unpackdir) - unpackdir = os.path.join(self.unpackdir, "npm") - self.assertTrue(os.path.exists(os.path.join(unpackdir, "package.json"))) - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_registry_invalid(self): - urls = ["npm://registry.invalid.org;package=@savoirfairelinux/node-server-example;version=1.0.0"] - fetcher = bb.fetch.Fetch(urls, self.d) - with self.assertRaises(bb.fetch2.FetchError): - fetcher.download() - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_package_invalid(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/invalid;version=1.0.0"] - fetcher = bb.fetch.Fetch(urls, self.d) - with self.assertRaises(bb.fetch2.FetchError): - fetcher.download() - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_version_invalid(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=invalid"] + urls = ["npm://registry.npmjs.org;package=@types/node;version=latest"] with self.assertRaises(bb.fetch2.ParameterError): - fetcher = bb.fetch.Fetch(urls, self.d) - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npm_registry_none(self): - urls = ["npm://;package=@savoirfairelinux/node-server-example;version=1.0.0"] - with self.assertRaises(bb.fetch2.MalformedUrl): - fetcher = bb.fetch.Fetch(urls, self.d) + bb.fetch.Fetch(urls, self.d) - @skipIfNoNpm() @skipIfNoNetwork() def test_npm_package_none(self): - urls = ["npm://registry.npmjs.org;version=1.0.0"] + urls = ["npm://registry.npmjs.org;version=22.10.2"] with self.assertRaises(bb.fetch2.MissingParameterError): fetcher = bb.fetch.Fetch(urls, self.d) - @skipIfNoNpm() @skipIfNoNetwork() def test_npm_version_none(self): - urls = ["npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example"] + urls = ["npm://registry.npmjs.org;package=@types/node"] with self.assertRaises(bb.fetch2.MissingParameterError): fetcher = bb.fetch.Fetch(urls, self.d) From patchwork Fri Dec 20 11:26:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A386E77195 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.106]) by mx.groups.io with SMTP id smtpd.web11.149743.1734693991344736713 for ; Fri, 20 Dec 2024 03:26:38 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=Q1oM/ffD; spf=pass (domain: weidmueller.com, ip: 40.107.22.106, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=bWX7NH2S3CURC3kerqcvDMFTL0HfmVIueamOVAh5o0t+TBT/uxcxWOYBVlfNrT6b5t8hmiSIRP023FthVViOLud7qMIR8D+GwfRvNEnIBOrM5RsyFJ1qLY6i4tI2PENqj3aaSNmhMKHIKjrfABMf3yd7/aWKxevrlbiOn+30zAnVvje4kANGWe/U4BsRZRVwpEzYcxCl4vUVU0bTN91eZGYruzj7JYOCQW8LNGyzfEySjuMZtLXOF2EwD7ySWq9YJq/YTrVP/Uw0zkao9Ec01/y+luYDdRMxtFrVsk57Y+Y9dwyeRjzND/MRM60FRYmyCYI+i/BuJKtSO99769Cxnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4XJfSBCTU3pIgNwk3SGlO6vmqo0NIaZAi32yAPJjF4Y=; b=HQ9ABuNjRMPJ4wBTU74UBbUKsNl3/2/HPZ1/59EdGMlCLOUZ2OPaOdQH8yEN8BwPhI5VgZJaJkplJkTFY1sRKJYKqlajaaD7xC1byZW154OtyvaBiRMXrzyg9EaFt6GKIPIJaYlwvY9CYaIsRYrltrsPRVniiu5fsvhN1XruKQfCE+5X5+7lECl+FXwwaeIfBOzo4wXrn06/JEZNt/bshSeIRzk9oJ9hsGCsH9jPL5hhyh8DWTjouaAx35za0z1R22//4JGkOj2s+82vZrahndamC+iOpCeCLNFejIHaj06uCThq1rGlfjs0lGC3kAZXKnwJRHrmYa/j3GaJLveaCw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4XJfSBCTU3pIgNwk3SGlO6vmqo0NIaZAi32yAPJjF4Y=; b=Q1oM/ffDTghqYq+3XCDR8O5WviN+lB4g0thClIviOwFzSyJwJXbevGcpP3INA7VDm7DvBq5ZaXLaBP6j8dm42GcqIE/kJMAQIJqISDBu56Y9hEwqyji64vBAxem2737p7ZnprScwjK5RGbWilkhNdrKQcm305RccAzjMbr3W6SCfuX2hVP+fmlv33hz/L5NXwdzOkUWOIy+tiNmgnbe5OLYGmYeiLYlfC4DB0J8nJrK5NswW1L9XrbItdlvR7uh3xGI0aWSUcOcu9LEXQ/INVcnCMImixIy7OG1BQ8Z/qoHtqpuF3oisRadhC3YAUwQ20hnJlHFOtDn+0XnMSTaCQw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8256.eurprd08.prod.outlook.com (2603:10a6:10:410::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.14; Fri, 20 Dec 2024 11:26:30 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:30 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 14/21] fetch: add dependency mixin Date: Fri, 20 Dec 2024 12:26:05 +0100 Message-ID: <20241220112613.22647-15-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|DU0PR08MB8256:EE_ X-MS-Office365-Filtering-Correlation-Id: 6980f98e-ef14-4bda-7353-08dd20e92590 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|52116014|1800799024|376014|38350700014; X-Microsoft-Antispam-Message-Info: ZSon4kxYYTYkzpVlQSaOZjfGBZe+BfnXpxVVYdxsQIhfc6hPT8NCRny1h6iIqca3fwCPfjru7Ps5hNsw94hC9qIed241kD4J657A1MFD3QkiDKAzGTtGGP0jsaAuru3eBuxVu/z+FXKZzdc8ov96BNkCzQd8DATDd/v0Z8lyTFCVVB9xAlNNQbdrIBE9wL6EOA+EXVFLKSatH97QuA9lAKglXaklloSxuN3+IE7JCFGTYKVRYWMyOSZh+zP6DQ8kW1syFScwm1CppCK/naO/aB7NhArh5O8hWLVwDSJAoZ5Pb580edqWudFCyrvMUovwOh78+b64waDpWEKf7hTKo8cLUjRiKpUAmLQzrUGlHCV8V15lTgAdUD8XdPKxwO4G43+D4IGiH4nTJePnMKz5dLgxwpv7KtoN7/WGiI66L4GjeqGh8My9ietX4q6xejb44ndlppaB1V4BlLabpyJo/+o1JVqa2/74RNseuieaD6zFufPQq971h/tcKhN5/ILOQlNWH33VC4qPMGDhYkT+Kq4xloWZDN5GTOpor5JyTxCrwBK45v+bEroe6Y5iell3fXfYQyIP2U7ODhj+gaHdTWCiYVOGzipSv5BTBlrUwK5shCbxFjJYVUK2jGZQd26GcRkD67WbuBMy0J+aBtqbD3XkWCvImjJOXtRGlln0/XuFOy/CJMIAwMILHGKcSMmXbeiSp9iQ3apxkrFoWS+j0Qp7ctCtvxxYl++4BhuJ3nUBF1Kfm5jGQwI/599I5oemffDcM9dAVgYYNpACIYvvK/+tQiWtK/o11xG4mgJB1oLA+N1JgY3ls+5cSbZrcagYaTpaA2PXMNJgwk/lNHJZAzXNWJ96J8lqwUuLcKL6eOrLsPNO6isF3qyVmcqD4kw9vikYjUMFqMB9sgy0FKayNuCSRst/C6ERdDZg/h/b/xd83n8XY3wagUXTMLQBkPtjk9iCS20Q2b3kerXsDn1VZHVQdweJPG+h/8+pwk3YhpngoeG216pyQVcYQXVPz0yK+Bweoot5y7vxC54h3ES8zhKIOlAPRiJ74d48Y0x+zsH5wbS1c1fg0YKNlfRrP4XjXcxyyGVY9IQuZEAlhEDNAbs+1s2Rcl3zgDm0o0cCSYSP+6GB4YK970es3oG028sUmkg9q/715RFhW0TovD2fe7OjwVJbDdoWCCf6aXYf1A2Rwd+OfNS75SkqVieYFaVxB2v9uRdmWhw8kUbWDn/VPzwAZj/iRQhJPcrgVz0cBhlHWRBFYUW3z5fXemTfLbgZWsXxotAs5RK5+vVcmizcgED6HQBfXxAKsPDeeiCELu4Eb2RkOmkxij2M8fOp0gyTlLt7pp40aE2zDVIu9dRApVFB/DQ842+xfO/stdE6pvue8XSDSJe9j5t9JMDn+btl X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(52116014)(1800799024)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: c43t+4Dyc8R6Psw4EC3xjnXpdK+2QaY6kmw3vEmnldeOv0dtbtfxx907ZTOGParllZh5FryCgrTTs7lxM2GdQrqBkcyY49OBgIuCOdAEWUhN9WymUZ0V7TJo5ta16+tZhqsVTPMKaysOJUtkigA9WqcH3uXslXipSPfJvLYxL8B1a4HWmEUjje6vqk4ckXuSJdh0l+sTqtGyYuUfNKx4OKSEUo65/+clMSW7rOPL8+LzGW84dP5ivV/TcjQO7DmbcOIzOPhgf+t3DdX5NmDSg8oUqcyks82R2y1fZb0lFYKYB92GpfYt5pYx6WJsoos/u0pDOYQqtHUXrZDc7ZtDXEeotlNWZVHq2RowmKulldF40WGWvj+aWb3PZZQh8lChoJA+eh19QZMTCMQKBq9ng+zA0gcP39ZXr8CLsFYciyOcNNWtd9CUd+bdE3foPg+b0f9wsUt9M16M2Lh2uJTldkIS6XnKzJO7AI1MyJuYH+Vgi7srm9hWKJSY9KEaABoD2IKy22/HIizuCfw6jV90EKCbgpOzjwNSd9n7I4+ckgQcw9nOoUZ/taVDTbQZFCJjn6vFN+4zDpbHgU4GF98XRDBKzNjbRcO2YV1rNSf0bFNSK7JPimpTh/L7i7MSwhmhDhMvEkTNJwesbD3BIKnzapsA5HyuwdzkOo8ntpixI8GNQ0zlS2x+hTanWplFsy0POUXrCDKPpu32lmZCNXqBSBbXJjhRbjl3k2Asa+fmxvCapeNy5wb5hadM6zg4T+e+UB4FLXDZnnmXb3hbDCXPieZ/pFVQvzfE73+itj2At98tI1LR/YirXMtyKDnQPSHRAipoANS0abMrrRKW9+bMfHllqwzrH1/dQ01f9zd7sbljPAzv40KOhzXLjSfaZ/V/WZAQ4v3kfUG2Es2S9Mj7vD2lLyMrJ/s/4seRd2zMQ8pRY6mVL/vhUbYvsl4IH8p7OSQrl52m3VegJyUwAx7ulY8BomdfduMQPiZ3/pfqjXpNhU7D7NeqGFX1dmEJO7Dg/GYBjenNBngB8oXMPeb1JoiYKwvHRr5oS+d/mSHbe4JVVCfEXxx728yNIast3NXySgv5lUO4BSZLBtCevbE0ZYxkfhRvfv7fHXfhZqCDdRkDKOEsOGTVLjqGLPfAAJWAkw5JC/2CIuzOiyCnLUl/S1aH+Y7Db9vZXxmBKtZ8iGBswsqL2JjWPmXKTnkTuHFUx3SYXNA5Gyhh6rzjbigRc7YPExkA6A4Ct6bu7OveT9C1J28I9Bdsm0lSaRiPS4nfXrhD/O7lcq7ejRmbdqRw6tpv+S1hDZA/hEkhczTWFkUQKLQPYXkpOdMKbSqFFcDj0zjBiYv10u8zbSMQcC+dF96i/2T/IE+5n6rxqiArHXO1kveYnoJP4k9xOrOTBitoty/dTnsE4zotKjNELneSPe/yWW5VpmrYdPqJfcDHKSoMiLlFRyVcmwKmAYrrF9hiKbmoMqVMvzPtnu0CaKmn8EOcnaOBAOhxHGXjnnwtmQ9HODsBjnm+I0emgcCcNVzilB2LFtkphtrQT3y2Q5EfWyLp0ncGpOyt1E+8LGH5ynUBEaj6CjlCHMkSNqQIEA3bNIjm8W2XlnhVGMjhzPzv/g== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6980f98e-ef14-4bda-7353-08dd20e92590 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:28.7559 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: V3sX4qq4q/+8u3BKT3+9HOZ1CoZYixplfqadoJnQ0/Jme4ONuxPnjrGJt815FLcbP8IU2ZAVeNj9ckqgjmatKg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8256 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16938 From: Stefan Herbrechtsmeier Add a dependency mixin classes and module. The mixin implements generic methods to fetch dependencies via a dependency specification file. The subclass must implement a resolve dependencies method. The method resolves the dependencies from a specification file into fetcher URLs. The module provides different fetcher subtypes: (Local) The fetcher uses a local specification file to fetch dependencies. SRC_URI = "://specification.txt" +https (Wget) The fetcher downloads a specification file or archive with a specification file in the root folder and uses the specification file to fetch dependencies. SRC_URI = "+http://example.com/specification.txt " SRC_URI = "+http://example.com/${BP}.tar.gz;striplevel=1;subdir=${BP}" +git (Git) The fetcher checkouts a git repository with a specification file to fetch dependencies. SRC_URI = "+git://example.com/${BPN}.git;protocol=https" Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/dependency.py | 175 ++++++++++++++++++++++++++++++++++++ 1 file changed, 175 insertions(+) create mode 100644 lib/bb/fetch2/dependency.py diff --git a/lib/bb/fetch2/dependency.py b/lib/bb/fetch2/dependency.py new file mode 100644 index 000000000..4acad8779 --- /dev/null +++ b/lib/bb/fetch2/dependency.py @@ -0,0 +1,175 @@ +# Copyright (C) 2024-2025 Weidmueller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +# SPDX-License-Identifier: MIT +# +""" +BitBake 'Fetch' mixin implementation for dependency specification files +""" + +import tempfile +import bb +from bb.fetch2 import Fetch +from bb.fetch2.git import Git +from bb.fetch2.local import Local +from bb.fetch2.wget import Wget +from bb.utils import lockfile, unlockfile + +class DependencyMixin: + """Class to fetch all dependencies resolved via foreign function""" + + def urldata_init(self, ud, d): + ud.type = ud.type.split("+")[-1] if "+" in ud.type else "file" + ud.url = ":".join((ud.type, ud.url.split(":", 1)[-1])) + super().urldata_init(ud, d) + ud.proxy = None + + def _init_proxy(self, ud, d): + if ud.proxy: + return + + urls = self.process_source(ud, d) + if urls: + ud.proxy = Fetch(urls, d) + + @staticmethod + def _foreach_proxy_method(ud, handle, d): + """Call method for each dependency""" + returns = [] + for proxy_url in ud.proxy.urls: + proxy_ud = ud.proxy.ud[proxy_url] + proxy_d = ud.proxy.d + proxy_ud.setup_localpath(proxy_d) + lf = lockfile(proxy_ud.lockfile) + returns.append(handle(proxy_ud.method, proxy_ud, proxy_d)) + unlockfile(lf) + return returns + + def verify_donestamp(self, ud, d): + """Verify the donestamp file""" + if not super().verify_donestamp(ud, d): + return False + + self._init_proxy(ud, d) + def handle(m, ud, d): + return m.verify_donestamp(ud, d) + return all(self._foreach_proxy_method(ud, handle, d)) + + def update_donestamp(self, ud, d): + """Update the donestamp file""" + super().update_donestamp(ud, d) + + self._init_proxy(ud, d) + def handle(m, ud, d): + m.update_donestamp(ud, d) + self._foreach_proxy_method(ud, handle, d) + + def need_update(self, ud, d): + """Force a fetch, even if localpath exists ?""" + if super().need_update(ud, d): + return True + + self._init_proxy(ud, d) + def handle(m, ud, d): + return m.need_update(ud, d) + return any(self._foreach_proxy_method(ud, handle, d)) + + def try_mirrors(self, fetch, ud, d, mirrors): + """Try to use a mirror""" + if not super().try_mirrors(fetch, ud, d, mirrors): + return False + + self._init_proxy(ud, d) + def handle(m, ud, d): + return m.try_mirrors(fetch, ud, d, mirrors) + return all(self._foreach_proxy_method(ud, handle, d)) + + def download(self, ud, d): + """Fetch url""" + super().download(ud, d) + self._init_proxy(ud, d) + ud.proxy.download() + + def unpack(self, ud, rootdir, d): + """Unpack the downloaded dependencies""" + super().unpack(ud, rootdir, d) + self._init_proxy(ud, d) + ud.proxy.unpack(ud.destdir) + + def clean(self, ud, d): + """Clean any existing full or partial download""" + self._init_proxy(ud, d) + ud.proxy.clean() + super().clean(ud, d) + + def done(self, ud, d): + """Is the download done ?""" + if not super().done(ud, d): + return False + + self._init_proxy(ud, d) + def _handle(m, ud, d): + return m.done(ud, d) + return all(self._foreach_proxy_method(ud, _handle, d)) + +class LocalDependency(DependencyMixin, Local): + """ + Abstract class to fetch all dependencies from a local specification file + """ + + def process_source(self, ud, d): + return self.resolve_dependencies(ud, ud.localpath, d) + +class WgetDependency(DependencyMixin, Wget): + """ + Abstract class to fetch all dependencies from a specification file inside an + archive + """ + + def process_source(self, ud, d): + with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir: + Wget.unpack(self, ud, tmpdir, d) + return self.resolve_dependencies(ud, ud.destdir, d) + +class GitDependency(DependencyMixin, Git): + """ + Abstract class to fetch all dependencies from a specification file inside a + git repository + """ + + def process_source(self, ud, d): + with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir: + Git.unpack(self, ud, tmpdir, d) + return self.resolve_dependencies(ud, ud.destdir, d) + +def create_methods(type, mixin): + class SpecificLocalDependency(mixin, LocalDependency): + """ + Specific class to fetch all dependencies from a local specification file + """ + + def supports(self, ud, d): + return ud.type == type + + class SpecificWgetDependency(mixin, WgetDependency): + """ + Specific class to fetch all dependencies from a specification file + inside an archive + """ + + def supports(self, ud, d): + return ud.type in [f"{type}+http", f"{type}+https"] + + class SpecificGitDependency(mixin, GitDependency): + """ + Specific class to fetch all dependencies from a specification file + inside a git repository + """ + + def supports(self, ud, d): + return ud.type == f"{type}+git" + + return [ + SpecificLocalDependency(), + SpecificWgetDependency(), + SpecificGitDependency()] From patchwork Fri Dec 20 11:26:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D50AE77199 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.114]) by mx.groups.io with SMTP id smtpd.web11.149745.1734693993938884026 for ; Fri, 20 Dec 2024 03:26:34 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=cvhNR8lt; spf=pass (domain: weidmueller.com, ip: 40.107.22.114, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=YNlEXEl42tqWUwhchRlGNZcTO6M6ui2m7xcFik+MPrzy09kf8xbeQYUVh/6cCfkULHetXRvuly3IbZ8eEDxabfHONCyg9O5Z33olfv3+7x9ucbkBJ4pmo9m96Lx2moQx8+05a9Gmy/Hfd6F7mRPu6zOWI6Gt+rt4iZZMhxvzgwBVHwf5fTmqHlrnEhL6HRzN2nEKQjAhPpjjpBoYu0FtsO+E7FrPM4OxtFrTR+EurbzkNilYpfMTwUbTKjFPTzD1gkClEVE+k+pqElhbjODj0n9tKZoXKWqVapZE3urYOhjArHrxXjuq401zQVc22vNCBviV8qsQJBqT2w0nQSGgNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+iZHi9BXyNE33KEPe5zk/Om8Ucv7r91bk7xCrTG1BUc=; b=dknPsFrEWlz922YkI07mv3aQRaNsKQGpFKulVHNRqtaUoq5uwFubnNaLX31/R9MRCmViCIZTnZz+TX4HEGB+/UqvoX5Fmn1HjXg+llHdUNIfJdV+r9xxPqNvT+G8drsCxRTNCRbd2M/3YpSmk9798RnSWoz/AjI/8Pi2IQZpBatZ0xo4cn0vfgd2wy7h51vglCAgEnsj/ULWLd2GhU5m7//EGx3kL4ERQkjniUwtuNM0R0yugCtvS+MQCRywAb8sDDfjoJIcUjhsz50K8waTT1wPn1R6HdajbDJ9EnhLuttRIAwg1aQ+pwkGaXIpHhlpKo8zw5O/Bok0u3LaRg81zg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+iZHi9BXyNE33KEPe5zk/Om8Ucv7r91bk7xCrTG1BUc=; b=cvhNR8ltG39O7w5UBM4C2prhO3Mh4WNMx71fFQ3YW+HnPbLD6v8Oexnh5uahdVc7wGs92Z6qt5QnXHGVHswE4cTXmpVL3VZPiJPp2cdKTyXAgqOYdqjeIW8cV/x03H5tzQOJPKnl1UQIg3Pwf3BreLoaWZ464Y5JPQKGJL4OhO8YsHQ1Y+4aXBro753gmYTy56gsbuhisNqihs7dLKeVLUiaQyWoTrZxFpA6WX72VCcZ09AqYe41yuPDq8W/NXLpWE3sw1kgW4l+B6Sb1TsubCVcprlr2ozEd+spuVS/EbEMq2jLxJBhYLidP0IBlzmLZ6c1NRve8OqsC4prLWyf0g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AM8PR08MB5684.eurprd08.prod.outlook.com (2603:10a6:20b:1dc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.16; Fri, 20 Dec 2024 11:26:31 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:31 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 15/21] tests: fetch: add test cases for dependency fetcher Date: Fri, 20 Dec 2024 12:26:06 +0100 Message-ID: <20241220112613.22647-16-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|AM8PR08MB5684:EE_ X-MS-Office365-Filtering-Correlation-Id: 2228ee38-7348-41d5-4870-08dd20e925c5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|1800799024|366016|38350700014; X-Microsoft-Antispam-Message-Info: PRgSGQNP9krG19JMx85IHsXkT3dErTXLqbbA1nKLrQV6bQ78+TYGu9JQL/0LcxGBfUfD5dGZc5OrLlR86jy6u01g4ovWhB9mEJHtWGuz2kujOJ7jAeJhGVlXkxIDopA9yXdNV7QHlMlkjmXiTDsXYx1MHMMJRvJyceHb0jJLWbxeZFd1k+1FJ3gXOaz7seFxw7QVFrgtnD4o9DzIQBZvOPDBS5BgWsVZUPgsRuzv0oDagCz5GoMlecFHsKXhGkKQ5yHJ14fYaMjOwttdPux4UHuwjlJpp3Ji32o2OZb2ekLKXI2hBXzb432INtTExr9wryFPsggbiy+9jnczfAEl/LoT8XsLW8QgDae6LOAAkPWzSXXDHWCtABGtjX5LBUXPcJ9s8EfzcU1n/KvVW6SJSN9lTP9Zt6bWh1vDoeSOcIpMn6n6ScYKm69NrkJ/ZOoW71YDR4txYj7bWv0L4MdGC2D/awFvZ2Rx/WZJFFirsLKeiAcF3ep+9VKN0Etr2I4TMF0bV2aZjAfhk23tpDXaVbvlZaH19Rv+9xU34kUOYGsn0LkxSy85eHDVM1iyYQQ9aNB7ENo3omZdYZR89tRVF0QfuHfOJ2wwSqGRdaiqelY8Hcj+PW9IpycL4oKDtR/fqlLhEptLTD08PDfmmcvWp3KuRI3F5XVZr4ucPH1X6U0GZnK9H9hY4A4EsiXRtYZECkY9cWvvsnypKoFHr8oB2Ivxx4Xkgrkf0Eo4Jc4rEvktRF+yOm2Nrv6t/cfIncVqeHa8vnX7cXrZLBiixYPmIG/yXFipij5OIZ6COfB6mX2VOkRhZLTTSUHm1emiwajG7JTqsh/hwHvSXoPmnsVPz2CQ+joYsg7tk2p9HpTuOC7UPn9e9KHbc9JUE/bVEpbkIGKkJZ/2joGYWE6q+h1xM/JGW2mC9ImmeaC12A849b7GkewpF66Fgtu/Aa3GFmXbmW0sUvBw3NqbyZAWhTIh4xb0b4ikehNAnxb3AfFwTYCrx/D192ErQ0crOfSXX7CPF/czGqA8BcrUwdvEh87R8ak7aBRLhqBONUSTVXsGqPMrXJaYrgV2Yh2NoYTnOWJEugYHyUB4U3bT2QkA2numsTeY5857wyKhEVZi0S4Q+OyuCTyYQCaZGmsAJ1md0kRNsqHnopb3A+N0ssL1VJ4reYHc7+rI9aNwZVnVY/icsoWFjK7z8zSoTPJIsXfXDhLk00BecfvSAasKISclc1BVhVYBsLLEZU0zmdqZiuXttypgLujnlw/PEES8+ARwZgj7PCYx/nQ6IEZ/q3ivLX3RBX9sMcus1jlzCGWlPzpzHLftXKKx7gP9c9oHeROcsfN77yXoAzFEgnbp8sIedg094Da3czEuFs3wgirmbBrxO6ivYRgvnE+PS55F2M+dfxDl8O5c+x3T0kCKdkJS70Rerg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(1800799024)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: HbM9eY+N4pEj7qZH4WgJDvMHzSEoz6VYpkMgr5DWDTiC3mDAJriXiAHkniwAEv1KdqvjE347J5AjGzkc7Fi0dNp0xR3IkNBVths4aAr0vtY+mMTwYcGkDsm1vZETfyJ94l3hdAdOkXuG8tp9B+znOfxNI1zo6a4FmF7FPfweF3Hc/oEGcsApXucs7F39+sk/AbFYhfs1Uc6RU4Sd7DuR5oTsw7pK+Iqj8tkeVhReslsioQygkHW5mCLoXkd3KDH13128EK7umTQXFjgFVjwfUcCrNUWnjehH3wKPG91gPLhWBgt8eyQNmRaVyjNGw3dLzTDXPsanlnZsP1fSFa55CCNTmXJ6xzX0mdsDQMRUyS/LHRB8QDa4D+1FJjSZMECKUkyNQUcJxjYDUHPAdJ1fO3B9UXChVrI8UC+4xv0cY5wvYbnjARtq5fcM8Qqm4lT5tnHh88YTvT9UlQLgVap4qnAKc6Zyhxj6tyPlsRg2Mylv737uORli5JVyZNfNJmIiDUdrq7lIyznF5ZWlQJWuQKsZDZ25qBReUsdg8hDgl2x2YS3IcVxUpno45dP63SCBVsZhovIWG3OXeXaA9uEnCxP4ksoORpoCeJNArFb4EwtM4f2XFxt5PKqZsIRFmPLI266EzvIKmLrOjF+p0HmqinvIqeoVWsQpKH5Fy92pElhfGVwRosQpGmKIm5N4AYndJBmz2AI6q6jsEU5CObKOcMA27JfiTJDu6njr0Vn2fbTB907dG9wt+1cBTyT9gh2oXw11HxT9Bk3xbBzkHHfSucPNodO/xj9/NWRWzPhdWSmt1htg6TVXi32scRRFsGVe49BHztW26AbYtLm4LUbi6yQA8oEUV857HXMBuGiyWIPTuF2iZCzp640iuWBz+UkruEe93DbTypQQhK1hvf7tVbh2zGMXqk2H1LKnvcE0mBoA7A8SEwBVApv53u8IKALfHSyD71rl50eZX65JxXm0zzpuciFZkC+uN1F2ID60IemWKwHqwbg/GgZwW8ymPHhCVPloyXyVP0Okm0NHdlZ5lrG0j4YPTuoIpWPxq7nmjnc3Xey0glu+Dc32vLHe8GDeqRf6iQolxwrd1EXCr1V8i0xaAuoMza5wA49034UUq4RKakv5QpCep2Wild0gSC6YRznOmCWBNDIC0RjAonO83FFUXUMbvpY+qOXesup79Ysbq71NmHNz+L6QYF/5k/xZyARCu/5trl4ph9r53W80+/+03Xt/9iOSa0fo9XDd+QbaT3xAY2yw6PgrOC4oqyE1BmUlRHxWxhT2kZbRQYxWV9AR9O8jpzxqSVU+cSC46PeCQyaDPbzVFRdilLxUqv8yEpju5kNMRSzK+VPW2veuqgeV4miUc6F4T4gm79RMhXbf4ZH55o408S7rh51m40uB6yfxBxzVLgESD20i6zZnsPaWt24lo7caU6e/vOONSpZHOL8MeAKFVeTT0Vgilc1GI3qqbNuE1FQdMsuQEF1vhmzgQZSGUieTzUbTcjm85eymrJ7N33cAu6vbb2U1WPrgTrkARK0ADhuvKEjSqDYxMwgFv3fNYr2x5UNmfPZ+mAxb/qLzrRdgyeQf7ZrFFObzQCHx4KD/eJxG4+Qrn4S6tQ== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2228ee38-7348-41d5-4870-08dd20e925c5 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:29.0692 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bhXwoM+wOZAqOu/CAzqOsvamQvloA5R6+tI0bj91DTZCUZcAkOa7Ogd1TQKEKSY+MDLxCdR0EaVC0oM700wsGA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5684 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16926 From: Stefan Herbrechtsmeier Add test cases for the dependency fetcher. The tests use a dummy fetcher because the dependency fetcher provides a mixin only and isn't self-contained. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/tests/fetch.py | 121 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 09f493f8a..903095746 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -3443,3 +3443,124 @@ class GoModGitTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod'))) self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod')), '0dc9ccc660ad21cebaffd548f2cc6efa27891c68b4fbc1f8a3893b00f1acec96') + +class DependencyTest(FetcherTest): + class DummyMixin: + def resolve_dependencies(self, ud, localpath, d): + urls = [] + if os.path.isdir(localpath): + localpath = os.path.join(localpath, "dummy.txt") + with open(localpath, "r") as f: + for line in f: + line = line.strip() + urls.append(line) + return urls + + def create_specification_file(self): + dummyfile = "dummy.txt" + lines = [ + "https://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", + "git://git.openembedded.org/bitbake;branch=master;protocol=https;rev=82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40" + ] + with open(os.path.join(self.srcdir, dummyfile), "w") as f: + f.write("\n".join(lines) + "\n") + return dummyfile + + def setUp(self): + from bb.fetch2.dependency import create_methods + super().setUp() + self.srcdir = os.path.join(self.tempdir, "src") + os.makedirs(self.srcdir) + bb.fetch.methods.extend(create_methods("dummy", self.DummyMixin)) + self.srcfilename = self.create_specification_file() + + @skipIfNoNetwork() + def test_dummy(self): + self.d.setVar("FILESPATH", self.srcdir) + fetcher = bb.fetch.Fetch([f"dummy://{self.srcfilename}"], self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "bitbake-1.0.tar.gz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "bitbake-1.0.tar.gz.done"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/git.openembedded.org.bitbake"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "dummy.txt"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "bitbake-1.0"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "git"))) + + @skipIfNoNetwork() + def test_dummy_git(self): + self.git_init(self.srcdir) + self.git(["add", "--all", "."], self.srcdir) + self.git(["commit", "-m", "Dummy commit"], self.srcdir) + rev = self.git(["rev-parse", "HEAD"], self.srcdir).strip() + urls = [ + f"dummy+git://{self.srcdir};branch=master;protocol=file;rev={rev}" + ] + fetcher = bb.fetch.Fetch(urls, self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "bitbake-1.0.tar.gz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "bitbake-1.0.tar.gz.done"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/git.openembedded.org.bitbake"))) + archivename = self.srcdir[1:].replace('/', '.') + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2", archivename))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "git/dummy.txt"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "git/bitbake-1.0"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "git/git"))) + + @skipIfNoNetwork() + def test_dummy_https_file(self): + archivename = "archive.tar.gz" + sha256sum = bb.utils.sha256_file(os.path.join(self.srcdir, self.srcfilename)) + server = HTTPService(self.srcdir, "127.0.0.1") + server.start() + port = server.port + try: + urls = [ + f"dummy+http://{server.host}:{server.port}/{self.srcfilename};" + f"sha256sum={sha256sum}" + ] + fetcher = bb.fetch.Fetch(urls, self.d) + fetcher.download() + finally: + server.stop() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "dummy.txt"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "bitbake-1.0.tar.gz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "bitbake-1.0.tar.gz.done"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/git.openembedded.org.bitbake"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "dummy.txt"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "bitbake-1.0"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "git"))) + + @skipIfNoNetwork() + def test_dummy_https_archive(self): + archivename = "archive.tar.gz" + projectname = "dummy" + projectdir = os.path.join(self.srcdir, projectname) + os.makedirs(projectdir) + os.rename(os.path.join(self.srcdir, self.srcfilename), + os.path.join(projectdir, self.srcfilename)) + bb.process.run(f"tar czf {archivename} -C {projectname} .", cwd=self.srcdir) + sha256sum = bb.utils.sha256_file(os.path.join(self.srcdir, archivename)) + server = HTTPService(self.srcdir, "127.0.0.1") + server.start() + port = server.port + try: + urls = [ + f"dummy+http://{server.host}:{server.port}/{archivename};" + f"sha256sum={sha256sum};striplevel=1;subdir={projectname}" + ] + fetcher = bb.fetch.Fetch(urls, self.d) + fetcher.download() + finally: + server.stop() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "archive.tar.gz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "archive.tar.gz.done"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "bitbake-1.0.tar.gz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "bitbake-1.0.tar.gz.done"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/git.openembedded.org.bitbake"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "dummy/dummy.txt"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "dummy/bitbake-1.0"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "dummy/git"))) From patchwork Fri Dec 20 11:26:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EF75E77188 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.114]) by mx.groups.io with SMTP id smtpd.web11.149745.1734693993938884026 for ; Fri, 20 Dec 2024 03:26:35 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=P+iHf2fr; spf=pass (domain: weidmueller.com, ip: 40.107.22.114, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=I5TuVP5KcR3LSQg7ovFexeWEqDvcRc3ZTdhR2YHannIUVELTvQjxbv+Qqb3w5GeVlCAX9kMVJAmB2WKVnMVjtV/zvLMUAnIE8P5iVNdLI81NrIiXjABNwrc7CxjQKZJDMqBWAzqPN56MtwH2ltkvU8Zg4FABer/BxLsSltFYzCSd1E04sOsrqU46BPXhP+HG4+mQyxklx4oOu7vScwfQ4HMnthZz+Op0splX7SwFEB21m1dk49WaugUi4ZkKjXgd7mBYIHgqp/6NJDMbii3ILY+GZBYIfAOgWVw4VPbl8Xjh+OIRWpCr9WCkeH1KASR4zF9TX4QontrkiaAe7xtTFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vvdvJvJB6+N3H9ozFzVfup0kjThedpLfqngTHxqOgM8=; b=g6LufJ/ONd9fqQEvF8S7/khtP/3oEbUtzbA9eW168TWKKK/ph7bDhgfSedoRFWdysjzaKArKV7ONoPCLM0TreVn9N20FTimjNLqDMC8Y02KLxeSGRpykRHrPypb/Yz33b08etHgeWYuV9PsCRc11GLbfT4LxY+Hn7fyWvyZIGe+zJeOVU+Am7DptkdPEOrqkRRpBCcdnf8u/+BDiquL1QJPRu1uAAKdAK3qg4O3UtTfsNtJWlhHTZxGLp0XLv9zXHolKuuZg0olRKI4aCscmaF7/o+KekSG8USQ/vSLjV+29XOlZ//D62kltxSBJA1v0mkf0zLH+DosiuA5KOB4TiA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vvdvJvJB6+N3H9ozFzVfup0kjThedpLfqngTHxqOgM8=; b=P+iHf2fr9UPNRyhKh/aVEM/iIa5BqeTtkZBZmv2VKA/U0rHBJ6Anwa344xEao5xdKosck4SeYXcm0zbrZHYjdamUxsI1bcVON8RExsH1A40R5F9ELPm/liq229RM0X0PCQDty7HgWCtinbQsk+/c4Qh5Od1PO1tU153EtIG4rXmfj3MpYIDq/fucA+IjKc4+ijDoLz9YB73sKRw3N+Vp22tn2h+7cp/S7eKsgp9IYN71owIHK6sN3cuU7z8tNaXIqT1kuoOFLd68jBkoABhibH8/Qxkcn1lSth/u/WZTJ0Uht0liwYVVlLkb8IzhnfiNErK1i/5F3Pvv1BNADm6n+g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AM8PR08MB5684.eurprd08.prod.outlook.com (2603:10a6:20b:1dc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.16; Fri, 20 Dec 2024 11:26:31 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:31 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 16/21] fetch: npmsw: migrate to dependency mixin Date: Fri, 20 Dec 2024 12:26:07 +0100 Message-ID: <20241220112613.22647-17-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|AM8PR08MB5684:EE_ X-MS-Office365-Filtering-Correlation-Id: a389a305-b96e-45e3-4b1c-08dd20e925f4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|1800799024|366016|38350700014; X-Microsoft-Antispam-Message-Info: y3huxe38ffn0XP53fvlVsBCIcJ3CagXN+uP+anoAOuJI6h4BudhqyLdZ/h4xBgPz4HEVjxqYil7xlgmH/PlLn2q+is0GnGhpKl3Gd1U+klKzRE5JF6UzX8o2wgW1dheCFfy7mNVDLjDuD5uD8ti/3FFGiYXH9ryE2WvzrRMus57X1QnocE3sf0OyLXRye4jKSbCj4GRlUDK6StPu0G3He0UL6s5ZxdjnpBrVZEf7lkB4zE2Dmwv0NVokYDDJMptIYOpNKsYXhoJ8rMZCFMZxrMt2G7j3W/vgrHQOGFIB8qxVRJKDOKt4kFe5b0Tg96r1a7+34Pff6Qfcmp756dBVyGRli+23WOU8+sGp8sC6YeFinIL+k+G+Jktdrj13I1ESgCGFDAuQEy63fuu/4wCAXc194FThuSkt6DJsPL6D/5SR9bN2zvw/N/0yjAHn/+NV9VQHXKO1o59Kwl6pL8CoSuj2vopISLlULEkVlKI59BaER3FNsS/k2EP2opyGyZdjxLHrG08qowjVsnR1jc33H5wNhPZ1OibKjkNu+aKUDtmTD7JFWprLVILedmhisILIkr/MNsR0e0xj05QsRX6q+GeTpbIRFbWUl4I2Kh8ntMwhVTeED2mCp9hrKLfzam75eiOmISyyQ6Wt3wVDQI5Ix36WupK3q1MYlne3J+tndUxF16OwQhQzdKVhWUauGIPV+MeQNxAKecXJ9j2D0Y/bklo6fQgdH1MgiXpvI+mXJ9X3zuVWbqHRbLGuSAtnZu15ImVBZeffDz05VLbcYlTownNHp7zxp2r9Ina+QNIfMu7cRPRcENKgT/t8asg0ZLViuY91sfMNGQcG2zaLLRqdbPlz3hFtUUHDJNc523lgm9bSFfuM0QFeQc27Qoyqxg/a9brFIQIU9vMEyq20dYc9EKvqRb/DKaTd8FR2IZ/Wb6zQePjIuAbT5FcG67hQy2fpP0k8wrx0zFIVUStIWJJnjLpOE3PKQqsBZcYrCfqlsqlqQSpLvfLqy5pkoeitXamvO557frioxNvvO7lGbPqcHdASLg5Ko0+dcaUYv8xlXIDIsGFvMmv+crNnyWU17VqOanuBP7Ms0JuzeqpPT1yBUwlf4dOlI5o12qgFrrW+zjMjFOI8wv3V7UF2rrroB2XcWlq6/3bX4k/mHtGPp13L10E2m8xVeeRW2MrEf6/qDbt6cu8GArtbQkRFNe4VR4glfv7/Ko0rl3K9WDXfA/7CXLfcDApvWc5ADofWe/x0poVHrRo6b7V7GVZt0omUX3IkmQcTZUhZ58D006pPm3avdoT9Yv59ntFaMMTlQHmtJ4+jvVwNrqjddCzr8Z7sNPzQVeHJRXqeJ87GZ0e87Zj2QXP+ciyQeIlKL0xxJABkVzYkLf/A54spIYELbBD3fgW2 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(1800799024)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: cPEEPENEtWnx/xpci+Yq6517UqGbNzDS+LJuDxWWtBQ/+JPOu8/I4RSl1ENfef8Bpuk1p4wDN9o6bX0oZ11pxv+EvFASUzimyQNjPc49yg0OjVC1GVy/DPexvBxzHbu3hMTn5/b2VEt0Nlmc+PJ1g/MY/9UKnDtXDAvXLpwXzpAWyagK7PIEFY92eyjmPA3QonXT+XeB98Qr3gM2CesTQtVmKZfIlOsZk1fYXu1jgE1sm3UnGsh0yKLfLNhBv6tnlpxl70ybG8dBun9DO3O/uEVP6FNLJMtaB1wHyfUB2PnKaNmNb9fVbckgl+I7FUBUcKj+balTmKv3JlUuegt2L43Bylc95H+oWp5VE+iEqeX2fzvT0XVoTJLkLJAeze49ZY3p6AitZktDYwTTM/Y8FubXZGDBnjbCMwZArWanvLxS4Iemgor6M7ezPg0vSJZuGbYlOYPwPH1bJ7WJpA6u4ibRqWG81PdUjfdZVHR5Nj1670b2x00Gwj2jygtO8Q8weo+jGV+dsw06ueLeylO57ixZZR/RnPb+RmijtHXcfhfn96gpFSjyNlaLp0VTf5sFB8aM38aVCh6PX2EaDIpZEbOCu8vzsubunSlNFiHedVNtQGdKDllq/1QCKIgvdw8V525zkYY+20LPD9nOAaY1EMG211boopdHj+DV8JDDPuBtfDrUVgX5KC3LlVHIrPdbdjgYNlOA3JzD6YD5LwTdBKxSIOkfUTtXDMCLphZHOjdDi/nM7jcHSYdA8gLuZuEdNwOBAc2nLGqqASTo0mveaA3XDXMBbm+yqmC99yR33DeJAyJ2CG96am2ToLarBPuk9OXqNM7aQuPb3RkB0jN/S/VJPUiXiHIsllSfhr+sYNXDa8DCoT0m2K/yDLNDZtOVCjJjLW266L8BpCWggSX9fp5h+ulDsjpOG3RqSlpnBlQtiOMlrTNEeF2CNBtoowi9B84iUX+hWAf6yD53ogdYC4kupzKT9JQ1UqE1++wLM1+G1oMMrUVr3UdkS7LxmZzGIPY2TyrYk2B8/PVX9+S2Ggbf2vmF4a6Dt1EI+Y2IWr6VtOq8kqr7ucrQNe/Q1Hs+n/BKG08UlxYQHaH7EvBhmbLhMMzsZXQLBaCUsMON1CBSMCjF6RQrmAaWgvQNFUHsRlEKOFP4nVgMiVOHThHWn2rFcPXIhExd5EJyIUgewyDrhhvqrNWfBvtEINUbNQec+TkS3iqyZu7cgnnt47A2v5gsPE/DSCxkxHEFr6GAmmZ6jq0ufLORsl0jG2BXiA5hDEuetH+3TbF2XkAl+YPVvCUfH3RLZ3l1ypbnBQ/pPQBSZ0tDcspRZhX9VWL2EO2hFvoFNV8wQ/6GAnKgZpbcewX/+Iy6Qy9Bx1qCwaXpiJDYdACyVIRqT7RHWtPmofwqQ/DGCOjwa2dYRTh7jK3lHgz+Yh1LkhP1PXlXJyWj3PbOQjNJhx6R9wCxmUqmIqxjVHuk2rOM3IYJB+GKzvJAPbFikwBsej1YdgSWTh3cgEZ9QwYFSTWcuWXLnuvchjQ4ig2mK33oEbCm/6UZmYM99CKfuZlHyGuR/WztTGYW9Af6J1fjFxlkWjxvV0n+eod/fFBGHFb2430sBPAjv3/XSg== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: a389a305-b96e-45e3-4b1c-08dd20e925f4 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:29.3882 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: LCAfQvzwLeJ8O5xWz+LREBxYJITMOGdarNbuc8kmje8kfGsAPWhSE92GNEqqq0cM2lIe3c9XLENpBc/IUT20VA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5684 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16929 From: Stefan Herbrechtsmeier Migrate npmsw fetcher to dependency mixin to support different fetcher subtypes. The migrated fetcher fetches dependencies via a npm-shrinkwrap.json file or if missing via package-lock.json file. It supports different types:s npmsw The fetcher uses a local npm-shrinkwrap.json or package-lock.json file to fetch dependencies. SRC_URI = "npmsw://npm-shrinkwrap.json" npmsw+https The fetcher downloads a npm-shrinkwrap.json or package-lock.json file or archive with a npm-shrinkwrap.json or package-lock.json file in the root folder and uses the npm-shrinkwrap.json or package-lock.json file to fetch dependencies. SRC_URI = "npmsw+http://example.com/ npm-shrinkwrap.json" SRC_URI = "npmsw+http://example.com/${BP}.tar.gz;striplevel=1;subdir=${BP}" npmsw+git The fetcher checkouts a git repository with a npm-shrinkwrap.json or package-lock.json file to fetch dependencies. SRC_URI = "npmsw+git://example.com/${BPN}.git;protocol=https" Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 2 +- lib/bb/fetch2/dependency.py | 8 -- lib/bb/fetch2/npmsw.py | 272 +++++++++--------------------------- 3 files changed, 66 insertions(+), 216 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 3a7030bf3..5dbc0598d 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -2134,9 +2134,9 @@ methods.append(osc.Osc()) methods.append(repo.Repo()) methods.append(clearcase.ClearCase()) methods.append(npm.Npm()) -methods.append(npmsw.NpmShrinkWrap()) methods.append(az.Az()) methods.append(crate.Crate()) methods.append(gcp.GCP()) methods.append(gomod.GoMod()) methods.append(gomod.GoModGit()) +methods.extend(npmsw.methods) diff --git a/lib/bb/fetch2/dependency.py b/lib/bb/fetch2/dependency.py index 4acad8779..e30d7fb73 100644 --- a/lib/bb/fetch2/dependency.py +++ b/lib/bb/fetch2/dependency.py @@ -46,7 +46,6 @@ class DependencyMixin: return returns def verify_donestamp(self, ud, d): - """Verify the donestamp file""" if not super().verify_donestamp(ud, d): return False @@ -56,7 +55,6 @@ class DependencyMixin: return all(self._foreach_proxy_method(ud, handle, d)) def update_donestamp(self, ud, d): - """Update the donestamp file""" super().update_donestamp(ud, d) self._init_proxy(ud, d) @@ -65,7 +63,6 @@ class DependencyMixin: self._foreach_proxy_method(ud, handle, d) def need_update(self, ud, d): - """Force a fetch, even if localpath exists ?""" if super().need_update(ud, d): return True @@ -75,7 +72,6 @@ class DependencyMixin: return any(self._foreach_proxy_method(ud, handle, d)) def try_mirrors(self, fetch, ud, d, mirrors): - """Try to use a mirror""" if not super().try_mirrors(fetch, ud, d, mirrors): return False @@ -85,25 +81,21 @@ class DependencyMixin: return all(self._foreach_proxy_method(ud, handle, d)) def download(self, ud, d): - """Fetch url""" super().download(ud, d) self._init_proxy(ud, d) ud.proxy.download() def unpack(self, ud, rootdir, d): - """Unpack the downloaded dependencies""" super().unpack(ud, rootdir, d) self._init_proxy(ud, d) ud.proxy.unpack(ud.destdir) def clean(self, ud, d): - """Clean any existing full or partial download""" self._init_proxy(ud, d) ud.proxy.clean() super().clean(ud, d) def done(self, ud, d): - """Is the download done ?""" if not super().done(ud, d): return False diff --git a/lib/bb/fetch2/npmsw.py b/lib/bb/fetch2/npmsw.py index 2f9599ee9..fffb2a102 100644 --- a/lib/bb/fetch2/npmsw.py +++ b/lib/bb/fetch2/npmsw.py @@ -1,37 +1,35 @@ # Copyright (C) 2020 Savoir-Faire Linux +# Copyright (C) 2024-2025 Weidmueller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier # # SPDX-License-Identifier: GPL-2.0-only # """ -BitBake 'Fetch' npm shrinkwrap implementation +BitBake 'Fetch' implementation for npm-shrinkwrap.json and package-lock.json -npm fetcher support the SRC_URI with format of: -SRC_URI = "npmsw://some.registry.url;OptionA=xxx;OptionB=xxx;..." +The npmsw, npmsw+https and npmsw+git fetchers are used to download npm package +dependencies via a npm-shrinkwrap.json and package-lock.json file. -Supported SRC_URI options are: +The fetcher support the SRC_URI with format of: +SRC_URI = "npmsw://npm-shrinkwrap.json" +SRC_URI = "npmsw+https://example.com/name-1.2.3.tar.gz" +SRC_URI = "npmsw+git://example.com/repo.git" + +Additional supported SRC_URI options are: - dev Set to 1 to also install devDependencies. - -- destsuffix - Specifies the directory to use to unpack the dependencies (default: ${S}). """ +import base64 import json import os import re import bb -from bb.fetch2 import Fetch -from bb.fetch2 import FetchMethod -from bb.fetch2 import ParameterError -from bb.fetch2 import runfetchcmd -from bb.fetch2 import URI -from bb.fetch2.npm import npm_integrity -from bb.fetch2.npm import npm_localfile -from bb.fetch2.npm import npm_unpack +from bb.fetch2 import FetchError, ParameterError, URI +from bb.fetch2.dependency import create_methods +from bb.fetch2.npm import construct_url_path from bb.utils import is_semver -from bb.utils import lockfile -from bb.utils import unlockfile def foreach_dependencies(shrinkwrap, callback=None, dev=False): """ @@ -58,40 +56,31 @@ def foreach_dependencies(shrinkwrap, callback=None, dev=False): name = location.split('node_modules/')[-1] callback(name, data, location) -class NpmShrinkWrap(FetchMethod): - """Class to fetch all package from a shrinkwrap file""" - - def supports(self, ud, d): - """Check if a given url can be fetched with npmsw""" - return ud.type in ["npmsw"] - +class NpmShrinkWrapMixin: def urldata_init(self, ud, d): """Init npmsw specific variables within url data""" - - # Get the 'shrinkwrap' parameter - ud.shrinkwrap_file = re.sub(r"^npmsw://", "", ud.url.split(";")[0]) - - # Get the 'dev' parameter + super().urldata_init(ud, d) ud.dev = bb.utils.to_boolean(ud.parm.get("dev"), False) - # Resolve the dependencies - ud.deps = [] + def resolve_dependencies(self, ud, localpath, d): + urls = [] - def _resolve_dependency(name, params, destsuffix): + def resolve_dependency(name, data, location): url = None - localpath = None - extrapaths = [] - unpack = True - integrity = params.get("integrity") - resolved = params.get("resolved") - version = params.get("version") - link = params.get("link", False) + integrity = data.get("integrity") + resolved = data.get("resolved") + version = data.get("version") + link = data.get("link", False) + + if integrity: + algorithm, value = integrity.split("-", maxsplit=1) + checksum_name = f"{algorithm}sum" + checksum_value = base64.b64decode(value).hex() - # Handle link sources + # Skip link sources if link: - localpath = resolved - unpack = False + return # Handle registry sources elif version and is_semver(version) and integrity: @@ -99,193 +88,62 @@ class NpmShrinkWrap(FetchMethod): if not resolved: return - localfile = npm_localfile(name, version) - uri = URI(resolved) - uri.params["downloadfilename"] = localfile - - checksum_name, checksum_expected = npm_integrity(integrity) - uri.params[checksum_name] = checksum_expected - + package_path = construct_url_path(name, version) + if uri.scheme == "https" and uri.path.endswith(package_path): + uri.scheme = "npm" + uri.path = uri.path[:-len(package_path)] + uri.params["dn"] = name + uri.params["dv"] = version + uri.params["destsuffix"] = location + else: + bb.warn(f"Please add support for the url to npm fetcher: {resolved}") + uri.params[checksum_name] = checksum_value url = str(uri) - localpath = os.path.join(d.getVar("DL_DIR"), localfile) - - # Create a resolve file to mimic the npm fetcher and allow - # re-usability of the downloaded file. - resolvefile = localpath + ".resolved" - - bb.utils.mkdirhier(os.path.dirname(resolvefile)) - with open(resolvefile, "w") as f: - f.write(url) - - extrapaths.append(resolvefile) - # Handle http tarball sources elif resolved.startswith("http") and integrity: - localfile = npm_localfile(os.path.basename(resolved)) - uri = URI(resolved) - uri.params["downloadfilename"] = localfile - - checksum_name, checksum_expected = npm_integrity(integrity) - uri.params[checksum_name] = checksum_expected - + uri.params["subdir"] = location + uri.params["striplevel"] = 1 + uri.params[checksum_name] = checksum_value url = str(uri) - localpath = os.path.join(d.getVar("DL_DIR"), localfile) - - # Handle local tarball sources + # Skip local tarball elif resolved.startswith("file"): - localpath = resolved[5:] + return # Handle git sources elif resolved.startswith("git"): - regex = re.compile(r""" - ^ - git\+ - (?P[a-z]+) - :// - (?P[^#]+) - \# - (?P[0-9a-f]+) - $ - """, re.VERBOSE) - - match = regex.match(resolved) - if not match: - raise ParameterError("Invalid git url: %s" % resolved, ud.url) - - groups = match.groupdict() - - uri = URI("git://" + str(groups["url"])) - uri.params["protocol"] = str(groups["protocol"]) - uri.params["rev"] = str(groups["rev"]) + url, _, rev = resolved.partition("#") + uri = URI(url) + scheme, _, protocol = uri.scheme.partition("+") + if protocol: + uri.params["protocol"] = protocol + uri.scheme = scheme + uri.params["rev"] = rev uri.params["nobranch"] = "1" - uri.params["destsuffix"] = destsuffix - + uri.params["destsuffix"] = location url = str(uri) else: - raise ParameterError("Unsupported dependency: %s" % name, ud.url) + raise ParameterError(f"Unsupported dependency: {name}", ud.url) - # name is needed by unpack tracer for module mapping - ud.deps.append({ - "name": name, - "url": url, - "localpath": localpath, - "extrapaths": extrapaths, - "destsuffix": destsuffix, - "unpack": unpack, - }) + urls.append(url) + if os.path.isdir(localpath): + localdir = localpath + localpath = os.path.join(localdir, "npm-shrinkwrap.json") + if not os.path.isfile(localpath): + localpath = os.path.join(localdir, "package-lock.json") try: - with open(ud.shrinkwrap_file, "r") as f: + with open(localpath, "r") as f: shrinkwrap = json.load(f) except Exception as e: raise ParameterError("Invalid shrinkwrap file: %s" % str(e), ud.url) - foreach_dependencies(shrinkwrap, _resolve_dependency, ud.dev) - - # Avoid conflicts between the environment data and: - # - the proxy url revision - # - the proxy url checksum - data = bb.data.createCopy(d) - data.delVar("SRCREV") - data.delVarFlags("SRC_URI") - - # This fetcher resolves multiple URIs from a shrinkwrap file and then - # forwards it to a proxy fetcher. The management of the donestamp file, - # the lockfile and the checksums are forwarded to the proxy fetcher. - shrinkwrap_urls = [dep["url"] for dep in ud.deps if dep["url"]] - if shrinkwrap_urls: - ud.proxy = Fetch(shrinkwrap_urls, data) - ud.needdonestamp = False - - @staticmethod - def _foreach_proxy_method(ud, handle): - returns = [] - #Check if there are dependencies before try to fetch them - if len(ud.deps) > 0: - for proxy_url in ud.proxy.urls: - proxy_ud = ud.proxy.ud[proxy_url] - proxy_d = ud.proxy.d - proxy_ud.setup_localpath(proxy_d) - lf = lockfile(proxy_ud.lockfile) - returns.append(handle(proxy_ud.method, proxy_ud, proxy_d)) - unlockfile(lf) - return returns - - def verify_donestamp(self, ud, d): - """Verify the donestamp file""" - def _handle(m, ud, d): - return m.verify_donestamp(ud, d) - return all(self._foreach_proxy_method(ud, _handle)) - - def update_donestamp(self, ud, d): - """Update the donestamp file""" - def _handle(m, ud, d): - m.update_donestamp(ud, d) - self._foreach_proxy_method(ud, _handle) - - def need_update(self, ud, d): - """Force a fetch, even if localpath exists ?""" - def _handle(m, ud, d): - return m.need_update(ud, d) - return all(self._foreach_proxy_method(ud, _handle)) - - def try_mirrors(self, fetch, ud, d, mirrors): - """Try to use a mirror""" - def _handle(m, ud, d): - return m.try_mirrors(fetch, ud, d, mirrors) - return all(self._foreach_proxy_method(ud, _handle)) - - def download(self, ud, d): - """Fetch url""" - ud.proxy.download() - - def unpack(self, ud, rootdir, d): - """Unpack the downloaded dependencies""" - destdir = rootdir - destsuffix = ud.parm.get("destsuffix") - if destsuffix: - destdir = os.path.join(rootdir, destsuffix) - ud.unpack_tracer.unpack("npm-shrinkwrap", destdir) - - bb.utils.mkdirhier(destdir) - bb.utils.copyfile(ud.shrinkwrap_file, - os.path.join(destdir, "npm-shrinkwrap.json")) - - auto = [dep["url"] for dep in ud.deps if not dep["localpath"]] - manual = [dep for dep in ud.deps if dep["localpath"]] - - if auto: - ud.proxy.unpack(destdir, auto) - - for dep in manual: - depdestdir = os.path.join(destdir, dep["destsuffix"]) - if dep["url"]: - npm_unpack(dep["localpath"], depdestdir, d) - else: - depsrcdir= os.path.join(destdir, dep["localpath"]) - if dep["unpack"]: - npm_unpack(depsrcdir, depdestdir, d) - else: - bb.utils.mkdirhier(depdestdir) - cmd = 'cp -fpPRH "%s/." .' % (depsrcdir) - runfetchcmd(cmd, d, workdir=depdestdir) - - def clean(self, ud, d): - """Clean any existing full or partial download""" - ud.proxy.clean() + foreach_dependencies(shrinkwrap, resolve_dependency, ud.dev) - # Clean extra files - for dep in ud.deps: - for path in dep["extrapaths"]: - bb.utils.remove(path) + return urls - def done(self, ud, d): - """Is the download done ?""" - def _handle(m, ud, d): - return m.done(ud, d) - return all(self._foreach_proxy_method(ud, _handle)) +methods = create_methods("npmsw", NpmShrinkWrapMixin) From patchwork Fri Dec 20 11:26:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66B2EE77190 for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.114]) by mx.groups.io with SMTP id smtpd.web11.149745.1734693993938884026 for ; Fri, 20 Dec 2024 03:26:36 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=bWvwy3aL; spf=pass (domain: weidmueller.com, ip: 40.107.22.114, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XT/8M/2cfZP4CUWOkpjxTCLt/ZrriMISP+lqDcKxl8R+lgQx6o74ToUpWXR+pBOpGVMllUq48WfQ5/q3kHwtIoszqVuwv8f1WjXOaZRoFvDK1WXDeg8m5creZukDfYfFGqpn48vy3lYgA1ZqoDm5ab/k9JuWBogilOoDfJocLAXq/DXg9mfGFo98ETOVwqvzFrYI3WuOybdsJfzRQv9uT1priUcC6f/iAFFlmhoFheoxt18Nq3bAgIwAB5rz/Iu+J8IdpH/e27YKIzpCB3nR+xXqEruR/uoMhBeGOpbAO+gitZ32bi6myyUBAA12Cgl119ixFtPp3Umh7ugyLsD8mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gF/1cmfyQar4RmGSIuctJzlIqWaTxq4P++PXjMRTXrU=; b=yVqxFq0KCZ7BP6dZNmeIedB+n1GxGny90erSgHoxlOXYT5OC6iSkWs3YfaojGhZlqGkY7MVRD9bjl42YFHpqRsKjlKLQpm7TlkSBs8n4rtyCvUT+w2LY/5ltE4uJJxg3uQsr/qYO1BgC5C+BA/V9S2Iyv86nFZc3DfSduhPu1vtF2Lj9f4BjyHSmXTqUp9ImnPw+drpIzaelIa+4goQCB00qrUqfU+/12KrVpnx0Z/Ggbv96LyK1TCijPuapXzx4s9SkczpOZmv27RlIoQOT4gbcGoPgXn9N1Jzk7QjbkWQclgPF+W5odGLdyMnQMDYMO6qBPUjW6KeEc8WkFTq7Vg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gF/1cmfyQar4RmGSIuctJzlIqWaTxq4P++PXjMRTXrU=; b=bWvwy3aL5cd5/errXLNJ4He3KGgxHdBEdb7TL3DQBn/n3Mw7kQP2mrblBy2yWAC/ZoEx1sIv36KOj32mK07bME93mtnUwntnxNRm/XF18vYHCQY8P78LBatyUo3pN3AxlJEczgz685uBbIqyfM7apUhrWXRiUiShn1Npf9htBO9JedxGhxKY2bSPnzydxtg2XZHwGtmtuHmLocPV+P8wqAwwWDPRlwHVeiePuAUWSGaSx2JJaXpennzS9fopM9qv3lnyrS7CRUMcSyEAWAf+rTIlXLenYzA12qENeVGVTEWM/3bqTIVAAvFE44ZD83OSpohM5Yq8w1IUskDqQiiOCw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AM8PR08MB5684.eurprd08.prod.outlook.com (2603:10a6:20b:1dc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.16; Fri, 20 Dec 2024 11:26:31 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:31 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 17/21] tests: fetch: adapt npmsw test cases Date: Fri, 20 Dec 2024 12:26:08 +0100 Message-ID: <20241220112613.22647-18-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|AM8PR08MB5684:EE_ X-MS-Office365-Filtering-Correlation-Id: 8d1481bb-bf7f-4a4a-eb8d-08dd20e92625 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|1800799024|366016|38350700014; X-Microsoft-Antispam-Message-Info: fVxrD1iLfWeaanXQQdEAGy4qv0G35PWwDDYCr4+kv+BI8H3OlveVvpJ0t+kW3TejxZhQjtgS1vjoGOAvYaHtbIqKz0Gh/mPq2LKdHWzjE/hIVtgp84Gi/2+azPQT6IahHndJEnt5VRgjLZZIsg4pjJLvRZW941vKZeSBZyy5G1bSXuZScH3mkAuwDFJeh8AbaaBWXd8mRdavwDyfiWHqfXrHYm5ytlckrP1PzG9XJBj4VsRMAih+zEt9XWOA7SrizGSL2YoKJd+pQ4B1pmn0LldBsNpSyV+oE1rvRWpfEMWJOf64bB41GasqpRgrQX/0ADMjKWmVHbMoXjEJJc5qhfUXKT8ZnbEG0FASXKf57YUY8mH+m5yxqkLHL+ZTzzl0GMcbBMqD58n5GGY7VXNYxKu7Aunc2wcCT+YPdFf3qtTS0i4s5+w5lc4aTV4WfJ8x3jzXhvpeQrSap/P+IX+MTOY0tC3PVORlU/tQbMu0nQxMc5ME79/Liic61d0Cxl6PcnbVjZlmG6jUmuiHCD8Ro8lHwfwzDRqGiVrnmQ/BmGdmyY1MXJ7IG2oc/bwBzepFUnWboKWwnNSc3OYDc2sNf1uiDUAOJq3lapr39P4MGh+Ht2qdobaI95jH0YNrh5nm7u9a6jmzRDWlLioMA7pgin9dcYAbA6f3rgEiFr3fIgAkHffCA5O9vTMGbVygw8k7IubfYl0pv8pRhrGeOB4PO8iL8IQGJA3AUyNNPZ77x2+7op3V8JEQKGW/HcHQJWeBR2levLQcFfRxUY1yXLPbkHV6oWzwu44p8IOXxKiJJcy/O1gGl7o337io7zt8gO2uWOIiGpYW6E/5HKAktLge2QPhSC2cn2vttoZz/Djz2FjgqfONDq8Prw4p32cR8Rz2e/S8kz9mPgRdbN476nAuklibSXpqgCKkRqBoZR6Uxo32sQRGW3KMyL9Pbhj5nyHYSfOCQ0rwkXEA87SLzJNCOWQ5ky+hFUQYi1k/ZjClXMNg+42uzLs8f2xmmMJfk+mpAOKCp7W4o1H5drv/V2o5HCZqIbBOScpfKgnm8ERnMAUMzN3vWn6d0cK+80ggc8WAF98oVq++biNP0X+TRqdMlG7I0s6fl12cEOfM8Pi1K6q6M4w+uIZ7oftwx+W0DTH6EtBuXdBuu3nBacp+c8A1uWHlFxbd7wztYAeVdNnQc2KK9jkA+8//RIXgt27v97J14r8f6Uy456RlV/eBtDqeMl35CXAzEtF5NbdpJIecmX7QyictOPtaDwZ6uPctJZLOXwCmtfx9NhhVSBoC2I9RqzdrX71ctwzeoPhLaXUxru4EEkFRPZLnfkaBAersnU+4iRVwNAxtw+T5MRYYdYLaaW9k8GPMOne5K53sPsKbuM+gxrlGkI2Oh2M7aGcftSSeiW90zzsmhwBFBKHQ9AdjTQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(1800799024)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: tzCeWBIHKc1GDHON7ficET7MP0Oq4X2OcVziPq/LFzLbWRpy4ZmQawFm4n4NpLvwijlkgqNqHz9Kw5NmXzljiLFjN8R1YtqnlaCAdposWXvfbEE2GGzeHRZZSfc2yaDn7j0/kwsMgOmW+wvkuVdgqfeBZ3aprp3GtljBQP4+azwBU5bgoYxuOGUcMgTEEhm2IRc9UYnXnEAkVqIc4TYOlq/u9d/g3vvRh7qLOkEbfXSPMI80Z+qQUf/ys0yktT2/9Hw6LTpYAUkqFXM78mB3RULyjsdtUeyCNr303LZk1hkqve9Zz78zru+Xd89KEcqoUVNLoca9ljL8iICGCbauEmoRRyFzodCW7Yy/FbeOV/aPBlUwTmH3z9nPpBtKP/RyAXTeCU6pggytA6I2fD90niwPPqf3vM+BpXlv4fJUaLzmmzfdBJw/65puC2tpGd20i8eZe2sTi0/xNlBs+DJSP1ZuncZn4vHvN8xVBloXJJ7hnfow20LAP7oDc9zaL62SLM+O6c7K4MI1keVVuqfyZZWyRWr0NoqUqzobKsdLYejHKQA75+i2r9brbBb47xIgMfXaj8E6ELrWzvuqHIwogmdkcyZlHnTAJDqTnzJ+xzXJ46vZD1JzlKx8jWHxgXTTMuUm+FEubNNUMjPVulsviE8QtJj4pXqlHNMSBP54pmDVFm3HEFS6eaSeN1nYtLp4ngc3soeZlfEJaaN/CHCvJ7W+rv70zaBmixOI6A37C613+dlovkyti+SGUAFsvOJ9Rd/MSMwlXrqrFeHn1sLry51PBmbkpxNpXEiPBxk/WMTAMaEQzU73vvp8CY3CfXdlFfCxy24g/38LJP8bCZ3M+ON2zsyivP/R3F7Zp/teGjnrKvnmPKkrNvhEORESLbRcs6H1VlmHwao6sgGE+1CMi0U/9Um0EwecXf5NBk/1Mphf8UAWmWRtu48WOwKxLJzjalrZw0XUsLbEUIQMv7zRij9U32JXun6CRVDoVYdj7rLZKQuEF5k+SSFTTWcoQjUEHwYUbjUJnG+mWGrAUfqu2wNOIqSvYQhbRLshlTURLum+KmlsTbObTmTZ3zg+SXq2u2bjMb81c5Aw+A2rmBcZ7jeQiPEwhRRg9BQwCfEHaguaYwepAr/ze0wWTMj4mF0n7u7pExD50xXueAxBjeJBJmi5w/8nnGJXm9Ra+CElRsh+0zFfuSOhGZJjy247BaKJdIvYYho3O5qpQFyjAMX22S8gjXHjrlvRqDLKsG1zuVdKJLULTirFo4zjx+h+yLNdwzTYqWcr2BHIObU9YLvehVQXGWj6HDhhSYrJihOmmDKD/txGzRNFqS0YqmCklVcsjcpGz+OrLtwxAka4LMGQedDU9Lt2CMPm1Cu/htiZGSWU6KxU2jPahL7ldba47JofHN2kRsa/jaeFpcDdtdrH2hnlsVBGKW5PX9khbeG29RE0cE4jxg9JwIyZ0U7amEXWlQHwqzaJZwr14pvFPOa9NQLvk5zMcUM0SFnjKT4qonAEIX93zqeVvcoRQW3r60qkJM7FbLuHxhtmV4rvHxHtCTzU1yO0y4YBkw8ZYO1gUwDfnH+RUGA9PLGfheMEJ9EuycmjQQ9wRRh1sTUz2r57cw== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8d1481bb-bf7f-4a4a-eb8d-08dd20e92625 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:29.7171 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cGXIuOfo1mwCe2q3RcVETkgdJsnqarlW/C0I/2QNoPyvbXkl5xBpsMyjM8nrWHFoSSi45yNlq1AQPJWXA2UjGA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5684 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16933 From: Stefan Herbrechtsmeier Adapt the npmsw test cases to the reworked npmsw fetcher: * Remove decorator for npm binary check * Define excepted file names * Remove test cases for (pre)mirrors, network and invalid urls because the reworked class uses the npm, wget and git fetcher. Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/tests/fetch.py | 255 +++++++++++------------------------------- 1 file changed, 65 insertions(+), 190 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 903095746..437571f1c 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2683,24 +2683,22 @@ class NPMTest(FetcherTest): fetcher = bb.fetch.Fetch(urls, self.d) class NPMSWTest(FetcherTest): - def skipIfNoNpm(): - import shutil - if not shutil.which("npm"): - return unittest.skip("npm not installed") - return lambda f: f + def setUp(self): + super().setUp() + self.localsrcdir = os.path.join(self.tempdir, "localsrc") + os.makedirs(self.localsrcdir) + self.d.setVar("FILESPATH", self.localsrcdir) def create_shrinkwrap_file(self, data): import json - datadir = os.path.join(self.tempdir, "data") - swfile = os.path.join(datadir, "npm-shrinkwrap.json") - bb.utils.mkdirhier(datadir) - with open(swfile, "w") as f: + filename = "npm-shrinkwrap.json" + with open(os.path.join(self.localsrcdir, filename), 'w') as f: json.dump(data, f) - return swfile + return filename @skipIfNoNetwork() def test_npmsw(self): - swfile = self.create_shrinkwrap_file({ + filename = self.create_shrinkwrap_file({ "packages": { "node_modules/array-flatten": { "version": "1.1.1", @@ -2720,13 +2718,19 @@ class NPMSWTest(FetcherTest): }, "node_modules/array-flatten/node_modules/content-type/node_modules/cookie": { "resolved": "git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09" + }, + "node_modules/karma": { + "resolved": "", + "link": True } } }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) + fetcher = bb.fetch.Fetch([f"npmsw://{filename}"], self.d) fetcher.download() self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz.done"))) self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz.done"))) self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/github.com.jshttp.cookie.git"))) fetcher.unpack(self.unpackdir) self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "npm-shrinkwrap.json"))) @@ -2735,21 +2739,8 @@ class NPMSWTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "node_modules/array-flatten/node_modules/content-type/node_modules/cookie/package.json"))) @skipIfNoNetwork() - def test_npmsw_git(self): - swfile = self.create_shrinkwrap_file({ - "packages": { - "node_modules/cookie": { - "resolved": "git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09" - } - } - }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) - fetcher.download() - self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/github.com.jshttp.cookie.git"))) - - @skipIfNoNetwork() - def test_npmsw_dev(self): - swfile = self.create_shrinkwrap_file({ + def test_npmsw_resolve_dev(self): + filename = self.create_shrinkwrap_file({ "packages": { "node_modules/array-flatten": { "version": "1.1.1", @@ -2765,19 +2756,23 @@ class NPMSWTest(FetcherTest): } }) # Fetch with dev disabled - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) + fetcher = bb.fetch.Fetch([f"npmsw://{filename}"], self.d) fetcher.download() self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz.done"))) self.assertFalse(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz"))) + self.assertFalse(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz.done"))) # Fetch with dev enabled - fetcher = bb.fetch.Fetch(["npmsw://" + swfile + ";dev=1"], self.d) + fetcher = bb.fetch.Fetch([f"npmsw://{filename};dev=1"], self.d) fetcher.download() self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz.done"))) self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz.done"))) @skipIfNoNetwork() - def test_npmsw_destsuffix(self): - swfile = self.create_shrinkwrap_file({ + def test_npmsw_subdir(self): + filename = self.create_shrinkwrap_file({ "packages": { "node_modules/array-flatten": { "version": "1.1.1", @@ -2786,170 +2781,17 @@ class NPMSWTest(FetcherTest): } } }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile + ";destsuffix=foo/bar"], self.d) + fetcher = bb.fetch.Fetch([f"npmsw://{filename};subdir=foo/bar"], self.d) fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz.done"))) fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "foo/bar/npm-shrinkwrap.json"))) self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "foo/bar/node_modules/array-flatten/package.json"))) - def test_npmsw_no_network_no_tarball(self): - swfile = self.create_shrinkwrap_file({ - "packages": { - "node_modules/array-flatten": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", - "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" - } - } - }) - self.d.setVar("BB_NO_NETWORK", "1") - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) - with self.assertRaises(bb.fetch2.NetworkAccess): - fetcher.download() - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npmsw_no_network_with_tarball(self): - # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(["npm://registry.npmjs.org;package=array-flatten;version=1.1.1"], self.d) - fetcher.download() - # Disable network access - self.d.setVar("BB_NO_NETWORK", "1") - # Fetch again - swfile = self.create_shrinkwrap_file({ - "packages": { - "node_modules/array-flatten": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", - "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" - } - } - }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) - fetcher.download() - fetcher.unpack(self.unpackdir) - self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "node_modules/array-flatten/package.json"))) - - @skipIfNoNetwork() - def test_npmsw_npm_reusability(self): - # Fetch once with npmsw - swfile = self.create_shrinkwrap_file({ - "packages": { - "node_modules/array-flatten": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", - "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" - } - } - }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) - fetcher.download() - # Disable network access - self.d.setVar("BB_NO_NETWORK", "1") - # Fetch again with npm - fetcher = bb.fetch.Fetch(["npm://registry.npmjs.org;package=array-flatten;version=1.1.1"], self.d) - fetcher.download() - fetcher.unpack(self.unpackdir) - self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "npm/package.json"))) - - @skipIfNoNetwork() - def test_npmsw_bad_checksum(self): - # Try to fetch with bad checksum - swfile = self.create_shrinkwrap_file({ - "packages": { - "node_modules/array-flatten": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", - "integrity": "sha1-gfNEp2hqgLTFKT6P3AsBYMgsBqg=" - } - } - }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) - with self.assertRaises(bb.fetch2.FetchError): - fetcher.download() - # Fetch correctly to get a tarball - swfile = self.create_shrinkwrap_file({ - "packages": { - "node_modules/array-flatten": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", - "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" - } - } - }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) - fetcher.download() - localpath = os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz") - self.assertTrue(os.path.exists(localpath)) - # Modify the tarball - bad = b"bad checksum" - with open(localpath, "wb") as f: - f.write(bad) - # Verify that the tarball is fetched again - fetcher.download() - badsum = hashlib.sha1(bad).hexdigest() - self.assertTrue(os.path.exists(localpath + "_bad-checksum_" + badsum)) - self.assertTrue(os.path.exists(localpath)) - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npmsw_premirrors(self): - # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(["npm://registry.npmjs.org;package=array-flatten;version=1.1.1"], self.d) - ud = fetcher.ud[fetcher.urls[0]] - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - # Setup the mirror - mirrordir = os.path.join(self.tempdir, "mirror") - bb.utils.mkdirhier(mirrordir) - os.replace(ud.localpath, os.path.join(mirrordir, os.path.basename(ud.localpath))) - self.d.setVar("PREMIRRORS", "https?$://.*/.* file://%s/" % mirrordir) - self.d.setVar("BB_FETCH_PREMIRRORONLY", "1") - # Fetch again - self.assertFalse(os.path.exists(ud.localpath)) - swfile = self.create_shrinkwrap_file({ - "packages": { - "node_modules/array-flatten": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", - "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" - } - } - }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - - @skipIfNoNpm() - @skipIfNoNetwork() - def test_npmsw_mirrors(self): - # Fetch once to get a tarball - fetcher = bb.fetch.Fetch(["npm://registry.npmjs.org;package=array-flatten;version=1.1.1"], self.d) - ud = fetcher.ud[fetcher.urls[0]] - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - # Setup the mirror - mirrordir = os.path.join(self.tempdir, "mirror") - bb.utils.mkdirhier(mirrordir) - os.replace(ud.localpath, os.path.join(mirrordir, os.path.basename(ud.localpath))) - self.d.setVar("MIRRORS", "https?$://.*/.* file://%s/" % mirrordir) - # Fetch again with invalid url - self.assertFalse(os.path.exists(ud.localpath)) - swfile = self.create_shrinkwrap_file({ - "packages": { - "node_modules/array-flatten": { - "version": "1.1.1", - "resolved": "https://invalid", - "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" - } - } - }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) - fetcher.download() - self.assertTrue(os.path.exists(ud.localpath)) - @skipIfNoNetwork() def test_npmsw_bundled(self): - swfile = self.create_shrinkwrap_file({ + filename = self.create_shrinkwrap_file({ "packages": { "node_modules/array-flatten": { "version": "1.1.1", @@ -2964,10 +2806,43 @@ class NPMSWTest(FetcherTest): } } }) - fetcher = bb.fetch.Fetch(["npmsw://" + swfile], self.d) + fetcher = bb.fetch.Fetch([f"npmsw://{filename}"], self.d) fetcher.download() self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/array-flatten-1.1.1.tgz.done"))) self.assertFalse(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz"))) + self.assertFalse(os.path.exists(os.path.join(self.dldir, "npm2/content-type-1.0.4.tgz.done"))) + + @skipIfNoNetwork() + def test_npmsw_git(self): + urls = [ + "npmsw+git://github.com/karma-runner/karma.git;protocol=https;" + "rev=84f85e7016efc2266fa6b3465f494a3fa151c85c" + ] + fetcher = bb.fetch.Fetch(urls, self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/github.com.karma-runner.karma.git"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/http-proxy-1.18.1.tgz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/http-proxy-1.18.1.tgz.done"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "git/package.json"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "git/node_modules/http-proxy"))) + + @skipIfNoNetwork() + def test_npmswhttps(self): + urls = [ + "npmsw+https://github.com/karma-runner/karma/archive/refs/tags/v6.4.4.tar.gz;" + "striplevel=1;subdir=karma-6.4.4;" + "sha256sum=3cbd3b72da3b0b8aa650a90ac3a97aa5d0995ad2415989b9b8b59d09c460a6bc" + ] + fetcher = bb.fetch.Fetch(urls, self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "v6.4.4.tar.gz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "v6.4.4.tar.gz.done"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "npm2/http-proxy-1.18.1.tgz"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "karma-6.4.4/package.json"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "karma-6.4.4/node_modules/http-proxy"))) class GitSharedTest(FetcherTest): def setUp(self): From patchwork Fri Dec 20 11:26:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 483E5E7718E for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.114]) by mx.groups.io with SMTP id smtpd.web11.149745.1734693993938884026 for ; Fri, 20 Dec 2024 03:26:36 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=dsXLOfrK; spf=pass (domain: weidmueller.com, ip: 40.107.22.114, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=F0kRTJ/j41cNul61tpu5kZGHPFZvQdabCvQSMRsJgCmvSlRSDJQzQrb/ojydk45mMcg4E3AeLoIeE7VvVKS2quz+LcsW+qKLBgQOBgaGJkmcspAtkDabSmdRGActhMW+4bn0+2rQ8lyS0f21fLnZsdafzXm4xBG4Vvb2rjZZnqdcFclkfZSGq8f89YhHQZZ1mq52FEYMVKOOMe5Xc6RIo1cpjRMJS5pxlfH+4GQNQU+sZTkE65kDh24O+x2N5Y8UaprxfZY1x6WN7ffwleKlFUSMlH8Z3WUxmPNo+cXzotX917lEEw4L/0HgOu07LrcVgC8PGTltS3OaZA0l49QyEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pcdW8MubFG0xs5axX8roi4aUXLdBiwpdbBx3JxYKD4s=; b=QCKeGta0MeWM31D63MHKeW9ukxbmOaqDjIuFlO3QePntYMRXcJjh5X3gcyTPKgrW6fcGx6kx/bIlwlAUffEjdx/YZiKVoVZVx1W2taO4bSKwEw+tafIfDng0bs0XRCwawcSA13VhX/RYhvoBEU4P5il/fr69szlXBpgc1HowmnyeOC197BEPeAZumErega97aMH6iD3u5W9kMeiyzjOxXFpv78s+ufB9TQBsQN/G1C8+QdYZrFbdAf1SJEQwbkxvsYM0FP3My/jgQ9Ap318kTA0PofOwXRaz0r8WemTgz6GB/tFsEvmqSw7QzBl9aYlu1nWXfHbdw9MEhvgl+GDXfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pcdW8MubFG0xs5axX8roi4aUXLdBiwpdbBx3JxYKD4s=; b=dsXLOfrKR+iS4mw09DyAYXLJcGlVGGvpII5haYzhSHyks85d7P94u/fADm5eYvU9ILKVkU+1gEsIy5MsYQLay4ClOnzrs7JoFJGn7aK5fTiXVgIkXh58jhBN7qGSLaYMP0aaeHM1us1lBoo1J/cll3R91ixr9OaV4HGdHhUNTv6B3YWaEHRvkJdrpeFKrYYhFmSi+9XNvRPACDfZAS7wK22cIpjayaPSUU18pznaBQcxeTtIk0ArSMUatT/mwiK7hUiOJ9aLOYXM9+fWSo+M/+caY9Ciziui7898bBi83+p+0F/vg8NRWtNmPHhyfc1P2/DPPIJplUDKBUTxw39gVQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AM8PR08MB5684.eurprd08.prod.outlook.com (2603:10a6:20b:1dc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.16; Fri, 20 Dec 2024 11:26:31 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:31 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 18/21] fetch: add gosum fetcher Date: Fri, 20 Dec 2024 12:26:09 +0100 Message-ID: <20241220112613.22647-19-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|AM8PR08MB5684:EE_ X-MS-Office365-Filtering-Correlation-Id: 3d79d5fa-8ca3-4889-5b9e-08dd20e92657 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|1800799024|366016|38350700014; X-Microsoft-Antispam-Message-Info: Fmq+GJumgtdsxwnyvqff6zt20c5JoZEi8W2/H/KNFvYnPiQj9eElKXmiFCJaGVSBwkwfK9jqSgG3/GbvxV7f7gefOpYNyEXXiXqRl2aRUMU8c7xp6YM0EqbcwaJyYKdXko2Tpar0ITbCI1DqKaZzYvaZnj3ciYfWrDThgl7zYoNHrSebZllDFLcRuUsoP4xRiLCiQL2IIo9TYpjOE/EdUiFTKwCsZKhhkHfzkAsqLhe1xXYD4zkwWK2APqztfrzeaZeZHgaaWeR9AjSqPXz2H2Wsfy58SWQx2pozjDBaRM2wxjfp8qjVL3kpKJfOV2VOwAF8aQse8LDWqf2CALofNWOcnwH4sIC8DAxppvTZd8yzKW8TNCZeZkFm5TGWoR60Rie7XpQMU3lT/6UM3x8huxAkWWMgDslAu1eRGDIEkeb1x0XknxYusJ96aL3yAkMGZaLvY3A/dH3v3Z6mC6bKPYaXXNkdCrnE6inGYVX8A8yX0D3gMoZPWeX7E5YhjgAPd8q4+W0fVsx07e+pDdvhEaPMltlFJo/x8syFcCiiKoxJNTT3crBjMBfFRgBIUc4AJ/A6OBMGNkGVgn5T9maoSrHSRNw1hxVcQ2kICxsSou6/p6oUA+hSzS9tmw2bRqQEZJoJ/SS9t4N/cW2a/4d5vZcseC8G6FGfQph2BtPVtsE4ZYsvbVuvVeP0vdCz0jHT4MHSWvpbqZ2sSeYt9XTGFtvohs0wiIlhhuESLpw478upGKidgcaFoNiw3Zmu0AFiYSCEdbz4Sasu3fUKXpf0xRR09W1+XHo/T6OAkIndrkIj+egJdTGKqJ83VrP2z3IqcRrY1JpcdlazW/TK2D6viZDwdt1V6yPeuyQQKNF0W677bjpUpmq+0PpGqQaBLoKmX97NxHt8B6XRqRZqP8NDKwX8zTsA82E68OS+mtrzDJvXD0FLuzutkGNiK8kl6mCI/SVGLZJMaUuErI3zWMEOJyRzrbYz0WX2ylkImsDS+WJ6NVNY8JMscGlf9FQK+tTWiIgzCtwFOKIIIVJZLEVJvVfcu0agR1O9s4X24Jq6xQCFBSvH1kDgjCl38r6IXQSu2itp1QHFYvfRJ4/pVTsnrOt9xNsK6z1+14O25p56xztntotvhC8p5OSvAI7p6ZBMSlLaM3ys5ZczpBahedtv5413U3jZmUmOm4A2wcEC2PCloDTpHAgAFY8tGHGruwx7KrluP+QYA5ex/3dWYSYctS6Sk20ZfRMzqS/IDJiasHlvB2D3hh/dZke/0yUQff55m2D6XcgrVkTEGtEqfsw9/oqtVhLq0KzWb91Qa3+zutxhi8e1LPcbvKGFi0aB7Mhwmokm4Dr7cJcZYr0vsNVl9CmHjSs6hFTOCS0xkZ9izSaedSYVbX20TBFKViqnqM7j X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(1800799024)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: fWx/u5appbVk27m4Nz5afaXNR2W/2frsCs2RZH5w/vJ+SQIVYKipP5vSldJcgL81X9Ws0fuZ8O3h57yiQO/AzbHf4LkWfeCYnNf2AKt8Ga1WQ6O4vHmNppsvRaOEq1NRm0bAMwyXowUBfEiYEAAkWm7ZRLpUEXemv22D9u0CE9A0uXD2QFCrIfE3sIyVbxN48o/+rbxd/XU7jISEe4GedCFG+qsmvGf7n+M88Hlts50oRc5yKqPVKDBpgRFtNCJ5eMBUgOr9rkJkQ55FWYDyeAaEHkVpBYfYT3OHeBaF3doa7gG11hojSg3mwtJjzt/H+blz50Dyjs7nTXujwEtY7W4tl0vLhAs7XC1gYhdNWZtCgvyQ08gHd9sVEM+bLqO6m/7Qypi0KXBuLCeNhWguPRaUEfNP8VxbTZm+qxOW2oP3zP4ex4uYsaPwTQzu2j8fC82Q68zNwZSkdPEer7AuexLKHAlY3wu+4evksVl88LWvBOZJllY8zAufU6ysYK9AHJlCiYsvjyJvLfRKjjoUrPTjyi4pJrF8msytjVGRIxn2A4xmfawjf4llSk1ZrELyEVZTyXPdHipTd4gXtEfgiJGzr5ARpquyJEY10ITu8/K7MnYHfthRIFbh+Ahfc2s4Un8s2gZiC+u0tv8aF1puOymMy1h9xX79bqJ4+Rx8Nt/O/Y9VHcZrqw06obKMK5cnQ/iFDfA1ptFMyZmH8g9qUWXGB0rmy7/YbamP5nc3lGv7jAiNgtO8lFJ55BGfvobel4NGCdWO+jJOz+IIm92pVgWmUEr09WV2NOwtxiJ/SQw9ZW2nl1ycntQYIDxUs9uBj42g5ac+vhTAc45Ij7mw7GuxRgq+zwTcp9XvcAaQ1aV/EQAlFC/d/0dKjd33aYULC7qPFh6IsEvUSlE0/38WpdnF+nI0yD8lUdzM/N5KQ1sXGc29oxGJ7x5LGlWAPzYkJ64Mv5meRd3WfROHf3zb9JsAtL/dhB+kWVdWJ+WqjfY+LuKqGQ5ozSynrD8MF7Z6j46C2Z++m8e6Tw+OYui5REHRrl57jCXBC+FwvAa/RDoVn+5UH2c5Hj2HT8npH4u/GtA+xOXSFO/qPFY3afeD1E/az1NAlKVoxjRBh14UrPhgAgY+0mHMxiIfonzddklE/NNcrxBJnjyj55jGnP0BmHzt5IB7r1GST2R/XWkKO095eHQTtqP1ZRI4Slv2BvYiKPplp56Rpbd8aJKdK97CLvqnlucf0EDR64ivVbJqHmAtSfCHUBlk1U/0N3emvgjKUf4gSPCAamTR2vhJ0XKsbL+0DQdQ4DOYWSYBeVviEg59XR1DtrL+IPqzS7trec6pVDFt5MY/gbHN0hw4g73ElsaC7H8Ele+WhhSJAeYze80XyB1H+e9Hv6NA+4lDx8eDkfMve/H0JbaZL9aiuDENNNz8aMT27cSeUcFg5/PRk+dxXDqaUxHf7q1g+v49OK5pL3m0hNA8ffbwe8ExEGUXDscTJ3M/weMKOf8fNaYCoFQX50Avhax3jVkwGuYO40ZWNLzwsKHYHaenMRKsF+8D4NfgVco3Nzg6+qqR5rbq018na4JloL92F29EmGRjMO1E0SRXl0Dcu8pHqSWPobTyYA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3d79d5fa-8ca3-4889-5b9e-08dd20e92657 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:30.0117 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cn2bOJ1dp5nb1CYb49cBWLGJpfcrB1XXdpBr4MSBV9vdo5riKMD9esHXrn70BCZ+3tarVBlVhutrmtSAakkD8w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5684 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16936 From: Stefan Herbrechtsmeier Add gosum fetcher to fetch dependencies via a go.sum file. The fetcher uses the dependency mixin and supports different types: gosum The fetcher uses a local go.sum file to fetch dependencies. SRC_URI = "gosum://go.sum" gosum+https The fetcher downloads a go.sum file or archive with a go.sum file in the root folder and uses the go.sum to fetch dependencies. SRC_URI = "gosum+http://example.com/go.sum" SRC_URI = "gosum+http://example.com/${BP}.tar.gz;striplevel=1;subdir=${BP}" gosum+git The fetcher checkouts a git repository with a go.sum file to fetch dependencies. SRC_URI = "gosum+git://example.com/${BPN}.git;protocol=https" Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 2 ++ lib/bb/fetch2/gosum.py | 51 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 lib/bb/fetch2/gosum.py diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 5dbc0598d..10b4cf24b 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -2116,6 +2116,7 @@ from . import az from . import crate from . import gcp from . import gomod +from . import gosum methods.append(local.Local()) methods.append(wget.Wget()) @@ -2140,3 +2141,4 @@ methods.append(gcp.GCP()) methods.append(gomod.GoMod()) methods.append(gomod.GoModGit()) methods.extend(npmsw.methods) +methods.extend(gosum.methods) diff --git a/lib/bb/fetch2/gosum.py b/lib/bb/fetch2/gosum.py new file mode 100644 index 000000000..001ef3304 --- /dev/null +++ b/lib/bb/fetch2/gosum.py @@ -0,0 +1,51 @@ +# Copyright (C) 2024-2025 Weidmueller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +# SPDX-License-Identifier: MIT +# +""" +BitBake 'Fetch' implementation for go.sum + +The gosum, gosum+https and gosum+git fetchers are used to download +Go mod dependencies via a go.sum file. +""" + +import os +import bb +import base64 +from bb.fetch2 import ParameterError +from bb.fetch2 import URI +from bb.fetch2.dependency import create_methods + +class GoSumMixin: + def resolve_dependencies(self, ud, localpath, d): + urls = [] + + def resolve_dependency(module_path, version, hash): + uri = URI(f"gomod://{module_path}") + if version.endswith("/go.mod"): + uri.params["version"] = version[:-7] + uri.params["mod"] = "1" + else: + uri.params["version"] = version + if hash.startswith("h1:"): + uri.params["h1sum"] = base64.b64decode(hash[3:]).hex() + else: + raise ParameterError(f"Invalid hash: {hash}", ud.url) + urls.append(str(uri)) + + if os.path.isdir(localpath): + localpath = os.path.join(localpath, "go.sum") + try: + with open(localpath, "r") as f: + for line in f: + fields = line.strip().split() + if len(fields) != 3: + raise ParameterError(f"Invalid go.sum line: {line}", ud.url) + resolve_dependency(*fields) + except Exception as e: + raise ParameterError(f"Invalid go.sum file: {str(e)}", ud.url) + + return urls + +methods = create_methods("gosum", GoSumMixin) From patchwork Fri Dec 20 11:26:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BCC2C3DA4A for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.114]) by mx.groups.io with SMTP id smtpd.web11.149745.1734693993938884026 for ; Fri, 20 Dec 2024 03:26:40 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=NTrdkvnI; spf=pass (domain: weidmueller.com, ip: 40.107.22.114, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KXlUIjGP2ZyO+zSLq0on+K2zqRZL96k2VR+OR+oFNXn7K4iAPnBUAV8+UR4gv9X05oXCn29c94Ma9YQCPKDG7nnwZ83M4J7CV0sGQQNMnJz5COU/eS/QBZCFXkiuMP9iCFizVJA1D3cLi18cvr1xuyYJoYlBO5vT17dD+5pKdHn95bUpwIigevpj0liCJWeQvt783ZaxB2a8BxgxBCzRc7g7FuKEI1hBMKKJCNsDEhqQn4A77wGUIHn76mIOGNv9Jn0pI63fpmkYkdLZLfAZzMzBlB7USoVFEcR48yCQweD/VymA8UUudSon0VAQ1epyekAYz8KNmYMhg5mSOZwiug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eLENERgwmHO7EYBIoxAN9UBOjHA8RShHsMMjX0MMi1A=; b=o7LZc3E4EXKREJ+C4fVe2bNNwWWAiDt6cgQm/3Tr4ZKIBiJnsZxXOi6VJQzW/qOj/YeHBmBl8uhxhGeQeKUYs7GStxSes5RmgdtlGJBAYXOjJ8J39eR2qgAEkZMX5GXKoLCZe5oxQ9QtVsMxp9b6vFYoLdGf45ED5Lrzd+5R3P8Bpop3j/GC6qN0WG10FZwEsDMhMoFJFTxNG9h/XwtjKTM5a9KT0RYwQtz5PGYZ7oEV0gtNI4xGy8QktGR9eJCysytsPXZciPVw2rXmZVC1SA6sQdObL3/i6JtlZ7lkcEBpwgWyXE6pESFSu8NlI9HmJ/cmApmszpZWkMJQq5Bqaw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eLENERgwmHO7EYBIoxAN9UBOjHA8RShHsMMjX0MMi1A=; b=NTrdkvnIyysR4PEliaM06auU36+y2BvftokwUJq2s4gIGwBnLzHCA0xbzIj/AecV/45Of6B5xVhgJ82gqhs4k/4kpMdVL7tpo+oP8CMmKgQTsmQSMf/GfDYZAEXrnz7PkwXvTpRlsjSRuVVn+kwg3IuUsa5auHty9zZ0Q7QJ3ZUNbYQmEW9/02lEwZ9PeArBGb92YV64a9YXZ8dGRI8GWdRKBoN874MfPzPFVDFLLskHVn6JrgXDlPnlcgnp0lV36wBhI7/cq96/TajnXJhnc3kL1JwoXtPy5+VH3T5Nmq+mni8hA66/6L1zowJX4SnD4vtHi608FcSU4LdKteBzjw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AM8PR08MB5684.eurprd08.prod.outlook.com (2603:10a6:20b:1dc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.16; Fri, 20 Dec 2024 11:26:31 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:31 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 19/21] tests: fetch: add test cases for gosum Date: Fri, 20 Dec 2024 12:26:10 +0100 Message-ID: <20241220112613.22647-20-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|AM8PR08MB5684:EE_ X-MS-Office365-Filtering-Correlation-Id: 238f62ed-c9e3-4a4c-964e-08dd20e92683 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|1800799024|366016|38350700014; X-Microsoft-Antispam-Message-Info: 2alrcqpwAHmOyeKcHUxyRwBl372NEK0TeC5RTzBPz0/+wWtb0vOSVdN6zeqVDvcN/VsvrFiRdB7crvqsmjVtZ/kK5O2WA+J64jO3NoDoOwIecP6AtFqgxOaKIt2zgfSvuffi6dc59cOgS3IimVk0LarQPKDH7Se5dAH3ULeauVaQh/c/ortiCM9SYc6gvO1Tf4rLFZrpqrMComN+MJ+2hmDu1n2z8MrAS8MVjzKEQ3OaVHUL3oTh2RFtB5gdDCCa2LUwN03FG3h7J1xUsAH6i6ZVlKJO4Xk5cyNkRWQXXR+b8kziSyoE2QR2IjHWi8zua/4t/07S4io6W0dXc+otcc1YztUhP42y70jpTqcn4/A/h4e/DoX7p75tUtLglaBlWPDBmmjBJw2ytbm32nKXlpnu5+4S410YpfD9AtEhPF+qeWTee1jolDDjT8UG5ZNeHkN2jsW3QA1JsoeEekG/tgUH62f7x4mT1iicAcTOw5TjfJSiA54bC8bzLJ30lzA2DrDOlDCuD+bUHr1pq1w1sELxlHklFRSiw0pwJkFmdtGsRCoVpSE6O+SxvdUyqUc4fHega9KcmpUPjX2/8EcG5B+VdF0yr2q9MmxtAI5qIn33uSYAlZqlaDpV7vWJfnlkbS/3PcFCu+SN3J0/LHQIhJgQQxCDoMAnLJf0T5MC1P387MSey+FNv6X8pEhkRFId72VPFNxu/MicLl6VQ23AO6FddmSNS99ncy/654gsTE72c/eIkBcny754/Czcxj3xgWzi5eP6XbC11yvI03ELm3ZJa49NMGRjRhe+sXq1IckqmFS4p6Dna2u678uloWLttUpUqUfWxxjNFurcU9gMnO7fQX4cxyA9cDwnYo7jJqk+Bq/GTi/6Jl3cyeVCRET+W1r5vG6W6/0lwxk103vOSgAB2KfA6JJjSolj+rpGOWxaDDW7zhlmnRdE99F7XkE4QrjFb/tVeiqJsgxN/tzUKBd40T4Zv9j5rCfDzcWzcH/9nZMcPnaNNfVgUkH0DLEbkoVNi0efw1mbiBQcwdiGg8tD0wwDmSLfgOyYa43NJH/xGvWndmyceR3kzHDaTGJmVIFERvC8SrBNxVvW/6hj+EZm3Zd4FZhVtrVTeJ4KUkhMt/EJjoR6qq1e4GqkpHAvA7pMgnahTgN61jzxqLgN+eg8W7v9zoMWyGGds3+pU1c/RrktMUzACVuiw8O7HURQA0rPMPgIRYEvGS79+IZ0sI2zPpjvgVGlcUoHScH4JM4XBBNQhqQsTV48qz4ZbZQcurHEdTk6otywTGwwTXpzJP5trcVhwYNsQqO9uiSCdtiuRZDKvC45MpQ3pJKan3y+Nlid8vhgdKOKXRTf/beBKOmntL4BMG9V4H89yNXUK+yFGYGSPA+t1ogDGTcS6AsD8hdz5dPh268Xqdulc2nbAg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(1800799024)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: fWgw/y5eV4jeiR6Bnmev0xaM8U6/dV8IJfL8Dfbd21t1eptKONK1qdXO6m+sHWunxwmlWVj2aWUrMcsuF6jFr9fNVKnup/vYF7NRRll3z46GXk0uc4A6zuFGoGzldYneCFEAEGTqUbdL7n+dvD1ZEt2hbLoe23qUBofnmBv/VxjJ8kOT0Ye1+n5mQ3TJM9NGySMYGTNlXAWdFMZ/GNDbeIzCPncBEe3fbSOv5Dteh6Yz2HrN8JVcP7dalCzrzydFmZKRgDlaY2e0uMrbfW3DgAEDUFh+MMBI+9+BBPpQIU9yTh4uO1by7CP/G+rE4gNYnsF0DqwBMX3acOmfle16a7aHyoSQyqsCgC1XGQSi2D6fK5TdEX2ydSEIsP1tjzad7ggJO8ZkqvCpqnhK8bUAYk1OEvGMVZzEax00qe5zkkjBGG2j8llSiT2azffv0ZKbMK7ySXlumR8l9Q+PKFpOEabHyQTyLtzqq+0JiCUVRJVTsZazSQ2x8x2sD/0eZ5+p4sUmL6hceZyv3J3B2XQbgaxyvtTKX9CGSLsRBHhacFrfaHmVy2bEF7DGdQ7XOfJga0iidpDbqUNWsN4ptb5Nz5XCwPNJaA9YWfTAdyRcYL+ZqJOyOwlx2rSMmSCf2lciUvS3TXQ8e5OeuvrV2aoddCKg6icq4HVjnIpSjTSFQZzm+XVfWIUVU5xMROr5olqcWIlTSBBATJRhvGVorTFi3Hoi045Voi9Q1yQEE+pUWbNNpVWgc0I2LJLtWM+XEYF/JRTBLoCRPP9C8p6Bh8X18UVPFDYKv8bkaQ45NkNPvZ5Ump8qtloom1zOKGNti89Cf/kvacNJMYrL8J75reKed6NEJwlQtLLIO9dKrETCe/UGWEnt3mJaDvf2Q3OFMAZoe6H0h+zNANoUazJjd6iHS6O+/ScxfOKlrlZq31DCzwqLL6yEfRHBTeWhxIvIfYw1Owz2u26xaZ3VrzDdEY4y8+o82u9owKQIdsmIHfj2zh9JNsZpmeeYzsTuYUImgQl/q7/ka7Dg0M2eYTbWDYybMfdxOAxmaPGtbeQjDtnmF5MC3oShT7XkV59rDFjQXW02IzD1F4lC76ywqiI/STXxhZPWMNR3Ry98QRv7lBlQHsJvEf2XvyV+JaOT7KLdFot/OF2Yuc1Uz97IYj5M3XX4gPYfgBOsDG2gcDapSojdyQ9h3zUGWb8Cfqph3TTsd1tIUgi1oaHYZn3XKp8SXPGrpmIiUh9aszZL2rR4kjUAbWEW2ltftxqYq22gcIE4ajdxyBunWuBdN0Y5Dybwc4swEqWxxJKeh5Vi+u5Yd4V6n1V2g0iSjllHpfLm6tQP80jQ/S0LCT5JUjD2f1yaInkkvfq38h2f2vt2fEY/bJqjc8MtWVMG9UqI17Rd3A7YXbcTeM5jkWkOKZ6HnCzkaHmsZFRauaxjBYs2WElkRDj4eWhiVaxyII4J/7cg4ee8jbZqPr7A6G7ne5CW+neQrOiUKe7YEhQSG0qMGiNdOXI39nIo7yr83D84XoFACLBNGWfaueo1NKURgwNmHMRA3vHgZFVDHWIFdUCUmZJol+sDD23W1XyMavoydtYSUdvsvK3f2K0yj3V6sgtfhsqY1JSi5Q== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 238f62ed-c9e3-4a4c-964e-08dd20e92683 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:30.3116 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: krhEqhsGFuOs0+3lJWYyXwodt/ohXCBxO2iOpQGkXALV9iRV1XROCq2M8Zjel3YO+AYBFzL2FHx/nN+XUj1CTA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5684 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16939 From: Stefan Herbrechtsmeier Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/tests/fetch.py | 56 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 437571f1c..30c821a67 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -3439,3 +3439,59 @@ class DependencyTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "dummy/dummy.txt"))) self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "dummy/bitbake-1.0"))) self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "dummy/git"))) + +class GoSumTest(FetcherTest): + def setUp(self): + super().setUp() + self.localsrcdir = os.path.join(self.tempdir, 'localsrc') + os.makedirs(self.localsrcdir) + self.d.setVar("FILESPATH", self.localsrcdir) + + def create_go_sum_file(self, data): + filename = "go.sum" + with open(os.path.join(self.localsrcdir, filename), 'w') as f: + for module_path, version, hash in data: + f.write(f"{module_path} {version} {hash}\n") + return filename + + @skipIfNoNetwork() + def test_gosum(self): + filename = self.create_go_sum_file([ + ( + "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob", + "v1.0.0", + "h1:u/LLAOFgsMv7HmNL4Qufg58y+qElGOt5qv0z1mURkRY=" + ), ( + "gopkg.in/ini.v1", + "v1.67.0/go.mod", + "h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=" + ) + ]) + fetcher = bb.fetch.Fetch(["gosum://" + filename], self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip.done"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "gopkg.in/ini.v1/@v/v1.67.0.mod"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "gopkg.in/ini.v1/@v/v1.67.0.mod.done"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "go.sum"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "pkg/mod/cache/download/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "pkg/mod/cache/download/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "pkg/mod/cache/download/gopkg.in/ini.v1/@v/v1.67.0.mod"))) + + @skipIfNoNetwork() + def test_gosum_git(self): + urls = [ + "gosum+git://github.com/Azure/azure-sdk-for-go.git;protocol=https;" + "nobranch=1;subpath=sdk/storage/azblob;" + "rev=ec928e0ed34db682b3f783d3739d1c538142e0c3" + ] + fetcher = bb.fetch.Fetch(urls, self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/github.com.Azure.azure-sdk-for-go.git"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "golang.org/x/net/@v/v0.0.0-20220425223048-2871e0cb64e4.zip"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "golang.org/x/net/@v/v0.0.0-20220425223048-2871e0cb64e4.zip.done"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "azblob/go.sum"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "azblob/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20220425223048-2871e0cb64e4.zip"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "azblob/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20220425223048-2871e0cb64e4.mod"))) From patchwork Fri Dec 20 11:26:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABFD8E7719C for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.114]) by mx.groups.io with SMTP id smtpd.web11.149745.1734693993938884026 for ; Fri, 20 Dec 2024 03:26:41 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=AKxmv8dc; spf=pass (domain: weidmueller.com, ip: 40.107.22.114, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Xpbo+9OdTFYOeScZidyeKRQYOwcCfcK2ausTRgvJ7wBA0t5cOgactJaewfy1nDMuVfdrL9vW0aHhphYnGE3Rx557ova6DEEZmMuUf2M22vowRfTtMxDTEnUZeeUVka57mhxTkWkMwRdjxaRLNPoZUHXadXM1qZtsrRyb+MGMAqRSiXYy9t0RXiqh40+pvxDKEhhe15rDdZQI9nCGffNSiOQeSuR0afR2/zN8rkfT0hQOo4DbRcbF/l2uwiSAeQ/OduM6Vb+7NByfwc3lvilx01qbkJ8NYHAKMUKbf1z1sEawk4WsexL8v5XoqUyDfueQdaAny8b81e30Ubf8G+jXbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NjSgsCSlyuNISja8yjNqfAPuN8YK7Pt60PQplva+QQo=; b=qyXHLQ9haR/fziAH/YFSKgsSije6Xrs2zgDpOdVi9tnt5+rDJEdR6OLUr1pud7PopWDeW2rpDvJAGcITd9h2fH7YQKfjq33jyJ9J9MG26dowjBmXIB+mIZ970hGgT80EV0cDLOsZr/4U+HEz24rcmrhdmz5cXDKyKXjb5ywD06d5AwX8JPZfpzUDPZTTzkpXm9788Odw1hncrgkEm6zPpqqsq+OpRRo6gjB2Qb8msSuAGJrYKqO5U91TtTKoUiquKvJRX9gM6u6IanZz0XDiZEHOIaSZPggDtVdyLZHnfJP1QFV1sPto7dzaP5IIO2MwdPsvIQiGc3Nd5WQv9XslSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NjSgsCSlyuNISja8yjNqfAPuN8YK7Pt60PQplva+QQo=; b=AKxmv8dcu9kK1gf19ivna63jBxg4vDRiUul5lrF+9fo8/762evRIeSWLEiN2roTART18t9OpnlgmgdQ+PWeR1Ww2EJy+totYrDU0zK9k8V3C5kdzyENjPVxhO7YeSu0sfZpf6KlmlDR32Iz7F898+Vf/yMSmiyKILNVshWcIA7kFPyQtyI23zgaO+EFAoIbsPwmmyEO5YO9nRkszbBzd8hHWwgi6jVn/CcSrnMRb4BKL3PlY/3kM478Ll44VCUNHQ3z1pnyWlnzFc2w+5E+9tbUPcyGudTB9pBEzSLPJDzrYE52mOOMoVzd73dKob+MvGqupQT67apkKsXSw7Psvwg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AM8PR08MB5684.eurprd08.prod.outlook.com (2603:10a6:20b:1dc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.16; Fri, 20 Dec 2024 11:26:32 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:32 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 20/21] fetch: add cargolock fetcher Date: Fri, 20 Dec 2024 12:26:11 +0100 Message-ID: <20241220112613.22647-21-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|AM8PR08MB5684:EE_ X-MS-Office365-Filtering-Correlation-Id: 3ab7e62b-08dd-43d7-91a2-08dd20e926b0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|1800799024|366016|38350700014; X-Microsoft-Antispam-Message-Info: tooRTjcqnbPpKhG7Kdt7LS0BVAS41kC+mFIs2SZxLwsN6EbhA2bh9msfF78yiF2PjjmEL56y/aCsjlfdHtdzAjSTGdDoFWE/W+G0N3SfFGCeAuC+wGXKuD1Zf2674/1CiZTAVrJuKjCn22KuPspoP03nKZ+XQiKGks6up7rDdQDVlYmDWPlkxnJkwJk/y97TYsg6V9Xl5Vq6EhthaGQ1sRsNHA+Cl/oRViwzoLWKwEEGCBZgZWbAjnjpSYrYY5opTlZN+p1JBMv+Fy5EzRGYHMrR2/fLXH8LNfm4JcMNpwgOrj6vPG0SWBEl6LgHkJOYu6taKxNUc3x5ldCAt26Mc0wWHsWqPcUaJAb7NpTAl2wcXrILvGnIktvVpbUDclfoa2jNh7OGXj7nLOS6d+RJMV3P+mRvl3E2I/8WJKbhEJyeOz9xSuKzrxrHWy65ePiik89sT2+Ks+4/+Qhw+PyEg0HtutPMgLCcHKsS2rUc22oimoRSV4pAHfVdlCxeoVYxGb43FQSRt2p5QyzwgTnNBzyrtGkkMOb5uD1xX9hF31TbGkraRSD+8QjSLXd2IAlGeZrtBKAhDW/XP8FlwgStftsVXh7uP71TU5gy8ochtwSmspdZu9f6rm64C406cEeR+xZKRoU5o9N8E/ThrC/r3397WXmgL6KvDwVGMhe8OFT50YrYpN2SaxHoH1sFQVvoKExwmgr3oxDWM+C3g/0Jb5/5vlMvycz3OO20wrJr4MtAkw/VrQnVeez6SHCa6/Jxv+Z12OYGPA/t1yhSxM69WHuUatyw+S2fJPcmlx3ZfvAHbnqVoGXelQU+dMW1WknSYoy2JQ1hN34DinLdMLHi36e3NnDIEQ7k84O+e+RSZZoUg2IABy14sScdgk25PNZK5/RgYYZRVIe4TjWPFAs95Mc81TD0W8XOWye1+WNwBEhh7p1qNFLNXKCMvztS3ksR/qt8X1HN5uTnI3Sq7IOkGL2lFGDW1HpEo5ETFntWrYMBVDkQRdh8uWepHHsHdEcPcQtuF8wTx43awE6/Ducxn1w29PztbNCv3DPFdirAKPeFyO5Zf9Yxrvw+Eimrad4TLbiGZuKBRaBydy6zmTef8cM3/XDNxz1SCOe89giGCVPwqPrMipWUht6WkTjQOGRWfeYqi9okGS71NGRphHz8F48TTbZ2dz+ri34X+ft102h1vcK6DL1bjb90W1R7qU9mNhZ4cXXdPe3yRRCabTJ+094R/gXT8wdziW4eN7zmoGKzJI1IttuyH8bVFLrg6m19gXsV/Gcoh8pUbzMnNfvC4M9j37N5JCcdnvD7K2VUMcN/eeaitIKyWIIePbKwPGlxofDlZRB0EF62HqvILoS2KD35Q4EVBw36948Ntg0klHeACirolTfoC6r6fVc1X6Js X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(1800799024)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pBbFdU1KMnAY3A3rYZYG5NhGXhS5R5zOOi6nDN5baNGJ57/h8C1yF94kHWLsz/DRuVtaw4eAZdf59R5RNyXxsLNV/EOP0xs9UCss4RHYSYmM7waqoauUuRYRfbeb7jBKuqUv1AeJ/YkSQdM5I6XzktO7Sux7HM85wLXV+2xdpd8WtVjsdNITU3h/HJRHdTvmSkmdiAJjoSWtJU14rYvJe5fRK1mZK8KvI2MkWmJUwNoelyjNAgNbunShrPWFUdd5IOlH4QztgbDNmQfaXOB2M8VJ76FsJlNHS5rOLMYPu2fh6ahcD+EdFrvQFrSTU/Gbj2mPaY7i0i+pwHJ9xHACnWGmn3rorSOuCZwP0HTLYFpSSyo9X0r1A2cteKvPA8hsBQfOWx1ZwnV9re8AlG17oc+6SB98zwIjfnpkGRbULjgB1OO3fDDV+hftEG7W5pt3yJb1rMwYeYsLoYhGWoAmYReZSeUKqjBU2PhX9pWHAvNtAmDHmlavf3kHGX7O0RoxD5WHi7Fhc/s5+ssRnag4ttMVnefZUph6HBgronU1Pa2cl6RTkeZZuUT5jqAQ+ufyknae7T89SK+UebLNYECFLFlwzMXrtbI1AFFJtluj70oAqq4kFmOqUTL8rs8bvaE4aKzIbklh1CUFy2InhZItjG2miavAxu+0fARHMfQDVVIJTJsWkfoDFdhR1087aELRc8CZA1jZR8aUXuZFtU4A/XFMqopUivDIY0Wn2ev4ouzAoeNJEUTsyxVLsjh8dlt4/Gq6H9lla4dFzVJRjM9lCUpaucQz33PTnNIG1uxpuTZ23tLkP800Echd3kGMiJD1d7AbqP3XPwxQPr4G0p3125LEZhbgW+v2MkInBwKUnqEqtwudMUp7+FKv3PvAkOHOYCArmJRpaWQHLtD/r+1KK9QnoU+kp7J9vFFnjeDqrYkafSZfF1FptDiU82Ow+i6XuiSVy0X+jJ8e1TIS0kqaHo+poTbfDFANV136396/LSxKJ1e9eAZitn7XUKDNSd4Z4R43HQx+ZiFI0zv0iSzdgJ0AoRas3GkgpJtkXiZX2+6jC0/rr0kEGPMzS5pAiKsgmrKiT8mmnrveokQBhsZMWp1qx2KCPrlgI1fVVYJmDP6P1dftsjMGWTOrHHNegGVOImJ9TkgrHVQQgCWK9KX7oW+bQeR2bgA1k9W5PxfuDXrz+XChKlgabsjYXVrYMir/pjNuqhTvOdr52YPGLlUQ1awgI92UhA9+TN4XP3Z7Z4gxaQQP23ns0mxeSdPofGK3F/QNkZHwk6va3SXZKWADhegUQVALFDOg0VHa3r2HmUyHK4x4ENYKbwL39UK+TiZUuL0CKHlHfyMHkQBjgaOyE6m1Y0SmKOt8OPWiDU+kfF2aVd+pVeRr0J+5c+NAVBlEaV/i0kbTesQxG0qeaAcVXx6603p3qUpcHM8hiyEpjx16Kw2ZYqAmkeXcLwzb/R4AKTMhJHpuzVl5JK0pN+IIBSP0yQom8toN5Ff/UqcRtDCyPMc+R3dEAjokvpunMgjICrr9jPvZDWFrFaK4s2e4AHFyD7xfdeHrm5qRl5ilTyyWP7kTSLZAKFcsrkNM6jwukzIXxJkFTFa7W863W3hwhQ== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3ab7e62b-08dd-43d7-91a2-08dd20e926b0 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:30.5963 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ijLJ4AZ7EfxWW3iz4wqjo+/hltuP4XC2aLKlF1s1j23jx+ZEkO4LLwRJA6bp8e1K+ZRG9ebOqfH0wZMxilqrpw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5684 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16940 From: Stefan Herbrechtsmeier Add cargolock fetcher to fetch dependencies via a cargo.lock file. The fetcher uses the dependency mixin and supports different types: cargolock The fetcher uses a local cargo.lock file to fetch dependencies. SRC_URI = "cargolock://cargo.lock" cargolock+https The fetcher downloads a cargo.lock file or archive with a cargo.lock file in the root folder and uses the cargo.lock file to fetch dependencies. SRC_URI = "cargolock+http://example.com/cargo.lock" SRC_URI = "cargolock+http://example.com/${BP}.tar.gz;striplevel=1;subdir=${BP}" cargolock+git The fetcher checkouts a git repository with a cargo.lock file to fetch dependencies. SRC_URI = "cargolock+git://example.com/${BPN}.git;protocol=https" Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 2 ++ lib/bb/fetch2/cargolock.py | 73 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) create mode 100644 lib/bb/fetch2/cargolock.py diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 10b4cf24b..012130ac6 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -2117,6 +2117,7 @@ from . import crate from . import gcp from . import gomod from . import gosum +from . import cargolock methods.append(local.Local()) methods.append(wget.Wget()) @@ -2142,3 +2143,4 @@ methods.append(gomod.GoMod()) methods.append(gomod.GoModGit()) methods.extend(npmsw.methods) methods.extend(gosum.methods) +methods.extend(cargolock.methods) diff --git a/lib/bb/fetch2/cargolock.py b/lib/bb/fetch2/cargolock.py new file mode 100644 index 000000000..18df160ed --- /dev/null +++ b/lib/bb/fetch2/cargolock.py @@ -0,0 +1,73 @@ +# Copyright (C) 2024-2025 Weidmueller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +# SPDX-License-Identifier: MIT +# +""" +BitBake 'Fetch' implementation for cargo.lock + +The cargolock, cargolock+https and cargolock+git fetchers are used to download +Cargo dependencies via a cargo.lock file. +""" + +import os +import tomllib +import bb +from bb.fetch2 import ParameterError +from bb.fetch2 import URI +from bb.fetch2.dependency import create_methods + +class CargoLockMixin: + def resolve_dependencies(self, ud, localpath, d): + if os.path.isdir(localpath): + localpath = os.path.join(localpath, "Cargo.lock") + try: + with open(localpath, "rb") as f: + crates = tomllib.load(f) + except Exception as e: + raise ParameterError("Invalid Cargo lock file: %s" % str(e), ud.url) + + urls = [] + for crate in crates.get("package", []): + name = crate.get('name') + version = crate.get('version') + source = crate.get("source") + + if not source: + continue + + if source.startswith("registry"): + uri = URI(source[9:]) + if (uri.scheme == "https" and uri.hostname == "github.com" + and uri.path == "/rust-lang/crates.io-index"): + uri.scheme = "crate" + uri.hostname = "crates.io" + uri.path = f"/{name}/{version}" + uri.params["dn"] = name + uri.params["dv"] = version + else: + bb.warn(f"Please add support for the url to crate fetcher: {source}") + uri.params["subdir"] = os.path.join("cargo_home", "bitbake") + uri.params["sha256sum"] = crate.get('checksum') + url = str(uri) + + elif source.startswith("git"): + url, _, rev = source.partition("#") + uri = URI(url) + scheme, _, protocol = uri.scheme.partition("+") + if protocol: + uri.params["protocol"] = protocol + uri.scheme = scheme + uri.params["rev"] = rev + uri.params["nobranch"] = "1" + uri.params["destsuffix"] = f"{name}-{version}" + uri.params["subdir"] = os.path.join("cargo_home", "bitbake") + url = str(uri) + else: + raise ParameterError(f"Unsupported dependency: {name}", ud.url) + + urls.append(url) + + return urls + +methods = create_methods("cargolock", CargoLockMixin) From patchwork Fri Dec 20 11:26:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC8C8E77188 for ; Fri, 20 Dec 2024 11:26:51 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.114]) by mx.groups.io with SMTP id smtpd.web11.149745.1734693993938884026 for ; Fri, 20 Dec 2024 03:26:42 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=tlmbFOax; spf=pass (domain: weidmueller.com, ip: 40.107.22.114, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=AKpWZ9S+RH8ffwbOpm+XLckqb54pcZttJB9bf+WzHWX5Td8+5YBxhhS5H6n9nXKUwpBanLhobIUW/GmoxSsYEU2cIDIEodmas+fjhRjCSYCCY6NHxyHmO6Ea1ZVbo9yCIZh9TAZ+Ij8ikzMYu0rFbpgN+9LdwVUcOTcyKKv0G0ndEZt3WvaYyIIkOrIP0Pm2ACC3lBizAqW0CB0TMkrH6sFCMPEnM8y9Ss5PgSeZubeargpDRZgTaLkMbI+0TBNYI6mjaMcuT/Na/l5spkivbPIcd/X2FEFL6j/w3RwVfInG+r/CT8r0Ww+IdF/QhYFKYUUFxpoZVrVgb8Z9TWZOAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jusmXzbEL2xi/yTDZ7sX+cOybecyn5mJcQGTzq32W1c=; b=aW1oX1mb5z27nfjOqp+fELd8U1tTe9MEVlmEm8knoR83r0g0X7S7T57yB0v+12j/KbS7FcZEn/BDmr3Y5CgcQgHoWbE+Oll+Vuj+lVnDHLaytuH6HRJ2dV76HOaoERrYSqHaTyRU1Q13tQUZs77yInK2CPsaFRpNeZ4b2o91OIhaKt7C84Cd9G71Mxui0JwWmspPfr9aOwE/Mi/MRsmpl+NE1vjaYcZp3wba2I7KX1rECgbM5TsAZsG0WStcXkW2DUvJAUVF3ysaqJWOcpDLuuPmKo7PfpSy/Xcc3iiCUmLHWhujY3PUShtB0HbT7Dy5FTOQgrXahe3D5s+tYfGyrA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jusmXzbEL2xi/yTDZ7sX+cOybecyn5mJcQGTzq32W1c=; b=tlmbFOaxqba8kc3XvyXtSTAk1dF3lJGCUtuuX/g5amNM2oZQOTnjFXS2AYKDfsxop+0hj7cw0/D2Uch2rY4jVjXLupriLanhv+BD1lVKZByAh4zU+/v/WGyHqrsnw00MppN3tJqLZhUH0W6Q1vmMP6kGi2/usMRP9oM/I9WpOZ7ssTB4bEUaMBm0Up6fExOrMoiS2SUuD4mJ8FiNz6o245ALcmJ62K7AgnfTR3sY3/8uO5T5m4xJYlRrD3MlMQqKFoLvAL+/aqr1URUQMh2/qRu4f7OSIL/sFxGI59d+x6AiYiO0KP1vhidAJMnhObYNMyyo2GyaxooJ/NzhkJrGeQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AM8PR08MB5684.eurprd08.prod.outlook.com (2603:10a6:20b:1dc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.16; Fri, 20 Dec 2024 11:26:32 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:32 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 21/21] tests: fetch: add test cases for cargolock Date: Fri, 20 Dec 2024 12:26:12 +0100 Message-ID: <20241220112613.22647-22-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|AM8PR08MB5684:EE_ X-MS-Office365-Filtering-Correlation-Id: f168832f-9c2a-478f-6295-08dd20e926de X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|1800799024|366016|38350700014; X-Microsoft-Antispam-Message-Info: Z/k0wDjBgto+yCzQKdYxcFkbGEszZUGc5Gm9TZZC0pPdoKJoOXDhH9uDagA+BwGqHiGRoha45RerDnjZaKROWkxjlax6L64nw37wPmv8xdt7d7xjOE2Vt8QZ4zM8UECIBxzaHwnTtnBXepNVqsdSqBd6AAmWD9X0Gxs/nnSVhihu0cNpJ/ST0Jmp67nEN1nOmvr92GmFtYjjr8xxy3JIsPi8P2N+0dk6juKOpeQmoEs/SG7bldp+roKD5kd71pCuG9FX2cmj/ZTEvbWRRMsXd3wKp0L085Hq6UzDJWSZN+mvF+qGLz1RiTXlNEYIgnS2fNmf1TqC4LGgFtcNNpX3jLWqui89OM5lgKopfup/qpRnyrLbuhRVdn6/66ULziPomG3MnYvSMy6TOl2WcOITdDXHzKONFdHa3BR9gQjWayN33061coRl2+VJEA6psYGt5ovzkILjayvsRwKFTW5EhyS4MX5vz9qidJK6L0ydBW9OLZDuZ2biUbCE//zPdhMPGl6izgJgfD4Kd9iZAryBYL4i6OtaFQB7ef/ryHGt6YB5ntIr7jwIZh8JJdDybiZ/5FYpKLLTi2q9fXEXC3id11Wo15Xs3GyGUsYghG2cYHvVNyKZ1Yp8aqzLVQzjDYPhMpOmmQ19TTmw1HM74GKQUOMlkfnmNMZbhpGrSQ2Eqa9nCIuR4gWVZ5R2plT+VjxN2YQPuZ1znp+blWU3S4nzz5zY7oEgcbqnkvuK7DsT+5I5WCcUXW42JznGUMJXf4lD28XLRI/56OpAQOOCOSoGk8jDSC3am6t9aWPGpKADS512RlOX6rXokQCVvwNGtrH7fdwqL+p8IbEBQ2DaQEoiWnqxXqYWIFdphFjFdpZ4/zsk7jeK4gIEfPD6KHJsvqkOSQKsYHksr3a+mYRh+S1NHxJBz2MwNvqhE3KbO3vljmvnSZOgXPCS+0NrWi7OlD8iGepasabk0EnsueFdNr1K5Lv6Df18Uer8XfPScygfQlIkGzuzWx68MW9WUdfEYT4m74kJAbxkPvear+sRzLL02LfnG/bwY6fuF6Te4HaYAhONCGdaPADj2qEbPB3XtgIGZ1VSIpgLkv9/Tun/3JqXimHaljKnrH/cbakFnJvoIUV5sLufBdwekxB8VunLJU3GRK49VZ5ZJ5FoUGrnUTyjCV5ovRx51k5XcaK4HadKPm3mocVm7HcPCa8PVDR0ohHhcHvLqxOgaM/TE+4p1UGrPcMc5j2fayVyLjLXnsB3d8/542yQu9drvPJ7ReNp7ABh10dwfY1UXQJtgWOkw1+niyjszVwZKr2qiWFlC98/RszY5ceg1Dk76vwkJTd6MpqGFGbqC5lxBFZA0LaKx+HWTpejP50fBbLpwDgXuNWb/op8RzgA8EPUZPVQv2q0/T3schLQz6D9WLTUPZAgwHV4Pw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(1800799024)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ETcggvnqcTVs7/tP6MYXjhFTZyDYRIQBvSZ90GHrYFpK8kG6DXV06u2m9cNvrhpNlPoH4xL4KcAW8I5QKtq4FHuQU4LquiTCIw4eVeuaDl9U9GXs0SlHdb829yP/srkTverNSBiz3j1SHreQ9p2Vxu7OZi3m3M/pXWtsVA0hxX1QgMLN8cVxTJojrCVJH26cgSr9nSsr87yGsfoo0UXXWVSBECpeP8BPnMs4YTqRJC/Hr0oxyBzLv5ezLrF8AC0le7NUPBOA3ejs4vZJGLGPngujt9X3I9gdH4Xap/sczTLcjZqqwpUOtw57ir7Nvgpp4FvbsTpp9c3ZqEgrsuwc+QcSiUk0giBSmfeefaFkWxmDX9c5vJKjMu5Vubxr986s2glg1dRVPnrpR62GNFzp3TsFDFAIcWIrH8Nr4jD4kZOg+j2HPDC1frlV3T/cyfm31FwYXIo6pn7rvf3SSDSvZWjACYE6W43asUg0D8fPrBCtXadV1vdSGZmWmKrtqwgLnoRwPo5DM0O7urGbD+JiY8YhkEU0EG0kVYmLb1kjjVSDOvMotuaONYp4WgyICF7LrvURJaGnqBk4CRrf+LtOiG518gcuW7G8f5w9azn8DI7aZ3LnhPMuHACzgxgvp+uoM0pbKRMs4e3VM4kb2/Lp4BJK8iq4b3kmBrvr93FTdYlyc781EBdFYvzLPxkyYS0hXns0AUtGaZoTNCt5FZWjpEqOVKwAwu6886l8bh4Y2SovIUZtEIjG0T2eIKF91wchNrPllMsgHhOPzNPm34XE/yksDpouJWmS0pwRhS/Y8jRtJm196PPBlTvpl1mrmEkYSPwanqd/Sn3urCnehnkZYzPNl1YcBBhyTMJw95Xr/kxKN2DaM/bns34Gv/ECWw3tljcUkXzR7K5Tue0w+duEPVbZ4cZ1Ac+0A4YAlsQw2tT0U7CxAcJDaKWahee3yhiWVQu9wD2YqNICnK23H7d+uBRUlusHmqgE1Vi/oxzjRP8136AHTPrgiz6uqxyCcU0JDiZvVN4ShY+MTyRVAtOVG7bzvgYRKuD8/lYeHqfhJV+Ps7S1oP+XQA3Sq5GX27HjTK6iWOAAE7lRK43cC/xaQKCKwibSIMV9VQd43QVXeU1YuK0nZ7lQbh89gLt5h0V0yAeUG/gmoI7raIi2DBhyPwu27oCFRhl4ZsS59AabXbeZlkJTj/XZNhT29bnTv4PPizXDUiNws9Yh3POdkcrhD6wI086efiJeBHViQ+X5T7mmH/Y4kgTTwcNlpnOE5A8b3n86JjoYW7J98VjU+5QAIhVfRVQEnQzQDKKRuOZY66ZX3hPY+mx1hL5QRpTNBU5JqAXRl76uYNBdmyAb3Ytn3mJV1o+1d98toDJUrhf9uFj/cCkOJL3fF7mZ6UtrQgdLgnqzHgY+iqB9GByj1lFLasiPhD8BFMINlYWm6SvVYb/Xm79ZfP7VmDvzZ4nUSnvId30Wv9xZFJQx+/gKa2ui43bhxuqRQSEu09vLfhLFNN/FrJ4OQpPljBiiGN1Q1/Xo6yJyNNj2sFX2RTHNds09nhAS1h+6PCf6GPfZ9z3xwz/xSGEb7IxuhlG6gYFK0nWsRWDJgUaOUJkTQbp6s774Kw== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: f168832f-9c2a-478f-6295-08dd20e926de X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:30.8972 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: sQ+d3w/ACy0LdfLBBYxq0mjCdHgbnQ/5kfNC03MsUFi/vur6h+uzf3Itt3AyqafNhav1MRx2zW6tf0jUd8jS4w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5684 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16941 From: Stefan Herbrechtsmeier Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/tests/fetch.py | 75 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 30c821a67..37146c087 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -3495,3 +3495,78 @@ class GoSumTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "azblob/go.sum"))) self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "azblob/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20220425223048-2871e0cb64e4.zip"))) self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "azblob/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20220425223048-2871e0cb64e4.mod"))) + +class CargoLockTest(FetcherTest): + def setUp(self): + super().setUp() + self.localsrcdir = os.path.join(self.tempdir, "localsrc") + os.makedirs(self.localsrcdir) + self.d.setVar("FILESPATH", self.localsrcdir) + + def create_cargo_lock_file(self, data): + import tomllib + filename = "Cargo.lock" + with open(os.path.join(self.localsrcdir, filename), "w") as f: + for package in data.get("package", []): + f.write("\n[[package]]\n") + for key in package.keys(): + f.write(f'{key} = "{package[key]}"\n') + return filename + + @skipIfNoNetwork() + def test_cargolock(self): + filename = self.create_cargo_lock_file({ + "package": [ + { + "name": "regex", + "version": "1.4.0", + "source": "registry+https://github.com/rust-lang/crates.io-index", + "checksum": "36f45b719a674bf4b828ff318906d6c133264c793eff7a41e30074a45b5099e2" + }, { + "name": "regex", + "version": "1.5.0", + "source": "git+https://github.com/rust-lang/regex.git#9f9f693768c584971a4d53bc3c586c33ed3a6831" + } + ] + }) + fetcher = bb.fetch.Fetch([f"cargolock://{filename}"], self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "regex-1.4.0.crate"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "regex-1.4.0.crate.done"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/github.com.rust-lang.regex.git"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "Cargo.lock"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "cargo_home/bitbake/regex-1.4.0/Cargo.toml"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "cargo_home/bitbake/regex-1.5.0/Cargo.toml"))) + + @skipIfNoNetwork() + def test_cargolock_https(self): + urls = [ + "cargolock+https://download.gnome.org/sources/librsvg/2.58/librsvg-2.58.2.tar.xz;" + "striplevel=1;subdir=librsvg-2.58.2;" + "sha256sum=18e9d70c08cf25f50d610d6d5af571561d67cf4179f962e04266475df6e2e224" + ] + fetcher = bb.fetch.Fetch(urls, self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "librsvg-2.58.2.tar.xz"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "librsvg-2.58.2.tar.xz.done"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "adler-1.0.2.crate"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "adler-1.0.2.crate.done"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "librsvg-2.58.2/Cargo.toml"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "librsvg-2.58.2/cargo_home/bitbake/adler-1.0.2/Cargo.toml"))) + + @skipIfNoNetwork() + def test_cargolock_git(self): + urls = [ + "cargolock+git://gitlab.gnome.org/GNOME/librsvg.git;protocol=https;" + "nobranch=1;rev=ef5c94d8362c35573d7eb651cf9a07c6df9df6da" + ] + fetcher = bb.fetch.Fetch(urls, self.d) + fetcher.download() + self.assertTrue(os.path.exists(os.path.join(self.dldir, "git2/gitlab.gnome.org.GNOME.librsvg.git"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "adler-1.0.2.crate"))) + self.assertTrue(os.path.exists(os.path.join(self.dldir, "adler-1.0.2.crate.done"))) + fetcher.unpack(self.unpackdir) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "git/Cargo.toml"))) + self.assertTrue(os.path.exists(os.path.join(self.unpackdir, "git/cargo_home/bitbake/adler-1.0.2/Cargo.toml")))