From patchwork Tue Feb 11 15:00:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 57101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B78EC021AB for ; Tue, 11 Feb 2025 15:01:04 +0000 (UTC) Received: from DU2PR03CU002.outbound.protection.outlook.com (DU2PR03CU002.outbound.protection.outlook.com [52.101.66.76]) by mx.groups.io with SMTP id smtpd.web11.1128.1739286052946746079 for ; Tue, 11 Feb 2025 07:00:58 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=q56IapI8; spf=pass (domain: weidmueller.com, ip: 52.101.66.76, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vwa7obfkBDVgg52R7//6N5RzJOqcm/K2xgKpf95WWXr061Hg3CyExVetNdc9jMZGRqYVo9RvY8Gu/y/K8mbj8EN/vUwhKsNMJsbapSmIChxMBul5YFrnVTk7cXrUuXLWyzkexxyU6vXitJ+cnVBD2EdGYfQOPGkFpsDGd40BXRVmKQMUvo2EgZMWirbrQiMhtKKRAo3roZzPlkIJ4wTN1elOU1/I1WvEtwIU5QUERq82AIPETMf9vLHlDxw5OjiSsZM9Pknkwe3goHV9CP7Aam2dWC4XEI3OzFYX0tQvWqH0DIg9k43VhR7m9j7TfR3drElQsCiQAr4vibFqig2CtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=J4JXh/EGRc4IpqrK0PJnmUqNSOFgHWS6OMviH3LpTAU=; b=HZp/gJZE2P8la17x+ZGQE4RisbMd6LjqmlXAzOdtMir3gSlAQJvR8hBtrfK6MYxmKAA1/R50AhcQNd3+5wSSpcFpc04LWFrUI/3jr/7OkQNukQ4Lc6FYjAeBX0oysOq8Pjs3E0qxw+rtIbPuZGjuM5U7eRnpOhTU+LuTDJIx4mgP1MKUDe2Z8tudhvomYWd/Zqy1GEja8P8er7Ky/HovwjRyI9jwf+pOF9AfhUPXjrV2+B16jOOteRmm+KXV/QY8GiVMK33sdP1L9zEOvVMABCCM7Ry4S3K+oVEMmZF4wcyBkN29eH/HLsbeF+WilL8wFF5wt0UnX/NuqrTYmrtMBg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J4JXh/EGRc4IpqrK0PJnmUqNSOFgHWS6OMviH3LpTAU=; b=q56IapI8Gf14cPk3Bny1XZoFqACQ5mbYo6g2DqMmJNJ/+jI231/yph3H2q4TgUZYZz5CxJHfe+ywgi2Kky+kNqoYsRHGRL1asqUOMIyU2gQj/usqtEzppTOoVxEULYA+IeMckwp94sGq7neWlejJ/6ob/1+LUPJIZLSOh1kM73WkYSoWT2cCiGn2s7sDyynWNBes8PTnVTVL32TVhz8fybPOEldIrhBiVqqmO2YVNyUtKc+Mrx5NGrSwgyVUySwiybAT6AIjnk8q4JMNrwy9I5KCrPmKsiJwbT4gAwelxPmHhV8rdrypQu8v0a8Zx8CKqdlxDDDu8u+YpzZN15zZrA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from GV1PR08MB8426.eurprd08.prod.outlook.com (2603:10a6:150:8a::17) by DU0PR08MB9396.eurprd08.prod.outlook.com (2603:10a6:10:423::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.20; Tue, 11 Feb 2025 15:00:53 +0000 Received: from GV1PR08MB8426.eurprd08.prod.outlook.com ([fe80::f9f5:b4bd:9e01:9013]) by GV1PR08MB8426.eurprd08.prod.outlook.com ([fe80::f9f5:b4bd:9e01:9013%7]) with mapi id 15.20.8422.015; Tue, 11 Feb 2025 15:00:53 +0000 From: Stefan Herbrechtsmeier To: openembedded-core@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 08/30] lib: oe: vendor: add npm support Date: Tue, 11 Feb 2025 16:00:12 +0100 Message-ID: <20250211150034.18696-9-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250211150034.18696-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20250211150034.18696-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0103.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a9::16) To GV1PR08MB8426.eurprd08.prod.outlook.com (2603:10a6:150:8a::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: GV1PR08MB8426:EE_|DU0PR08MB9396:EE_ X-MS-Office365-Filtering-Correlation-Id: a2d05ec9-f981-4936-1fbf-08dd4aacdf25 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|52116014|366016|38350700014; X-Microsoft-Antispam-Message-Info: gs9O+HQleI0pGDxaXi4ci04E5PJyFUnK/NHXbEErt6+5lHG3dwWSbWm0CpH7RLmZFlmmeJuVsH9W7HGYApAJ8L1YS6mleAhX66iLli27lC0fV8DwwRNs4YssIkxYpUImxguoIfwkLMbiVyPZXzlZ7iJ9XiAnl8Ud+Eyht0+dLlNJ+MwB95VttHuJOBBM/be1HB4sLZstoljBDbiZSWk80WT7xK33NLc9YsRz055OkdMj0OYs1KFBWnG9R6jEc/b9CRHWxWsx36zKzt44kqTu6MFH7SW6FWAwgTjlzrjsizwKXqCfb3b/PzcvjlQPeVYb6wqlPHBBcTwxhgoXM1TZgn5amuf0TPz6Dz5MG6B2om5EaMiSM7s/cnJiU4A8NpLepwpC93K+GSO+KV0npt+QwPvGu7WtQ0j+mXXTeIghqqcsLEpw/mlkNT+oqPA3vfyK4RgvrlW/qAf8n9+56yMWzwYvoU5YFA+1CrSeyepd4Ol0iab9F5qWNKLq1W97RrclCqHNiKHveKr9kqoMS49WJKB2Eh7tBD/l5bw8KRbkI1RlQJc4+Iq0pxhCPyyNbdZqqcs+sLG0d9q2X5KN+HpswQyk8/vQDEXoNMQzWk80+aotPPv2ch4Y+KNwPK4IOlvlOwhWlTLajOFHD0ytMmlkiQwzN7pPxtdbszbLFeIQyYltBGGDsFR4uhf4Uo/YkwVEVJOABgCtB4Funzgz31zUWQXXBwLkkHSqZz5/MOLQqf+snpMby08e79BIaaBxmF4gaSpgjGIfgTSUuFc2uPsW/PfA5VP1K+o9CTfMngvBBuUYUTvc+T5FaleCTkh2v/OsLKyhBXQWx+MP4NCvN5q2Hud1eTum0PB1s1N2zFdWjvCIKbWqrJtLNuTG2FTKJklDjKLxWBdVptpD8OvWL8dKKr+BhvIUyRkMW9JrvTEwOGckAmjBxLaYmrVLGgOkeFw5aQ3jfbZBmB7HUIE2beCTwFcKqZzD/nD928r9DhsRS4cjdCtt29u3sCnjFxXKfPQ1HdbViAPh53Q6OturOjepp3m8ZEX5I5NTQaggoza4vvEz0HQoWkjay1bcQJG6q/6M1JAIHq2Poettv5s+XjkM9GSUOzdGaZdyVtYx1mMsZO4OYkoJ1TwJcqIJapYQCfqbnpiuZnkDPFvuYMdj6MDVxei659BYFj8h2+Wzi+QZxVAvqk7XYoBpBXOrm2c9mv+MiaAMYbK4Lavt/jSbSUgrAXwSQ87wJ+cJMhkTNMGmH0JWem832z85M8AWh52mTopFshgSYp7IC7tKJ0j8Qp6jGviTqbTqzM3jZSdGgiFlZcq1j1mXQrsnwcR9ABlQy/Z+c3FT6mqXHo128OBRigcyFzuiwL4qSO8AcuD/GBujMvClCzKGF/kpoVSRfLJnvIGm X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:GV1PR08MB8426.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(52116014)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 97NdLSPiXn5f3XfXLtIKFgMfJYa/Q4wyqYl7giiHZKekXB/pBZRYoyymF5wk653QRQQBPSQun/QAvO5idtN4LgPWCinLszyG26RTUu+nkuP/zhHccKW/TIGKF5a/qj/d1W933O+pjwEikaMPhdA/5dH/kIjrnzfbrEG9ZRybTqn+y1wimG7YjILmMPazXCe/+jlaossaWL8Awrpg5/VwYmr1Iwnsy3XLqAHl8aL+pTLrr0KDWU04kwLG6lleP5Z15ORUHwgrozn3KprlRasj90Hajmmz1k18cUN5qLlrx1Xb4xdzxHUh4ii/iYJnl5k2amGmOCAqoRv1LmACMefhYE9UK7I6KsbYaGP6//ZlyvX3C7BLezwyKAkCUs5L/ie+dvJ/I5vdlscZWb9Gb2nP5W7238XuhF4LAFIEuFkM2XYVTLDWfZzeMfd/NrZQ3lwr2A4cEJnpPNII4BX3mJTnvX9XX5XGqAw8ZT5L+786Wb0Jhlor2XiCRvHlyURVPWHs/qyMFh4tShQjqz78fFSt86SEUT3FPmFmHYmEhPtkoRXNmBNomuyesZfgYAayuNTo0/7HMPiT7fscL7ZpQ0RrJjtOHvhDdnF+kK95SEtrduAbQJeFiuzonPf4Db1Nym/X5bn4AbxBwqTAva5H7MhEg3WWa0KL4Clwd0HbzqkXOYBl6fjixJjq6FDNJ7kX0NUhh4OlDXpuwsxoaQlvSJqhhB+8jYsKOYfbjxA+9mTvL+E7pFfx8wYb1RnwZVcgGMtmOesoL+ZhuC59CfPv0dNpA8KrNVC5sVMKuifyjUMEEFfEiEjb+evW8SPQJSxrvMA2f1fxgHq8r5DHL93ggnC8Fp3YQ+XdCVG+qKgW1Xcz7By4GGtU2dfa7NoGcAgn5AvX/Lah/yPiQpVM8/pkHQJ4tEnXTzv9SzXRbpdH+ix7GOG1OQAtgdPcg91UTMVbZzqsan2TxUnOgav2TdZV3ElTFapDC6fY+FohcXt8PvEmnUmD1UzTxmCbDNrw1sKH9Gz7TJKxQITbL2jfpS3PJHM9qTzUm+l824p5kQM3cl4FHJZdX1xvr1Wx/krOfIhQWfp50FSrFkdC7jTBxPi220KIG04UgTNHZnYQAC7p9uAd/Ra7aDPeJvfLCkYizoRU17TAKPNx8rW5wAy2A1TyefcmE7z4vcBhCDhhpPOYdEIrKfjxuVfXG+k1qWWeXgketyknXhUpYNcy+t8iaH7iFH2PoUeX120fthW7JHMxK6jdYm8KjHTzyaN4MnYTbSYwmDpIiMIjYt+5V3+9/bQOyjxOYNNiUDE/6XzS+xA1sjG8xNAMVLcRbpbGY62acxhALn1cDNzWfrEssNtYpbDSOdgGogN+8UFxA4OvXcFgO0Q63Ao+QD3cg3nybU91HJE0Szhy7+T0uSDRkZEMthAJvdjW2oOMHtdyN4A/byf2ki6/en7wgNxvq5NzFMkfbF1rakfvDICQhS0HExjMU+TPtrZELxC2YhfSnAfpqYWMOONZG77dUbajkMPBcqaFY4O2reP0LuTtBeRQV7fRm2rbQjo4OoLd+TZz9bkyrA1PNOhePpqLLSRSY7hmeq2ZOoARBD7s2jcpz3L88G9FbO5U8QY4IQ== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: a2d05ec9-f981-4936-1fbf-08dd4aacdf25 X-MS-Exchange-CrossTenant-AuthSource: GV1PR08MB8426.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Feb 2025 15:00:49.6329 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qrWE5IO3dQWqhBznBQlGMaknLGXl9FrSzD9IUNCTcTmxoVgQIJQ58jZduzpnph2gcBLXQuIuH5zYFRbRrHwQuw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB9396 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 11 Feb 2025 15:01:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/211133 From: Stefan Herbrechtsmeier Add a vendor module for npm to resolve dependencies from a package-lock.json file. Signed-off-by: Stefan Herbrechtsmeier --- meta/lib/oe/vendor/npm.py | 141 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 meta/lib/oe/vendor/npm.py diff --git a/meta/lib/oe/vendor/npm.py b/meta/lib/oe/vendor/npm.py new file mode 100644 index 0000000000..6dcd756dcd --- /dev/null +++ b/meta/lib/oe/vendor/npm.py @@ -0,0 +1,141 @@ +# Copyright (C) 2024-2025 Weidmueller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +# SPDX-License-Identifier: MIT +# +import base64 +import glob +import json +import os +import shutil +import urllib.parse +import bb +import oe.vendor +from bb.fetch2 import URI +from . import ResolveError + +DEFAULT_REGISTRY = "https://registry.npmjs.org" +VENDOR_TYPE = "npm" + +def determine_uri_path(path, name, version): + return f"{path.rstrip('/')}/{name}/-/{name.split('/')[-1]}-{version}.tgz" + +def determine_downloadfilename(name, version): + filename = f"{name.replace('/', '-')}-{version}.tgz" + return oe.vendor.determine_downloadfilename(VENDOR_TYPE, filename) + +def extend_uri(uri, name, version, subdir, checksum_name=None, + checksum_value=None): + params = uri.params + params["subdir"] = subdir + params["downloadfilename"] = determine_downloadfilename(name, version) + params["striplevel"] = "1" + if checksum_name and checksum_value: + params[checksum_name] = checksum_value + +def determine_src_uri(registry, name, version, subdir): + uri = URI(registry) + uri.path = determine_uri_path(uri.path, name, version) + extend_uri(uri, name, version, subdir) + return str(uri) + +def parse_lock_file(lock_file, function, dev, bundle): + try: + with open(lock_file, "r") as f: + package_lock = json.load(f) + except Exception as e: + raise ResolveError(f"Invalid file: {str(e)}", lock_file) + + packages = package_lock.get("packages") + if not packages: + raise ResolveError("Invalid file format", lock_file) + + for location, data in packages.items(): + # Skip empty main and local link target packages + if not location.startswith('node_modules/'): + continue + elif not dev and data.get("dev", False): + continue + elif not bundle and data.get("inBundle", False): + continue + name = location.split('node_modules/')[-1] + function(name, data, location) + +def resolve_src_uris(lock_file, registry, base_subdir, dev=False): + src_uris = [] + + def resolve_src_uri(name, data, location): + integrity = data.get("integrity") + resolved = data.get("resolved") + name = data.get("name", name) + version = data.get("version") + link = data.get("link", False) + + if integrity: + algorithm, value = integrity.split("-", maxsplit=1) + checksum_name = f"{algorithm}sum" + checksum_value = base64.b64decode(value).hex() + + if resolved.startswith(DEFAULT_REGISTRY): + resolved = resolved.replace(DEFAULT_REGISTRY, registry) + + subdir = os.path.join(base_subdir, location) + + # Skip link sources + if link: + return + + # Handle registry sources + elif version and integrity: + # Handle duplicate dependencies without url + if not resolved: + return + + uri = URI(resolved) + params = uri.params + params["name"] = name + params["version"] = version + params["vendor"] = VENDOR_TYPE + extend_uri(uri, name, version, subdir, checksum_name, + checksum_value) + + # Handle http tarball sources + elif resolved.startswith("http") and integrity: + uri = URI(resolved) + params = uri.params + params["name"] = name + params["subdir"] = subdir + params["striplevel"] = "1" + params[checksum_name] = checksum_value + + # Skip local tarball + elif resolved.startswith("file"): + return + + # Handle git sources + elif resolved.startswith("git"): + resolved = resolved.replace("+ssh://git@github.com", "+https://github.com") + repository, _, revision = resolved.partition("#") + uri = URI(repository) + params = uri.params + scheme, _, protocol = uri.scheme.partition("+") + if protocol: + if protocol == "ssh" and uri.user == "git": + protocol = "https" + uri.user = "" + params["protocol"] = protocol + uri.scheme = scheme + params["nobranch"] = "1" + params["subdir"] = subdir + params["rev"] = revision + + else: + raise ResolveError(f"Unsupported dependency: {name}", lock_file) + + src_uri = str(uri) + src_uri = urllib.parse.unquote(src_uri) + src_uris.append(src_uri) + + parse_lock_file(lock_file, resolve_src_uri, dev, False) + + return src_uris