From patchwork Fri Dec 20 11:26:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 54457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABFD8E7719C for ; Fri, 20 Dec 2024 11:26:41 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.114]) by mx.groups.io with SMTP id smtpd.web11.149745.1734693993938884026 for ; Fri, 20 Dec 2024 03:26:41 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=AKxmv8dc; spf=pass (domain: weidmueller.com, ip: 40.107.22.114, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Xpbo+9OdTFYOeScZidyeKRQYOwcCfcK2ausTRgvJ7wBA0t5cOgactJaewfy1nDMuVfdrL9vW0aHhphYnGE3Rx557ova6DEEZmMuUf2M22vowRfTtMxDTEnUZeeUVka57mhxTkWkMwRdjxaRLNPoZUHXadXM1qZtsrRyb+MGMAqRSiXYy9t0RXiqh40+pvxDKEhhe15rDdZQI9nCGffNSiOQeSuR0afR2/zN8rkfT0hQOo4DbRcbF/l2uwiSAeQ/OduM6Vb+7NByfwc3lvilx01qbkJ8NYHAKMUKbf1z1sEawk4WsexL8v5XoqUyDfueQdaAny8b81e30Ubf8G+jXbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NjSgsCSlyuNISja8yjNqfAPuN8YK7Pt60PQplva+QQo=; b=qyXHLQ9haR/fziAH/YFSKgsSije6Xrs2zgDpOdVi9tnt5+rDJEdR6OLUr1pud7PopWDeW2rpDvJAGcITd9h2fH7YQKfjq33jyJ9J9MG26dowjBmXIB+mIZ970hGgT80EV0cDLOsZr/4U+HEz24rcmrhdmz5cXDKyKXjb5ywD06d5AwX8JPZfpzUDPZTTzkpXm9788Odw1hncrgkEm6zPpqqsq+OpRRo6gjB2Qb8msSuAGJrYKqO5U91TtTKoUiquKvJRX9gM6u6IanZz0XDiZEHOIaSZPggDtVdyLZHnfJP1QFV1sPto7dzaP5IIO2MwdPsvIQiGc3Nd5WQv9XslSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NjSgsCSlyuNISja8yjNqfAPuN8YK7Pt60PQplva+QQo=; b=AKxmv8dcu9kK1gf19ivna63jBxg4vDRiUul5lrF+9fo8/762evRIeSWLEiN2roTART18t9OpnlgmgdQ+PWeR1Ww2EJy+totYrDU0zK9k8V3C5kdzyENjPVxhO7YeSu0sfZpf6KlmlDR32Iz7F898+Vf/yMSmiyKILNVshWcIA7kFPyQtyI23zgaO+EFAoIbsPwmmyEO5YO9nRkszbBzd8hHWwgi6jVn/CcSrnMRb4BKL3PlY/3kM478Ll44VCUNHQ3z1pnyWlnzFc2w+5E+9tbUPcyGudTB9pBEzSLPJDzrYE52mOOMoVzd73dKob+MvGqupQT67apkKsXSw7Psvwg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AM8PR08MB5684.eurprd08.prod.outlook.com (2603:10a6:20b:1dc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.16; Fri, 20 Dec 2024 11:26:32 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::3b1:b329:1ed9:dad4%3]) with mapi id 15.20.8272.013; Fri, 20 Dec 2024 11:26:32 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 20/21] fetch: add cargolock fetcher Date: Fri, 20 Dec 2024 12:26:11 +0100 Message-ID: <20241220112613.22647-21-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0106.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR08MB6969:EE_|AM8PR08MB5684:EE_ X-MS-Office365-Filtering-Correlation-Id: 3ab7e62b-08dd-43d7-91a2-08dd20e926b0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|1800799024|366016|38350700014; X-Microsoft-Antispam-Message-Info: tooRTjcqnbPpKhG7Kdt7LS0BVAS41kC+mFIs2SZxLwsN6EbhA2bh9msfF78yiF2PjjmEL56y/aCsjlfdHtdzAjSTGdDoFWE/W+G0N3SfFGCeAuC+wGXKuD1Zf2674/1CiZTAVrJuKjCn22KuPspoP03nKZ+XQiKGks6up7rDdQDVlYmDWPlkxnJkwJk/y97TYsg6V9Xl5Vq6EhthaGQ1sRsNHA+Cl/oRViwzoLWKwEEGCBZgZWbAjnjpSYrYY5opTlZN+p1JBMv+Fy5EzRGYHMrR2/fLXH8LNfm4JcMNpwgOrj6vPG0SWBEl6LgHkJOYu6taKxNUc3x5ldCAt26Mc0wWHsWqPcUaJAb7NpTAl2wcXrILvGnIktvVpbUDclfoa2jNh7OGXj7nLOS6d+RJMV3P+mRvl3E2I/8WJKbhEJyeOz9xSuKzrxrHWy65ePiik89sT2+Ks+4/+Qhw+PyEg0HtutPMgLCcHKsS2rUc22oimoRSV4pAHfVdlCxeoVYxGb43FQSRt2p5QyzwgTnNBzyrtGkkMOb5uD1xX9hF31TbGkraRSD+8QjSLXd2IAlGeZrtBKAhDW/XP8FlwgStftsVXh7uP71TU5gy8ochtwSmspdZu9f6rm64C406cEeR+xZKRoU5o9N8E/ThrC/r3397WXmgL6KvDwVGMhe8OFT50YrYpN2SaxHoH1sFQVvoKExwmgr3oxDWM+C3g/0Jb5/5vlMvycz3OO20wrJr4MtAkw/VrQnVeez6SHCa6/Jxv+Z12OYGPA/t1yhSxM69WHuUatyw+S2fJPcmlx3ZfvAHbnqVoGXelQU+dMW1WknSYoy2JQ1hN34DinLdMLHi36e3NnDIEQ7k84O+e+RSZZoUg2IABy14sScdgk25PNZK5/RgYYZRVIe4TjWPFAs95Mc81TD0W8XOWye1+WNwBEhh7p1qNFLNXKCMvztS3ksR/qt8X1HN5uTnI3Sq7IOkGL2lFGDW1HpEo5ETFntWrYMBVDkQRdh8uWepHHsHdEcPcQtuF8wTx43awE6/Ducxn1w29PztbNCv3DPFdirAKPeFyO5Zf9Yxrvw+Eimrad4TLbiGZuKBRaBydy6zmTef8cM3/XDNxz1SCOe89giGCVPwqPrMipWUht6WkTjQOGRWfeYqi9okGS71NGRphHz8F48TTbZ2dz+ri34X+ft102h1vcK6DL1bjb90W1R7qU9mNhZ4cXXdPe3yRRCabTJ+094R/gXT8wdziW4eN7zmoGKzJI1IttuyH8bVFLrg6m19gXsV/Gcoh8pUbzMnNfvC4M9j37N5JCcdnvD7K2VUMcN/eeaitIKyWIIePbKwPGlxofDlZRB0EF62HqvILoS2KD35Q4EVBw36948Ntg0klHeACirolTfoC6r6fVc1X6Js X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6969.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(1800799024)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pBbFdU1KMnAY3A3rYZYG5NhGXhS5R5zOOi6nDN5baNGJ57/h8C1yF94kHWLsz/DRuVtaw4eAZdf59R5RNyXxsLNV/EOP0xs9UCss4RHYSYmM7waqoauUuRYRfbeb7jBKuqUv1AeJ/YkSQdM5I6XzktO7Sux7HM85wLXV+2xdpd8WtVjsdNITU3h/HJRHdTvmSkmdiAJjoSWtJU14rYvJe5fRK1mZK8KvI2MkWmJUwNoelyjNAgNbunShrPWFUdd5IOlH4QztgbDNmQfaXOB2M8VJ76FsJlNHS5rOLMYPu2fh6ahcD+EdFrvQFrSTU/Gbj2mPaY7i0i+pwHJ9xHACnWGmn3rorSOuCZwP0HTLYFpSSyo9X0r1A2cteKvPA8hsBQfOWx1ZwnV9re8AlG17oc+6SB98zwIjfnpkGRbULjgB1OO3fDDV+hftEG7W5pt3yJb1rMwYeYsLoYhGWoAmYReZSeUKqjBU2PhX9pWHAvNtAmDHmlavf3kHGX7O0RoxD5WHi7Fhc/s5+ssRnag4ttMVnefZUph6HBgronU1Pa2cl6RTkeZZuUT5jqAQ+ufyknae7T89SK+UebLNYECFLFlwzMXrtbI1AFFJtluj70oAqq4kFmOqUTL8rs8bvaE4aKzIbklh1CUFy2InhZItjG2miavAxu+0fARHMfQDVVIJTJsWkfoDFdhR1087aELRc8CZA1jZR8aUXuZFtU4A/XFMqopUivDIY0Wn2ev4ouzAoeNJEUTsyxVLsjh8dlt4/Gq6H9lla4dFzVJRjM9lCUpaucQz33PTnNIG1uxpuTZ23tLkP800Echd3kGMiJD1d7AbqP3XPwxQPr4G0p3125LEZhbgW+v2MkInBwKUnqEqtwudMUp7+FKv3PvAkOHOYCArmJRpaWQHLtD/r+1KK9QnoU+kp7J9vFFnjeDqrYkafSZfF1FptDiU82Ow+i6XuiSVy0X+jJ8e1TIS0kqaHo+poTbfDFANV136396/LSxKJ1e9eAZitn7XUKDNSd4Z4R43HQx+ZiFI0zv0iSzdgJ0AoRas3GkgpJtkXiZX2+6jC0/rr0kEGPMzS5pAiKsgmrKiT8mmnrveokQBhsZMWp1qx2KCPrlgI1fVVYJmDP6P1dftsjMGWTOrHHNegGVOImJ9TkgrHVQQgCWK9KX7oW+bQeR2bgA1k9W5PxfuDXrz+XChKlgabsjYXVrYMir/pjNuqhTvOdr52YPGLlUQ1awgI92UhA9+TN4XP3Z7Z4gxaQQP23ns0mxeSdPofGK3F/QNkZHwk6va3SXZKWADhegUQVALFDOg0VHa3r2HmUyHK4x4ENYKbwL39UK+TiZUuL0CKHlHfyMHkQBjgaOyE6m1Y0SmKOt8OPWiDU+kfF2aVd+pVeRr0J+5c+NAVBlEaV/i0kbTesQxG0qeaAcVXx6603p3qUpcHM8hiyEpjx16Kw2ZYqAmkeXcLwzb/R4AKTMhJHpuzVl5JK0pN+IIBSP0yQom8toN5Ff/UqcRtDCyPMc+R3dEAjokvpunMgjICrr9jPvZDWFrFaK4s2e4AHFyD7xfdeHrm5qRl5ilTyyWP7kTSLZAKFcsrkNM6jwukzIXxJkFTFa7W863W3hwhQ== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3ab7e62b-08dd-43d7-91a2-08dd20e926b0 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 11:26:30.5963 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ijLJ4AZ7EfxWW3iz4wqjo+/hltuP4XC2aLKlF1s1j23jx+ZEkO4LLwRJA6bp8e1K+ZRG9ebOqfH0wZMxilqrpw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5684 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 20 Dec 2024 11:26:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16940 From: Stefan Herbrechtsmeier Add cargolock fetcher to fetch dependencies via a cargo.lock file. The fetcher uses the dependency mixin and supports different types: cargolock The fetcher uses a local cargo.lock file to fetch dependencies. SRC_URI = "cargolock://cargo.lock" cargolock+https The fetcher downloads a cargo.lock file or archive with a cargo.lock file in the root folder and uses the cargo.lock file to fetch dependencies. SRC_URI = "cargolock+http://example.com/cargo.lock" SRC_URI = "cargolock+http://example.com/${BP}.tar.gz;striplevel=1;subdir=${BP}" cargolock+git The fetcher checkouts a git repository with a cargo.lock file to fetch dependencies. SRC_URI = "cargolock+git://example.com/${BPN}.git;protocol=https" Signed-off-by: Stefan Herbrechtsmeier --- lib/bb/fetch2/__init__.py | 2 ++ lib/bb/fetch2/cargolock.py | 73 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) create mode 100644 lib/bb/fetch2/cargolock.py diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 10b4cf24b..012130ac6 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -2117,6 +2117,7 @@ from . import crate from . import gcp from . import gomod from . import gosum +from . import cargolock methods.append(local.Local()) methods.append(wget.Wget()) @@ -2142,3 +2143,4 @@ methods.append(gomod.GoMod()) methods.append(gomod.GoModGit()) methods.extend(npmsw.methods) methods.extend(gosum.methods) +methods.extend(cargolock.methods) diff --git a/lib/bb/fetch2/cargolock.py b/lib/bb/fetch2/cargolock.py new file mode 100644 index 000000000..18df160ed --- /dev/null +++ b/lib/bb/fetch2/cargolock.py @@ -0,0 +1,73 @@ +# Copyright (C) 2024-2025 Weidmueller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +# SPDX-License-Identifier: MIT +# +""" +BitBake 'Fetch' implementation for cargo.lock + +The cargolock, cargolock+https and cargolock+git fetchers are used to download +Cargo dependencies via a cargo.lock file. +""" + +import os +import tomllib +import bb +from bb.fetch2 import ParameterError +from bb.fetch2 import URI +from bb.fetch2.dependency import create_methods + +class CargoLockMixin: + def resolve_dependencies(self, ud, localpath, d): + if os.path.isdir(localpath): + localpath = os.path.join(localpath, "Cargo.lock") + try: + with open(localpath, "rb") as f: + crates = tomllib.load(f) + except Exception as e: + raise ParameterError("Invalid Cargo lock file: %s" % str(e), ud.url) + + urls = [] + for crate in crates.get("package", []): + name = crate.get('name') + version = crate.get('version') + source = crate.get("source") + + if not source: + continue + + if source.startswith("registry"): + uri = URI(source[9:]) + if (uri.scheme == "https" and uri.hostname == "github.com" + and uri.path == "/rust-lang/crates.io-index"): + uri.scheme = "crate" + uri.hostname = "crates.io" + uri.path = f"/{name}/{version}" + uri.params["dn"] = name + uri.params["dv"] = version + else: + bb.warn(f"Please add support for the url to crate fetcher: {source}") + uri.params["subdir"] = os.path.join("cargo_home", "bitbake") + uri.params["sha256sum"] = crate.get('checksum') + url = str(uri) + + elif source.startswith("git"): + url, _, rev = source.partition("#") + uri = URI(url) + scheme, _, protocol = uri.scheme.partition("+") + if protocol: + uri.params["protocol"] = protocol + uri.scheme = scheme + uri.params["rev"] = rev + uri.params["nobranch"] = "1" + uri.params["destsuffix"] = f"{name}-{version}" + uri.params["subdir"] = os.path.join("cargo_home", "bitbake") + url = str(uri) + else: + raise ParameterError(f"Unsupported dependency: {name}", ud.url) + + urls.append(url) + + return urls + +methods = create_methods("cargolock", CargoLockMixin)