From patchwork Tue Feb 11 15:00:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 57107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61D89C021AA for ; Tue, 11 Feb 2025 15:01:04 +0000 (UTC) Received: from DU2PR03CU002.outbound.protection.outlook.com (DU2PR03CU002.outbound.protection.outlook.com [52.101.66.76]) by mx.groups.io with SMTP id smtpd.web11.1128.1739286052946746079 for ; Tue, 11 Feb 2025 07:00:57 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=F1vt8QyD; spf=pass (domain: weidmueller.com, ip: 52.101.66.76, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=JVJsfj1PIpkUAlsOR0w2ylDC7G20jkQTQ1NguX3IJie8wGN0REVclamV7N2RgAGV6jympyAj3P2X5jcQdsC9MOUed3CQoQNQ0rhEn3C1UJ6d4M8jm86nlnH+ne/usCPudBqkibZX/Scrn+PVO7ke5NvCR3oxo+paTs1bU3fWvkVPPg6bpMQ4uT3hhH4y/yut4hirSwe9x1iz0YTrjp8E5ADX4mLYE4i3hiDpzH9pWLqTNYkecGOEu6yDthFmVb0ll0n5DIudPd63gZvmdDlBmf9gaLGz8O61Aq33oImCk5boGpOYnguTc4OpLpiyJan8kbF5k1dsZ1fjTfD55dUJmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TxoSYnPx9KB8dZPCT8lLGGfADzGCDjolU+ZWJRwbVJk=; b=QwzcYe4IOZdL4qRYw1pGmaF2gnRGXEtqtzX8LfKFdMtgJWjM3fti0gLuqXkIvxgydyvBsZc9msyIYIoAC0RwVoVMKHPUQbwAlNEdWC80UDAD9oFRLotfhds1MEQrgZfyDIjm3N7Fohefsx9kFFO9AlUCh/+bKweyX+h7H7GcMtKqZqc/DdameUBpotu0lkLq95JRcPEZD8+J9ov9FHtDIiJbzlEc45O8OCIk//n2q1u3eV5qBgm/Eevi7mkPHFsa+t0pXgW5L4zWyFJHQm2nNcSbVf09qfnZF/P0eiS4n9Yx/Nwjd/mVZCU9KqWFRyLTNZiaubNQ2fxesui+fMgZoA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TxoSYnPx9KB8dZPCT8lLGGfADzGCDjolU+ZWJRwbVJk=; b=F1vt8QyDbwxMk+jAACWdEkJ4jwR5q8feH8ItfuZ1fXOkzkG2PacdqQNQ0WHXTkVpV2f4eVi/eNwCiHi6wsuTHu/izi6/gPT0fhcfqrx7P/oXKUbIQIKG9TMdn75n+sHoHU16gyCio0coEWWzDLp526IEoFUXtnw0XBCzt3E8MAUbLvYk6m9j7ooF1HMiEJc5IPWp94sb0ITxJWGmhshqu1zgRZwMPVHQxafbSboGLENVqeCawktgNW7VinfznHdfRafPRS7ZHrJy6a8WAdHouucZHJzVEsd/r2Dri/2P+vinnXGf0jT/oHDP1O9C02q5mgAukN4Gn3MOWnn7AGA09w== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from GV1PR08MB8426.eurprd08.prod.outlook.com (2603:10a6:150:8a::17) by DU0PR08MB9396.eurprd08.prod.outlook.com (2603:10a6:10:423::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.20; Tue, 11 Feb 2025 15:00:52 +0000 Received: from GV1PR08MB8426.eurprd08.prod.outlook.com ([fe80::f9f5:b4bd:9e01:9013]) by GV1PR08MB8426.eurprd08.prod.outlook.com ([fe80::f9f5:b4bd:9e01:9013%7]) with mapi id 15.20.8422.015; Tue, 11 Feb 2025 15:00:52 +0000 From: Stefan Herbrechtsmeier To: openembedded-core@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [RFC PATCH 06/30] lib: oe: vendor: add cargo support Date: Tue, 11 Feb 2025 16:00:10 +0100 Message-ID: <20250211150034.18696-7-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250211150034.18696-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20250211150034.18696-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0103.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a9::16) To GV1PR08MB8426.eurprd08.prod.outlook.com (2603:10a6:150:8a::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: GV1PR08MB8426:EE_|DU0PR08MB9396:EE_ X-MS-Office365-Filtering-Correlation-Id: a3473e45-c145-4ccd-d4b2-08dd4aacdea3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|52116014|366016|38350700014; X-Microsoft-Antispam-Message-Info: a1N+0LRPK2XYzW2qkofbzLxvnBP8LgycfU6yRkxWSxLwutpeEyBztmlgaaLXS4P6Ur3QkOtxAbNLm6lYtWyDjXPC+ZgER/a6506z7nKRnqy9l+O13jIGHgyUqJeZ5avLAFHa70p5aiyEHhX+ZxViEqSJMeVkz8Rkopku+fsZpQmA91RavyXP2aEzA/y3JUJAH9s7fNerk+En0nbGMZECgUT1MW+MZb4LRz1HmNygH5H/GvNmprtl7ngcg5FwIH1YIxxD4do4934Wrf0oJwNDwDeYX8M8skyzlZROdgeQudzvfK7nwW3bb3nXCqn4rUljLuJmFxKxqxjFX2OBF2ugggwAHvzBVw9QGf8d1hUa6YxmSeenSuLh5TtmYE2eosK7rTCRJ+yQEgaKm9Djik86++ltoXmA867k2o44H4659Vu+RYpwRYXtufJSDMBjr0p2uuFGigtmPiSHlk0DepHjf+SSS51Gxkrwyoa6ZX7ShN7Ys1wdGwCUuKdytsA7R/+it9PZREHfGCaWra8bb/d6QDfpM4cr6wExvL0MzX3Z6ul/WTG6HAjsgTembletqwu0JMruK1963G+sH/qSkEfAOD4AErozq7EPp25X5ojS9VYC4eLI4MLBjf7xSW1trMK42QuRYQqfBAYGgw3DYSl8FjP3FduZn8/u69cgxL9f2zAr5MgNJMpcPpDVMx4mvd+vl3hHnmSK2blexzEigj9TIcuud8Q2h+bJc3v6OtqoReJrWHwoMR9FYdzdapLfrYOziAOFtFoegZOD71r6uxTxP8tqn3YwiGsYMDL1kZ5l5nKScET9CVTi/XY0WeIQaqTYZ3me0+gcHK4NdorjbXk6wqE4/yTvEHk+1AAn93/QQo+0RkqH4gAlwXdmi7pfvE3sXFKbN4CQdYvPyrKbF0CenVtd1i+neqmhhYDZB4IdKdnLhxpxTWHI06elYEYMRb8a9FSaSsft3EN8+1vIf5AYHBF8VLObH8qtoFXPeoPqwSOJ7itqNlh3L7EMyztjMAxP45ZFbDy2ixCbcEaRp0gFFW+NMGb+PoclsdNSyYlxM55RUQOQPVwirlEE+R7OhxNJqd9gpQcjVFHmMThnO4Gd0xgc2Mkni0OXCAbSZOklo2qVYpB4+8Cka0oympM8EE/3DXuD7z0OfHghZ1Z0vJlYLGWXoxC/gg1gw/mx2m9pBZn7fNhA4INIDFwL+RdTAzHOyAxhMvvbigPYxMbTo9HWrPeRu3ZZ4OpZ3GzBfDDpyTqtYosFEUR0J+uOai1pHHcIvKKQpfZ7UY8jjv1tKI62FchHE3hLKADVo47KK9fB7ggLQ5xE3N/C9HJh+jUUbHatXqlAyYBJ4ldNHYfL0JpwoUUI4ulZFwQjP1/u6pE2UxYe6VpvPnR7HwzimY+Y19Ge+uVs4xQXPDM/apXp8jYCaw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:GV1PR08MB8426.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(52116014)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: XkhYjuMIIN+vou/TOTMDY1UMyjn8NgzMy+pLs1tMtfuwixWo0ebbKtJN1pL64FA/D0tC3bqvS1MzDP/zhMlekkctn7rRjRhQ5b7ZKHn+i66LbZdC4bT1PZvo1BEsBpNSPY1ciRf5hmqPDoAB6AlCgfWdJGVcQskZlff/8jU7pkciUu/xuBC6rPhSF9pUdWyX2XN2jydf0akUHCXdg6ru+DNUr2Ir/E8xtWJmZaPoraHKjIE8mwejfSLMHMiwfSDE9jSdiPikJ9Ye4Jy03SCoxmkq2hu2PGs0o0Ej6y4jmPjfCy7MIvwpNvzBwheveoorYiDbJFDICGkjBk5wJlx7oE785udotNhhcv8Ujul9er+Z40Zy0b7w+ukIqKJETICYAyVQkPSOxJm4haYANQKaRBeRTDuerL+Q0pkWe3WROwaLexmcC4cjnv5H3GOcJXV1hARgN/K3IRRhPDwS1OPhgn5cj9NHRcX5pXt2Ybbx9gzTCbBoRHMCOZDCRd8JIVp+htqjY/S+jBqWfE0rNDkKB8UmNzbyjUfP04qIwQhy1WqAnqi5FBMHmcKGCmcOq+zcAukD/aLp2r55A1k/FDHbUVbfmjfLcd81nQSZ+ICYyMX9Rz4ODsyf1IPPaSy2dBBIuNtsuTVDxwIXBX2Y7hBFCiN3+v85L9UQ/L3AX+YD/oRfNL+1vwN6skaG3C53GiTySoW++SYBoJ/Nc5JMJ/5xRa8bwSdtGv5uSGKF3m/gRZCyM0aiHsytOT97Og5HBZ5H0CQRO38h5fAXXQszUSVTLHmmThwqZMO4tVAu8E4dC1oQhHSlBM+UnTFUamz1PqmJRxlE9ubBXSjjYKTG+HY8WnhLqxlMlDjRbNLVdEialn5eu/tbU4hFPh7LaZzebwoWCTSX3mTtr1i3wg2vkjS7oxiBhteagpvBuYdjPpS213xORLLzDQMUg2C7yxfmCzEDEJgqjPjroniwed5JNnB6Asl5iDkx8yDsmCgINBRGw1QbYEk/oNJIRqc+L5S8rAFDnsIpZWmRIZfQgO6g+V56jWSXoN9IEB30LttNlgxibtNMc0njXuKvm4QBvEgaJH9DaJW5ohP1rroq/SgiY5M2umGw7fi2xhZRkUJRDtYimImLZ0LgbB5YPPpcTHYwtQnNdreBXdu4X85ODo4BCERywL/ddfzAdKYe57OQPjKLJzvzDkgrFrM9r18ITtK5sRHHF5cQMQl3ue50Zp4IBnDVb77c+7YOacm18yutZlw7IrLXxeYi+MibWV4GyIzjpxdhwKWHvpUPl2sXVNalwlINtFo50cEjWeVggYpyEh8ZND1PuFy2zFLxMHagqWj0dxEOBE6ZEsqpaZbc5wqke+b56YMcVm48hqoDtda4yAfWYnD21T+SBm42JPSOn29TelNcXJn9eUP8WggvkTAwhB9MnxRoBqgVYxUybla8Se23ggP1zMMsg2CjMDjguHRVWEMySkqX9A7/EYKRlxjGrn/icJ7OaBGxNW/qX3sTmMD/Q5ePkmOOzYcSD4MPuvrnTcwtNFf3PHahhm5kfUf7F2vclf++/RKE5VgdneweHREowGU7v1DSTc0nPxg8pX0RlWkU3mKGqAbxKRH+IazxaVZO5Q== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: a3473e45-c145-4ccd-d4b2-08dd4aacdea3 X-MS-Exchange-CrossTenant-AuthSource: GV1PR08MB8426.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Feb 2025 15:00:48.7771 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 19mB3bdn4H8uCz8Ld5loTxSXYoI7otnH42F+M9W4in71n7PSUzMDLQOzCzipVwrP2apNrDl7kuKhggLrgcxZRQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB9396 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 11 Feb 2025 15:01:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/211132 From: Stefan Herbrechtsmeier Add a vendor module for cargo to resolve dependencies and populate vendor directories from a Cargo.lock file. Signed-off-by: Stefan Herbrechtsmeier --- meta/lib/oe/vendor/cargo.py | 121 ++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 meta/lib/oe/vendor/cargo.py diff --git a/meta/lib/oe/vendor/cargo.py b/meta/lib/oe/vendor/cargo.py new file mode 100644 index 0000000000..4d0a0034f3 --- /dev/null +++ b/meta/lib/oe/vendor/cargo.py @@ -0,0 +1,121 @@ +# Copyright (C) 2024-2025 Weidmueller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +# SPDX-License-Identifier: MIT +# +import json +import os +import tomllib +import bb +import oe.vendor +from bb.fetch2 import URI +from . import ResolveError + +VENDOR_TYPE = "cargo" + +VENDOR_DIR = "vendor/cargo" + +def escape(path): + return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path) + +def determine_subdir(name, version): + return f"{name}-{version}" + +def determine_uri_path(path, name, version): + path = path.rstrip("/") + return f"{path}/api/v1/crates/{name}/{version}/download" + +def determine_downloadfilename(name, version): + filename = f"{name}-{version}.crate" + return oe.vendor.determine_downloadfilename(VENDOR_TYPE, filename) + +def extend_uri(uri, name, version, subdir, checksum_name=None, + checksum_value=None): + uri.path = determine_uri_path(uri.path, name, version) + params = uri.params + params["subdir"] = subdir + params["downloadfilename"] = determine_downloadfilename(name, version) + if checksum_name and checksum_value: + params[checksum_name] = checksum_value + +def determine_src_uri(registry, name, version, subdir): + uri = URI(registry) + extend_uri(uri, name, version, subdir) + return str(uri) + +def parse_lock_file(lock_file, function): + try: + with open(lock_file, "rb") as f: + crates = tomllib.load(f) + except Exception as e: + raise ResolveError(f"Invalid file: {str(e)}", lock_file) + + for data in crates.get("package", []): + if "source" not in data: + continue + + function(data) + +def resolve_src_uris(lock_file, registry, base_subdir, vendor_subdir): + src_uris = [] + + def resolve_src_uri(data): + name = data.get('name') + version = data.get('version') + source = data.get("source") + + if source.startswith("registry"): + checksum_name = "sha256sum" + checksum_value = data.get('checksum') + uri = URI(source[9:]) + if (source[9:] == "https://github.com/rust-lang/crates.io-index"): + uri = URI(registry) + params = uri.params + params["name"] = name + params["version"] = version + params["type"] = VENDOR_TYPE + subdir = os.path.join(base_subdir, vendor_subdir) + extend_uri(uri, name, version, subdir, checksum_name, + checksum_value) + else: + raise ResolveError(f"Unsupported cargo registry: {source}", + lock_file) + + elif source.startswith("git"): + repository, _, revision = source.partition("#") + uri = URI(repository) + params = uri.params + scheme, _, protocol = uri.scheme.partition("+") + if protocol: + params["protocol"] = protocol + uri.scheme = scheme + params["nobranch"] = "1" + subdir = determine_subdir(name, version) + params["subdir"] = os.path.join(base_subdir, vendor_subdir, subdir) + params["rev"] = revision + else: + raise ResolveError(f"Unsupported dependency: {name}", lock_file) + + src_uris.append(str(uri)) + + parse_lock_file(lock_file, resolve_src_uri) + + return src_uris + +def populate_vendor(lock_file, rootdir): + def populate_checksum(data): + name = data.get('name') + version = data.get('version') + source = data.get("source") + chechsum = data.get('checksum') + + if source.startswith("registry"): + subdir = determine_subdir(name, version) + filepath = os.path.join(rootdir, subdir, ".cargo-checksum.json") + with open(filepath, "w") as f: + json.dump({ + "files": {}, + "package": chechsum + }, f) + + parse_lock_file(lock_file, populate_checksum)