| Message ID | 20250211150034.18696-7-stefan.herbrechtsmeier-oss@weidmueller.com |
|---|---|
| State | New |
| Headers | show |
| Series | Add vendor support for go, npm and rust | expand |
On Tue, 11 Feb 2025 at 16:01, Stefan Herbrechtsmeier via lists.openembedded.org <stefan.herbrechtsmeier-oss=weidmueller.com@lists.openembedded.org> wrote: > > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > > Add a vendor module for cargo to resolve dependencies and populate > vendor directories from a Cargo.lock file. It isn't clear what is a public API here and what is internal to the module. Perhaps you could nest the internal functions and constants into the public functions that use them? Or use the _ prefix in their names. Alex
On Tue, 11 Feb 2025 at 16:01, Stefan Herbrechtsmeier via lists.openembedded.org <stefan.herbrechtsmeier-oss= weidmueller.com@lists.openembedded.org> wrote: > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > > Add a vendor module for cargo to resolve dependencies and populate > vendor directories from a Cargo.lock file. > > > Hello, Have you looked at vendor subcommand of cargo : https://doc.rust-lang.org/cargo/commands/cargo-vendor.html ? I didn't read the series but the commit message makes me think that "cargo vendor" already exists for resolving dependencies and populating a vendor directory.
Am 12.02.2025 um 13:45 schrieb Frédéric Martinsons: > > On Tue, 11 Feb 2025 at 16:01, Stefan Herbrechtsmeier via > lists.openembedded.org <http://lists.openembedded.org> > <stefan.herbrechtsmeier-oss=weidmueller.com@lists.openembedded.org> wrote: > > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > > Add a vendor module for cargo to resolve dependencies and populate > vendor directories from a Cargo.lock file. > > > > Hello, > > Have you looked at vendor subcommand of cargo : > https://doc.rust-lang.org/cargo/commands/cargo-vendor.html ? > I didn't read the series but the commit message makes me think that > "cargo vendor" already exists for resolving dependencies > and populating a vendor directory. > The class uses the bitbake fetcher (wget / git) to create a vendor directory and emulate the cargo vendor command.
Le mer. 12 févr. 2025, 17:29, Stefan Herbrechtsmeier < stefan.herbrechtsmeier-oss@weidmueller.com> a écrit : > Am 12.02.2025 um 13:45 schrieb Frédéric Martinsons: > > > On Tue, 11 Feb 2025 at 16:01, Stefan Herbrechtsmeier via > lists.openembedded.org <stefan.herbrechtsmeier-oss= > weidmueller.com@lists.openembedded.org> wrote: > >> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> >> >> Add a vendor module for cargo to resolve dependencies and populate >> vendor directories from a Cargo.lock file. >> >> >> > Hello, > > Have you looked at vendor subcommand of cargo : > https://doc.rust-lang.org/cargo/commands/cargo-vendor.html ? > I didn't read the series but the commit message makes me think that "cargo > vendor" already exists for resolving dependencies > and populating a vendor directory. > > The class uses the bitbake fetcher (wget / git) to create a vendor > directory and emulate the cargo vendor command. > Yes, I understood that. Let me rephrase my question. Is it desirable/needed to mimick an existing (upstream and maintained) command, considering that we already use native cargo to build rust binaries? Is the new flow you suggest can use the "cargo vendor" call? Maybe there is room for less custom code here. Sorry if I missed something obvious, I just react to some keywords in area that I contributed 2 years ago.
Am 12.02.2025 um 18:48 schrieb Frédéric Martinsons: > > > Le mer. 12 févr. 2025, 17:29, Stefan Herbrechtsmeier > <stefan.herbrechtsmeier-oss@weidmueller.com> a écrit : > > Am 12.02.2025 um 13:45 schrieb Frédéric Martinsons: >> >> On Tue, 11 Feb 2025 at 16:01, Stefan Herbrechtsmeier via >> lists.openembedded.org <http://lists.openembedded.org> >> <stefan.herbrechtsmeier-oss=weidmueller.com@lists.openembedded.org> >> wrote: >> >> From: Stefan Herbrechtsmeier >> <stefan.herbrechtsmeier@weidmueller.com> >> >> Add a vendor module for cargo to resolve dependencies and >> populate >> vendor directories from a Cargo.lock file. >> >> >> >> Hello, >> >> Have you looked at vendor subcommand of cargo : >> https://doc.rust-lang.org/cargo/commands/cargo-vendor.html ? >> I didn't read the series but the commit message makes me think >> that "cargo vendor" already exists for resolving dependencies >> and populating a vendor directory. >> > The class uses the bitbake fetcher (wget / git) to create a vendor > directory and emulate the cargo vendor command. > > > Yes, I understood that. Let me rephrase my question. > Is it desirable/needed to mimick an existing (upstream and maintained) > command, considering that we already use native cargo to build rust > binaries? We already mimick the cargo vendor command. But at the moment the code is hidden inside the crate fetcher. We need to bypass the download of crates via cargo to support download mirrors and offline build. Therefore we download the files via wget and populate a local crate repository. > Is the new flow you suggest can use the "cargo vendor" call? It is useless because we already create a vendor folder manual (CARGO_VENDORING_DIRECTORY). > Maybe there is room for less custom code here. Sorry if I missed > something obvious, I just react to some keywords in area that I > contributed 2 years ago. I mainly extend existing code from crate fetcher and cargo-update-recipe-crate class. We need to extract download information from the lock file to create SRC_URIs.
diff --git a/meta/lib/oe/vendor/cargo.py b/meta/lib/oe/vendor/cargo.py new file mode 100644 index 0000000000..4d0a0034f3 --- /dev/null +++ b/meta/lib/oe/vendor/cargo.py @@ -0,0 +1,121 @@ +# Copyright (C) 2024-2025 Weidmueller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> +# +# SPDX-License-Identifier: MIT +# +import json +import os +import tomllib +import bb +import oe.vendor +from bb.fetch2 import URI +from . import ResolveError + +VENDOR_TYPE = "cargo" + +VENDOR_DIR = "vendor/cargo" + +def escape(path): + return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path) + +def determine_subdir(name, version): + return f"{name}-{version}" + +def determine_uri_path(path, name, version): + path = path.rstrip("/") + return f"{path}/api/v1/crates/{name}/{version}/download" + +def determine_downloadfilename(name, version): + filename = f"{name}-{version}.crate" + return oe.vendor.determine_downloadfilename(VENDOR_TYPE, filename) + +def extend_uri(uri, name, version, subdir, checksum_name=None, + checksum_value=None): + uri.path = determine_uri_path(uri.path, name, version) + params = uri.params + params["subdir"] = subdir + params["downloadfilename"] = determine_downloadfilename(name, version) + if checksum_name and checksum_value: + params[checksum_name] = checksum_value + +def determine_src_uri(registry, name, version, subdir): + uri = URI(registry) + extend_uri(uri, name, version, subdir) + return str(uri) + +def parse_lock_file(lock_file, function): + try: + with open(lock_file, "rb") as f: + crates = tomllib.load(f) + except Exception as e: + raise ResolveError(f"Invalid file: {str(e)}", lock_file) + + for data in crates.get("package", []): + if "source" not in data: + continue + + function(data) + +def resolve_src_uris(lock_file, registry, base_subdir, vendor_subdir): + src_uris = [] + + def resolve_src_uri(data): + name = data.get('name') + version = data.get('version') + source = data.get("source") + + if source.startswith("registry"): + checksum_name = "sha256sum" + checksum_value = data.get('checksum') + uri = URI(source[9:]) + if (source[9:] == "https://github.com/rust-lang/crates.io-index"): + uri = URI(registry) + params = uri.params + params["name"] = name + params["version"] = version + params["type"] = VENDOR_TYPE + subdir = os.path.join(base_subdir, vendor_subdir) + extend_uri(uri, name, version, subdir, checksum_name, + checksum_value) + else: + raise ResolveError(f"Unsupported cargo registry: {source}", + lock_file) + + elif source.startswith("git"): + repository, _, revision = source.partition("#") + uri = URI(repository) + params = uri.params + scheme, _, protocol = uri.scheme.partition("+") + if protocol: + params["protocol"] = protocol + uri.scheme = scheme + params["nobranch"] = "1" + subdir = determine_subdir(name, version) + params["subdir"] = os.path.join(base_subdir, vendor_subdir, subdir) + params["rev"] = revision + else: + raise ResolveError(f"Unsupported dependency: {name}", lock_file) + + src_uris.append(str(uri)) + + parse_lock_file(lock_file, resolve_src_uri) + + return src_uris + +def populate_vendor(lock_file, rootdir): + def populate_checksum(data): + name = data.get('name') + version = data.get('version') + source = data.get("source") + chechsum = data.get('checksum') + + if source.startswith("registry"): + subdir = determine_subdir(name, version) + filepath = os.path.join(rootdir, subdir, ".cargo-checksum.json") + with open(filepath, "w") as f: + json.dump({ + "files": {}, + "package": chechsum + }, f) + + parse_lock_file(lock_file, populate_checksum)