mbox series

[RFC,00/21] Concept for tightly coupled package manager (Node.js, Go, Rust)

Message ID 20241220112613.22647-1-stefan.herbrechtsmeier-oss@weidmueller.com
Headers show
Series Concept for tightly coupled package manager (Node.js, Go, Rust) | expand

Message

Stefan Herbrechtsmeier Dec. 20, 2024, 11:25 a.m. UTC 
From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

The patch series improves the fetcher support for tightly coupled
package manager (npm, go and cargo). It adds support for embedded
dependency fetcher via a common dependency mixin. The patch series
reworks the npm-shrinkwrap.json (package-lock.json) support and adds a
fetcher for go.sum and cargo.lock files. The dependency mixin contains
two stages. The first stage locates a local specification file or
fetches an archive or git repository with a specification file. The
second stage resolves the dependency URLs from the specification file
and fetches the dependencies.

SRC_URI = "<type>://npm-shrinkwrap.json"
SRC_URI = "<type>+http://example.com/ npm-shrinkwrap.json"
SRC_URI = "<type>+http://example.com/${BP}.tar.gz;striplevel=1;subdir=${BP}"
SRC_URI = "<type>+git://example.com/${BPN}.git;protocol=https"

Additionally, the patch series reworks the npm fetcher to work without a
npm binary and external package repository. It adds support for a common
dependency name and version schema to integrate the dependencies into
the SBOM.

= Background
Bitbake has diverse concepts and drawbacks for different tightly coupled
package manager. The Python support uses a recipe per dependency and
generates common fetcher URLs via a python function. The other languages
embed the dependencies inside the recipe. The Node.js support offers a
npmsw fetcher which uses a lock file beside the recipe to generates
multiple common fetcher URLs on the fly and thereby hides the real
download sources. This leads to a single source in the SBOM for example.
The Go support contains two parallel implementations. A vendor-based
solution with a common fetcher and a go-mod-based solution with a gomod
fetcher. The vendor-based solution includes the individual dependencies
into the SRC_URI of the recipe and uses a python function to generate
common fetcher URLs which additional information for the vendor task.The
gomod fetcher uses a proprietary gomod URL. It translates the URL into a
common URL and prepares meta data during unpack. The Rust support
includes the individual dependencies in the SRC_URI of the recipe and
uses proprietary crate URLs. The crate fetcher translates a proprietary
URL into a common fetcher URL and prepares meta data during unpack. The
recipetool does not support the crate and the gomod fetcher. This leads
to missing licenses of the dependencies in the recipe for example
librsvg.

The steps needed to fetch dependencies for Node.js, Go and Rust are
similar:
1. Extract the dependencies from a specification file (name, version,
   checksum and URL)
2. Generate proprietary fetcher URIs
  a. npm://registry.npmjs.org/;package=glob;version= 10.3.15
  b. gomod://golang.org/x/net;version=v0.9.0
     gomodgit://golang.org/x/net;version=v0.9.0;repo=go.googlesource.com/net
  c. crate://crates.io/glob/0.3.1
3. Generate wget or git fetcher URIs
  a. https://registry.npmjs.org/glob/-/glob-10.3.15.tgz;downloadfilename=…
  b. https://proxy.golang.org/golang.org/x/net/@v/v0.9.0.zip;downloadfilename=…
     git://go.googlesource.com/net;protocol=https; subdir=…
  c. https://crates.io/api/v1/crates/glob/0.3.1/download;downloadfilename=…
4. Unpack
5. Create meta files
  a. Update lockfile and create tar.gz archives
  b. Create go.mod file
     Create info, go.mod file and zip archives
  c. Create .cargo-checksum.json files

It looks like the recipetool is not widely used and therefore this patch
series integrates the dependency resolving into the fetcher. After an
agreement on a concept the fetcher could be extended. The fetcher could
download the license information per package and a new build task could
run the license cruncher from the recipetool.

= Open questions

* Where should we download dependencies?
** Should we use a folder per fetcher (ex. git and npm)?
** Should we use the main folder (ex. crate)?
** Should we translate the name into folder (ex. gomod)?
** Should we integrate the name into the filename (ex. git)?
* Where should we unpack the dependencies?
** Should we use a folder inside the parent folder (ex. node_modules)?
** Should we use a fixed folder inside unpackdir
   (ex. go/pkg/mod/cache/download and cargo_home/bitbake)?
* How should we treat archives for package manager caches?
** Should we unpack the archives to support patching (ex. npm)?
** Should we copy the packed archive to avoid unpacking and packaging
   (ex. gomod)?

This patch series depends on patch series
20241209103158.20833-1-stefan.herbrechtsmeier-oss@weidmueller.com
("[1/4] tests: fetch: adapt npmsw tests to fixed unpack behavior").


Stefan Herbrechtsmeier (21):
  tests: fetch: update npmsw tests to new lockfile format
  fetch2: npmsw: remove old lockfile format support
  tests: fetch: replace [url] with urls for npm
  fetch2: do not prefix embedded checksums
  fetch2: read checksum from SRC_URI flag for npm
  fetch2: introduce common package manager metadata
  fetch2: add unpack support for npm archives
  utils: add Go mod h1 checksum support
  fetch2: add destdir to FetchData
  fetch: npm: rework
  tests: fetch: adapt style in npm(sw) class
  tests: fetch: move npmsw test cases into npmsw test class
  tests: fetch: adapt npm test cases
  fetch: add dependency mixin
  tests: fetch: add test cases for dependency fetcher
  fetch: npmsw: migrate to dependency mixin
  tests: fetch: adapt npmsw test cases
  fetch: add gosum fetcher
  tests: fetch: add test cases for gosum
  fetch: add cargolock fetcher
  tests: fetch: add test cases for cargolock

 lib/bb/fetch2/__init__.py   |  35 +-
 lib/bb/fetch2/cargolock.py  |  73 +++
 lib/bb/fetch2/dependency.py | 167 +++++++
 lib/bb/fetch2/gomod.py      |   5 +-
 lib/bb/fetch2/gosum.py      |  51 +++
 lib/bb/fetch2/npm.py        | 244 +++-------
 lib/bb/fetch2/npmsw.py      | 347 ++++----------
 lib/bb/tests/fetch.py       | 880 +++++++++++++++++-------------------
 lib/bb/utils.py             |  25 +
 9 files changed, 916 insertions(+), 911 deletions(-)
 create mode 100644 lib/bb/fetch2/cargolock.py
 create mode 100644 lib/bb/fetch2/dependency.py
 create mode 100644 lib/bb/fetch2/gosum.py