| Message ID | 20250211150034.18696-1-stefan.herbrechtsmeier-oss@weidmueller.com |
|---|---|
| Headers | show |
| Series | Add vendor support for go, npm and rust | expand |
In message: [bitbake-devel] [RFC PATCH 00/30] Add vendor support for go, npm and rust on 11/02/2025 Stefan Herbrechtsmeier via lists.openembedded.org wrote: > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > > The series adds on-the-fly support for package manager specific > dependencies and vendor directories. It contains the following changes: > 1. Adds an early fetch, unpack and patch task to unpack and patch source > code with an embedded lock file for dependencies. > 2. Parse the go.sum, Cargo.lock and package-lock.json lock files and > resolve the dependencies to SRC_URIs. > 3. Save the SRC_URIs in a file and adapt all SRC_URIs users to handle > the SRC_URI files beside the SRC_URIs in the recipe. I made a few comments, and will have another / better look at the series tomorrow. There's a lot here, and it is hard to wrap my head around everything that is changing. I have one specific question below (from the point of view of go). I've been looking through the series, and can't pick out where #3 is done. I see patch 14 using SRC_URI_FILES, but where are those files written ? Is that in patch 18 (vendor_go_do_vendor_resolve ?) What is written to those files ? The concept that I'm not understanding (and that's just me not being familiar with things, I'll continue reading the series) is that when we suggested we'd like to have a mode where the dependencies could clearly be listed in the SRC_URI, at least I was just thinking about a way run the fetch/module elements that you were adding, write them to a file and then have the recipe include it. I can't tell if in the series those files are written each time, and that there would be no way to edit those SRC_URI_FILES .. but I'll look again tomorrow. That file would manipulate the standard SRC_URI. In other words still support a mode that is like the .inc files with crate://. So someone could either have the lockfile parsed and fetched, or have a way to run the parsing and fetching via a task, write a file and include the file in their recipe to short circuit the processing of the lockfile. (meaning the expanded and end fetches that are done once you've processed the file are simply listed as a series of fetches that are carried out without extra processing .. and "unrolled" dependency file pointing at the "sources" git, crate, mod, whatever) If that just doesn't make sense, then if there was a way to copy the lockfile out of the recipe and have it overlayed onto the fetched one .. maybe breaking out the individual fetch lines isn't required, since they could be individually manipulated in that lockfile. Bruce > 4. Create a package manager specific vendor directory during unpack to > support additional patching of the dependencies. > 5. Add the dependency name and version to the SBOM. > 6. Simplify the npm support > > > Stefan Herbrechtsmeier (30): > classes: create-spdx-2.2: use expanded FetchData for downloaded > packages > lib: spdx30_tasks: use expanded FetchData for download files > classes: create-spdx-2.2: use name and version for download > dependencies > lib: bb: fetch2: add support to unpack .crate files > lib: oe: add vendor module > lib: oe: vendor: add cargo support > lib: oe: vendor: add go support > lib: oe: vendor: add npm support > oeqa: oelib: add vendor tests > conf: bitbake: add SRC_URI_FILES variable > classes: go: make source directory configurable > classes: go-mod: make class customizable > classes: add nodejs-arch class > classes: base: add get_src_uris and unpack_src_uris functions > classes: add early fetch, unpack and patch support > classes: add vendor class > classes: add vendor class for cargo > classes: add vendor class for go > classes: add vendor class for npm > classes: add vendor_npm_build class > python3-bcrypt: mirgrate to vendor cargo class > python3-cryptography: mirgrate to vendor cargo class > python3-maturin: mirgrate to vendor cargo class > python3-rpds-py: mirgrate to vendor cargo class > librsvg: mirgrate to vendor cargo class > librsvg: update dependecies to fix RUSTSEC-2024-0421 > [DO NOT MERGE] recipes: add crucible go demo > [DO NOT MERGE] recipes: add node-red npm demo > [DO NOT MERGE] recipes: add nucleoidai npm demo > [DO NOT MERGE] classes: spdx: use version 2.2 > > bitbake/lib/bb/fetch2/__init__.py | 2 +- > .../crucible/crucible2_2023.11.02.bb | 18 + > .../node-red/node-red/package-lock.json | 6096 +++++++++++++++++ > .../node-red/node-red_4.0.8.bb | 14 + > .../nucleoidai/nucleoidai_0.7.10.bb | 11 + > meta/classes-global/base.bbclass | 47 +- > meta/classes-global/patch.bbclass | 17 +- > meta/classes-recipe/early.bbclass | 61 + > meta/classes-recipe/go-mod.bbclass | 10 +- > meta/classes-recipe/go.bbclass | 22 +- > meta/classes-recipe/nodejs-arch.bbclass | 15 + > meta/classes-recipe/vendor.bbclass | 28 + > meta/classes-recipe/vendor_cargo.bbclass | 46 + > meta/classes-recipe/vendor_go.bbclass | 59 + > meta/classes-recipe/vendor_npm.bbclass | 115 + > meta/classes-recipe/vendor_npm_build.bbclass | 50 + > meta/classes/archiver.bbclass | 4 +- > meta/classes/buildhistory.bbclass | 4 +- > meta/classes/copyleft_compliance.bbclass | 2 +- > meta/classes/create-spdx-2.2.bbclass | 14 +- > meta/classes/create-spdx.bbclass | 2 +- > meta/classes/externalsrc.bbclass | 2 +- > meta/conf/bitbake.conf | 1 + > meta/lib/oe/patch.py | 10 +- > meta/lib/oe/spdx30_tasks.py | 5 +- > meta/lib/oe/vendor/__init__.py | 28 + > meta/lib/oe/vendor/cargo.py | 121 + > meta/lib/oe/vendor/go.py | 96 + > meta/lib/oe/vendor/npm.py | 141 + > meta/lib/oeqa/selftest/cases/oelib/vendor.py | 237 + > .../python/python3-bcrypt-crates.inc | 84 - > .../python/python3-bcrypt_4.2.1.bb | 4 +- > .../python/python3-cryptography-crates.inc | 76 - > .../python/python3-cryptography.bb | 4 +- > .../python/python3-maturin-crates.inc | 712 -- > .../python/python3-maturin_1.8.1.bb | 4 +- > .../python/python3-rpds-py-crates.inc | 54 - > .../python/python3-rpds-py_0.22.3.bb | 4 +- > meta/recipes-gnome/librsvg/librsvg-crates.inc | 590 -- > ...-to-get-an-updated-idna-rustsec-2024.patch | 398 ++ > meta/recipes-gnome/librsvg/librsvg_2.59.2.bb | 7 +- > 41 files changed, 7633 insertions(+), 1582 deletions(-) > create mode 100644 meta-selftest/recipes-support/crucible/crucible2_2023.11.02.bb > create mode 100644 meta-selftest/recipes-support/node-red/node-red/package-lock.json > create mode 100644 meta-selftest/recipes-support/node-red/node-red_4.0.8.bb > create mode 100644 meta-selftest/recipes-support/nucleoidai/nucleoidai_0.7.10.bb > create mode 100644 meta/classes-recipe/early.bbclass > create mode 100644 meta/classes-recipe/nodejs-arch.bbclass > create mode 100644 meta/classes-recipe/vendor.bbclass > create mode 100644 meta/classes-recipe/vendor_cargo.bbclass > create mode 100644 meta/classes-recipe/vendor_go.bbclass > create mode 100644 meta/classes-recipe/vendor_npm.bbclass > create mode 100644 meta/classes-recipe/vendor_npm_build.bbclass > create mode 100644 meta/lib/oe/vendor/__init__.py > create mode 100644 meta/lib/oe/vendor/cargo.py > create mode 100644 meta/lib/oe/vendor/go.py > create mode 100644 meta/lib/oe/vendor/npm.py > create mode 100644 meta/lib/oeqa/selftest/cases/oelib/vendor.py > delete mode 100644 meta/recipes-devtools/python/python3-bcrypt-crates.inc > delete mode 100644 meta/recipes-devtools/python/python3-cryptography-crates.inc > delete mode 100644 meta/recipes-devtools/python/python3-maturin-crates.inc > delete mode 100644 meta/recipes-devtools/python/python3-rpds-py-crates.inc > delete mode 100644 meta/recipes-gnome/librsvg/librsvg-crates.inc > create mode 100644 meta/recipes-gnome/librsvg/librsvg/0001-update-url-crate-to-get-an-updated-idna-rustsec-2024.patch > > -- > 2.39.5 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#17192): https://lists.openembedded.org/g/bitbake-devel/message/17192 > Mute This Topic: https://lists.openembedded.org/mt/111123517/1050810 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [bruce.ashfield@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Am 12.02.2025 um 00:14 schrieb Bruce Ashfield via lists.openembedded.org: > In message: [bitbake-devel] [RFC PATCH 00/30] Add vendor support for go, npm and rust > on 11/02/2025 Stefan Herbrechtsmeier via lists.openembedded.org wrote: > >> From: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com> >> >> The series adds on-the-fly support for package manager specific >> dependencies and vendor directories. It contains the following changes: >> 1. Adds an early fetch, unpack and patch task to unpack and patch source >> code with an embedded lock file for dependencies. >> 2. Parse the go.sum, Cargo.lock and package-lock.json lock files and >> resolve the dependencies to SRC_URIs. >> 3. Save the SRC_URIs in a file and adapt all SRC_URIs users to handle >> the SRC_URI files beside the SRC_URIs in the recipe. > I made a few comments, and will have another / better look at the > series tomorrow. There's a lot here, and it is hard to wrap my head > around everything that is changing. > > I have one specific question below (from the point of view of go). > > I've been looking through the series, and can't pick out where #3 is > done. The files are created in the package manager specific vendor classes (17 - 19) in the do_vendor_resolve task. The parsing is implemented in the package manager specific modules in the vendor package (6-8). > I see patch 14 using SRC_URI_FILES, but where are those files > written ? Is that in patch 18 (vendor_go_do_vendor_resolve ?) What is > written to those files ? The file is written into the vendor directory below WORKDIR. The approach is similar to other classes which need to share data between tasks. The files contain individual SRC_URIs per line. > The concept that I'm not understanding (and that's just me not being > familiar with things, I'll continue reading the series) is that when > we suggested we'd like to have a mode where the dependencies could > clearly be listed in the SRC_URI, at least I was just thinking about a > way run the fetch/module elements that you were adding, write them to > a file and then have the recipe include it. The additional SRC_URIs are dynamically added to the recipe without the indirection over an include file. The generated SRC_URIs could be inspected via the SRC_URI_FILES and manipulated via patches for the lock files or an separated lock file. > I can't tell if in the series those files are written each time, and > that there would be no way to edit those SRC_URI_FILES .. but I'll > look again tomorrow. The files are written if the do_vendor_resolve task need to be rerun. It isn't possible to edit the generated SRC_URIs. Instead the lock file could be patched. > That file would manipulate the standard SRC_URI. In other words still > support a mode that is like the .inc files with crate://. So someone > could either have the lockfile parsed and fetched, or have a way to > run the parsing and fetching via a task, write a file and include the > file in their recipe to short circuit the processing of the lockfile. > (meaning the expanded and end fetches that are done once you've > processed the file are simply listed as a series of fetches that are > carried out without extra processing .. and "unrolled" dependency > file pointing at the "sources" git, crate, mod, whatever) What is the motivation "to short circuit the processing of the lockfile"? Like any task the processing and its output are cached via the sstate cache. It is possible to use the vendor package to create a include file but we should have good reasons to support two solutions for the same problem. > If that just doesn't make sense, then if there was a way to copy > the lockfile out of the recipe and have it overlayed onto the fetched > one .. maybe breaking out the individual fetch lines isn't required, > since they could be individually manipulated in that lockfile. I haven't test it but this should be possible. You can place a lock file beside the recipe, add it to the SRC_URI and change the XXX_LOCK_PATH. Regards Stefan
On Wed, Feb 12, 2025 at 3:41 AM Stefan Herbrechtsmeier < stefan.herbrechtsmeier-oss@weidmueller.com> wrote: > Am 12.02.2025 um 00:14 schrieb Bruce Ashfield via lists.openembedded.org: > > In message: [bitbake-devel] [RFC PATCH 00/30] Add vendor support for go, npm and rust > on 11/02/2025 Stefan Herbrechtsmeier via lists.openembedded.org wrote: > > > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> <stefan.herbrechtsmeier@weidmueller.com> > > The series adds on-the-fly support for package manager specific > dependencies and vendor directories. It contains the following changes: > 1. Adds an early fetch, unpack and patch task to unpack and patch source > code with an embedded lock file for dependencies. > 2. Parse the go.sum, Cargo.lock and package-lock.json lock files and > resolve the dependencies to SRC_URIs. > 3. Save the SRC_URIs in a file and adapt all SRC_URIs users to handle > the SRC_URI files beside the SRC_URIs in the recipe. > > I made a few comments, and will have another / better look at the > series tomorrow. There's a lot here, and it is hard to wrap my head > around everything that is changing. > > I have one specific question below (from the point of view of go). > > I've been looking through the series, and can't pick out where #3 is > done. > > The files are created in the package manager specific vendor classes (17 - > 19) in the do_vendor_resolve task. The parsing is implemented in the > package manager specific modules in the vendor package (6-8). > > aha. I'll have a closer look in a bit! > I see patch 14 using SRC_URI_FILES, but where are those files > written ? Is that in patch 18 (vendor_go_do_vendor_resolve ?) What is > written to those files ? > > The file is written into the vendor directory below WORKDIR. The approach > is similar to other classes which need to share data between tasks. > > The files contain individual SRC_URIs per line. > I may have missed that in the patches, but if that isn't described in the commit logs, then that description would be a nice addition. Along with a high level description for each supported language package manager about what is in the files and what uses it to resolve the dependencies. > The concept that I'm not understanding (and that's just me not being > familiar with things, I'll continue reading the series) is that when > we suggested we'd like to have a mode where the dependencies could > clearly be listed in the SRC_URI, at least I was just thinking about a > way run the fetch/module elements that you were adding, write them to > a file and then have the recipe include it. > > The additional SRC_URIs are dynamically added to the recipe without the > indirection over an include file. The generated SRC_URIs could be inspected > via the SRC_URI_FILES and manipulated via patches for the lock files or an > separated lock file. > The issue with patching those dynamic files .. is just that, they are dynamic and they could be regenerated (but hopefully without a SRCREV change they are identically generated each time, so maybe it isn't a big deal) and patching could fail. Much like we don't patch generated automake files, we patch the .am and .in files that generate the outputs. > > I can't tell if in the series those files are written each time, and > that there would be no way to edit those SRC_URI_FILES .. but I'll > look again tomorrow. > > The files are written if the do_vendor_resolve task need to be rerun. It > isn't possible to edit the generated SRC_URIs. Instead the lock file could > be patched. > ok. So they are really just a way to dump what is being built to disk and out of "memory", it really isn't what I was looking for, but at least I understand what you were trying to do with the writing and later use of the files now. I was hoping for something that I could include "statically" (for lack of a better term) in my recipes, and then have the handlers for the protocols do the fetching .. again, much like the crate:// fetches in the .inc files. To be clear, I'm not suggesting that mode would be the only mode, just that the way the generation of the fetches -> actual fetching was constructed such that I could avoid "short circuit" the handling of the lock file and simply list the dependencies for fetching (much like how I can go get all of the SRCREVS for a gitsm:// fetch, and put them into my recipes as pure git:// fetches if I'm having issues with the git submodles). That file would manipulate the standard SRC_URI. In other words still > support a mode that is like the .inc files with crate://. So someone > could either have the lockfile parsed and fetched, or have a way to > run the parsing and fetching via a task, write a file and include the > file in their recipe to short circuit the processing of the lockfile. > (meaning the expanded and end fetches that are done once you've > processed the file are simply listed as a series of fetches that are > carried out without extra processing .. and "unrolled" dependency > file pointing at the "sources" git, crate, mod, whatever) > > What is the motivation "to short circuit the processing of the lockfile"? > Like any task the processing and its output are cached via the sstate cache. > See above. It isn't about re-running it a later time and re-using the artifacts, it is about having a way to make sure nothing is dynamically resolved when bitbake <foo> is called, just fetch and the rest of the standard tasks. > > It is possible to use the vendor package to create a include file but we > should have good reasons to support two solutions for the same problem. > > If that just doesn't make sense, then if there was a way to copy > the lockfile out of the recipe and have it overlayed onto the fetched > one .. maybe breaking out the individual fetch lines isn't required, > since they could be individually manipulated in that lockfile. > > I haven't test it but this should be possible. You can place a lock file > beside the recipe, add it to the SRC_URI and change the XXX_LOCK_PATH. > Something like that is a big step towards having that finer grained control in a recipe that I'm looking for. I'd still like some sort of line by line "dumped dependency list" if there's a lower level protocol at play (but the lock file really also is just that line by line list for many of the languages). Bruce > Regards Stefan >
Am 12.02.2025 um 15:11 schrieb Bruce Ashfield: > > > On Wed, Feb 12, 2025 at 3:41 AM Stefan Herbrechtsmeier > <stefan.herbrechtsmeier-oss@weidmueller.com> wrote: > > Am 12.02.2025 um 00:14 schrieb Bruce Ashfield via > lists.openembedded.org <http://lists.openembedded.org>: >> In message: [bitbake-devel] [RFC PATCH 00/30] Add vendor support for go, npm and rust >> on 11/02/2025 Stefan Herbrechtsmeier vialists.openembedded.org <http://lists.openembedded.org> wrote: >> >>> From: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com> <mailto:stefan.herbrechtsmeier@weidmueller.com> >>> >>> The series adds on-the-fly support for package manager specific >>> dependencies and vendor directories. It contains the following changes: >>> 1. Adds an early fetch, unpack and patch task to unpack and patch source >>> code with an embedded lock file for dependencies. >>> 2. Parse the go.sum, Cargo.lock and package-lock.json lock files and >>> resolve the dependencies to SRC_URIs. >>> 3. Save the SRC_URIs in a file and adapt all SRC_URIs users to handle >>> the SRC_URI files beside the SRC_URIs in the recipe. >> I made a few comments, and will have another / better look at the >> series tomorrow. There's a lot here, and it is hard to wrap my head >> around everything that is changing. >> >> I have one specific question below (from the point of view of go). >> >> I've been looking through the series, and can't pick out where #3 is >> done. > > The files are created in the package manager specific vendor > classes (17 - 19) in the do_vendor_resolve task. The parsing is > implemented in the package manager specific modules in the vendor > package (6-8). > > > aha. I'll have a closer look in a bit! > >> I see patch 14 using SRC_URI_FILES, but where are those files >> written ? Is that in patch 18 (vendor_go_do_vendor_resolve ?) What is >> written to those files ? > > The file is written into the vendor directory below WORKDIR. The > approach is similar to other classes which need to share data > between tasks. > > The files contain individual SRC_URIs per line. > > I may have missed that in the patches, but if that isn't described in > the commit logs, then > that description would be a nice addition. Along with a high level > description for each > supported language package manager about what is in the files and what > uses it to > resolve the dependencies. I will extend the commit messages. > >> The concept that I'm not understanding (and that's just me not being >> familiar with things, I'll continue reading the series) is that when >> we suggested we'd like to have a mode where the dependencies could >> clearly be listed in the SRC_URI, at least I was just thinking about a >> way run the fetch/module elements that you were adding, write them to >> a file and then have the recipe include it. > > The additional SRC_URIs are dynamically added to the recipe > without the indirection over an include file. The generated > SRC_URIs could be inspected via the SRC_URI_FILES and manipulated > via patches for the lock files or an separated lock file. > > The issue with patching those dynamic files .. is just that, they are > dynamic and > they could be regenerated (but hopefully without a SRCREV change they > are identically > generated each time, so maybe it isn't a big deal) and patching could > fail. Much like we > don't patch generated automake files, we patch the .am and .in files > that generate the > outputs. You can't compare lock files with automake files. The lock files is generated but it contains additional information. The only difference is that a program and not a human write the file. If an update creates too much noise in the file we should discuss with upstream how to minimize it. The lock file is designed to be under revision control and shouldn't change without reason. Dependabot automatically creates updates and pull requests for dependency updates. This will not work if the lock file isn't deterministic. > >> I can't tell if in the series those files are written each time, and >> that there would be no way to edit those SRC_URI_FILES .. but I'll >> look again tomorrow. > > The files are written if the do_vendor_resolve task need to be > rerun. It isn't possible to edit the generated SRC_URIs. Instead > the lock file could be patched. > > > ok. So they are really just a way to dump what is being built to disk > and out of "memory", > it really isn't what I was looking for, but at least I understand what > you were trying to > do with the writing and later use of the files now. > > I was hoping for something that I could include "statically" (for lack > of a better term) in my > recipes, and then have the handlers for the protocols do the fetching > .. again, much like > the crate:// fetches in the .inc files. I don't use the fetcher because of its drawbacks and limitations. I use the same wget URL without the indirection over the package manager specific fetchers. I can follow the pypi approach to generate SRC_URIs which call a python function: ${@crate_src_uri(d, 'xyz', '1.2.3')} But why you need this? In any case the manipulation of a dependency could lead to interdependence between dependencies. > To be clear, I'm not suggesting that mode would be the only mode, just > that the way the > generation of the fetches -> actual fetching was constructed such that > I could avoid > "short circuit" the handling of the lock file and simply list the > dependencies for fetching > (much like how I can go get all of the SRCREVS for a gitsm:// fetch, > and put them into > my recipes as pure git:// fetches if I'm having issues with the git > submodles). Add a function to create this file isn't a problem, but I can't ensure that this will work in any case and doesn't create problems because of the discrepancy between the manifest / lock file and the available sources. >> That file would manipulate the standard SRC_URI. In other words still >> support a mode that is like the .inc files with crate://. So someone >> could either have the lockfile parsed and fetched, or have a way to >> run the parsing and fetching via a task, write a file and include the >> file in their recipe to short circuit the processing of the lockfile. >> (meaning the expanded and end fetches that are done once you've >> processed the file are simply listed as a series of fetches that are >> carried out without extra processing .. and "unrolled" dependency >> file pointing at the "sources" git, crate, mod, whatever) > > What is the motivation "to short circuit the processing of the > lockfile"? Like any task the processing and its output are cached > via the sstate cache. > > > See above. It isn't about re-running it a later time and re-using the > artifacts, it is about > having a way to make sure nothing is dynamically resolved when bitbake > <foo> is called, > just fetch and the rest of the standard tasks. The resolve always create the same URL like the crate fetcher. It only changes if you change the resolve function or the crate fetcher. Every needed information is defined inside the lock file. > > It is possible to use the vendor package to create a include file > but we should have good reasons to support two solutions for the > same problem. > >> If that just doesn't make sense, then if there was a way to copy >> the lockfile out of the recipe and have it overlayed onto the fetched >> one .. maybe breaking out the individual fetch lines isn't required, >> since they could be individually manipulated in that lockfile. > > I haven't test it but this should be possible. You can place a > lock file beside the recipe, add it to the SRC_URI and change the > XXX_LOCK_PATH. > > > Something like that is a big step towards having that finer grained > control in a recipe that > I'm looking for. I'd still like some sort of line by line "dumped > dependency list" if there's a > lower level protocol at play (but the lock file really also is just > that line by line list for many > of the languages). Why you need this finer grained control in the recipe? You have the finer grained control in the lock file and additional tools to manage the file and all its interdependence. The resolve is deterministic and the .inc is only another representation. Manipulation of the representation could lead to problems and complicate the process. It's like you translate the .am and .in files into a .inc file to influence the generated automake files. Would it help if we have a devtool shell so that you can run "cargo update" inside the folder created by "devtool modify" to update dependencies via the common tools and create a patch?
On Thu, Feb 13, 2025 at 3:36 AM Stefan Herbrechtsmeier < stefan.herbrechtsmeier-oss@weidmueller.com> wrote: > Am 12.02.2025 um 15:11 schrieb Bruce Ashfield: > > > > On Wed, Feb 12, 2025 at 3:41 AM Stefan Herbrechtsmeier < > stefan.herbrechtsmeier-oss@weidmueller.com> wrote: > >> Am 12.02.2025 um 00:14 schrieb Bruce Ashfield via lists.openembedded.org: >> >> In message: [bitbake-devel] [RFC PATCH 00/30] Add vendor support for go, npm and rust >> on 11/02/2025 Stefan Herbrechtsmeier via lists.openembedded.org wrote: >> >> >> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> <stefan.herbrechtsmeier@weidmueller.com> >> >> The series adds on-the-fly support for package manager specific >> dependencies and vendor directories. It contains the following changes: >> 1. Adds an early fetch, unpack and patch task to unpack and patch source >> code with an embedded lock file for dependencies. >> 2. Parse the go.sum, Cargo.lock and package-lock.json lock files and >> resolve the dependencies to SRC_URIs. >> 3. Save the SRC_URIs in a file and adapt all SRC_URIs users to handle >> the SRC_URI files beside the SRC_URIs in the recipe. >> >> I made a few comments, and will have another / better look at the >> series tomorrow. There's a lot here, and it is hard to wrap my head >> around everything that is changing. >> >> I have one specific question below (from the point of view of go). >> >> I've been looking through the series, and can't pick out where #3 is >> done. >> >> The files are created in the package manager specific vendor classes (17 >> - 19) in the do_vendor_resolve task. The parsing is implemented in the >> package manager specific modules in the vendor package (6-8). >> >> > aha. I'll have a closer look in a bit! > > > >> I see patch 14 using SRC_URI_FILES, but where are those files >> written ? Is that in patch 18 (vendor_go_do_vendor_resolve ?) What is >> written to those files ? >> >> The file is written into the vendor directory below WORKDIR. The approach >> is similar to other classes which need to share data between tasks. >> >> The files contain individual SRC_URIs per line. >> > I may have missed that in the patches, but if that isn't described in the > commit logs, then > that description would be a nice addition. Along with a high level > description for each > supported language package manager about what is in the files and what > uses it to > resolve the dependencies. > > I will extend the commit messages. > > >> The concept that I'm not understanding (and that's just me not being >> familiar with things, I'll continue reading the series) is that when >> we suggested we'd like to have a mode where the dependencies could >> clearly be listed in the SRC_URI, at least I was just thinking about a >> way run the fetch/module elements that you were adding, write them to >> a file and then have the recipe include it. >> >> The additional SRC_URIs are dynamically added to the recipe without the >> indirection over an include file. The generated SRC_URIs could be inspected >> via the SRC_URI_FILES and manipulated via patches for the lock files or an >> separated lock file. >> > The issue with patching those dynamic files .. is just that, they are > dynamic and > they could be regenerated (but hopefully without a SRCREV change they are > identically > generated each time, so maybe it isn't a big deal) and patching could > fail. Much like we > don't patch generated automake files, we patch the .am and .in files that > generate the > outputs. > > You can't compare lock files with automake files. The lock files is > generated but it contains additional information. The only difference is > that a program and not a human write the file. If an update creates too > much noise in the file we should discuss with upstream how to minimize it. > The lock file is designed to be under revision control and shouldn't change > without reason. Dependabot automatically creates updates and pull requests > for dependency updates. This will not work if the lock file isn't > deterministic. > I may be using different terminology, but I do know what the lock files are (I interchange the dependency and lock files at times though), patching *anything* that is generated is a bad idea, that's my point (so the comparison is valid). > >> I can't tell if in the series those files are written each time, and >> that there would be no way to edit those SRC_URI_FILES .. but I'll >> look again tomorrow. >> >> The files are written if the do_vendor_resolve task need to be rerun. It >> isn't possible to edit the generated SRC_URIs. Instead the lock file could >> be patched. >> > > ok. So they are really just a way to dump what is being built to disk and > out of "memory", > it really isn't what I was looking for, but at least I understand what you > were trying to > do with the writing and later use of the files now. > > I was hoping for something that I could include "statically" (for lack of > a better term) in my > recipes, and then have the handlers for the protocols do the fetching .. > again, much like > the crate:// fetches in the .inc files. > > I don't use the fetcher because of its drawbacks and limitations. I use > the same wget URL without the indirection over the package manager specific > fetchers. I can follow the pypi approach to generate SRC_URIs which call a > python function: > Can you elaborate on the limitations ? Although from your description, I'm fine calling wget a "fetcher". I'm just talking about something that pulls in the dependency source code, whether it be git, wget, or some other protocol, it's the concept that is the same. > ${@crate_src_uri(d, 'xyz', '1.2.3')} > > But why you need this? In any case the manipulation of a dependency could > lead to interdependence between dependencies.\ > Because I need to redirect and update them manually. Anything that runs in python to determine the fetch is also hard to debug, since I'd be interested in the output of that processing if there's an issue. I'm describing something like (and I'm describing it badly .. but I'll try anyway: <language file that describes the dependencies> <process that file into a specific source revision and fetch/download location> <have a language / protocol specific fetcher get the modules / source code iteratively from a list of expanded dependencies> <put them into place for the build> <arrange for the build to use those fetched depdendencies> wget would fall into that "language specific fetcher", as would git or "go vendor" (Although I don't really like that, since it isn't transparent where the source code is being pulled from). Copying the files into vendor, or again "go mod vendor" (doing both the fetch and placement) would be the "put them in place for the build". I realize that is what your series is doing. The processing, fetching, placement stages could run fully behind the scenes from a dependency/lock file, or be skipped if broken out into the list of fetches (from the SCR_URI (via .inc file or whatever)). Those are the two modes I'm describing, where the "processing the file into a specific source revision / fetch steps" can be avoided. > > > To be clear, I'm not suggesting that mode would be the only mode, just > that the way the > generation of the fetches -> actual fetching was constructed such that I > could avoid > "short circuit" the handling of the lock file and simply list the > dependencies for fetching > (much like how I can go get all of the SRCREVS for a gitsm:// fetch, and > put them into > my recipes as pure git:// fetches if I'm having issues with the git > submodles). > > Add a function to create this file isn't a problem, but I can't ensure > that this will work in any case and doesn't create problems because of the > discrepancy between the manifest / lock file and the available sources. > Right, and that is fine. If someone bumps a revision to the point where the dependency now has different dependencies that are not in the lock file, then they've taken things fully into their control and need to deal with any issues. That obviously isn't the default, but does allow an advanced user/developer the possibility to have fine grained control and debug. > > That file would manipulate the standard SRC_URI. In other words still >> support a mode that is like the .inc files with crate://. So someone >> could either have the lockfile parsed and fetched, or have a way to >> run the parsing and fetching via a task, write a file and include the >> file in their recipe to short circuit the processing of the lockfile. >> (meaning the expanded and end fetches that are done once you've >> processed the file are simply listed as a series of fetches that are >> carried out without extra processing .. and "unrolled" dependency >> file pointing at the "sources" git, crate, mod, whatever) >> >> What is the motivation "to short circuit the processing of the lockfile"? >> Like any task the processing and its output are cached via the sstate cache. >> > > See above. It isn't about re-running it a later time and re-using the > artifacts, it is about > having a way to make sure nothing is dynamically resolved when bitbake > <foo> is called, > just fetch and the rest of the standard tasks. > > The resolve always create the same URL like the crate fetcher. It only > changes if you change the resolve function or the crate fetcher. Every > needed information is defined inside the lock file. > That's good, and if the lock file is present (versus just a dependency file), would the processing and creation of the lock file be skipped ? > > >> It is possible to use the vendor package to create a include file but we >> should have good reasons to support two solutions for the same problem. >> >> If that just doesn't make sense, then if there was a way to copy >> the lockfile out of the recipe and have it overlayed onto the fetched >> one .. maybe breaking out the individual fetch lines isn't required, >> since they could be individually manipulated in that lockfile. >> >> I haven't test it but this should be possible. You can place a lock file >> beside the recipe, add it to the SRC_URI and change the XXX_LOCK_PATH. >> > > Something like that is a big step towards having that finer grained > control in a recipe that > I'm looking for. I'd still like some sort of line by line "dumped > dependency list" if there's a > lower level protocol at play (but the lock file really also is just that > line by line list for many > of the languages). > > Why you need this finer grained control in the recipe? You have the finer > grained control in the lock file and additional tools to manage the file > and all its interdependence. The resolve is deterministic and the .inc is > only another representation. Manipulation of the representation could lead > to problems and complicate the process. It's like you translate the .am and > .in files into a .inc file to influence the generated automake files. > The tools that I'm talking about are a text editor, not anything language specific. I don't want or need tools to manipulate those files, I want to see the base fetches of the dependencies, it is those that I'm manipulating. I'm simply describing what I've done in the past to fix issues with the builds and pull in fixes, so yes, it is something that I need. > Would it help if we have a devtool shell so that you can run "cargo > update" inside the folder created by "devtool modify" to update > dependencies via the common tools and create a patch? > Maybe, but I don't use devtool at all for my recipe maintenance, and I would push back against it being a requirement to do so. Bruce
From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> The series adds on-the-fly support for package manager specific dependencies and vendor directories. It contains the following changes: 1. Adds an early fetch, unpack and patch task to unpack and patch source code with an embedded lock file for dependencies. 2. Parse the go.sum, Cargo.lock and package-lock.json lock files and resolve the dependencies to SRC_URIs. 3. Save the SRC_URIs in a file and adapt all SRC_URIs users to handle the SRC_URI files beside the SRC_URIs in the recipe. 4. Create a package manager specific vendor directory during unpack to support additional patching of the dependencies. 5. Add the dependency name and version to the SBOM. 6. Simplify the npm support Stefan Herbrechtsmeier (30): classes: create-spdx-2.2: use expanded FetchData for downloaded packages lib: spdx30_tasks: use expanded FetchData for download files classes: create-spdx-2.2: use name and version for download dependencies lib: bb: fetch2: add support to unpack .crate files lib: oe: add vendor module lib: oe: vendor: add cargo support lib: oe: vendor: add go support lib: oe: vendor: add npm support oeqa: oelib: add vendor tests conf: bitbake: add SRC_URI_FILES variable classes: go: make source directory configurable classes: go-mod: make class customizable classes: add nodejs-arch class classes: base: add get_src_uris and unpack_src_uris functions classes: add early fetch, unpack and patch support classes: add vendor class classes: add vendor class for cargo classes: add vendor class for go classes: add vendor class for npm classes: add vendor_npm_build class python3-bcrypt: mirgrate to vendor cargo class python3-cryptography: mirgrate to vendor cargo class python3-maturin: mirgrate to vendor cargo class python3-rpds-py: mirgrate to vendor cargo class librsvg: mirgrate to vendor cargo class librsvg: update dependecies to fix RUSTSEC-2024-0421 [DO NOT MERGE] recipes: add crucible go demo [DO NOT MERGE] recipes: add node-red npm demo [DO NOT MERGE] recipes: add nucleoidai npm demo [DO NOT MERGE] classes: spdx: use version 2.2 bitbake/lib/bb/fetch2/__init__.py | 2 +- .../crucible/crucible2_2023.11.02.bb | 18 + .../node-red/node-red/package-lock.json | 6096 +++++++++++++++++ .../node-red/node-red_4.0.8.bb | 14 + .../nucleoidai/nucleoidai_0.7.10.bb | 11 + meta/classes-global/base.bbclass | 47 +- meta/classes-global/patch.bbclass | 17 +- meta/classes-recipe/early.bbclass | 61 + meta/classes-recipe/go-mod.bbclass | 10 +- meta/classes-recipe/go.bbclass | 22 +- meta/classes-recipe/nodejs-arch.bbclass | 15 + meta/classes-recipe/vendor.bbclass | 28 + meta/classes-recipe/vendor_cargo.bbclass | 46 + meta/classes-recipe/vendor_go.bbclass | 59 + meta/classes-recipe/vendor_npm.bbclass | 115 + meta/classes-recipe/vendor_npm_build.bbclass | 50 + meta/classes/archiver.bbclass | 4 +- meta/classes/buildhistory.bbclass | 4 +- meta/classes/copyleft_compliance.bbclass | 2 +- meta/classes/create-spdx-2.2.bbclass | 14 +- meta/classes/create-spdx.bbclass | 2 +- meta/classes/externalsrc.bbclass | 2 +- meta/conf/bitbake.conf | 1 + meta/lib/oe/patch.py | 10 +- meta/lib/oe/spdx30_tasks.py | 5 +- meta/lib/oe/vendor/__init__.py | 28 + meta/lib/oe/vendor/cargo.py | 121 + meta/lib/oe/vendor/go.py | 96 + meta/lib/oe/vendor/npm.py | 141 + meta/lib/oeqa/selftest/cases/oelib/vendor.py | 237 + .../python/python3-bcrypt-crates.inc | 84 - .../python/python3-bcrypt_4.2.1.bb | 4 +- .../python/python3-cryptography-crates.inc | 76 - .../python/python3-cryptography.bb | 4 +- .../python/python3-maturin-crates.inc | 712 -- .../python/python3-maturin_1.8.1.bb | 4 +- .../python/python3-rpds-py-crates.inc | 54 - .../python/python3-rpds-py_0.22.3.bb | 4 +- meta/recipes-gnome/librsvg/librsvg-crates.inc | 590 -- ...-to-get-an-updated-idna-rustsec-2024.patch | 398 ++ meta/recipes-gnome/librsvg/librsvg_2.59.2.bb | 7 +- 41 files changed, 7633 insertions(+), 1582 deletions(-) create mode 100644 meta-selftest/recipes-support/crucible/crucible2_2023.11.02.bb create mode 100644 meta-selftest/recipes-support/node-red/node-red/package-lock.json create mode 100644 meta-selftest/recipes-support/node-red/node-red_4.0.8.bb create mode 100644 meta-selftest/recipes-support/nucleoidai/nucleoidai_0.7.10.bb create mode 100644 meta/classes-recipe/early.bbclass create mode 100644 meta/classes-recipe/nodejs-arch.bbclass create mode 100644 meta/classes-recipe/vendor.bbclass create mode 100644 meta/classes-recipe/vendor_cargo.bbclass create mode 100644 meta/classes-recipe/vendor_go.bbclass create mode 100644 meta/classes-recipe/vendor_npm.bbclass create mode 100644 meta/classes-recipe/vendor_npm_build.bbclass create mode 100644 meta/lib/oe/vendor/__init__.py create mode 100644 meta/lib/oe/vendor/cargo.py create mode 100644 meta/lib/oe/vendor/go.py create mode 100644 meta/lib/oe/vendor/npm.py create mode 100644 meta/lib/oeqa/selftest/cases/oelib/vendor.py delete mode 100644 meta/recipes-devtools/python/python3-bcrypt-crates.inc delete mode 100644 meta/recipes-devtools/python/python3-cryptography-crates.inc delete mode 100644 meta/recipes-devtools/python/python3-maturin-crates.inc delete mode 100644 meta/recipes-devtools/python/python3-rpds-py-crates.inc delete mode 100644 meta/recipes-gnome/librsvg/librsvg-crates.inc create mode 100644 meta/recipes-gnome/librsvg/librsvg/0001-update-url-crate-to-get-an-updated-idna-rustsec-2024.patch