| Message ID | 20250211150034.18696-18-stefan.herbrechtsmeier-oss@weidmueller.com |
|---|---|
| State | New |
| Headers | show |
| Series | Add vendor support for go, npm and rust | expand |
In message: [OE-core] [RFC PATCH 18/30] classes: add vendor class for go on 11/02/2025 Stefan Herbrechtsmeier via lists.openembedded.org wrote: > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > > Add a vendor class for go to resolve the dependency SRC_URIs from a > go.sum file and run populate the go mod vendor folder. > > Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > --- > > meta/classes-recipe/vendor_go.bbclass | 59 +++++++++++++++++++++++++++ > 1 file changed, 59 insertions(+) > create mode 100644 meta/classes-recipe/vendor_go.bbclass > > diff --git a/meta/classes-recipe/vendor_go.bbclass b/meta/classes-recipe/vendor_go.bbclass > new file mode 100644 > index 0000000000..dc5f8d1d8d > --- /dev/null > +++ b/meta/classes-recipe/vendor_go.bbclass > @@ -0,0 +1,59 @@ > +# Copyright (C) 2025 Weidmueller Interface GmbH & Co. KG > +# Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > +# > +# SPDX-License-Identifier: MIT > +# > + > +# The directory of the go.mod file relative to the root directory, per default > +# assume there's a file directly in the root directory > +GO_SRC_DIR ?= "" > + > +# The path to the go.mod file > +GO_MANIFEST_DIR ?= "${GO_SRC_PATH}/go.mod" > + > +# The path to go.sum file > +GO_LOCK_DIR ?= "${@os.path.join(os.path.dirname(d.getVar('GO_MANIFEST_DIR')), 'go.sum')}" > + > +# The URL of the go proxy > +GO_PROXY ?= "https://proxy.golang.org" > + > +GO_SRC_PATH = "${S}/${GO_SRC_DIR}" > +GO_SRC_SUBDIR = "${@os.path.relpath(d.getVar('CARGO_SRC_PATH'), d.getVar('WORKDIR'))}" > +GO_SRC_URI_FILE = "${VENDOR_DIR}/go-source-uris.txt" > +SRC_URI_FILES:append = " ${GO_SRC_URI_FILE}" > + > +inherit go-mod vendor > + > +GO_INSTALL_PREFIX = "." > +GO_SRC_PATH = "${S}" > +GO_SRCURI_DESTSUFFIX = "" > + > +GOMODCACHE = "invalid" > +GO_MOD_CACHE_DIR = "go/pkg/mod" > +GO_WORKPATH = "${GO_SRC_PATH}" > +GO_MOD_RECRDEPTASK = "" > + > +GOBUILDFLAGS:append = " -mod=vendor" > + > +python vendor_go_do_vendor_resolve() { > + import oe.vendor > + import oe.vendor.go > + > + lock_file_dir = d.getVar("GO_LOCK_DIR") > + lock_file_subdir = get_early_source_dir(d, lock_file_dir) > + proxy = d.getVar("GO_PROXY") > + cache_subdir = d.getVar("GO_MOD_CACHE_DIR") > + src_uris = oe.vendor.go.resolve_src_uris(lock_file_subdir, proxy, cache_subdir) > + with open(d.getVar("GO_SRC_URI_FILE"), "w") as f: > + oe.vendor.dump(f, src_uris) > +} > + > +run_go_mod_vendor() { > + cd ${GO_SRC_PATH} > + export GOMODCACHE="${UNPACKDIR}/${GO_MOD_CACHE_DIR}" > + ${GO} mod vendor What happens if a module isn't in the unpackdir ? I assume it goes to the go infrastructure and fetches the module ? Or is it impossible for one to not be in the cache (i.e. is it always fully pre-poulated via the vendor_go_do_vendor_resolve ?) It is unclear to me how the vendor'd modules get into an archive that can be saved for future reproducible builds (can you point me at that glue?). Bruce > +} > +do_unpack[postfuncs] += "run_go_mod_vendor" > +do_unpack[depends] += "${@oe.utils.build_depends_string(d.getVar('DEPENDS_GOLANG'), 'do_populate_sysroot')}" > + > +EXPORT_FUNCTIONS do_vendor_resolve > -- > 2.39.5 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#211143): https://lists.openembedded.org/g/openembedded-core/message/211143 > Mute This Topic: https://lists.openembedded.org/mt/111123541/1050810 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Am 11.02.2025 um 23:59 schrieb Bruce Ashfield: > In message: [OE-core] [RFC PATCH 18/30] classes: add vendor class for go > on 11/02/2025 Stefan Herbrechtsmeier via lists.openembedded.org wrote: > >> From: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com> >> >> Add a vendor class for go to resolve the dependency SRC_URIs from a >> go.sum file and run populate the go mod vendor folder. >> >> Signed-off-by: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com> >> --- >> >> meta/classes-recipe/vendor_go.bbclass | 59 +++++++++++++++++++++++++++ >> 1 file changed, 59 insertions(+) >> create mode 100644 meta/classes-recipe/vendor_go.bbclass >> >> diff --git a/meta/classes-recipe/vendor_go.bbclass b/meta/classes-recipe/vendor_go.bbclass >> new file mode 100644 >> index 0000000000..dc5f8d1d8d >> --- /dev/null >> +++ b/meta/classes-recipe/vendor_go.bbclass >> @@ -0,0 +1,59 @@ >> +# Copyright (C) 2025 Weidmueller Interface GmbH & Co. KG >> +# Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com> >> +# >> +# SPDX-License-Identifier: MIT >> +# >> + >> +# The directory of the go.mod file relative to the root directory, per default >> +# assume there's a file directly in the root directory >> +GO_SRC_DIR ?= "" >> + >> +# The path to the go.mod file >> +GO_MANIFEST_DIR ?= "${GO_SRC_PATH}/go.mod" >> + >> +# The path to go.sum file >> +GO_LOCK_DIR ?="${@os.path.join(os.path.dirname(d.getVar('GO_MANIFEST_DIR')), >> 'go.sum')}" >> + >> +# The URL of the go proxy >> +GO_PROXY ?="https://proxy.golang.org" >> + >> +GO_SRC_PATH = "${S}/${GO_SRC_DIR}" >> +GO_SRC_SUBDIR ="${@os.path.relpath(d.getVar('CARGO_SRC_PATH'), d.getVar('WORKDIR'))}" >> +GO_SRC_URI_FILE = "${VENDOR_DIR}/go-source-uris.txt" >> +SRC_URI_FILES:append = " ${GO_SRC_URI_FILE}" >> + >> +inherit go-mod vendor >> + >> +GO_INSTALL_PREFIX = "." >> +GO_SRC_PATH = "${S}" >> +GO_SRCURI_DESTSUFFIX = "" >> + >> +GOMODCACHE = "invalid" >> +GO_MOD_CACHE_DIR = "go/pkg/mod" >> +GO_WORKPATH = "${GO_SRC_PATH}" >> +GO_MOD_RECRDEPTASK = "" >> + >> +GOBUILDFLAGS:append = " -mod=vendor" >> + >> +python vendor_go_do_vendor_resolve() { >> + import oe.vendor >> + import oe.vendor.go >> + >> + lock_file_dir = d.getVar("GO_LOCK_DIR") >> + lock_file_subdir = get_early_source_dir(d, lock_file_dir) >> + proxy = d.getVar("GO_PROXY") >> + cache_subdir = d.getVar("GO_MOD_CACHE_DIR") >> + src_uris = oe.vendor.go.resolve_src_uris(lock_file_subdir, proxy, cache_subdir) >> + with open(d.getVar("GO_SRC_URI_FILE"), "w") as f: >> + oe.vendor.dump(f, src_uris) >> +} >> + >> +run_go_mod_vendor() { >> + cd ${GO_SRC_PATH} >> + export GOMODCACHE="${UNPACKDIR}/${GO_MOD_CACHE_DIR}" >> + ${GO} mod vendor > What happens if a module isn't in the unpackdir ? I assume it goes > to the go infrastructure and fetches the module ? Or is it > impossible for one to not be in the cache (i.e. is it > always fully pre-poulated via the vendor_go_do_vendor_resolve ?) A missing packages leads to an build error because the network is disabled. But this shouldn't happen because the resolve task add all modules to the dynamic SRC_URI. > It is unclear to me how the vendor'd modules get into an archive > that can be saved for future reproducible builds (can you point > me at that glue?). The do_vendor_resolve task add all modules from the go.sum to the dynamic SRC_URI. The fetch and unpack task fetch the modules and unpack it to the cache. The go mod vendor command create a vendor directory from the populated cache. Depending on the archiver class function the archive contains the created vendor directory in S or the recipe and dynamic SRC_URI archives.
diff --git a/meta/classes-recipe/vendor_go.bbclass b/meta/classes-recipe/vendor_go.bbclass new file mode 100644 index 0000000000..dc5f8d1d8d --- /dev/null +++ b/meta/classes-recipe/vendor_go.bbclass @@ -0,0 +1,59 @@ +# Copyright (C) 2025 Weidmueller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> +# +# SPDX-License-Identifier: MIT +# + +# The directory of the go.mod file relative to the root directory, per default +# assume there's a file directly in the root directory +GO_SRC_DIR ?= "" + +# The path to the go.mod file +GO_MANIFEST_DIR ?= "${GO_SRC_PATH}/go.mod" + +# The path to go.sum file +GO_LOCK_DIR ?= "${@os.path.join(os.path.dirname(d.getVar('GO_MANIFEST_DIR')), 'go.sum')}" + +# The URL of the go proxy +GO_PROXY ?= "https://proxy.golang.org" + +GO_SRC_PATH = "${S}/${GO_SRC_DIR}" +GO_SRC_SUBDIR = "${@os.path.relpath(d.getVar('CARGO_SRC_PATH'), d.getVar('WORKDIR'))}" +GO_SRC_URI_FILE = "${VENDOR_DIR}/go-source-uris.txt" +SRC_URI_FILES:append = " ${GO_SRC_URI_FILE}" + +inherit go-mod vendor + +GO_INSTALL_PREFIX = "." +GO_SRC_PATH = "${S}" +GO_SRCURI_DESTSUFFIX = "" + +GOMODCACHE = "invalid" +GO_MOD_CACHE_DIR = "go/pkg/mod" +GO_WORKPATH = "${GO_SRC_PATH}" +GO_MOD_RECRDEPTASK = "" + +GOBUILDFLAGS:append = " -mod=vendor" + +python vendor_go_do_vendor_resolve() { + import oe.vendor + import oe.vendor.go + + lock_file_dir = d.getVar("GO_LOCK_DIR") + lock_file_subdir = get_early_source_dir(d, lock_file_dir) + proxy = d.getVar("GO_PROXY") + cache_subdir = d.getVar("GO_MOD_CACHE_DIR") + src_uris = oe.vendor.go.resolve_src_uris(lock_file_subdir, proxy, cache_subdir) + with open(d.getVar("GO_SRC_URI_FILE"), "w") as f: + oe.vendor.dump(f, src_uris) +} + +run_go_mod_vendor() { + cd ${GO_SRC_PATH} + export GOMODCACHE="${UNPACKDIR}/${GO_MOD_CACHE_DIR}" + ${GO} mod vendor +} +do_unpack[postfuncs] += "run_go_mod_vendor" +do_unpack[depends] += "${@oe.utils.build_depends_string(d.getVar('DEPENDS_GOLANG'), 'do_populate_sysroot')}" + +EXPORT_FUNCTIONS do_vendor_resolve