| Message ID | 20250211150034.18696-6-stefan.herbrechtsmeier-oss@weidmueller.com |
|---|---|
| State | New |
| Headers | show |
| Series | Add vendor support for go, npm and rust | expand |
On Tue, 2025-02-11 at 16:00 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote: > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > > Add a vendor package as base for package manager specific > implementations to resolve dependencies and populate vendor directories. > Add common dump and load function for SRC_URI files. > > Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > --- > > meta/lib/oe/vendor/__init__.py | 28 ++++++++++++++++++++++++++++ > 1 file changed, 28 insertions(+) > create mode 100644 meta/lib/oe/vendor/__init__.py Initially I was going to ask there to be a "fetch" or "download" in the namespacing. That then made me wonder if in the interests of clarity and namepacing, these vendor classes should go into lib/bb/fetch2/vendor in bitbake? I appreciate the class in OE will use them and separation into bitbake can be a bit awkward but it would mean we don't confuse this with any other kind of vendoring, it becomes fetch specific and it places them alongside the fetch code, which it will need to remain tied to. I do feel we need to namespace them though as vendor is not specific enough to be directly in lib/oe. Cheers, Richard
Am 11.02.2025 um 22:31 schrieb Richard Purdie: > On Tue, 2025-02-11 at 16:00 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote: >> From: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com> >> >> Add a vendor package as base for package manager specific >> implementations to resolve dependencies and populate vendor directories. >> Add common dump and load function for SRC_URI files. >> >> Signed-off-by: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com> >> --- >> >> meta/lib/oe/vendor/__init__.py | 28 ++++++++++++++++++++++++++++ >> 1 file changed, 28 insertions(+) >> create mode 100644 meta/lib/oe/vendor/__init__.py > Initially I was going to ask there to be a "fetch" or "download" in the > namespacing. That then made me wonder if in the interests of clarity > and namepacing, these vendor classes should go into > lib/bb/fetch2/vendor in bitbake? The package isn't limited to fetch / download. In some cases it populate the vendor directory and could be used for the license extraction in future. I think it is a bad idea to move code into bitbake which isn't used by bitbake. It will complicate future development without any benefit. > I appreciate the class in OE will use them and separation into bitbake > can be a bit awkward but it would mean we don't confuse this with any > other kind of vendoring What do you mean by "any other kind of vendoring"? > , it becomes fetch specific and it places them > alongside the fetch code, which it will need to remain tied to. It isn't fetch specific. The modules in the package parse lock files and extract it content or populate the vendor directory. > I do feel we need to namespace them though as vendor is not specific > enough to be directly in lib/oe. I chose vendor because it is the common name in the package manager to create a folder which contains all package manager specific dependencies.
On Wed, 2025-02-12 at 10:27 +0100, Stefan Herbrechtsmeier wrote: > Am 11.02.2025 um 22:31 schrieb Richard Purdie: > > On Tue, 2025-02-11 at 16:00 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote: > > > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > > > > > > Add a vendor package as base for package manager specific > > > implementations to resolve dependencies and populate vendor directories. > > > Add common dump and load function for SRC_URI files. > > > > > > Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > > > --- > > > > > > meta/lib/oe/vendor/__init__.py | 28 ++++++++++++++++++++++++++++ > > > 1 file changed, 28 insertions(+) > > > create mode 100644 meta/lib/oe/vendor/__init__.py > > > > > > > > > > > > Initially I was going to ask there to be a "fetch" or "download" in the > > namespacing. That then made me wonder if in the interests of clarity > > and namepacing, these vendor classes should go into > > lib/bb/fetch2/vendor in bitbake? > > > > > The package isn't limited to fetch / download. In some cases it > populate the vendor directory and could be used for the license > extraction in future. "populate" and "extraction" sound related to the fetcher. > I think it is a bad idea to move code into bitbake which isn't used > by bitbake. It will complicate future development without any > benefit. You could argue that about the fetch module. It does encourage strong/stable APIs to the modules, which has been of benefit. I can see the arguments either way though. > > I appreciate the class in OE will use them and separation into bitbake > > can be a bit awkward but it would mean we don't confuse this with any > > other kind of vendoring > > What do you mean by "any other kind of vendoring"? When someone customises their BSP, the original might be from "the vendor" so perhaps this code handles that? OSV as in software vendor is going to get confused with this. I'd imagine there will be other ways people could read it too. We have TARGET_VENDOR, HOST_VENDOR, SDK_VENDOR and so on in the triplets. My point is that if someone sees a "vendor" module in isolation, they aren't going to know what it relates to. You're really close to this topic so it is obvious to you, it is not going to be obvious to others, or perhaps even you in a few years time. Cheers, Richard
Am 12.02.2025 um 10:38 schrieb Richard Purdie: > On Wed, 2025-02-12 at 10:27 +0100, Stefan Herbrechtsmeier wrote: >> Am 11.02.2025 um 22:31 schrieb Richard Purdie: >>> On Tue, 2025-02-11 at 16:00 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote: >>>> From: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com> >>>> >>>> Add a vendor package as base for package manager specific >>>> implementations to resolve dependencies and populate vendor directories. >>>> Add common dump and load function for SRC_URI files. >>>> >>>> Signed-off-by: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com> >>>> --- >>>> >>>> meta/lib/oe/vendor/__init__.py | 28 ++++++++++++++++++++++++++++ >>>> 1 file changed, 28 insertions(+) >>>> create mode 100644 meta/lib/oe/vendor/__init__.py >>>> >>>> >>> >>> >>> Initially I was going to ask there to be a "fetch" or "download" in the >>> namespacing. That then made me wonder if in the interests of clarity >>> and namepacing, these vendor classes should go into >>> lib/bb/fetch2/vendor in bitbake? >>> >>> >> The package isn't limited to fetch / download. In some cases it >> populate the vendor directory and could be used for the license >> extraction in future. > "populate" and "extraction" sound related to the fetcher. > >> I think it is a bad idea to move code into bitbake which isn't used >> by bitbake. It will complicate future development without any >> benefit. > You could argue that about the fetch module. It does encourage > strong/stable APIs to the modules, which has been of benefit. I can see > the arguments either way though. At the moment there is no common api between the different resolve functions. The different package manager need different details. But it is possible to unify the functions parameters. Where would you place the code in bitbake? >>> I appreciate the class in OE will use them and separation into bitbake >>> can be a bit awkward but it would mean we don't confuse this with any >>> other kind of vendoring >> What do you mean by "any other kind of vendoring"? > When someone customises their BSP, the original might be from "the > vendor" so perhaps this code handles that? OSV as in software vendor is > going to get confused with this. I'd imagine there will be other ways > people could read it too. We have TARGET_VENDOR, HOST_VENDOR, > SDK_VENDOR and so on in the triplets. What name do you recommended? > My point is that if someone sees a "vendor" module in isolation, they > aren't going to know what it relates to. You're really close to this > topic so it is obvious to you, it is not going to be obvious to others, > or perhaps even you in a few years time. I fear names like dependencies, package manager or resolver are misleading too.
diff --git a/meta/lib/oe/vendor/__init__.py b/meta/lib/oe/vendor/__init__.py new file mode 100644 index 0000000000..d9a22a91ca --- /dev/null +++ b/meta/lib/oe/vendor/__init__.py @@ -0,0 +1,28 @@ +from dataclasses import dataclass +from typing import List + +class VendorError(Exception): + def __init__(self, message): + self.msg = message + super().__init__(message) + + def __str__(self): + return self.msg + +class ResolveError(VendorError): + def __init__(self, message, localpath): + msg = "Resolve dependency failure for file: '%s'. %s" % (localpath, message) + self.localpath = localpath + super().__init__(msg) + self.args = (message, localpath) + +def dump(fp, uris): + for uri in uris: + fp.write(uri) + fp.write("\n") + +def load(fp): + return [line.rstrip() for line in fp] + +def determine_downloadfilename(type, filename): + return f"{type}/{filename}"