| Message ID | 20221222150246.91423-1-uvv.mail@gmail.com |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [langdale,kirkstone] go-mod.bbclass: Allow network in do_compile | expand |
On Thu, 2022-12-22 at 16:02 +0100, Vyacheslav Yurkov wrote: > This is a workaround to allow using go-mod until a proper solution is in > place. Possible solutions were discussued ealier this year in these > threads: > > https://lists.openembedded.org/g/openembedded-core/topic/89464905 > https://lists.openembedded.org/g/openembedded-architecture/topic/90782883 > > Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> > --- > meta/classes-recipe/go-mod.bbclass | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/meta/classes-recipe/go-mod.bbclass b/meta/classes-recipe/go-mod.bbclass > index 61571596bc..da9a583288 100644 > --- a/meta/classes-recipe/go-mod.bbclass > +++ b/meta/classes-recipe/go-mod.bbclass > @@ -24,3 +24,8 @@ inherit go > > GO_WORKDIR ?= "${GO_IMPORT}" > do_compile[dirs] += "${B}/src/${GO_WORKDIR}" > +do_compile[network] = "1" > + > +do_compile:prepend() { > + bbwarn "Using go-mod enables network during compilation to be able to fetch dependencies. This breaks reproducibility of the build!" > +} This patch is not in master and is not a backport therefore not eligible for kirkstone/langdale. Cheers, Richard
On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Thu, 2022-12-22 at 16:02 +0100, Vyacheslav Yurkov wrote: > > This is a workaround to allow using go-mod until a proper solution is in > > place. Possible solutions were discussued ealier this year in these > > threads: > > > > https://lists.openembedded.org/g/openembedded-core/topic/89464905 > > https://lists.openembedded.org/g/openembedded-architecture/topic/90782883 > > > > Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> > > --- > > meta/classes-recipe/go-mod.bbclass | 5 +++++ > > 1 file changed, 5 insertions(+) > > > > diff --git a/meta/classes-recipe/go-mod.bbclass b/meta/classes-recipe/go-mod.bbclass > > index 61571596bc..da9a583288 100644 > > --- a/meta/classes-recipe/go-mod.bbclass > > +++ b/meta/classes-recipe/go-mod.bbclass > > @@ -24,3 +24,8 @@ inherit go > > > > GO_WORKDIR ?= "${GO_IMPORT}" > > do_compile[dirs] += "${B}/src/${GO_WORKDIR}" > > +do_compile[network] = "1" > > + > > +do_compile:prepend() { > > + bbwarn "Using go-mod enables network during compilation to be able to fetch dependencies. This breaks reproducibility of the build!" > > +} > > This patch is not in master and is not a backport therefore not > eligible for kirkstone/langdale. > 'nor should it go to master. If someone wants to allow go to fetch sources during builds, it needs to be done in their own layers. Bruce > Cheers, > > Richard > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#174968): https://lists.openembedded.org/g/openembedded-core/message/174968 > Mute This Topic: https://lists.openembedded.org/mt/95827446/1050810 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 22.12.2022 17:48, Bruce Ashfield wrote: > On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: >> This patch is not in master and is not a backport therefore not >> eligible for kirkstone/langdale. My bad, the intention was to have it in all three branches: master, kirkstone, and langdale. > 'nor should it go to master. > > If someone wants to allow go to fetch sources during builds, it needs > to be done in their own layers. > > Bruce Probably I misunderstood the outcome of the discussion in my first link. Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad practice in general (I'm aware of a few more build systems with the same shortcoming). My point is, until a proper solution is in place, it should be at least documented somewhere that this should be done in own layer/recipe. Any suggestions where this can be documented if not in OE-Core? Vyacheslav
On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <uvv.mail@gmail.com> wrote: > On 22.12.2022 17:48, Bruce Ashfield wrote: > > On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > >> This patch is not in master and is not a backport therefore not > >> eligible for kirkstone/langdale. > > My bad, the intention was to have it in all three branches: master, > kirkstone, and langdale. > > > 'nor should it go to master. > > > > If someone wants to allow go to fetch sources during builds, it needs > > to be done in their own layers. > > > > Bruce > > Probably I misunderstood the outcome of the discussion in my first link. > Bruce and Richard, you both suggested to have it in go-mod with the > warning. I agree that it breaks reproducibility, and is considered a bad > practice in general (I'm aware of a few more build systems with the same > shortcoming). > > My point is, until a proper solution is in place, it should be at least > documented somewhere that this should be done in own layer/recipe. Any > suggestions where this can be documented if not in OE-Core? > Have you tried the changes submitted by Lukas/Stefan in: https://lists.openembedded.org/g/openembedded-architecture/message/1539 ? It's not perfect, I was testing it on https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed. But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well). Lukas/Stefan: is there something we can do to help with v2 of your patches? I wanted to submit better review on submitted patches after more debugging, but here it is: 1) to resolve 2 deps: {'Path': 'gonum.org/v1/gonum', 'Version': 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found" {'Path': 'code.cloudfoundry.org/clock', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org'. (_ssl.c:996)>" parse 2) some issues with LICENSEs: INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replace `Unknown` with the license: c4807c5b058a56c02f592207b4e056b1,Unknown Traceback (most recent call last): File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 111, in <module> ret = main() ^^^^^^ File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 100, in main ret = args.func(args) ^^^^^^^^^^^^^^^ File "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, in create_recipe handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues) File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process self._rewrite_lic_uri(lines_before) File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in _rewrite_lic_uri updated, newlines = bb.utils.edit_metadata(lines_before, ['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in edit_metadata if handle_var_end(): ^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in handle_var_end (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, op, newlines) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfunc uri, chksum = license.split(';', 1) ^^^^^^^^^^^ ValueError: not enough values to unpack (expected 2, got 1) (the license was just '/') 3) some SRCREVs set to None leading to parsing failure: File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in _latest_revision raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH ERROR: Parsing newly created recipe failed, moving recipe to /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed for reference. If this looks to be caused by the recipe itself, please report this error. It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.
Martin, I did see that patch series, but I was not sure about its state, therefore I didn't try it. If I understood correctly, that partially solves the issue, but the proper solution should involve the fetcher, that's why it's not yet merged to master. Am I missing something? Regards, Slava On 22.12.2022 20:08, Martin Jansa wrote: > > Have you tried the changes submitted by Lukas/Stefan in: > https://lists.openembedded.org/g/openembedded-architecture/message/1539 > ? > > It's not perfect, I was testing it on > https://github.com/influxdata/telegraf/blob/master/go.mod and I've > found some corner cases where it failed. > > But it seems like very good start and we should work with Lukas/Stefan > to get it merged in master. Then all branches could consume recipes > created in master and only the exceptions would need to have network > access (instead of the bbclass enabling it for every go-mod user - > go-vendor inherits go-mod as well). > > Lukas/Stefan: is there something we can do to help with v2 of your > patches? > > I wanted to submit better review on submitted patches after more > debugging, but here it is: > > 1) to resolve 2 deps: > {'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version': > 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP > Error 404: Not Found" > > {'Path': 'code.cloudfoundry.org/clock > <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', > 'Indirect': True} due to this being redirect to github.com > <http://github.com> and failing with "ERROR: Error while fetching > redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] > certificate verify failed: Hostname mismatch, certificate is not valid > for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. (_ssl.c:996)>" > parse > > 2) some issues with LICENSEs: > INFO: Please add the following line for > 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' > and replace `Unknown` with the license: > c4807c5b058a56c02f592207b4e056b1,Unknown > Traceback (most recent call last): > File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line > 111, in <module> > ret = main() > ^^^^^^ > File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line > 100, in main > ret = args.func(args) > ^^^^^^^^^^^^^^^ > File > "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line > 746, in create_recipe > handler.process(srctree_use, classes, lines_before, lines_after, > handled, extravalues) > File > "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", > line 349, in process > self._rewrite_lic_uri(lines_before) > File > "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", > line 380, in _rewrite_lic_uri > updated, newlines = bb.utils.edit_metadata(lines_before, > ['LIC_FILES_CHKSUM'], varfunc) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in > edit_metadata > if handle_var_end(): > ^^^^^^^^^^^^^^^^ > File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in > handle_var_end > (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, > op, newlines) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File > "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", > line 372, in varfunc > uri, chksum = license.split(';', 1) > ^^^^^^^^^^^ > ValueError: not enough values to unpack (expected 2, got 1) > > (the license was just '/') > > 3) some SRCREVs set to None leading to parsing failure: > > File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in > _latest_revision > raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git > repository in git ls-remote output for %s" % \ > bb.data_smart.ExpansionError: Failure expanding variable SRCPV, > expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception > FetchError: Fetcher failure: Unable to resolve 'None' in upstream git > repository in git ls-remote output for > github.com/googleapis/google-cloud-go > <http://github.com/googleapis/google-cloud-go> > The variable dependency chain for the failure is: SRCPV -> PV -> BP -> > FILESPATH > > ERROR: Parsing newly created recipe failed, moving recipe to > /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed > for reference. If this looks to be caused by the recipe itself, please > report this error. > > It takes long time to fetch all telegraf dependencies, so I'll create > some test recipe to more easily debug these 3 issues in isolation. >
On Mon, Dec 26, 2022 at 10:38 AM Vyacheslav Yurkov <uvv.mail@gmail.com> wrote: > Martin, I did see that patch series, but I was not sure about its state, > therefore I didn't try it. If I understood correctly, that partially solves > the issue, but the proper solution should involve the fetcher, that's why > it's not yet merged to master. Am I missing something? > That's not my understanding. The proposed changes for recipetool had some cosmetic and functional review comments and it doesn't work correctly for every possible go dependency (e.g. the 3 issues I've mentioned from telegram recipe), but it's very good starting point and we should be able to get it merged in master (and then everybody can more easily contribute fixes for the corner cases where it might not work well yet). Even missing documentation IMHO shouldn't block merging it at this point, because there is a clear need this (as there are already 3 partial implementations to fix this issue, neither 100% complete and nothing merged in master - the recipetool changes are IMHO the best of 3 as it integrates with devtool as well and seems to be most complete). go fetcher in bitbake is another possibility for future, but generating the long deps in SRC_URI is much better solution then enabling network in every go-mod recipe. Please try recipetool changes on your go-mod recipes and provide feedback. Regards,
On 26.12.2022 10:59, Martin Jansa wrote: > > Please try recipetool changes on your go-mod recipes and provide feedback. > Coincidentally, I also needed grpc-web recipe, which original patchset was tested on. I applied the series on my kirkstone branch of poky and tried to run the command: % recipetool create -S c1971f71a28950d212b7ddc8df07b5563723031d -B master https://github.com/improbable-eng/grpc-web.git Unfortunately this failed with the following log: WARNING: go.mod files generated by Go < 1.17 might have incomplete indirect dependencies. Traceback (most recent call last): File "/home/uvv/projects/poky/scripts/recipetool", line 111, in <module> ret = main() File "/home/uvv/projects/poky/scripts/recipetool", line 100, in main ret = args.func(args) File "/home/uvv/projects/poky/scripts/lib/recipetool/create.py", line 746, in create_recipe handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues) File "/home/uvv/projects/poky/scripts/lib/recipetool/create_go.py", line 332, in process go_mod = self._go_mod_patch(patchfile, go_import, srctree, localfilesdir, extravalues, d) File "/home/uvv/projects/poky/scripts/lib/recipetool/create_go.py", line 291, in _go_mod_patch bb.process.run("go mod tidy", stderr=subprocess.STDOUT, env=runenv, shell=True, cwd=srctree) File "/home/uvv/projects/poky/bitbake/lib/bb/process.py", line 189, in run raise ExecutionError(cmd, pipe.returncode, stdout, stderr) bb.process.ExecutionError: Execution of 'go mod tidy' failed with exit code 1: go: finding module for package google.golang.org/grpc/metadata go: finding module for package google.golang.org/grpc go: finding module for package github.com/golang/protobuf/protoc-gen-go go: finding module for package golang.org/x/net/context go: finding module for package google.golang.org/grpc/codes go: finding module for package google.golang.org/grpc/grpclog go: downloading golang.org/x/net v0.4.0 go: downloading google.golang.org/grpc v1.51.0 go: downloading github.com/golang/protobuf v1.5.2 go: finding module for package github.com/cenkalti/backoff/v4 go: downloading github.com/cenkalti/backoff/v4 v4.2.0 go: finding module for package github.com/desertbit/timer go: downloading github.com/cenkalti/backoff v2.2.1+incompatible go: finding module for package github.com/rs/cors go: finding module for package golang.org/x/net/http2 go: finding module for package google.golang.org/grpc/health/grpc_health_v1 go: finding module for package google.golang.org/grpc/status go: finding module for package nhooyr.io/websocket go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus go: downloading github.com/rs/cors v1.8.3 go: finding module for package github.com/grpc-ecosystem/go-grpc-prometheus go: finding module for package github.com/mwitkow/go-conntrack go: downloading nhooyr.io/websocket v1.8.7 go: finding module for package github.com/mwitkow/go-conntrack/connhelpers go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 go: finding module for package github.com/mwitkow/grpc-proxy/proxy go: finding module for package github.com/prometheus/client_golang/prometheus/promhttp go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f go: finding module for package github.com/sirupsen/logrus go: finding module for package github.com/spf13/pflag go: finding module for package golang.org/x/net/trace go: finding module for package google.golang.org/grpc/credentials go: finding module for package github.com/golang/protobuf/ptypes/empty go: downloading github.com/mwitkow/grpc-proxy v0.0.0-20220126150247-db34e7bfee32 go: finding module for package github.com/golang/protobuf/proto go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/sirupsen/logrus v1.9.0 go: finding module for package github.com/stretchr/testify/assert go: finding module for package github.com/stretchr/testify/require go: finding module for package github.com/stretchr/testify/suite go: finding module for package google.golang.org/grpc/health go: finding module for package google.golang.org/protobuf/reflect/protoreflect go: downloading github.com/prometheus/client_golang v1.14.0 go: downloading github.com/stretchr/testify v1.8.1 go: finding module for package google.golang.org/protobuf/runtime/protoimpl go: finding module for package google.golang.org/protobuf/types/known/emptypb go: downloading google.golang.org/protobuf v1.28.1 go: found github.com/golang/protobuf/protoc-gen-go in github.com/golang/protobuf v1.5.2 go: found golang.org/x/net/context in golang.org/x/net v0.4.0 go: found google.golang.org/grpc in google.golang.org/grpc v1.51.0 go: found google.golang.org/grpc/codes in google.golang.org/grpc v1.51.0 go: found google.golang.org/grpc/grpclog in google.golang.org/grpc v1.51.0 go: found google.golang.org/grpc/metadata in google.golang.org/grpc v1.51.0 go: found github.com/cenkalti/backoff/v4 in github.com/cenkalti/backoff/v4 v4.2.0 go: found github.com/desertbit/timer in github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: found github.com/rs/cors in github.com/rs/cors v1.8.3 go: found golang.org/x/net/http2 in golang.org/x/net v0.4.0 go: found google.golang.org/grpc/health/grpc_health_v1 in google.golang.org/grpc v1.51.0 go: found google.golang.org/grpc/status in google.golang.org/grpc v1.51.0 go: found nhooyr.io/websocket in nhooyr.io/websocket v1.8.7 go: found github.com/grpc-ecosystem/go-grpc-middleware in github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 go: found github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus in github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 go: found github.com/grpc-ecosystem/go-grpc-prometheus in github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: found github.com/mwitkow/go-conntrack in github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f go: found github.com/mwitkow/go-conntrack/connhelpers in github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f go: found github.com/mwitkow/grpc-proxy/proxy in github.com/mwitkow/grpc-proxy v0.0.0-20220126150247-db34e7bfee32 go: found github.com/prometheus/client_golang/prometheus/promhttp in github.com/prometheus/client_golang v1.14.0 go: found github.com/sirupsen/logrus in github.com/sirupsen/logrus v1.9.0 go: found github.com/spf13/pflag in github.com/spf13/pflag v1.0.5 go: found golang.org/x/net/trace in golang.org/x/net v0.4.0 go: found google.golang.org/grpc/credentials in google.golang.org/grpc v1.51.0 go: found github.com/golang/protobuf/ptypes/empty in github.com/golang/protobuf v1.5.2 go: found github.com/golang/protobuf/proto in github.com/golang/protobuf v1.5.2 go: found github.com/stretchr/testify/assert in github.com/stretchr/testify v1.8.1 go: found github.com/stretchr/testify/require in github.com/stretchr/testify v1.8.1 go: found github.com/stretchr/testify/suite in github.com/stretchr/testify v1.8.1 go: found google.golang.org/grpc/health in google.golang.org/grpc v1.51.0 go: found google.golang.org/protobuf/reflect/protoreflect in google.golang.org/protobuf v1.28.1 go: found google.golang.org/protobuf/runtime/protoimpl in google.golang.org/protobuf v1.28.1 go: found google.golang.org/protobuf/types/known/emptypb in google.golang.org/protobuf v1.28.1 go: downloading google.golang.org/genproto v0.0.0-20210401141331-865547bb08e2 go: downloading github.com/google/go-cmp v0.5.6 go: downloading github.com/klauspost/compress v1.10.3 go: downloading github.com/gin-gonic/gin v1.6.3 go: downloading github.com/gobwas/ws v1.0.2 go: downloading github.com/gorilla/websocket v1.4.1 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading github.com/pmezard/go-difflib v1.0.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/jpillora/backoff v1.0.0 go: downloading github.com/prometheus/client_model v0.3.0 go: downloading github.com/prometheus/common v0.37.0 go: downloading golang.org/x/sys v0.3.0 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/mattn/go-isatty v0.0.12 go: downloading github.com/gobwas/httphead v0.0.0-20180130184737-2c6c146eadee go: downloading github.com/gobwas/pool v0.2.0 go: downloading golang.org/x/text v0.5.0 go: downloading gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.1.2 go: downloading github.com/prometheus/procfs v0.8.0 go: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1 go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 go: downloading github.com/go-playground/validator/v10 v10.2.0 go: downloading github.com/ugorji/go/codec v1.1.7 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/ugorji/go v1.1.7 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/go-playground/universal-translator v0.17.0 go: downloading github.com/leodido/go-urn v1.2.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/go-playground/locales v0.13.0 go: downloading github.com/google/go-cmp v0.5.8 go: downloading golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f go: downloading gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/kr/pretty v0.1.0 go: downloading github.com/kr/text v0.1.0 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 github.com/improbable-eng/grpc-web/go/grpcwebproxy imports github.com/mwitkow/grpc-proxy/proxy tested by github.com/mwitkow/grpc-proxy/proxy.test imports github.com/mwitkow/grpc-proxy/testservice imports golang.org/x/sync/errgroup loaded from golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f, but go 1.16 would select v0.0.0-20220722155255-886fb9371eb4 To upgrade to the versions selected by go 1.16: go mod tidy -go=1.16 && go mod tidy -go=1.17 If reproducibility with go 1.16 is not needed: go mod tidy -compat=1.17 For other options, see: https://golang.org/doc/modules/pruning Kirkstone uses go1.17, but for some reason it's not detected properly. Perhaps I'm doing something wrong? Thanks, Slava
Hi Martin, Am 26.12.2022 um 10:59 schrieb Martin Jansa via lists.openembedded.org: > On Mon, Dec 26, 2022 at 10:38 AM Vyacheslav Yurkov > <uvv.mail@gmail.com> wrote: > > Martin, I did see that patch series, but I was not sure about its > state, therefore I didn't try it. If I understood correctly, that > partially solves the issue, but the proper solution should involve > the fetcher, that's why it's not yet merged to master. Am I > missing something? > > > That's not my understanding. > > The proposed changes for recipetool had some cosmetic and functional > review comments and it doesn't work correctly for every possible go > dependency (e.g. the 3 issues I've mentioned from telegram recipe), > but it's very good starting point and we should be able to get it > merged in master (and then everybody can more easily contribute fixes > for the corner cases where it might not work well yet). > > Even missing documentation IMHO shouldn't block merging it at this > point, because there is a clear need this (as there are already 3 > partial implementations to fix this issue, neither 100% complete and > nothing merged in master - the recipetool changes are IMHO the best of > 3 as it integrates with devtool as well and seems to be most complete). > > go fetcher in bitbake is another possibility for future, but > generating the long deps in SRC_URI is much better solution then > enabling network in every go-mod recipe. Does an agreement exist if we should use a package manager class which creates a wget fetcher URI (ex. pipi) or a specific fetcher (ex. crate and npm)? At the moment both specific fetchers have different URI styles, create their specific tar commands and doesn't reuse the wget fetcher. Furthermore, the crate fetcher doesn't check the integrity during fetch. The fetcher inside bitbake have the advantage that you can fetch the latest version (ex. AUTOREV), but it brings the risk of code duplication and missing features. If we have an agreement and a common style its easy to replace the function with a specific fetcher. Regards Stefan
Martin,
first of all: thank you for testing the patches. As usual the corner
cases are the most tricky ones.
tl;dr: I'm working on an updated patch series to address the reported
issues.
I've looked into the issues and would like to give some explanation.
1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for
resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control
systems'). The documentation states that there should be a html page,
queried by "?go-get=1", with a <meta>-tag which contains the original
URL to the source-code repository. For 'gonum' there is only a 404-page.
This page contains javascript-code that redirects to the actual gonum
package page. Unnecessary hard to process. I'll open an issue for that.
The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I
already opend an issue on github for this.
I addressed this issue above by maintaining a list of modules and their
actual repo URLs inside the go-recipetool as an absolute fallback. Would
do you think about this solution?
2) yes, my bad :)
3) The problem is that some repositories have a prefix in their tags.
The tags usually only contain the semantic version string. I'm curretly
working on this issue.
Best regards
Lukas
Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:
> On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <uvv.mail@gmail.com>
> wrote:
>
> On 22.12.2022 17:48, Bruce Ashfield wrote:
> > On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> >> This patch is not in master and is not a backport therefore not
> >> eligible for kirkstone/langdale.
>
> My bad, the intention was to have it in all three branches: master,
> kirkstone, and langdale.
>
> > 'nor should it go to master.
> >
> > If someone wants to allow go to fetch sources during builds, it
> needs
> > to be done in their own layers.
> >
> > Bruce
>
> Probably I misunderstood the outcome of the discussion in my first
> link.
> Bruce and Richard, you both suggested to have it in go-mod with the
> warning. I agree that it breaks reproducibility, and is considered
> a bad
> practice in general (I'm aware of a few more build systems with
> the same
> shortcoming).
>
> My point is, until a proper solution is in place, it should be at
> least
> documented somewhere that this should be done in own layer/recipe.
> Any
> suggestions where this can be documented if not in OE-Core?
>
>
> Have you tried the changes submitted by Lukas/Stefan in:
> https://lists.openembedded.org/g/openembedded-architecture/message/1539
> ?
>
> It's not perfect, I was testing it on
> https://github.com/influxdata/telegraf/blob/master/go.mod and I've
> found some corner cases where it failed.
>
> But it seems like very good start and we should work with Lukas/Stefan
> to get it merged in master. Then all branches could consume recipes
> created in master and only the exceptions would need to have network
> access (instead of the bbclass enabling it for every go-mod user -
> go-vendor inherits go-mod as well).
>
> Lukas/Stefan: is there something we can do to help with v2 of your
> patches?
>
> I wanted to submit better review on submitted patches after more
> debugging, but here it is:
>
> 1) to resolve 2 deps:
> {'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version':
> 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP
> Error 404: Not Found"
>
> {'Path': 'code.cloudfoundry.org/clock
> <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0',
> 'Indirect': True} due to this being redirect to github.com
> <http://github.com> and failing with "ERROR: Error while fetching
> redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
> certificate verify failed: Hostname mismatch, certificate is not valid
> for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. (_ssl.c:996)>"
> parse
>
> 2) some issues with LICENSEs:
> INFO: Please add the following line for
> 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv'
> and replace `Unknown` with the license:
> c4807c5b058a56c02f592207b4e056b1,Unknown
> Traceback (most recent call last):
> File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
> 111, in <module>
> ret = main()
> ^^^^^^
> File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
> 100, in main
> ret = args.func(args)
> ^^^^^^^^^^^^^^^
> File
> "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line
> 746, in create_recipe
> handler.process(srctree_use, classes, lines_before, lines_after,
> handled, extravalues)
> File
> "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py",
> line 349, in process
> self._rewrite_lic_uri(lines_before)
> File
> "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py",
> line 380, in _rewrite_lic_uri
> updated, newlines = bb.utils.edit_metadata(lines_before,
> ['LIC_FILES_CHKSUM'], varfunc)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in
> edit_metadata
> if handle_var_end():
> ^^^^^^^^^^^^^^^^
> File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
> handle_var_end
> (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value,
> op, newlines)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py",
> line 372, in varfunc
> uri, chksum = license.split(';', 1)
> ^^^^^^^^^^^
> ValueError: not enough values to unpack (expected 2, got 1)
>
> (the license was just '/')
>
> 3) some SRCREVs set to None leading to parsing failure:
>
> File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
> _latest_revision
> raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git
> repository in git ls-remote output for %s" % \
> bb.data_smart.ExpansionError: Failure expanding variable SRCPV,
> expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
> FetchError: Fetcher failure: Unable to resolve 'None' in upstream git
> repository in git ls-remote output for
> github.com/googleapis/google-cloud-go
> <http://github.com/googleapis/google-cloud-go>
> The variable dependency chain for the failure is: SRCPV -> PV -> BP ->
> FILESPATH
>
> ERROR: Parsing newly created recipe failed, moving recipe to
> /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed
> for reference. If this looks to be caused by the recipe itself, please
> report this error.
>
> It takes long time to fetch all telegraf dependencies, so I'll create
> some test recipe to more easily debug these 3 issues in isolation.
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#174971): https://lists.openembedded.org/g/openembedded-core/message/174971
> Mute This Topic: https://lists.openembedded.org/mt/95827446/7344993
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [lukas.funke-oss@weidmueller.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
I also tested using telegraf v1.21.4. Kirkstone uses go1.17, so I tested this version. Unfortunately, it failed with the following log. Did I have something mistakes? ~$ recipetool create -S 583ee20a093acaa0781508b7840deb97a0fda622 -B release-1.21 https://github.com/influxdata/telegraf.git Initialising tasks: 100% |################################################################################################| Time: 0:00:00 Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% match, 0% complete) NOTE: No setscene tasks NOTE: Executing Tasks NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded. NOTE: Writing buildhistory NOTE: Writing buildhistory took: 4 seconds INFO: Resolved download path for import 'cloud.google.com/go/bigquery' => https://github.com/googleapis/google-cloud-go INFO: Resolved download path for import 'cloud.google.com/go/monitoring' => https://github.com/googleapis/google-cloud-go INFO: Resolved download path for import 'cloud.google.com/go/pubsub' => https://github.com/googleapis/google-cloud-go INFO: Resolved download path for import 'collectd.org' => https://github.com/collectd/go-collectd INFO: Resolved download path for import 'github.com/Azure/azure-event-hubs-go/v3' => https://github.com/Azure/azure-event-hubs-go INFO: Resolved download path for import 'github.com/Azure/azure-kusto-go' => https://github.com/Azure/azure-kusto-go INFO: Resolved download path for import 'github.com/Azure/azure-storage-queue-go' => https://github.com/Azure/azure-storage-queue-go INFO: Resolved download path for import 'github.com/Azure/go-autorest/autorest' => https://github.com/Azure/go-autorest INFO: Resolved download path for import 'github.com/Azure/go-autorest/autorest/adal' => https://github.com/Azure/go-autorest INFO: Resolved download path for import 'github.com/Azure/go-autorest/autorest/azure/auth' => https://github.com/Azure/go-autorest INFO: Resolved download path for import 'github.com/BurntSushi/toml' => https://github.com/BurntSushi/toml INFO: Resolved download path for import 'github.com/Mellanox/rdmamap' => https://github.com/Mellanox/rdmamap INFO: Resolved download path for import 'github.com/Shopify/sarama' => https://github.com/Shopify/sarama INFO: Resolved download path for import 'github.com/aerospike/aerospike-client-go' => https://github.com/aerospike/aerospike-client-go INFO: Resolved download path for import 'github.com/alecthomas/units' => https://github.com/alecthomas/units INFO: Resolved download path for import 'github.com/aliyun/alibaba-cloud-sdk-go' => https://github.com/aliyun/alibaba-cloud-sdk-go INFO: Resolved download path for import 'github.com/amir/raidman' => https://github.com/amir/raidman INFO: Resolved download path for import 'github.com/antchfx/jsonquery' => https://github.com/antchfx/jsonquery INFO: Resolved download path for import 'github.com/antchfx/xmlquery' => https://github.com/antchfx/xmlquery INFO: Resolved download path for import 'github.com/antchfx/xpath' => https://github.com/antchfx/xpath INFO: Resolved download path for import 'github.com/apache/thrift' => https://github.com/apache/thrift INFO: Resolved download path for import 'github.com/aristanetworks/goarista' => https://github.com/aristanetworks/goarista INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/config' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/credentials' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/feature/ec2/imds' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/cloudwatch' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/dynamodb' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/ec2' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/kinesis' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/sts' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/timestreamwrite' => https://github.com/aws/aws-sdk-go-v2 INFO: Resolved download path for import 'github.com/aws/smithy-go' => https://github.com/aws/smithy-go INFO: Resolved download path for import 'github.com/benbjohnson/clock' => https://github.com/benbjohnson/clock INFO: Resolved download path for import 'github.com/bmatcuk/doublestar/v3' => https://github.com/bmatcuk/doublestar INFO: Resolved download path for import 'github.com/caio/go-tdigest' => https://github.com/caio/go-tdigest INFO: Resolved download path for import 'github.com/cisco-ie/nx-telemetry-proto' => https://github.com/cisco-ie/nx-telemetry-proto INFO: Resolved download path for import 'github.com/coreos/go-semver' => https://github.com/coreos/go-semver INFO: Resolved download path for import 'github.com/couchbase/go-couchbase' => https://github.com/couchbase/go-couchbase INFO: Resolved download path for import 'github.com/denisenkom/go-mssqldb' => https://github.com/denisenkom/go-mssqldb INFO: Resolved download path for import 'github.com/dimchansky/utfbom' => https://github.com/dimchansky/utfbom INFO: Resolved download path for import 'github.com/djherbis/times' => https://github.com/djherbis/times INFO: Resolved download path for import 'github.com/docker/docker' => https://github.com/docker/docker INFO: Resolved download path for import 'github.com/doclambda/protobufquery' => https://github.com/doclambda/protobufquery INFO: Resolved download path for import 'github.com/dynatrace-oss/dynatrace-metric-utils-go' => https://github.com/dynatrace-oss/dynatrace-metric-utils-go INFO: Resolved download path for import 'github.com/eclipse/paho.mqtt.golang' => https://github.com/eclipse/paho.mqtt.golang INFO: Resolved download path for import 'github.com/fatih/color' => https://github.com/fatih/color INFO: Resolved download path for import 'github.com/ghodss/yaml' => https://github.com/ghodss/yaml INFO: Resolved download path for import 'github.com/go-ldap/ldap/v3' => https://github.com/go-ldap/ldap INFO: Resolved download path for import 'github.com/go-logfmt/logfmt' => https://github.com/go-logfmt/logfmt INFO: Resolved download path for import 'github.com/go-ping/ping' => https://github.com/go-ping/ping INFO: Resolved download path for import 'github.com/go-redis/redis' => https://github.com/go-redis/redis INFO: Resolved download path for import 'github.com/go-sql-driver/mysql' => https://github.com/go-sql-driver/mysql INFO: Resolved download path for import 'github.com/gobwas/glob' => https://github.com/gobwas/glob INFO: Resolved download path for import 'github.com/gofrs/uuid' => https://github.com/gofrs/uuid INFO: Resolved download path for import 'github.com/golang-jwt/jwt/v4' => https://github.com/golang-jwt/jwt INFO: Resolved download path for import 'github.com/golang/geo' => https://github.com/golang/geo INFO: Resolved download path for import 'github.com/golang/snappy' => https://github.com/golang/snappy INFO: Resolved download path for import 'github.com/google/go-cmp' => https://github.com/google/go-cmp INFO: Resolved download path for import 'github.com/google/go-github/v32' => https://github.com/google/go-github INFO: Resolved download path for import 'github.com/gopcua/opcua' => https://github.com/gopcua/opcua INFO: Resolved download path for import 'github.com/gophercloud/gophercloud' => https://github.com/gophercloud/gophercloud INFO: Resolved download path for import 'github.com/gorilla/mux' => https://github.com/gorilla/mux INFO: Resolved download path for import 'github.com/gorilla/websocket' => https://github.com/gorilla/websocket INFO: Resolved download path for import 'github.com/gosnmp/gosnmp' => https://github.com/gosnmp/gosnmp INFO: Resolved download path for import 'github.com/grid-x/modbus' => https://github.com/grid-x/modbus INFO: Resolved download path for import 'github.com/gwos/tcg/sdk' => https://github.com/gwos/tcg INFO: Resolved download path for import 'github.com/harlow/kinesis-consumer' => https://github.com/harlow/kinesis-consumer INFO: Resolved download path for import 'github.com/hashicorp/consul/api' => https://github.com/hashicorp/consul INFO: Resolved download path for import 'github.com/hashicorp/go-uuid' => https://github.com/hashicorp/go-uuid INFO: Resolved download path for import 'github.com/influxdata/go-syslog/v3' => https://github.com/influxdata/go-syslog INFO: Resolved download path for import 'github.com/influxdata/influxdb-observability/common' => https://github.com/influxdata/influxdb-observability INFO: Resolved download path for import 'github.com/influxdata/influxdb-observability/influx2otel' => https://github.com/influxdata/influxdb-observability INFO: Resolved download path for import 'github.com/influxdata/influxdb-observability/otel2influx' => https://github.com/influxdata/influxdb-observability INFO: Resolved download path for import 'github.com/influxdata/tail' => https://github.com/influxdata/tail INFO: Resolved download path for import 'github.com/influxdata/toml' => https://github.com/influxdata/toml INFO: Resolved download path for import 'github.com/influxdata/wlog' => https://github.com/influxdata/wlog INFO: Resolved download path for import 'github.com/intel/iaevents' => https://github.com/intel/iaevents INFO: Resolved download path for import 'github.com/jackc/pgx/v4' => https://github.com/jackc/pgx INFO: Resolved download path for import 'github.com/james4k/rcon' => https://github.com/james4k/rcon INFO: Resolved download path for import 'github.com/jhump/protoreflect' => https://github.com/jhump/protoreflect INFO: Resolved download path for import 'github.com/jmespath/go-jmespath' => https://github.com/jmespath/go-jmespath INFO: Resolved download path for import 'github.com/kardianos/service' => https://github.com/kardianos/service INFO: Resolved download path for import 'github.com/karrick/godirwalk' => https://github.com/karrick/godirwalk INFO: Resolved download path for import 'github.com/kballard/go-shellquote' => https://github.com/kballard/go-shellquote INFO: Resolved download path for import 'github.com/matttproud/golang_protobuf_extensions' => https://github.com/matttproud/golang_protobuf_extensions INFO: Resolved download path for import 'github.com/mdlayher/apcupsd' => https://github.com/mdlayher/apcupsd INFO: Resolved download path for import 'github.com/microsoft/ApplicationInsights-Go' => https://github.com/microsoft/ApplicationInsights-Go INFO: Resolved download path for import 'github.com/miekg/dns' => https://github.com/miekg/dns INFO: Resolved download path for import 'github.com/moby/ipvs' => https://github.com/moby/ipvs INFO: Resolved download path for import 'github.com/multiplay/go-ts3' => https://github.com/multiplay/go-ts3 INFO: Resolved download path for import 'github.com/nats-io/nats-server/v2' => https://github.com/nats-io/nats-server INFO: Resolved download path for import 'github.com/nats-io/nats.go' => https://github.com/nats-io/nats.go INFO: Resolved download path for import 'github.com/newrelic/newrelic-telemetry-sdk-go' => https://github.com/newrelic/newrelic-telemetry-sdk-go INFO: Resolved download path for import 'github.com/nsqio/go-nsq' => https://github.com/nsqio/go-nsq INFO: Resolved download path for import 'github.com/olivere/elastic' => https://github.com/olivere/elastic INFO: Resolved download path for import 'github.com/openconfig/gnmi' => https://github.com/openconfig/gnmi INFO: Resolved download path for import 'github.com/opentracing/opentracing-go' => https://github.com/opentracing/opentracing-go INFO: Resolved download path for import 'github.com/openzipkin-contrib/zipkin-go-opentracing' => https://github.com/openzipkin-contrib/zipkin-go-opentracing INFO: Resolved download path for import 'github.com/openzipkin/zipkin-go' => https://github.com/openzipkin/zipkin-go INFO: Resolved download path for import 'github.com/pion/dtls/v2' => https://github.com/pion/dtls INFO: Resolved download path for import 'github.com/pkg/errors' => https://github.com/pkg/errors INFO: Resolved download path for import 'github.com/prometheus/client_golang' => https://github.com/prometheus/client_golang INFO: Resolved download path for import 'github.com/prometheus/client_model' => https://github.com/prometheus/client_model INFO: Resolved download path for import 'github.com/prometheus/common' => https://github.com/prometheus/common INFO: Resolved download path for import 'github.com/prometheus/procfs' => https://github.com/prometheus/procfs INFO: Resolved download path for import 'github.com/prometheus/prometheus' => https://github.com/prometheus/prometheus INFO: Resolved download path for import 'github.com/riemann/riemann-go-client' => https://github.com/riemann/riemann-go-client INFO: Resolved download path for import 'github.com/safchain/ethtool' => https://github.com/safchain/ethtool INFO: Resolved download path for import 'github.com/sensu/sensu-go/api/core/v2' => https://github.com/sensu/sensu-go INFO: Resolved download path for import 'github.com/shirou/gopsutil/v3' => https://github.com/shirou/gopsutil INFO: Resolved download path for import 'github.com/showwin/speedtest-go' => https://github.com/showwin/speedtest-go INFO: Resolved download path for import 'github.com/signalfx/golib/v3' => https://github.com/signalfx/golib INFO: Resolved download path for import 'github.com/sirupsen/logrus' => https://github.com/sirupsen/logrus INFO: Resolved download path for import 'github.com/sleepinggenius2/gosmi' => https://github.com/sleepinggenius2/gosmi INFO: Resolved download path for import 'github.com/snowflakedb/gosnowflake' => https://github.com/snowflakedb/gosnowflake INFO: Resolved download path for import 'github.com/streadway/amqp' => https://github.com/streadway/amqp INFO: Resolved download path for import 'github.com/stretchr/testify' => https://github.com/stretchr/testify INFO: Resolved download path for import 'github.com/tbrandon/mbserver' => https://github.com/tbrandon/mbserver INFO: Resolved download path for import 'github.com/testcontainers/testcontainers-go' => https://github.com/testcontainers/testcontainers-go INFO: Resolved download path for import 'github.com/tidwall/gjson' => https://github.com/tidwall/gjson INFO: Resolved download path for import 'github.com/tinylib/msgp' => https://github.com/tinylib/msgp INFO: Resolved download path for import 'github.com/vapourismo/knx-go' => https://github.com/vapourismo/knx-go INFO: Resolved download path for import 'github.com/vjeantet/grok' => https://github.com/vjeantet/grok INFO: Resolved download path for import 'github.com/vmware/govmomi' => https://github.com/vmware/govmomi INFO: Resolved download path for import 'github.com/wavefronthq/wavefront-sdk-go' => https://github.com/wavefronthq/wavefront-sdk-go INFO: Resolved download path for import 'github.com/wvanbergen/kafka' => https://github.com/wvanbergen/kafka INFO: Resolved download path for import 'github.com/xdg/scram' => https://github.com/xdg/scram INFO: Resolved download path for import 'go.mongodb.org/mongo-driver' => https://github.com/mongodb/mongo-go-driver.git INFO: Resolved download path for import 'go.opentelemetry.io/collector/model' => https://github.com/open-telemetry/opentelemetry-collector INFO: Resolved download path for import 'go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc' => https://github.com/open-telemetry/opentelemetry-go INFO: Resolved download path for import 'go.opentelemetry.io/otel/metric' => https://github.com/open-telemetry/opentelemetry-go INFO: Resolved download path for import 'go.opentelemetry.io/otel/sdk/metric' => https://github.com/open-telemetry/opentelemetry-go INFO: Resolved download path for import 'go.starlark.net' => https://github.com/google/starlark-go INFO: Resolved download path for import 'golang.org/x/net' => https://go.googlesource.com/net INFO: Resolved download path for import 'golang.org/x/oauth2' => https://go.googlesource.com/oauth2 INFO: Resolved download path for import 'golang.org/x/sync' => https://go.googlesource.com/sync INFO: Resolved download path for import 'golang.org/x/sys' => https://go.googlesource.com/sys INFO: Resolved download path for import 'golang.org/x/text' => https://go.googlesource.com/text INFO: Resolved download path for import 'golang.zx2c4.com/wireguard/wgctrl' => https://github.com/WireGuard/wgctrl-go INFO: Resolved download path for import 'google.golang.org/api' => https://github.com/googleapis/google-api-go-client INFO: Resolved download path for import 'google.golang.org/genproto' => https://github.com/googleapis/go-genproto INFO: Resolved download path for import 'google.golang.org/grpc' => https://github.com/grpc/grpc-go INFO: Resolved download path for import 'google.golang.org/protobuf' => https://go.googlesource.com/protobuf INFO: Resolved download path for import 'gopkg.in/gorethink/gorethink.v3' => https://gopkg.in/gorethink/gorethink.v3 INFO: Resolved download path for import 'gopkg.in/olivere/elastic.v5' => https://gopkg.in/olivere/elastic.v5 INFO: Resolved download path for import 'gopkg.in/tomb.v1' => https://gopkg.in/tomb.v1 INFO: Resolved download path for import 'gopkg.in/yaml.v2' => https://gopkg.in/yaml.v2 INFO: Resolved download path for import 'k8s.io/api' => https://github.com/kubernetes/api INFO: Resolved download path for import 'k8s.io/apimachinery' => https://github.com/kubernetes/apimachinery INFO: Resolved download path for import 'k8s.io/client-go' => https://github.com/kubernetes/client-go INFO: Resolved download path for import 'modernc.org/sqlite' => https://gitlab.com/cznic/sqlite INFO: Resolved download path for import 'cloud.google.com/go' => https://github.com/googleapis/google-cloud-go INFO: Resolved download path for import 'cloud.google.com/go/compute' => https://github.com/googleapis/google-cloud-go INFO: Resolved download path for import 'cloud.google.com/go/iam' => https://github.com/googleapis/google-cloud-go Traceback (most recent call last): File "/oe-core/scripts/recipetool", line 111, in <module> ret = main() File "/oe-core/scripts/recipetool", line 100, in main ret = args.func(args) File "/oe-core/scripts/lib/recipetool/create.py", line 746, in create_recipe handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues) File "/oe-core/scripts/lib/recipetool/create_go.py", line 336, in process dep_src_uris, src_revs = self._handle_dependencies(d, srctree, go_mod) File "/oe-core/scripts/lib/recipetool/create_go.py", line 261, in _handle_dependencies repodata = self._resolve_repository(module_path) File "/oe-core/scripts/lib/recipetool/create_go.py", line 184, in _resolve_repository repodata = self._resolve_repository_dynamic(modulepath) File "/oe-core/scripts/lib/recipetool/create_go.py", line 176, in _resolve_repository_dynamic return GoImport(parser.rootpath, parser.vcs, parser.repourl, None) File "/oe-core/scripts/lib/recipetool/create_go.py", line 151, in rootpath return self.__srv[0] IndexError: list index out of range
On Tue, Jan 3, 2023 at 5:03 PM Lukas Funke <lukas.funke-oss@weidmueller.com> wrote: > Martin, > > first of all: thank you for testing the patches. As usual the corner > cases are the most tricky ones. > > tl;dr: I'm working on an updated patch series to address the reported > issues. > Hi Lukas, and update on this series? It doesn't have to be perfect, other people can help with remaining corner cases and right now there is nothing for this in oe-core, so any version will be big improvement :) and it will make it easier for others to submit incremental improvements - I have some as well, but haven't sent them as I don't know what you've already updated locally, so I'm waiting for v2. Mickledore is already in feature freeze, but maybe RP will make an exception as this is new important functionality which isn't likely to break other existing code. Kind Regards, I've looked into the issues and would like to give some explanation. > > 1) the first module you mention ('gonum.org/v1/gonum > <http://gonum.org/v1/gonum>') does not obey the go documentation for > resolving the modules source-code repository (see > https://go.dev/ref/mod#go-mod-download , Section 'Version control > systems'). The documentation states that there should be a html page, > queried by "?go-get=1", with a <meta>-tag which contains the original > URL to the source-code repository. For 'gonum' there is only a 404-page. > This page contains javascript-code that redirects to the actual gonum > package page. Unnecessary hard to process. I'll open an issue for that. > The second module 'code.cloudfoundry.org/clock > <http://code.cloudfoundry.org/clock>' has a broken certificate. I > already opend an issue on github for this. > > I addressed this issue above by maintaining a list of modules and their > actual repo URLs inside the go-recipetool as an absolute fallback. Would > do you think about this solution? > Yes, that's what I was starting to do in: https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8 https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00 > 2) yes, my bad :) > as work around I was using https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547 3) The problem is that some repositories have a prefix in their tags. > The tags usually only contain the semantic version string. I'm curretly > working on this issue. > > Best regards > > Lukas > > Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org: > > On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <uvv.mail@gmail.com> > > wrote: > > > > On 22.12.2022 17:48, Bruce Ashfield wrote: > > > On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie > > > <richard.purdie@linuxfoundation.org> wrote: > > >> This patch is not in master and is not a backport therefore not > > >> eligible for kirkstone/langdale. > > > > My bad, the intention was to have it in all three branches: master, > > kirkstone, and langdale. > > > > > 'nor should it go to master. > > > > > > If someone wants to allow go to fetch sources during builds, it > > needs > > > to be done in their own layers. > > > > > > Bruce > > > > Probably I misunderstood the outcome of the discussion in my first > > link. > > Bruce and Richard, you both suggested to have it in go-mod with the > > warning. I agree that it breaks reproducibility, and is considered > > a bad > > practice in general (I'm aware of a few more build systems with > > the same > > shortcoming). > > > > My point is, until a proper solution is in place, it should be at > > least > > documented somewhere that this should be done in own layer/recipe. > > Any > > suggestions where this can be documented if not in OE-Core? > > > > > > Have you tried the changes submitted by Lukas/Stefan in: > > https://lists.openembedded.org/g/openembedded-architecture/message/1539 > > ? > > > > It's not perfect, I was testing it on > > https://github.com/influxdata/telegraf/blob/master/go.mod and I've > > found some corner cases where it failed. > > > > But it seems like very good start and we should work with Lukas/Stefan > > to get it merged in master. Then all branches could consume recipes > > created in master and only the exceptions would need to have network > > access (instead of the bbclass enabling it for every go-mod user - > > go-vendor inherits go-mod as well). > > > > Lukas/Stefan: is there something we can do to help with v2 of your > > patches? > > > > I wanted to submit better review on submitted patches after more > > debugging, but here it is: > > > > 1) to resolve 2 deps: > > {'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version': > > 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP > > Error 404: Not Found" > > > > {'Path': 'code.cloudfoundry.org/clock > > <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', > > 'Indirect': True} due to this being redirect to github.com > > <http://github.com> and failing with "ERROR: Error while fetching > > redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] > > certificate verify failed: Hostname mismatch, certificate is not valid > > for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. > (_ssl.c:996)>" > > parse > > > > 2) some issues with LICENSEs: > > INFO: Please add the following line for > > 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' > > and replace `Unknown` with the license: > > c4807c5b058a56c02f592207b4e056b1,Unknown > > Traceback (most recent call last): > > File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line > > 111, in <module> > > ret = main() > > ^^^^^^ > > File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line > > 100, in main > > ret = args.func(args) > > ^^^^^^^^^^^^^^^ > > File > > "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line > > 746, in create_recipe > > handler.process(srctree_use, classes, lines_before, lines_after, > > handled, extravalues) > > File > > "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", > > line 349, in process > > self._rewrite_lic_uri(lines_before) > > File > > "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", > > line 380, in _rewrite_lic_uri > > updated, newlines = bb.utils.edit_metadata(lines_before, > > ['LIC_FILES_CHKSUM'], varfunc) > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in > > edit_metadata > > if handle_var_end(): > > ^^^^^^^^^^^^^^^^ > > File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in > > handle_var_end > > (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, > > op, newlines) > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > File > > "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", > > line 372, in varfunc > > uri, chksum = license.split(';', 1) > > ^^^^^^^^^^^ > > ValueError: not enough values to unpack (expected 2, got 1) > > > > (the license was just '/') > > > > 3) some SRCREVs set to None leading to parsing failure: > > > > File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in > > _latest_revision > > raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git > > repository in git ls-remote output for %s" % \ > > bb.data_smart.ExpansionError: Failure expanding variable SRCPV, > > expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception > > FetchError: Fetcher failure: Unable to resolve 'None' in upstream git > > repository in git ls-remote output for > > github.com/googleapis/google-cloud-go > > <http://github.com/googleapis/google-cloud-go> > > The variable dependency chain for the failure is: SRCPV -> PV -> BP -> > > FILESPATH > > > > ERROR: Parsing newly created recipe failed, moving recipe to > > /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed > > for reference. If this looks to be caused by the recipe itself, please > > report this error. > > > > It takes long time to fetch all telegraf dependencies, so I'll create > > some test recipe to more easily debug these 3 issues in isolation. > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#174971): > https://lists.openembedded.org/g/openembedded-core/message/174971 > > Mute This Topic: https://lists.openembedded.org/mt/95827446/7344993 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > lukas.funke-oss@weidmueller.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > >
On Sun, Mar 12, 2023 at 10:15 AM Martin Jansa via lists.openembedded.org <Martin.Jansa=gmail.com@lists.openembedded.org> wrote: > On Tue, Jan 3, 2023 at 5:03 PM Lukas Funke < > lukas.funke-oss@weidmueller.com> wrote: > >> Martin, >> >> first of all: thank you for testing the patches. As usual the corner >> cases are the most tricky ones. >> >> tl;dr: I'm working on an updated patch series to address the reported >> issues. >> > > Hi Lukas, Stefan, > > and update on this series? > > It doesn't have to be perfect, other people can help with remaining corner > cases and right now there is nothing for this in oe-core, so any version > will be big improvement :) and it will make it easier for others to submit > incremental improvements - I have some as well, but haven't sent them as I > don't know what you've already updated locally, so I'm waiting for v2. > > Mickledore is already in feature freeze, but maybe RP will make an > exception as this is new important functionality which isn't likely to > break other existing code. > Hello Lukas, any progress on updated patch series? Nanbield release is near and I don't remember seeing the updated patch series in last couple months. Is there anything I can do to help? The last version looked reasonably well and IMHO could be used as starting point for everybody to help fixing those corner cases where it didn't work well. Regards, > Kind Regards, > > > I've looked into the issues and would like to give some explanation. >> >> 1) the first module you mention ('gonum.org/v1/gonum >> <http://gonum.org/v1/gonum>') does not obey the go documentation for >> resolving the modules source-code repository (see >> https://go.dev/ref/mod#go-mod-download , Section 'Version control >> systems'). The documentation states that there should be a html page, >> queried by "?go-get=1", with a <meta>-tag which contains the original >> URL to the source-code repository. For 'gonum' there is only a 404-page. >> This page contains javascript-code that redirects to the actual gonum >> package page. Unnecessary hard to process. I'll open an issue for that. >> The second module 'code.cloudfoundry.org/clock >> <http://code.cloudfoundry.org/clock>' has a broken certificate. I >> already opend an issue on github for this. >> >> I addressed this issue above by maintaining a list of modules and their >> actual repo URLs inside the go-recipetool as an absolute fallback. Would >> do you think about this solution? >> > > Yes, that's what I was starting to do in: > > https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8 > > https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00 > > >> 2) yes, my bad :) >> > > as work around I was using > > https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547 > > > 3) The problem is that some repositories have a prefix in their tags. >> The tags usually only contain the semantic version string. I'm curretly >> working on this issue. >> >> Best regards >> >> Lukas >> >> Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org: >> > On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <uvv.mail@gmail.com> >> > wrote: >> > >> > On 22.12.2022 17:48, Bruce Ashfield wrote: >> > > On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie >> > > <richard.purdie@linuxfoundation.org> wrote: >> > >> This patch is not in master and is not a backport therefore not >> > >> eligible for kirkstone/langdale. >> > >> > My bad, the intention was to have it in all three branches: master, >> > kirkstone, and langdale. >> > >> > > 'nor should it go to master. >> > > >> > > If someone wants to allow go to fetch sources during builds, it >> > needs >> > > to be done in their own layers. >> > > >> > > Bruce >> > >> > Probably I misunderstood the outcome of the discussion in my first >> > link. >> > Bruce and Richard, you both suggested to have it in go-mod with the >> > warning. I agree that it breaks reproducibility, and is considered >> > a bad >> > practice in general (I'm aware of a few more build systems with >> > the same >> > shortcoming). >> > >> > My point is, until a proper solution is in place, it should be at >> > least >> > documented somewhere that this should be done in own layer/recipe. >> > Any >> > suggestions where this can be documented if not in OE-Core? >> > >> > >> > Have you tried the changes submitted by Lukas/Stefan in: >> > https://lists.openembedded.org/g/openembedded-architecture/message/1539 >> > ? >> > >> > It's not perfect, I was testing it on >> > https://github.com/influxdata/telegraf/blob/master/go.mod and I've >> > found some corner cases where it failed. >> > >> > But it seems like very good start and we should work with Lukas/Stefan >> > to get it merged in master. Then all branches could consume recipes >> > created in master and only the exceptions would need to have network >> > access (instead of the bbclass enabling it for every go-mod user - >> > go-vendor inherits go-mod as well). >> > >> > Lukas/Stefan: is there something we can do to help with v2 of your >> > patches? >> > >> > I wanted to submit better review on submitted patches after more >> > debugging, but here it is: >> > >> > 1) to resolve 2 deps: >> > {'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version': >> > 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP >> > Error 404: Not Found" >> > >> > {'Path': 'code.cloudfoundry.org/clock >> > <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', >> > 'Indirect': True} due to this being redirect to github.com >> > <http://github.com> and failing with "ERROR: Error while fetching >> > redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] >> > certificate verify failed: Hostname mismatch, certificate is not valid >> > for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. >> (_ssl.c:996)>" >> > parse >> > >> > 2) some issues with LICENSEs: >> > INFO: Please add the following line for >> > 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' >> > and replace `Unknown` with the license: >> > c4807c5b058a56c02f592207b4e056b1,Unknown >> > Traceback (most recent call last): >> > File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line >> > 111, in <module> >> > ret = main() >> > ^^^^^^ >> > File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line >> > 100, in main >> > ret = args.func(args) >> > ^^^^^^^^^^^^^^^ >> > File >> > "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line >> > 746, in create_recipe >> > handler.process(srctree_use, classes, lines_before, lines_after, >> > handled, extravalues) >> > File >> > "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", >> > line 349, in process >> > self._rewrite_lic_uri(lines_before) >> > File >> > "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", >> > line 380, in _rewrite_lic_uri >> > updated, newlines = bb.utils.edit_metadata(lines_before, >> > ['LIC_FILES_CHKSUM'], varfunc) >> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> > File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in >> > edit_metadata >> > if handle_var_end(): >> > ^^^^^^^^^^^^^^^^ >> > File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in >> > handle_var_end >> > (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, >> > op, newlines) >> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> > File >> > "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", >> > line 372, in varfunc >> > uri, chksum = license.split(';', 1) >> > ^^^^^^^^^^^ >> > ValueError: not enough values to unpack (expected 2, got 1) >> > >> > (the license was just '/') >> > >> > 3) some SRCREVs set to None leading to parsing failure: >> > >> > File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in >> > _latest_revision >> > raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git >> > repository in git ls-remote output for %s" % \ >> > bb.data_smart.ExpansionError: Failure expanding variable SRCPV, >> > expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception >> > FetchError: Fetcher failure: Unable to resolve 'None' in upstream git >> > repository in git ls-remote output for >> > github.com/googleapis/google-cloud-go >> > <http://github.com/googleapis/google-cloud-go> >> > The variable dependency chain for the failure is: SRCPV -> PV -> BP -> >> > FILESPATH >> > >> > ERROR: Parsing newly created recipe failed, moving recipe to >> > >> /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed >> > for reference. If this looks to be caused by the recipe itself, please >> > report this error. >> > >> > It takes long time to fetch all telegraf dependencies, so I'll create >> > some test recipe to more easily debug these 3 issues in isolation. >> > >> > >> > >> > >> > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#178380): > https://lists.openembedded.org/g/openembedded-core/message/178380 > Mute This Topic: https://lists.openembedded.org/mt/95827446/3617156 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > Martin.Jansa@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On 10.09.2023 23:26, Martin Jansa wrote: > On Sun, Mar 12, 2023 at 10:15 AM Martin Jansa via lists.openembedded.org > <Martin.Jansa=gmail.com@lists.openembedded.org> wrote: > >> On Tue, Jan 3, 2023 at 5:03 PM Lukas Funke < >> lukas.funke-oss@weidmueller.com> wrote: >> >>> Martin, >>> >>> first of all: thank you for testing the patches. As usual the corner >>> cases are the most tricky ones. >>> >>> tl;dr: I'm working on an updated patch series to address the reported >>> issues. >>> >> >> Hi Lukas, Stefan, >> >> and update on this series? >> >> It doesn't have to be perfect, other people can help with remaining corner >> cases and right now there is nothing for this in oe-core, so any version >> will be big improvement :) and it will make it easier for others to submit >> incremental improvements - I have some as well, but haven't sent them as I >> don't know what you've already updated locally, so I'm waiting for v2. >> >> Mickledore is already in feature freeze, but maybe RP will make an >> exception as this is new important functionality which isn't likely to >> break other existing code. >> > > Hello Lukas, > > any progress on updated patch series? Nanbield release is near and I don't > remember seeing the updated patch series in last couple months. > > Is there anything I can do to help? The last version looked reasonably well > and IMHO could be used as starting point for everybody to help fixing those > corner cases where it didn't work well. > > Regards, Hello Martin, I'm currently working on an update, but can't make any promises regarding the timeline. Thanks for offering help! Reviewing and testing is always helpful when the updated series is on it's way. Regards Lukas > > >> Kind Regards, >> >> >> I've looked into the issues and would like to give some explanation. >>> >>> 1) the first module you mention ('gonum.org/v1/gonum >>> <http://gonum.org/v1/gonum>') does not obey the go documentation for >>> resolving the modules source-code repository (see >>> https://go.dev/ref/mod#go-mod-download , Section 'Version control >>> systems'). The documentation states that there should be a html page, >>> queried by "?go-get=1", with a <meta>-tag which contains the original >>> URL to the source-code repository. For 'gonum' there is only a 404-page. >>> This page contains javascript-code that redirects to the actual gonum >>> package page. Unnecessary hard to process. I'll open an issue for that. >>> The second module 'code.cloudfoundry.org/clock >>> <http://code.cloudfoundry.org/clock>' has a broken certificate. I >>> already opend an issue on github for this. >>> >>> I addressed this issue above by maintaining a list of modules and their >>> actual repo URLs inside the go-recipetool as an absolute fallback. Would >>> do you think about this solution? >>> >> >> Yes, that's what I was starting to do in: >> >> https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8 >> >> https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00 >> >> >>> 2) yes, my bad :) >>> >> >> as work around I was using >> >> https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547 >> >> >> 3) The problem is that some repositories have a prefix in their tags. >>> The tags usually only contain the semantic version string. I'm curretly >>> working on this issue. >>> >>> Best regards >>> >>> Lukas >>> >>> Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org: >>>> On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <uvv.mail@gmail.com> >>>> wrote: >>>> >>>> On 22.12.2022 17:48, Bruce Ashfield wrote: >>>> > On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie >>>> > <richard.purdie@linuxfoundation.org> wrote: >>>> >> This patch is not in master and is not a backport therefore not >>>> >> eligible for kirkstone/langdale. >>>> >>>> My bad, the intention was to have it in all three branches: master, >>>> kirkstone, and langdale. >>>> >>>> > 'nor should it go to master. >>>> > >>>> > If someone wants to allow go to fetch sources during builds, it >>>> needs >>>> > to be done in their own layers. >>>> > >>>> > Bruce >>>> >>>> Probably I misunderstood the outcome of the discussion in my first >>>> link. >>>> Bruce and Richard, you both suggested to have it in go-mod with the >>>> warning. I agree that it breaks reproducibility, and is considered >>>> a bad >>>> practice in general (I'm aware of a few more build systems with >>>> the same >>>> shortcoming). >>>> >>>> My point is, until a proper solution is in place, it should be at >>>> least >>>> documented somewhere that this should be done in own layer/recipe. >>>> Any >>>> suggestions where this can be documented if not in OE-Core? >>>> >>>> >>>> Have you tried the changes submitted by Lukas/Stefan in: >>>> https://lists.openembedded.org/g/openembedded-architecture/message/1539 >>>> ? >>>> >>>> It's not perfect, I was testing it on >>>> https://github.com/influxdata/telegraf/blob/master/go.mod and I've >>>> found some corner cases where it failed. >>>> >>>> But it seems like very good start and we should work with Lukas/Stefan >>>> to get it merged in master. Then all branches could consume recipes >>>> created in master and only the exceptions would need to have network >>>> access (instead of the bbclass enabling it for every go-mod user - >>>> go-vendor inherits go-mod as well). >>>> >>>> Lukas/Stefan: is there something we can do to help with v2 of your >>>> patches? >>>> >>>> I wanted to submit better review on submitted patches after more >>>> debugging, but here it is: >>>> >>>> 1) to resolve 2 deps: >>>> {'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version': >>>> 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP >>>> Error 404: Not Found" >>>> >>>> {'Path': 'code.cloudfoundry.org/clock >>>> <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', >>>> 'Indirect': True} due to this being redirect to github.com >>>> <http://github.com> and failing with "ERROR: Error while fetching >>>> redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] >>>> certificate verify failed: Hostname mismatch, certificate is not valid >>>> for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. >>> (_ssl.c:996)>" >>>> parse >>>> >>>> 2) some issues with LICENSEs: >>>> INFO: Please add the following line for >>>> 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' >>>> and replace `Unknown` with the license: >>>> c4807c5b058a56c02f592207b4e056b1,Unknown >>>> Traceback (most recent call last): >>>> File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line >>>> 111, in <module> >>>> ret = main() >>>> ^^^^^^ >>>> File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line >>>> 100, in main >>>> ret = args.func(args) >>>> ^^^^^^^^^^^^^^^ >>>> File >>>> "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line >>>> 746, in create_recipe >>>> handler.process(srctree_use, classes, lines_before, lines_after, >>>> handled, extravalues) >>>> File >>>> "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", >>>> line 349, in process >>>> self._rewrite_lic_uri(lines_before) >>>> File >>>> "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", >>>> line 380, in _rewrite_lic_uri >>>> updated, newlines = bb.utils.edit_metadata(lines_before, >>>> ['LIC_FILES_CHKSUM'], varfunc) >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>> File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in >>>> edit_metadata >>>> if handle_var_end(): >>>> ^^^^^^^^^^^^^^^^ >>>> File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in >>>> handle_var_end >>>> (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, >>>> op, newlines) >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>> File >>>> "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", >>>> line 372, in varfunc >>>> uri, chksum = license.split(';', 1) >>>> ^^^^^^^^^^^ >>>> ValueError: not enough values to unpack (expected 2, got 1) >>>> >>>> (the license was just '/') >>>> >>>> 3) some SRCREVs set to None leading to parsing failure: >>>> >>>> File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in >>>> _latest_revision >>>> raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git >>>> repository in git ls-remote output for %s" % \ >>>> bb.data_smart.ExpansionError: Failure expanding variable SRCPV, >>>> expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception >>>> FetchError: Fetcher failure: Unable to resolve 'None' in upstream git >>>> repository in git ls-remote output for >>>> github.com/googleapis/google-cloud-go >>>> <http://github.com/googleapis/google-cloud-go> >>>> The variable dependency chain for the failure is: SRCPV -> PV -> BP -> >>>> FILESPATH >>>> >>>> ERROR: Parsing newly created recipe failed, moving recipe to >>>> >>> /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed >>>> for reference. If this looks to be caused by the recipe itself, please >>>> report this error. >>>> >>>> It takes long time to fetch all telegraf dependencies, so I'll create >>>> some test recipe to more easily debug these 3 issues in isolation. >>>> >>>> >>>> >>>> >>> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#178380): >> https://lists.openembedded.org/g/openembedded-core/message/178380 >> Mute This Topic: https://lists.openembedded.org/mt/95827446/3617156 >> Group Owner: openembedded-core+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ >> Martin.Jansa@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >> >> >
On Wed, Sep 27, 2023 at 9:34 AM Lukas Funke <lukas.funke-oss@weidmueller.com> wrote: > On 10.09.2023 23:26, Martin Jansa wrote: > > On Sun, Mar 12, 2023 at 10:15 AM Martin Jansa via lists.openembedded.org > > <Martin.Jansa=gmail.com@lists.openembedded.org> wrote: > > > >> On Tue, Jan 3, 2023 at 5:03 PM Lukas Funke < > >> lukas.funke-oss@weidmueller.com> wrote: > >> > >>> Martin, > >>> > >>> first of all: thank you for testing the patches. As usual the corner > >>> cases are the most tricky ones. > >>> > >>> tl;dr: I'm working on an updated patch series to address the reported > >>> issues. > >>> > >> > >> Hi Lukas, Stefan, > >> > >> and update on this series? > >> > >> It doesn't have to be perfect, other people can help with remaining > corner > >> cases and right now there is nothing for this in oe-core, so any version > >> will be big improvement :) and it will make it easier for others to > submit > >> incremental improvements - I have some as well, but haven't sent them > as I > >> don't know what you've already updated locally, so I'm waiting for v2. > >> > >> Mickledore is already in feature freeze, but maybe RP will make an > >> exception as this is new important functionality which isn't likely to > >> break other existing code. > >> > > > > Hello Lukas, > > > > any progress on updated patch series? Nanbield release is near and I > don't > > remember seeing the updated patch series in last couple months. > > > > Is there anything I can do to help? The last version looked reasonably > well > > and IMHO could be used as starting point for everybody to help fixing > those > > corner cases where it didn't work well. > > > > Regards, > > Hello Martin, > > I'm currently working on an update, but can't make any promises > regarding the timeline. > Just FYI: on meta-virt master there is a reworked script that is used to update the .inc files for go projects in the layer. It might be helpful, it might not be! Bruce > > Thanks for offering help! Reviewing and testing is always helpful when > the updated series is on it's way. > > Regards > Lukas > > > > > > > >> Kind Regards, > >> > >> > >> I've looked into the issues and would like to give some explanation. > >>> > >>> 1) the first module you mention ('gonum.org/v1/gonum > >>> <http://gonum.org/v1/gonum>') does not obey the go documentation for > >>> resolving the modules source-code repository (see > >>> https://go.dev/ref/mod#go-mod-download , Section 'Version control > >>> systems'). The documentation states that there should be a html page, > >>> queried by "?go-get=1", with a <meta>-tag which contains the original > >>> URL to the source-code repository. For 'gonum' there is only a > 404-page. > >>> This page contains javascript-code that redirects to the actual gonum > >>> package page. Unnecessary hard to process. I'll open an issue for that. > >>> The second module 'code.cloudfoundry.org/clock > >>> <http://code.cloudfoundry.org/clock>' has a broken certificate. I > >>> already opend an issue on github for this. > >>> > >>> I addressed this issue above by maintaining a list of modules and their > >>> actual repo URLs inside the go-recipetool as an absolute fallback. > Would > >>> do you think about this solution? > >>> > >> > >> Yes, that's what I was starting to do in: > >> > >> > https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8 > >> > >> > https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00 > >> > >> > >>> 2) yes, my bad :) > >>> > >> > >> as work around I was using > >> > >> > https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547 > >> > >> > >> 3) The problem is that some repositories have a prefix in their tags. > >>> The tags usually only contain the semantic version string. I'm curretly > >>> working on this issue. > >>> > >>> Best regards > >>> > >>> Lukas > >>> > >>> Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org > : > >>>> On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <uvv.mail@gmail.com > > > >>>> wrote: > >>>> > >>>> On 22.12.2022 17:48, Bruce Ashfield wrote: > >>>> > On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie > >>>> > <richard.purdie@linuxfoundation.org> wrote: > >>>> >> This patch is not in master and is not a backport therefore > not > >>>> >> eligible for kirkstone/langdale. > >>>> > >>>> My bad, the intention was to have it in all three branches: > master, > >>>> kirkstone, and langdale. > >>>> > >>>> > 'nor should it go to master. > >>>> > > >>>> > If someone wants to allow go to fetch sources during builds, it > >>>> needs > >>>> > to be done in their own layers. > >>>> > > >>>> > Bruce > >>>> > >>>> Probably I misunderstood the outcome of the discussion in my > first > >>>> link. > >>>> Bruce and Richard, you both suggested to have it in go-mod with > the > >>>> warning. I agree that it breaks reproducibility, and is > considered > >>>> a bad > >>>> practice in general (I'm aware of a few more build systems with > >>>> the same > >>>> shortcoming). > >>>> > >>>> My point is, until a proper solution is in place, it should be at > >>>> least > >>>> documented somewhere that this should be done in own > layer/recipe. > >>>> Any > >>>> suggestions where this can be documented if not in OE-Core? > >>>> > >>>> > >>>> Have you tried the changes submitted by Lukas/Stefan in: > >>>> > https://lists.openembedded.org/g/openembedded-architecture/message/1539 > >>>> ? > >>>> > >>>> It's not perfect, I was testing it on > >>>> https://github.com/influxdata/telegraf/blob/master/go.mod and I've > >>>> found some corner cases where it failed. > >>>> > >>>> But it seems like very good start and we should work with Lukas/Stefan > >>>> to get it merged in master. Then all branches could consume recipes > >>>> created in master and only the exceptions would need to have network > >>>> access (instead of the bbclass enabling it for every go-mod user - > >>>> go-vendor inherits go-mod as well). > >>>> > >>>> Lukas/Stefan: is there something we can do to help with v2 of your > >>>> patches? > >>>> > >>>> I wanted to submit better review on submitted patches after more > >>>> debugging, but here it is: > >>>> > >>>> 1) to resolve 2 deps: > >>>> {'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version': > >>>> 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP > >>>> Error 404: Not Found" > >>>> > >>>> {'Path': 'code.cloudfoundry.org/clock > >>>> <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', > >>>> 'Indirect': True} due to this being redirect to github.com > >>>> <http://github.com> and failing with "ERROR: Error while fetching > >>>> redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] > >>>> certificate verify failed: Hostname mismatch, certificate is not valid > >>>> for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. > >>> (_ssl.c:996)>" > >>>> parse > >>>> > >>>> 2) some issues with LICENSEs: > >>>> INFO: Please add the following line for > >>>> 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' > >>>> and replace `Unknown` with the license: > >>>> c4807c5b058a56c02f592207b4e056b1,Unknown > >>>> Traceback (most recent call last): > >>>> File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line > >>>> 111, in <module> > >>>> ret = main() > >>>> ^^^^^^ > >>>> File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line > >>>> 100, in main > >>>> ret = args.func(args) > >>>> ^^^^^^^^^^^^^^^ > >>>> File > >>>> "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line > >>>> 746, in create_recipe > >>>> handler.process(srctree_use, classes, lines_before, lines_after, > >>>> handled, extravalues) > >>>> File > >>>> "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", > >>>> line 349, in process > >>>> self._rewrite_lic_uri(lines_before) > >>>> File > >>>> "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", > >>>> line 380, in _rewrite_lic_uri > >>>> updated, newlines = bb.utils.edit_metadata(lines_before, > >>>> ['LIC_FILES_CHKSUM'], varfunc) > >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > >>>> File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in > >>>> edit_metadata > >>>> if handle_var_end(): > >>>> ^^^^^^^^^^^^^^^^ > >>>> File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in > >>>> handle_var_end > >>>> (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, > >>>> op, newlines) > >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > >>>> File > >>>> "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", > >>>> line 372, in varfunc > >>>> uri, chksum = license.split(';', 1) > >>>> ^^^^^^^^^^^ > >>>> ValueError: not enough values to unpack (expected 2, got 1) > >>>> > >>>> (the license was just '/') > >>>> > >>>> 3) some SRCREVs set to None leading to parsing failure: > >>>> > >>>> File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in > >>>> _latest_revision > >>>> raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream > git > >>>> repository in git ls-remote output for %s" % \ > >>>> bb.data_smart.ExpansionError: Failure expanding variable SRCPV, > >>>> expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception > >>>> FetchError: Fetcher failure: Unable to resolve 'None' in upstream git > >>>> repository in git ls-remote output for > >>>> github.com/googleapis/google-cloud-go > >>>> <http://github.com/googleapis/google-cloud-go> > >>>> The variable dependency chain for the failure is: SRCPV -> PV -> BP -> > >>>> FILESPATH > >>>> > >>>> ERROR: Parsing newly created recipe failed, moving recipe to > >>>> > >>> > /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed > >>>> for reference. If this looks to be caused by the recipe itself, please > >>>> report this error. > >>>> > >>>> It takes long time to fetch all telegraf dependencies, so I'll create > >>>> some test recipe to more easily debug these 3 issues in isolation. > >>>> > >>>> > >>>> > >>>> > >>> > >> > >> -=-=-=-=-=-=-=-=-=-=-=- > >> Links: You receive all messages sent to this group. > >> View/Reply Online (#178380): > >> https://lists.openembedded.org/g/openembedded-core/message/178380 > >> Mute This Topic: https://lists.openembedded.org/mt/95827446/3617156 > >> Group Owner: openembedded-core+owner@lists.openembedded.org > >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > >> Martin.Jansa@gmail.com] > >> -=-=-=-=-=-=-=-=-=-=-=- > >> > >> > > > >
diff --git a/meta/classes-recipe/go-mod.bbclass b/meta/classes-recipe/go-mod.bbclass index 61571596bc..da9a583288 100644 --- a/meta/classes-recipe/go-mod.bbclass +++ b/meta/classes-recipe/go-mod.bbclass @@ -24,3 +24,8 @@ inherit go GO_WORKDIR ?= "${GO_IMPORT}" do_compile[dirs] += "${B}/src/${GO_WORKDIR}" +do_compile[network] = "1" + +do_compile:prepend() { + bbwarn "Using go-mod enables network during compilation to be able to fetch dependencies. This breaks reproducibility of the build!" +}
This is a workaround to allow using go-mod until a proper solution is in place. Possible solutions were discussued ealier this year in these threads: https://lists.openembedded.org/g/openembedded-core/topic/89464905 https://lists.openembedded.org/g/openembedded-architecture/topic/90782883 Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> --- meta/classes-recipe/go-mod.bbclass | 5 +++++ 1 file changed, 5 insertions(+)