| Message ID | 20230823210438.2172797-1-rs@ti.com |
|---|---|
| State | New |
| Headers | show |
| Series | bitbake-worker: export_proxies if network flag = 1 | expand |
Something doesn’t compute here. How did it work all this time in tasks like fetch? Alex On Wed 23. Aug 2023 at 23.04, Randolph Sapp via lists.openembedded.org <rs= ti.com@lists.openembedded.org> wrote: > From: Randolph Sapp <rs@ti.com> > > Call the export_proxies() function if the network flag is set. Not sure > why anyone would want to enable access without passing proxy variables > if they are set. > > Signed-off-by: Randolph Sapp <rs@ti.com> > --- > bin/bitbake-worker | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/bin/bitbake-worker b/bin/bitbake-worker > index 451e6926..ba199ef9 100755 > --- a/bin/bitbake-worker > +++ b/bin/bitbake-worker > @@ -285,6 +285,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, > extraconfigdata, runtask): > for e, v in exports: > os.environ[e] = v > > + # if network is true we should assume proxy variables are > to be passed through > + if the_data.getVarFlag(taskname, 'network', False): > + bb.utils.export_proxies(the_data) > + > for e in fakeenv: > os.environ[e] = fakeenv[e] > the_data.setVar(e, fakeenv[e]) > -- > 2.41.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#14959): > https://lists.openembedded.org/g/bitbake-devel/message/14959 > Mute This Topic: https://lists.openembedded.org/mt/100924096/1686489 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Because fetchers explicitly sets proxies through a different mechanism, partially because different fetch mechanisms may have different ways to set proxies. The variables set by export_proxies are the most common, widely accepted ones though so I figured they should be set by default for tasks that need network access. On 8/23/23 16:20, Alexander Kanavin wrote: > Something doesn’t compute here. How did it work all this time in tasks > like fetch? > > Alex > > On Wed 23. Aug 2023 at 23.04, Randolph Sapp via lists.openembedded.org > <http://lists.openembedded.org> <rs=ti.com@lists.openembedded.org > <mailto:ti.com@lists.openembedded.org>> wrote: > > From: Randolph Sapp <rs@ti.com <mailto:rs@ti.com>> > > Call the export_proxies() function if the network flag is set. Not sure > why anyone would want to enable access without passing proxy variables > if they are set. > > Signed-off-by: Randolph Sapp <rs@ti.com <mailto:rs@ti.com>> > --- > bin/bitbake-worker | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/bin/bitbake-worker b/bin/bitbake-worker > index 451e6926..ba199ef9 100755 > --- a/bin/bitbake-worker > +++ b/bin/bitbake-worker > @@ -285,6 +285,10 @@ def fork_off_task(cfg, data, databuilder, > workerdata, extraconfigdata, runtask): > for e, v in exports: > os.environ[e] = v > > + # if network is true we should assume proxy > variables are to be passed through > + if the_data.getVarFlag(taskname, 'network', False): > + bb.utils.export_proxies(the_data) > + > for e in fakeenv: > os.environ[e] = fakeenv[e] > the_data.setVar(e, fakeenv[e]) > -- > 2.41.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#14959): > https://lists.openembedded.org/g/bitbake-devel/message/14959 > <https://lists.openembedded.org/g/bitbake-devel/message/14959> > Mute This Topic: https://lists.openembedded.org/mt/100924096/1686489 > <https://lists.openembedded.org/mt/100924096/1686489> > Group Owner: bitbake-devel+owner@lists.openembedded.org > <mailto:bitbake-devel%2Bowner@lists.openembedded.org> > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub > <https://lists.openembedded.org/g/bitbake-devel/unsub> > [alex.kanavin@gmail.com <mailto:alex.kanavin@gmail.com>] > -=-=-=-=-=-=-=-=-=-=-=- >
On Wed, 2023-08-23 at 23:20 +0200, Alexander Kanavin wrote: > Something doesn’t compute here. How did it work all this time in > tasks like fetch? Before we had network isolation, we did ensure we only set the network proxy information before network access code. This meant the things would break for "unknown" network access outside of the points in the code where we expected it on systems that used a proxy. I've mixed feelings on "simplifying" things like this as we'd lose that markup of the code and a second level of checking, even if that check does need a proxy setup. Was there a specific place where the proxy information was needed but not present? Cheers, Richard
On 8/23/23 16:25, Richard Purdie wrote: > On Wed, 2023-08-23 at 23:20 +0200, Alexander Kanavin wrote: >> Something doesn’t compute here. How did it work all this time in >> tasks like fetch? > > Before we had network isolation, we did ensure we only set the network > proxy information before network access code. This meant the things > would break for "unknown" network access outside of the points in the > code where we expected it on systems that used a proxy. > > I've mixed feelings on "simplifying" things like this as we'd lose that > markup of the code and a second level of checking, even if that check > does need a proxy setup. > > Was there a specific place where the proxy information was needed but > not present? Rust, Go, Flutter, Matter, and all of the other new frameworks and languages that decided to be special and make their own build systems that reach out during the do_compile phase. Though, it could be argued that this is just a cover-up for the real issue, which is lack of proper integration of these build systems into bitbake. I still believe this is useful for compilations behind a corporate proxy for those few recipes still *cheating*.
On 8/23/23 16:34, Randolph Sapp via lists.openembedded.org wrote: > On 8/23/23 16:25, Richard Purdie wrote: >> On Wed, 2023-08-23 at 23:20 +0200, Alexander Kanavin wrote: >>> Something doesn’t compute here. How did it work all this time in >>> tasks like fetch? >> >> Before we had network isolation, we did ensure we only set the network >> proxy information before network access code. This meant the things >> would break for "unknown" network access outside of the points in the >> code where we expected it on systems that used a proxy. >> >> I've mixed feelings on "simplifying" things like this as we'd lose that >> markup of the code and a second level of checking, even if that check >> does need a proxy setup. >> >> Was there a specific place where the proxy information was needed but >> not present? > > Rust, Go, Flutter, Matter, and all of the other new frameworks and > languages that decided to be special and make their own build systems > that reach out during the do_compile phase. > > Though, it could be argued that this is just a cover-up for the real > issue, which is lack of proper integration of these build systems into > bitbake. I still believe this is useful for compilations behind a > corporate proxy for those few recipes still *cheating*. Also, any custom tasks people add that have network dependencies. I just think it's a little rough having to play whack-a-mole with task and proxy oddities.
On Wed, 2023-08-23 at 16:34 -0500, Randolph Sapp wrote: > On 8/23/23 16:25, Richard Purdie wrote: > > On Wed, 2023-08-23 at 23:20 +0200, Alexander Kanavin wrote: > > > Something doesn’t compute here. How did it work all this time in > > > tasks like fetch? > > > > Before we had network isolation, we did ensure we only set the network > > proxy information before network access code. This meant the things > > would break for "unknown" network access outside of the points in the > > code where we expected it on systems that used a proxy. > > > > I've mixed feelings on "simplifying" things like this as we'd lose that > > markup of the code and a second level of checking, even if that check > > does need a proxy setup. > > > > Was there a specific place where the proxy information was needed but > > not present? > > Rust, Go, Flutter, Matter, and all of the other new frameworks and > languages that decided to be special and make their own build systems > that reach out during the do_compile phase. > > Though, it could be argued that this is just a cover-up for the real > issue, which is lack of proper integration of these build systems into > bitbake. I still believe this is useful for compilations behind a > corporate proxy for those few recipes still *cheating*. Do we really want to make it easier for people to ignore that set of problems? :/ People seem very very reluctant to try and fix any of these issues properly and with the kinds of regulations coming in, this is a serious issue whether people like it or not... Cheers, Richard
On Wed, 2023-08-23 at 16:38 -0500, Randolph Sapp wrote: > On 8/23/23 16:34, Randolph Sapp via lists.openembedded.org wrote: > > On 8/23/23 16:25, Richard Purdie wrote: > > > On Wed, 2023-08-23 at 23:20 +0200, Alexander Kanavin wrote: > > > > Something doesn’t compute here. How did it work all this time in > > > > tasks like fetch? > > > > > > Before we had network isolation, we did ensure we only set the network > > > proxy information before network access code. This meant the things > > > would break for "unknown" network access outside of the points in the > > > code where we expected it on systems that used a proxy. > > > > > > I've mixed feelings on "simplifying" things like this as we'd lose that > > > markup of the code and a second level of checking, even if that check > > > does need a proxy setup. > > > > > > Was there a specific place where the proxy information was needed but > > > not present? > > > > Rust, Go, Flutter, Matter, and all of the other new frameworks and > > languages that decided to be special and make their own build systems > > that reach out during the do_compile phase. > > > > Though, it could be argued that this is just a cover-up for the real > > issue, which is lack of proper integration of these build systems into > > bitbake. I still believe this is useful for compilations behind a > > corporate proxy for those few recipes still *cheating*. > > Also, any custom tasks people add that have network dependencies. I just > think it's a little rough having to play whack-a-mole with task and > proxy oddities. The idea is that you realise something is not right and then fix the issue properly, which would be proper SRC_URI support which then means we can cover SPDX manifests, license auditing, source mirroring and reproducibility amongst other things. Sadly most people just want to ignore those things and go for a quick/easy fix. I'm therefore torn on the patch. How easily do we want people to ignore these issues? Cheers, Richard
On 8/23/23 16:40, Richard Purdie wrote: > On Wed, 2023-08-23 at 16:34 -0500, Randolph Sapp wrote: >> On 8/23/23 16:25, Richard Purdie wrote: >>> On Wed, 2023-08-23 at 23:20 +0200, Alexander Kanavin wrote: >>>> Something doesn’t compute here. How did it work all this time in >>>> tasks like fetch? >>> >>> Before we had network isolation, we did ensure we only set the network >>> proxy information before network access code. This meant the things >>> would break for "unknown" network access outside of the points in the >>> code where we expected it on systems that used a proxy. >>> >>> I've mixed feelings on "simplifying" things like this as we'd lose that >>> markup of the code and a second level of checking, even if that check >>> does need a proxy setup. >>> >>> Was there a specific place where the proxy information was needed but >>> not present? >> >> Rust, Go, Flutter, Matter, and all of the other new frameworks and >> languages that decided to be special and make their own build systems >> that reach out during the do_compile phase. >> >> Though, it could be argued that this is just a cover-up for the real >> issue, which is lack of proper integration of these build systems into >> bitbake. I still believe this is useful for compilations behind a >> corporate proxy for those few recipes still *cheating*. > > Do we really want to make it easier for people to ignore that set of > problems? :/ I suppose not. Though it is a little rough to add a fetcher and class for every one-off build system that Google introduces for their shinny new product :P Feel free to drop this patch. Just figured I would test the waters.
On 8/23/23 16:40, Richard Purdie wrote: > On Wed, 2023-08-23 at 16:34 -0500, Randolph Sapp wrote: >> On 8/23/23 16:25, Richard Purdie wrote: >>> On Wed, 2023-08-23 at 23:20 +0200, Alexander Kanavin wrote: >>>> Something doesn’t compute here. How did it work all this time in >>>> tasks like fetch? >>> >>> Before we had network isolation, we did ensure we only set the network >>> proxy information before network access code. This meant the things >>> would break for "unknown" network access outside of the points in the >>> code where we expected it on systems that used a proxy. >>> >>> I've mixed feelings on "simplifying" things like this as we'd lose that >>> markup of the code and a second level of checking, even if that check >>> does need a proxy setup. >>> >>> Was there a specific place where the proxy information was needed but >>> not present? >> >> Rust, Go, Flutter, Matter, and all of the other new frameworks and >> languages that decided to be special and make their own build systems >> that reach out during the do_compile phase. >> >> Though, it could be argued that this is just a cover-up for the real >> issue, which is lack of proper integration of these build systems into >> bitbake. I still believe this is useful for compilations behind a >> corporate proxy for those few recipes still *cheating*. > > Do we really want to make it easier for people to ignore that set of > problems? :/ > > People seem very very reluctant to try and fix any of these issues > properly and with the kinds of regulations coming in, this is a serious > issue whether people like it or not... I would argue that as long as the 'network' flag exists people will just continue to ignore the real issue. The current proxy problem just makes it slightly more annoying for a small percentage of the people currently circumventing the issue.
diff --git a/bin/bitbake-worker b/bin/bitbake-worker index 451e6926..ba199ef9 100755 --- a/bin/bitbake-worker +++ b/bin/bitbake-worker @@ -285,6 +285,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask): for e, v in exports: os.environ[e] = v + # if network is true we should assume proxy variables are to be passed through + if the_data.getVarFlag(taskname, 'network', False): + bb.utils.export_proxies(the_data) + for e in fakeenv: os.environ[e] = fakeenv[e] the_data.setVar(e, fakeenv[e])