| Message ID | 20220706182329.3085564-1-alex@linutronix.de |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] scripts/oe-setup-builddir: copy site.conf.sample out of template directories (if it exists) | expand |
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded- > core@lists.openembedded.org> On Behalf Of Alexander Kanavin > Sent: den 6 juli 2022 20:23 > To: openembedded-core@lists.openembedded.org > Cc: Alexander Kanavin <alex@linutronix.de> > Subject: [OE-core] [PATCH 1/2] scripts/oe-setup-builddir: copy > site.conf.sample out of template directories (if it exists) > > This allows: > > 1. Showing users where and how to define these settings correctly when setting up > a build from templates in poky (meta-poky/conf/site.conf.sample has commented > out examples and was previously unused). > > 2. Distributing site-specific settings with template configurations in other layers, > so there's no need to set them up separately. > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > scripts/oe-setup-builddir | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir > index 54048e62ec..5ad6dd4138 100755 > --- a/scripts/oe-setup-builddir > +++ b/scripts/oe-setup-builddir > @@ -64,6 +64,7 @@ if [ -n "$TEMPLATECONF" ]; then > fi > OECORELAYERCONF="$TEMPLATECONF/bblayers.conf.sample" > OECORELOCALCONF="$TEMPLATECONF/local.conf.sample" > + OECORESITECONF="$TEMPLATECONF/site.conf.sample" > OECORENOTESCONF="$TEMPLATECONF/conf-notes.txt" > fi > > @@ -77,9 +78,11 @@ You had no conf/local.conf file. This configuration > file has therefore been > created for you with some default values. You may wish to edit it to, for > example, select a different MACHINE (target hardware). See conf/local.conf > for more information as common configuration options are commented. > - > +Also check conf/site.conf for site specific settings such as proxies and > +download cache locations. > EOM > cp -f "$OECORELOCALCONF" "$BUILDDIR/conf/local.conf" > + cp -f "$OECORESITECONF" "$BUILDDIR/conf/site.conf" || true > SHOWYPDOC=yes > fi > > @@ -107,6 +110,7 @@ fi > # Prevent disturbing a new GIT clone in same console > unset OECORELOCALCONF > unset OECORELAYERCONF > +unset OECORESITECONF > > # Ending the first-time run message. Show the YP Documentation banner. > if [ ! -z "$SHOWYPDOC" ]; then > -- > 2.30.2 While I can't speak for others, this change will break our build environment. We have a global site.conf file in an NFS directory that everyone use. It contains all proxy settings and similar that are required to build inside the corporate environment. Copying a dummy site.conf file into the local build directory will prevent the global file from being used. //Peter
On Thu, 7 Jul 2022 at 15:26, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > While I can't speak for others, this change will break our build environment. > We have a global site.conf file in an NFS directory that everyone use. It > contains all proxy settings and similar that are required to build inside the > corporate environment. Copying a dummy site.conf file into the local build > directory will prevent the global file from being used. Then you need to create a configuration template that is specific to your company first, and there's a convenient command to do that provided in patch 2/2. Setting up a build directory from standard poky or oe-core templates (which I presume is how you set up builds) does not hold a promise that what is being copied will not change in potentially breaking ways. Alex
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 7 juli 2022 15:36 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: openembedded-core@lists.openembedded.org; Alexander Kanavin > <alex@linutronix.de> > Subject: Re: [OE-core] [PATCH 1/2] scripts/oe-setup-builddir: copy > site.conf.sample out of template directories (if it exists) > > On Thu, 7 Jul 2022 at 15:26, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > > While I can't speak for others, this change will break our build environment. > > We have a global site.conf file in an NFS directory that everyone use. It > > contains all proxy settings and similar that are required to build inside the > > corporate environment. Copying a dummy site.conf file into the local build > > directory will prevent the global file from being used. > > Then you need to create a configuration template that is specific to > your company first, and there's a convenient command to do that > provided in patch 2/2. While I haven't actually tried your new command, based on what you and others have written, it does not at all seem compatible with how our environment is setup and what it expects from poky. :( > Setting up a build directory from standard poky > or oe-core templates (which I presume is how you set up builds) does > not hold a promise that what is being copied will not change in > potentially breaking ways. While I may be able to fix the environment for builds that use some of our layers, setting up pure poky builds will no longer work without manual intervention. > Alex //Peter
On Thu, 7 Jul 2022 at 15:45, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > > Setting up a build directory from standard poky > > or oe-core templates (which I presume is how you set up builds) does > > not hold a promise that what is being copied will not change in > > potentially breaking ways. > > While I may be able to fix the environment for builds that use some of > our layers, setting up pure poky builds will no longer work without > manual intervention. Wait. How are you setting up pure poky builds right now, so that your magic site.conf is correctly inserted into builds? Alex
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded- > core@lists.openembedded.org> On Behalf Of Alexander Kanavin > Sent: den 7 juli 2022 15:56 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: openembedded-core@lists.openembedded.org; Alexander Kanavin > <alex@linutronix.de> > Subject: Re: [OE-core] [PATCH 1/2] scripts/oe-setup-builddir: copy > site.conf.sample out of template directories (if it exists) > > On Thu, 7 Jul 2022 at 15:45, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > > > Setting up a build directory from standard poky > > > or oe-core templates (which I presume is how you set up builds) does > > > not hold a promise that what is being copied will not change in > > > potentially breaking ways. > > > > While I may be able to fix the environment for builds that use some of > > our layers, setting up pure poky builds will no longer work without > > manual intervention. > > Wait. How are you setting up pure poky builds right now, so that your > magic site.conf is correctly inserted into builds? > > Alex Actually, you can forget what I said. I had a look at our wrapper for oe-init-build-env, which is responsible for creating our templateconf directory, and it will of course not be affected by your change. (It was six years since I wrote that code, so my memory was obviously a bit fuzzy.) And in case you wonder what that wrapper actually does, it creates the bblayers.conf.sample file based on all layers that are found in the top directory. It also creates a local.conf.sample file by combining fragment files in the layers (i.e., local.conf.sample.XX where XX is a number between 00 and 99). The local.conf.sample from meta-poky is considered to have XX == 50. The reason for this is that we quickly realized that the static templateconf directory setup was unusable for us when we want to be able to use layers in different combinations. //Peter
On Thu, 7 Jul 2022 at 16:38, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > And in case you wonder what that wrapper actually does, it creates the > bblayers.conf.sample file based on all layers that are found in the top > directory. It also creates a local.conf.sample file by combining fragment > files in the layers (i.e., local.conf.sample.XX where XX is a number > between 00 and 99). The local.conf.sample from meta-poky is considered > to have XX == 50. The reason for this is that we quickly realized that > the static templateconf directory setup was unusable for us when we > want to be able to use layers in different combinations. Cheers. I'd say if you need to place lots of customizations into local.conf that's not a recommended setup to begin with: as said elsewhere it should have precisely two definitions: distro and machine, and nothing else. What kind of things are in those fragments? Should they be consolidated in machine and distro definitions? Alex
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 7 juli 2022 16:48 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: openembedded-core@lists.openembedded.org; Alexander Kanavin > <alex@linutronix.de> > Subject: Re: [OE-core] [PATCH 1/2] scripts/oe-setup-builddir: copy > site.conf.sample out of template directories (if it exists) > > On Thu, 7 Jul 2022 at 16:38, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > > And in case you wonder what that wrapper actually does, it creates the > > bblayers.conf.sample file based on all layers that are found in the top > > directory. It also creates a local.conf.sample file by combining > fragment > > files in the layers (i.e., local.conf.sample.XX where XX is a number > > between 00 and 99). The local.conf.sample from meta-poky is considered > > to have XX == 50. The reason for this is that we quickly realized that > > the static templateconf directory setup was unusable for us when we > > want to be able to use layers in different combinations. > > Cheers. I'd say if you need to place lots of customizations into There actually isn't lots of configuration in those fragments. > local.conf that's not a recommended setup to begin with: as said > elsewhere it should have precisely two definitions: distro and > machine, and nothing else. What kind of things are in those fragments? In our BSP layer there is some configuration we want even if our distro layer is not used (e.g., SSTATE_DIR). It also sets a default MACHINE that matches one from our BSPs. In the distro layer, it sets DISTRO. It also adds some extra, commented variables that our users may want to enable in their local.conf files, much like the examples in the local.conf.sample from meta-poky. Part of this is of course related to that we use repo to fetch the layers we need and thus we do not have the luxury of bitbake specific tools like kas and whisk that can add bitbake configuration in addition to the actual layers. On the other hand, our environment is very similar to the default experience with poky from a user's perspective. > Should they be consolidated in machine and distro definitions? > > Alex //Peter
On Wed, 2022-07-06 at 20:23 +0200, Alexander Kanavin wrote: > This allows: > > 1. Showing users where and how to define these settings correctly when setting up > a build from templates in poky (meta-poky/conf/site.conf.sample has commented > out examples and was previously unused). > > 2. Distributing site-specific settings with template configurations in other layers, > so there's no need to set them up separately. > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > scripts/oe-setup-builddir | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) We'd always intended that site.conf was more something the user would setup once for their environment and not need to change. It was therefore deliberately left out of the auto-setup piece. I hadn't remembered we have an sample version of it. I think a sample is fine as an example of how a site might setup a shared downloads directory or common proxy information but I don't think the layer should be installing it, particularly if it can trample an existing file which I think the patch could as it stands. As such I don't think this patch is the right thing to do. Cheers, Richard
On Fri, 15 Jul 2022 at 16:42, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > We'd always intended that site.conf was more something the user would > setup once for their environment and not need to change. It was > therefore deliberately left out of the auto-setup piece. > I hadn't remembered we have an sample version of it. I think a sample > is fine as an example of how a site might setup a shared downloads > directory or common proxy information but I don't think the layer > should be installing it, particularly if it can trample an existing > file which I think the patch could as it stands. Ah I see where the problem is: the sample file is installed subject to existence of local.conf in the destination build directory, which is ok most of the time, but not all of the time. It should check for existence of site.conf. The idea is to serve two use cases: - when setting up a build from the poky template, there's a sample file installed. Same as local.conf.sample, we give users something to play with via commented out examples. They'll never find it in meta-poky/conf, even you had no idea it's there :) - when setting up a build from an organizational template, the right servers get configured without any extra steps. Ross pointed out that can be achieved with 'meta-product/conf/site.conf', but that can't be edited when needed, and can't be used to configure different settings for different builds (e.g. where the caches are - such things can be under access control restrictions). I think this is worth reworking and making the commit message clearer. Or we should come up with a different way to provide 'official' way to set up site.conf, because otherwise it's back to 'write a custom script'. Alex
On Fri, 15 Jul 2022 at 16:42, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > We'd always intended that site.conf was more something the user would > setup once for their environment and not need to change. It was > therefore deliberately left out of the auto-setup piece. I don't think I understand this point in particular. If site.conf is copied once into a newly created, empty build directory, why is that problematic? Alex
On Fri, 2022-07-15 at 18:15 +0200, Alexander Kanavin wrote: > On Fri, 15 Jul 2022 at 16:42, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > We'd always intended that site.conf was more something the user would > > setup once for their environment and not need to change. It was > > therefore deliberately left out of the auto-setup piece. > > I don't think I understand this point in particular. If site.conf is > copied once into a newly created, empty build directory, why is that > problematic? Originally, we actually wanted this to be something like from the user's homedir, i.e. things they really would setup just once. Doing that has downsides in that if not careful, things can "leak" into a build unexpectedly too so we never did end up enabling it from $HOME. I do still think of it that way though. In that sense, it was something a user would provide, not something that would be provisioned by automated scripting. If there are settings to be provisioned by a layer, we have local.conf for that? Why do we need another file too? Cheers, Richard
On Fri, 15 Jul 2022 at 18:24, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > In that sense, it was something a user would provide, not something > that would be provisioned by automated scripting. If there are settings > to be provisioned by a layer, we have local.conf for that? Why do we > need another file too? Right, I'm ok with organizational proxies and servers going into local.conf. Then site.conf.sample and local.conf.extended.sample should be promoted in the banner when setting up from the meta-poky template. Alex
On 15/07/2022 18:00:57+0200, Alexander Kanavin wrote: > On Fri, 15 Jul 2022 at 16:42, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > We'd always intended that site.conf was more something the user would > > setup once for their environment and not need to change. It was > > therefore deliberately left out of the auto-setup piece. > > I hadn't remembered we have an sample version of it. I think a sample > > is fine as an example of how a site might setup a shared downloads > > directory or common proxy information but I don't think the layer > > should be installing it, particularly if it can trample an existing > > file which I think the patch could as it stands. > > Ah I see where the problem is: the sample file is installed subject to > existence of local.conf in the destination build directory, which is > ok most of the time, but not all of the time. It should check for > existence of site.conf. > > The idea is to serve two use cases: > - when setting up a build from the poky template, there's a sample > file installed. Same as local.conf.sample, we give users something to > play with via commented out examples. They'll never find it in > meta-poky/conf, even you had no idea it's there :) > - when setting up a build from an organizational template, the right > servers get configured without any extra steps. Ross pointed out that > can be achieved with 'meta-product/conf/site.conf', but that can't be > edited when needed, and can't be used to configure different settings > for different builds (e.g. where the caches are - such things can be > under access control restrictions). > > I think this is worth reworking and making the commit message clearer. > Or we should come up with a different way to provide 'official' way to > set up site.conf, because otherwise it's back to 'write a custom > script'. > I don't think you need a custom script for site.conf, it is searched for relative to BBPATH? I usually just put it directly as conf/site.conf in the custom layer so it is in version control and not copied to BUILDDIR. > Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#168121): https://lists.openembedded.org/g/openembedded-core/message/168121 > Mute This Topic: https://lists.openembedded.org/mt/92212803/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Fri, 15 Jul 2022 at 23:57, Alexandre Belloni <alexandre.belloni@bootlin.com> wrote: > I don't think you need a custom script for site.conf, it is searched for > relative to BBPATH? I usually just put it directly as conf/site.conf in > the custom layer so it is in version control and not copied to BUILDDIR. I was thinking of situations where the settings that go there vary between build configurations. But we settled with RP that then they should be in local.conf. Alex
diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir index 54048e62ec..5ad6dd4138 100755 --- a/scripts/oe-setup-builddir +++ b/scripts/oe-setup-builddir @@ -64,6 +64,7 @@ if [ -n "$TEMPLATECONF" ]; then fi OECORELAYERCONF="$TEMPLATECONF/bblayers.conf.sample" OECORELOCALCONF="$TEMPLATECONF/local.conf.sample" + OECORESITECONF="$TEMPLATECONF/site.conf.sample" OECORENOTESCONF="$TEMPLATECONF/conf-notes.txt" fi @@ -77,9 +78,11 @@ You had no conf/local.conf file. This configuration file has therefore been created for you with some default values. You may wish to edit it to, for example, select a different MACHINE (target hardware). See conf/local.conf for more information as common configuration options are commented. - +Also check conf/site.conf for site specific settings such as proxies and +download cache locations. EOM cp -f "$OECORELOCALCONF" "$BUILDDIR/conf/local.conf" + cp -f "$OECORESITECONF" "$BUILDDIR/conf/site.conf" || true SHOWYPDOC=yes fi @@ -107,6 +110,7 @@ fi # Prevent disturbing a new GIT clone in same console unset OECORELOCALCONF unset OECORELAYERCONF +unset OECORESITECONF # Ending the first-time run message. Show the YP Documentation banner. if [ ! -z "$SHOWYPDOC" ]; then
This allows: 1. Showing users where and how to define these settings correctly when setting up a build from templates in poky (meta-poky/conf/site.conf.sample has commented out examples and was previously unused). 2. Distributing site-specific settings with template configurations in other layers, so there's no need to set them up separately. Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- scripts/oe-setup-builddir | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)