| Message ID | 20220922124708.2218995-2-pkj@axis.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] .templateconf: Mark "meta/conf" as an invalid path for templates | expand |
On Thu, 2022-09-22 at 14:47 +0200, Peter Kjellerstedt wrote: > Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded > as invalid paths for templates. However, this is suboptimal for other > distros that are setup similarly to Poky. Instead, add support for a > new variable INVALID_TEMPLATECONFS, which takes a list of invalid > paths. It is expected that this variable is set in the .templateconf > file together with the default value for TEMPLATECONF, typically at > the same time that TEMPLATECONF is updated to match the new > requirements. > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > --- > scripts/oe-setup-builddir | 21 +++++++++++++-------- > 1 file changed, 13 insertions(+), 8 deletions(-) > > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir > index 225919be92..c1148daf4f 100755 > --- a/scripts/oe-setup-builddir > +++ b/scripts/oe-setup-builddir > @@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD > > cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!" > > -if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then > - TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg") > - # The following two are no longer valid; unsetting them will automatically get them replaced > - # with correct ones. > - if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then > - unset TEMPLATECONF > - rm "$BUILDDIR/conf/templateconf.cfg" > - fi > +TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg" > +if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then > + TEMPLATECONF=$(cat "$TEMPLATECONF_CFG") > + # Unset TEMPLATECONF if it is set to a known invalid value to have it > + # automatically replaced with a correct one. > + for dir in $INVALID_TEMPLATECONFS; do > + if [ "$TEMPLATECONF" = "$dir" ]; then > + echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'" > + unset TEMPLATECONF > + rm "$TEMPLATECONF_CFG" > + break; > + fi > + done > fi I've been trying to keep out this discussion but I really don't like piling in more complexity and more things we have to support because of a legacy code path we're trying to obsolete. I also suspect this is going to be used to "subvert" the move to try and standardise and if that happens, we may as well not have bothered making changes at all :(. As such I'm worried about this direction. Cheers, Richard
> -----Original Message----- > From: Richard Purdie <richard.purdie@linuxfoundation.org> > Sent: den 22 september 2022 15:14 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-core@lists.openembedded.org > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates > > On Thu, 2022-09-22 at 14:47 +0200, Peter Kjellerstedt wrote: > > Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded > > as invalid paths for templates. However, this is suboptimal for other > > distros that are setup similarly to Poky. Instead, add support for a > > new variable INVALID_TEMPLATECONFS, which takes a list of invalid > > paths. It is expected that this variable is set in the .templateconf > > file together with the default value for TEMPLATECONF, typically at > > the same time that TEMPLATECONF is updated to match the new > > requirements. > > > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > > --- > > scripts/oe-setup-builddir | 21 +++++++++++++-------- > > 1 file changed, 13 insertions(+), 8 deletions(-) > > > > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir > > index 225919be92..c1148daf4f 100755 > > --- a/scripts/oe-setup-builddir > > +++ b/scripts/oe-setup-builddir > > @@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD > > > > cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!" > > > > -if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then > > - TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg") > > - # The following two are no longer valid; unsetting them will automatically get them replaced > > - # with correct ones. > > - if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then > > - unset TEMPLATECONF > > - rm "$BUILDDIR/conf/templateconf.cfg" > > - fi > > +TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg" > > +if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then > > + TEMPLATECONF=$(cat "$TEMPLATECONF_CFG") > > + # Unset TEMPLATECONF if it is set to a known invalid value to have it > > + # automatically replaced with a correct one. > > + for dir in $INVALID_TEMPLATECONFS; do > > + if [ "$TEMPLATECONF" = "$dir" ]; then > > + echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'" > > + unset TEMPLATECONF > > + rm "$TEMPLATECONF_CFG" > > + break; > > + fi > > + done > > fi > > I've been trying to keep out this discussion but I really don't like > piling in more complexity and more things we have to support because of > a legacy code path we're trying to obsolete. > > I also suspect this is going to be used to "subvert" the move to try > and standardise and if that happens, we may as well not have bothered > making changes at all :(. > > As such I'm worried about this direction. > > Cheers, > > Richard All I want is to remove the hardcoded superficial limitations and allow us that provide distros setup just as Poky is setup to continue to do so. //Peter
.templateconf is not the way to specify the location of custom templates, and never was. It got added to documentation by misunderstanding, this has now been removed, and so you need to update your tooling so that templates are set with TEMPLATECONF going forward. Alex On Thu, 22 Sept 2022 at 15:26, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > > > -----Original Message----- > > From: Richard Purdie <richard.purdie@linuxfoundation.org> > > Sent: den 22 september 2022 15:14 > > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-core@lists.openembedded.org > > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates > > > > On Thu, 2022-09-22 at 14:47 +0200, Peter Kjellerstedt wrote: > > > Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded > > > as invalid paths for templates. However, this is suboptimal for other > > > distros that are setup similarly to Poky. Instead, add support for a > > > new variable INVALID_TEMPLATECONFS, which takes a list of invalid > > > paths. It is expected that this variable is set in the .templateconf > > > file together with the default value for TEMPLATECONF, typically at > > > the same time that TEMPLATECONF is updated to match the new > > > requirements. > > > > > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > > > --- > > > scripts/oe-setup-builddir | 21 +++++++++++++-------- > > > 1 file changed, 13 insertions(+), 8 deletions(-) > > > > > > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir > > > index 225919be92..c1148daf4f 100755 > > > --- a/scripts/oe-setup-builddir > > > +++ b/scripts/oe-setup-builddir > > > @@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD > > > > > > cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!" > > > > > > -if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then > > > - TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg") > > > - # The following two are no longer valid; unsetting them will automatically get them replaced > > > - # with correct ones. > > > - if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then > > > - unset TEMPLATECONF > > > - rm "$BUILDDIR/conf/templateconf.cfg" > > > - fi > > > +TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg" > > > +if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then > > > + TEMPLATECONF=$(cat "$TEMPLATECONF_CFG") > > > + # Unset TEMPLATECONF if it is set to a known invalid value to have it > > > + # automatically replaced with a correct one. > > > + for dir in $INVALID_TEMPLATECONFS; do > > > + if [ "$TEMPLATECONF" = "$dir" ]; then > > > + echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'" > > > + unset TEMPLATECONF > > > + rm "$TEMPLATECONF_CFG" > > > + break; > > > + fi > > > + done > > > fi > > > > I've been trying to keep out this discussion but I really don't like > > piling in more complexity and more things we have to support because of > > a legacy code path we're trying to obsolete. > > > > I also suspect this is going to be used to "subvert" the move to try > > and standardise and if that happens, we may as well not have bothered > > making changes at all :(. > > > > As such I'm worried about this direction. > > > > Cheers, > > > > Richard > > All I want is to remove the hardcoded superficial limitations and > allow us that provide distros setup just as Poky is setup to continue > to do so. > > //Peter > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#170981): https://lists.openembedded.org/g/openembedded-core/message/170981 > Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 22 september 2022 17:53 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; openembedded- > core@lists.openembedded.org > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode > invalid paths for templates > > .templateconf is not the way to specify the location of custom > templates, and never was. It got added to documentation by > misunderstanding, this has now been removed, and so you need to update > your tooling so that templates are set with TEMPLATECONF going > forward. > > Alex [Irritated rant below, and I apologize in advance. Sorry. I know you want to improve things, but I just have to state my opinion.] Ok, this is silly. .templateconf _is_ the documented way to specify a default TEMPLATECONF and has been so for a long time. Just because you don't use it or like it does not mean that you can just rip it out and say we now have to use some (for us) inferior method. We have 500+ developers and a huge number of CI/CD pipelines configured and built thousand times a day. They all expect the current setup, one they have used for ten years. The _exact same setup_ you get if you only fetch Poky and start a build. All I want to do is remove hardcoded values in favor of remaining useful for all users of Yocto. Why is that a problem? I simply cannot understand it. To me, hardcoded values should never be used when a generic solution is possible. Why is it better to hardcode knowledge about Poky in OE-Core, when there is no reason to do so? I know (or rather hope) you have some plan going forward for how to setup the build environment, but it is not there in Langdale. Also, I want to be able to continue with fetching the layers ourselves the way we always have done it and then continue to tell bitbake where to find them as we always have done without having to go through hoops or having to duplicate a lot of bitbake's internals. And I believe this was actually said during all discussions about how to add support for a common way of setting up the build environment, that it should be possible for us that already have ways to do it (be they repo, submodules, kas, whisk or whatever) to continue to do so. So please, please be mindful of simple changes that allow us to continue to use the processes we already have in place when it really doesn't cost anything. //Peter > On Thu, 22 Sept 2022 at 15:26, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > > > > -----Original Message----- > > > From: Richard Purdie <richard.purdie@linuxfoundation.org> > > > Sent: den 22 september 2022 15:14 > > > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded- > core@lists.openembedded.org > > > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode > invalid paths for templates > > > > > > On Thu, 2022-09-22 at 14:47 +0200, Peter Kjellerstedt wrote: > > > > Previously, the paths "meta/conf" and "meta-poky/conf" were > hardcoded > > > > as invalid paths for templates. However, this is suboptimal for > other > > > > distros that are setup similarly to Poky. Instead, add support for a > > > > new variable INVALID_TEMPLATECONFS, which takes a list of invalid > > > > paths. It is expected that this variable is set in the > .templateconf > > > > file together with the default value for TEMPLATECONF, typically at > > > > the same time that TEMPLATECONF is updated to match the new > > > > requirements. > > > > > > > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > > > > --- > > > > scripts/oe-setup-builddir | 21 +++++++++++++-------- > > > > 1 file changed, 13 insertions(+), 8 deletions(-) > > > > > > > > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir > > > > index 225919be92..c1148daf4f 100755 > > > > --- a/scripts/oe-setup-builddir > > > > +++ b/scripts/oe-setup-builddir > > > > @@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo > "WARNING: unable to chmod $BUILDD > > > > > > > > cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!" > > > > > > > > -if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" > ]; then > > > > - TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg") > > > > - # The following two are no longer valid; unsetting them will > automatically get them replaced > > > > - # with correct ones. > > > > - if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta- > poky/conf ]; then > > > > - unset TEMPLATECONF > > > > - rm "$BUILDDIR/conf/templateconf.cfg" > > > > - fi > > > > +TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg" > > > > +if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then > > > > + TEMPLATECONF=$(cat "$TEMPLATECONF_CFG") > > > > + # Unset TEMPLATECONF if it is set to a known invalid value to > have it > > > > + # automatically replaced with a correct one. > > > > + for dir in $INVALID_TEMPLATECONFS; do > > > > + if [ "$TEMPLATECONF" = "$dir" ]; then > > > > + echo "WARNING: Removing $TEMPLATECONF_CFG as it > contained the invalid value '$TEMPLATECONF'" > > > > + unset TEMPLATECONF > > > > + rm "$TEMPLATECONF_CFG" > > > > + break; > > > > + fi > > > > + done > > > > fi > > > > > > I've been trying to keep out this discussion but I really don't like > > > piling in more complexity and more things we have to support because > of > > > a legacy code path we're trying to obsolete. > > > > > > I also suspect this is going to be used to "subvert" the move to try > > > and standardise and if that happens, we may as well not have bothered > > > making changes at all :(. > > > > > > As such I'm worried about this direction. > > > > > > Cheers, > > > > > > Richard > > > > All I want is to remove the hardcoded superficial limitations and > > allow us that provide distros setup just as Poky is setup to continue > > to do so. > > > > //Peter > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#170981): > https://lists.openembedded.org/g/openembedded-core/message/170981 > > Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub > [alex.kanavin@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- > >
On Thu, 2022-09-22 at 17:52 +0200, Alexander Kanavin wrote: > .templateconf is not the way to specify the location of custom > templates, and never was. That much isn't true. It was a way this has been done, it was suggested as a way people do it and it was even documented as such. The original commit does make the intent clear: https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9 I think the key point is that it has been of limited use as it requires a custom repo due to the use of OEROOT. Where you had a custom repo, it was fine to use. Poky was one example. I haven't read back through the discussions but I suspect Peter must have some kind of custom repo setup like that to be able to alter files in OEROOT. > It got added to documentation by misunderstanding, Again, I'm not sure about that. I think at the time, when layers were much newer things and technology was more in flux, we did think this was a valid customisation. At the time we were suggesting people might want to use tools like combo-layer to mess with repos. What the documentation doesn't make clear is that this trick only works for repo alterations as far as I can see. Since we don't recommend such repo alterations now and combo-layer is pretty dead, the code is of limited use and I do think it probably should be removed. I therefore don't like adding complexity to something which really shouldn't be there at all. I appreciate this does looks like it creates a challenge for Peter's situation though. I have put off touching this particular area of code since I know there will be a ton of problems exactly like this. The success or failure of such changes does depend on how the migration is handled. A large part of this is communication and to some extend negotiation too. The fact I'm having to start trying to understand all the details is the situation I didn't really want to be in. I suspect to get anywhere I now need to go and read all the previous threads to then try and unravel this :/. Peter: is there a pointer you can give to the setup you're using? Cheers, Richard
First of all, I would like to understand why writing to .templateconf cannot be replaced with specifying TEMPLATECONF= in your setup. Then we can maybe proceed to the similar question asked earlier about replacing writing to bblayer.conf with using bitbake-layers to adjust the layer list. There was also a question about removing the step of creating a custom template altogether, and just writing a build configuration directly. Where are the answers to those, Peter? Alex On Thu, 22 Sept 2022 at 22:32, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Thu, 2022-09-22 at 17:52 +0200, Alexander Kanavin wrote: > > .templateconf is not the way to specify the location of custom > > templates, and never was. > > That much isn't true. It was a way this has been done, it was suggested > as a way people do it and it was even documented as such. The original > commit does make the intent clear: > > https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9 > > I think the key point is that it has been of limited use as it requires > a custom repo due to the use of OEROOT. Where you had a custom repo, it > was fine to use. Poky was one example. > > I haven't read back through the discussions but I suspect Peter must > have some kind of custom repo setup like that to be able to alter files > in OEROOT. > > > It got added to documentation by misunderstanding, > > Again, I'm not sure about that. I think at the time, when layers were > much newer things and technology was more in flux, we did think this > was a valid customisation. At the time we were suggesting people might > want to use tools like combo-layer to mess with repos. > > What the documentation doesn't make clear is that this trick only works > for repo alterations as far as I can see. > > Since we don't recommend such repo alterations now and combo-layer is > pretty dead, the code is of limited use and I do think it probably > should be removed. I therefore don't like adding complexity to > something which really shouldn't be there at all. I appreciate this > does looks like it creates a challenge for Peter's situation though. > > I have put off touching this particular area of code since I know there > will be a ton of problems exactly like this. The success or failure of > such changes does depend on how the migration is handled. A large part > of this is communication and to some extend negotiation too. The fact > I'm having to start trying to understand all the details is the > situation I didn't really want to be in. > > I suspect to get anywhere I now need to go and read all the previous > threads to then try and unravel this :/. > > Peter: is there a pointer you can give to the setup you're using? > > Cheers, > > Richard > > > > >
I've just written and sent a sort of 'introduction' to the official layer setup tooling: https://lists.openembedded.org/g/openembedded-core/message/171070 Build configuration management is a separate concern, I have an introduction to that too, but not sending it yet, as particularly oe-setup-build needs to be reviewed and discussed and (hopefully) merged. Alex On Thu, 22 Sept 2022 at 23:54, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > First of all, I would like to understand why writing to .templateconf > cannot be replaced with specifying TEMPLATECONF= in your setup. > > Then we can maybe proceed to the similar question asked earlier about > replacing writing to bblayer.conf with using bitbake-layers to adjust > the layer list. > There was also a question about removing the step of creating a custom > template altogether, and just writing a build configuration directly. > > Where are the answers to those, Peter? > > Alex > > On Thu, 22 Sept 2022 at 22:32, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Thu, 2022-09-22 at 17:52 +0200, Alexander Kanavin wrote: > > > .templateconf is not the way to specify the location of custom > > > templates, and never was. > > > > That much isn't true. It was a way this has been done, it was suggested > > as a way people do it and it was even documented as such. The original > > commit does make the intent clear: > > > > https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9 > > > > I think the key point is that it has been of limited use as it requires > > a custom repo due to the use of OEROOT. Where you had a custom repo, it > > was fine to use. Poky was one example. > > > > I haven't read back through the discussions but I suspect Peter must > > have some kind of custom repo setup like that to be able to alter files > > in OEROOT. > > > > > It got added to documentation by misunderstanding, > > > > Again, I'm not sure about that. I think at the time, when layers were > > much newer things and technology was more in flux, we did think this > > was a valid customisation. At the time we were suggesting people might > > want to use tools like combo-layer to mess with repos. > > > > What the documentation doesn't make clear is that this trick only works > > for repo alterations as far as I can see. > > > > Since we don't recommend such repo alterations now and combo-layer is > > pretty dead, the code is of limited use and I do think it probably > > should be removed. I therefore don't like adding complexity to > > something which really shouldn't be there at all. I appreciate this > > does looks like it creates a challenge for Peter's situation though. > > > > I have put off touching this particular area of code since I know there > > will be a ton of problems exactly like this. The success or failure of > > such changes does depend on how the migration is handled. A large part > > of this is communication and to some extend negotiation too. The fact > > I'm having to start trying to understand all the details is the > > situation I didn't really want to be in. > > > > I suspect to get anywhere I now need to go and read all the previous > > threads to then try and unravel this :/. > > > > Peter: is there a pointer you can give to the setup you're using? > > > > Cheers, > > > > Richard > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#170995): https://lists.openembedded.org/g/openembedded-core/message/170995 > Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
> -----Original Message----- > From: Richard Purdie <richard.purdie@linuxfoundation.org> > Sent: den 22 september 2022 22:33 > To: Alexander Kanavin <alex.kanavin@gmail.com>; Peter Kjellerstedt > <peter.kjellerstedt@axis.com> > Cc: openembedded-core@lists.openembedded.org > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode > invalid paths for templates > > On Thu, 2022-09-22 at 17:52 +0200, Alexander Kanavin wrote: > > .templateconf is not the way to specify the location of custom > > templates, and never was. > > That much isn't true. It was a way this has been done, it was suggested > as a way people do it and it was even documented as such. The original > commit does make the intent clear: > > https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9 > > I think the key point is that it has been of limited use as it requires > a custom repo due to the use of OEROOT. Where you had a custom repo, it > was fine to use. Poky was one example. > > I haven't read back through the discussions but I suspect Peter must > have some kind of custom repo setup like that to be able to alter files > in OEROOT. Yes, and it is very simple to do when using `repo` like we do. Just add a <copyfile> line in the manifest that tells `repo` to copy a .templateconf file from a suitable repository and place it in the root. > > It got added to documentation by misunderstanding, > > Again, I'm not sure about that. I think at the time, when layers were > much newer things and technology was more in flux, we did think this > was a valid customisation. At the time we were suggesting people might > want to use tools like combo-layer to mess with repos. > > What the documentation doesn't make clear is that this trick only works > for repo alterations as far as I can see. See above. At least in the case of using `repo` to fetch the layers, this way of specifying the default template makes perfect sense. > Since we don't recommend such repo alterations now and combo-layer is > pretty dead, the code is of limited use and I do think it probably > should be removed. I therefore don't like adding complexity to > something which really shouldn't be there at all. I appreciate this > does looks like it creates a challenge for Peter's situation though. The expectation I have on Yocto is that whatever is fine and works for Poky should also be fine and work for anyone else setting up the environment in the same way. I.e., for Poky I expect the following: * Fetch the distribution (poky). In this case a simple git clone with the URL to the poky repository and the wanted version. * Change to the newly fetched directory. * Run `. ./ oe-init-build-env`. Which in our case corresponds to: * Create and change to a new directory. * Run `repo init` with the URL to the manifest repository, its version and the manifest to use, followed by `repo sync`. * Run `. ./ oe-init-build-env`. So, as long as the layers with their correct versions have been fetched, all I should have to do is run `. ./ oe-init-build-env`. That is what we have always done, and I am not going to change that. Changing that would be a huge undertaking for no gain to us. Now, the above works since I have modified our wrapper to create the template files in "templateconf/templates/axis" rather than the old "templateconf" and to create a dummy layer.conf file so the new requirements are satisfied. The problem is that because how the templateconf.cfg file is read, its value overrides the default in the .templateconf file and thus anyone updating an existing build tree to Langdale faces an error because the old templates path is unlikely to satisfy the new requirements. This is why the code in scripts/oe-setup-builddir was changed to remove the old templateconf.cfg file if it contains a known bad path. Since I of course do not want for my developers to face this error when they update, I have two options: 1) I can remove the old templateconf.cfg file in our wrapper. This means I have to duplicate the code in scripts/oe-buildenv-internal that calculates the BUILDDIR variable. It also means I will have to keep and maintain this code for the foreseeable future. 2) Change the code in scripts/oe-setup-builddir so that it is not hardcoded to only support OE and Poky, but instead is generic and works the same for anyone. From my point of view, option 2 should be the natural choice. > I have put off touching this particular area of code since I know there > will be a ton of problems exactly like this. The success or failure of > such changes does depend on how the migration is handled. A large part > of this is communication and to some extend negotiation too. The fact > I'm having to start trying to understand all the details is the > situation I didn't really want to be in. > > I suspect to get anywhere I now need to go and read all the previous > threads to then try and unravel this :/. > > Peter: is there a pointer you can give to the setup you're using? I gave an example in https://lists.openembedded.org/g/openembedded-core/message/170677. Is that what you were looking for? > > Cheers, > > Richard //Peter
On Tue, 27 Sept 2022 at 16:56, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > Since I of course do not want for my developers to face this > error when they update, I have two options: > > 1) I can remove the old templateconf.cfg file in our wrapper. > This means I have to duplicate the code in > scripts/oe-buildenv-internal that calculates the BUILDDIR > variable. It also means I will have to keep and maintain this > code for the foreseeable future. > 2) Change the code in scripts/oe-setup-builddir so that it is > not hardcoded to only support OE and Poky, but instead is > generic and works the same for anyone. > > From my point of view, option 2 should be the natural choice. Three options. You can also instruct your developers to simply remove their build directories. Is that such a catastrophe? We do not promise that build directories stay backwards compatible and never explode on distro upgrades, we try out best, but sometimes they just don't. Alex
Now it's my time to rant at you, Peter. Brace yourself. Let me just say this, you don't get to have expectations and entitlements, if none of your '500 developers' are working fulltime to help the core. I think you've been in the room last week when Nico and Philip presented the numbers and the downward trend of who is doing the work in core; what are you doing to address that situation, given the resources your organization has? We have a small handful of exhausted maintainers, I in particular have to squeeze my maintenance work between fixing various customer BSPs, and so it is on you to help us first. Otherwise, yes, I will say this: fix your scripts, and stop complaining. I have a large amount of manual recipe updates I need to do, because no one else wants to do it, and so I will not be arguing with you anymore. Alex On Tue, 27 Sept 2022 at 17:29, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > On Tue, 27 Sept 2022 at 16:56, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > Since I of course do not want for my developers to face this > > error when they update, I have two options: > > > > 1) I can remove the old templateconf.cfg file in our wrapper. > > This means I have to duplicate the code in > > scripts/oe-buildenv-internal that calculates the BUILDDIR > > variable. It also means I will have to keep and maintain this > > code for the foreseeable future. > > 2) Change the code in scripts/oe-setup-builddir so that it is > > not hardcoded to only support OE and Poky, but instead is > > generic and works the same for anyone. > > > > From my point of view, option 2 should be the natural choice. > > Three options. You can also instruct your developers to simply remove > their build directories. Is that such a catastrophe? We do not promise > that build directories stay backwards compatible and never explode on > distro upgrades, we try out best, but sometimes they just don't. > > Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#171107): https://lists.openembedded.org/g/openembedded-core/message/171107 > Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 27 september 2022 17:42 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core > <openembedded-core@lists.openembedded.org> > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode > invalid paths for templates > > Now it's my time to rant at you, Peter. Brace yourself. > > Let me just say this, you don't get to have expectations and > entitlements, if none of your '500 developers' are working fulltime to > help the core. I think you've been in the room last week when Nico and > Philip presented the numbers and the downward trend of who is doing > the work in core; what are you doing to address that situation, given > the resources your organization has? I (and the small team I am part of) is the proxy between them and Yocto. Thus I try to address their needs by making sure new versions of Yocto continues to work for them without having 500+ developers facing the same problems. I.e., it is my job to work with upstream, which is what I am trying to do here. > We have a small handful of exhausted maintainers, I in particular have > to squeeze my maintenance work between fixing various customer BSPs, > and so it is on you to help us first. Which is exactly what I am trying to do. I am trying to make sure that what has worked, continues to work, when it is not in opposition to the actual changes that you are making. You made this exact change (that my patch addresses) for OE and Poky because you realized that there was an error that could be avoided in a simple way. And I just want to extend that to be possible to do for all distributions besides Poky. This is typically something that affect those of us who have existing distributions and not users of the new way to setup the environment. > Otherwise, yes, I will say this: > fix your scripts, and stop complaining. I can (and will) of course do that, but since I believe my fix is of interest not only to us, but others who setup their environment using the .templateconf file, my primary goal is to try to improve upstream first. > I have a large amount of > manual recipe updates I need to do, because no one else wants to do > it, and so I will not be arguing with you anymore. > > Alex //Peter > On Tue, 27 Sept 2022 at 17:29, Alexander Kanavin via > lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> > wrote: > > > > On Tue, 27 Sept 2022 at 16:56, Peter Kjellerstedt > > <peter.kjellerstedt@axis.com> wrote: > > > Since I of course do not want for my developers to face this > > > error when they update, I have two options: > > > > > > 1) I can remove the old templateconf.cfg file in our wrapper. > > > This means I have to duplicate the code in > > > scripts/oe-buildenv-internal that calculates the BUILDDIR > > > variable. It also means I will have to keep and maintain this > > > code for the foreseeable future. > > > 2) Change the code in scripts/oe-setup-builddir so that it is > > > not hardcoded to only support OE and Poky, but instead is > > > generic and works the same for anyone. > > > > > > From my point of view, option 2 should be the natural choice. > > > > Three options. You can also instruct your developers to simply remove > > their build directories. Is that such a catastrophe? We do not promise > > that build directories stay backwards compatible and never explode on > > distro upgrades, we try out best, but sometimes they just don't. > > > > Alex > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#171107): > https://lists.openembedded.org/g/openembedded-core/message/171107 > > Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub > [alex.kanavin@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- > >
On Tue, 27 Sept 2022 at 18:06, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > > Otherwise, yes, I will say this: > > fix your scripts, and stop complaining. > > I can (and will) of course do that, but since I believe my fix is of > interest not only to us, but others who setup their environment > using the .templateconf file, my primary goal is to try to improve > upstream first. Yes, but: 1. There is no evidence anyone else except Axis is actually using .templateconf. 2. .templateconf is now officially retired as a way to set the template location. The patch you've sent is completely contrary to that: rather it makes the incorrect impression that it is still supported, and the implication is that the core upstream would still have to accommodate and think of those who are using it going forward. Yes it's mildly annoying to delete the build directory, but can you just do it and move on? And yes, make a plan for replacing it with TEMPLATECONF, which is what I've been trying to squeeze out of you for I don't know how many times. Alex
On Tue, 27 Sept 2022 at 18:06, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > I (and the small team I am part of) is the proxy between them and Yocto. > Thus I try to address their needs by making sure new versions of Yocto > continues to work for them without having 500+ developers facing the > same problems. I.e., it is my job to work with upstream, which is what > I am trying to do here. Excuse me, but your job is not to 'work with upstream'. Your job is to make damn sure 'upstream' continues to exist. Which means: go to your management, describe the situation we're in (you have the numbers), and ask that they allocate at least one developer to work in upstream full time. Not drive-by patches to fix your specific product problems, but upstream proper: something that makes the project sustainable, e.g. fixing medium+/high bugs, working on new features, upgrading recipes, sending pending patches upstream. This works never ends, and no one wants to do it. If you can do this, I promise to be more sympathetic to your particular issues, but as things stand, it's rather difficult. Alex
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 27 september 2022 18:17 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core <openembedded-core@lists.openembedded.org> > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates > > On Tue, 27 Sept 2022 at 18:06, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > > > Otherwise, yes, I will say this: > > > fix your scripts, and stop complaining. > > > > I can (and will) of course do that, but since I believe my fix is of > > interest not only to us, but others who setup their environment > > using the .templateconf file, my primary goal is to try to improve > > upstream first. > > Yes, but: > > 1. There is no evidence anyone else except Axis is actually using > .templateconf. Since we obviously use our own distribution, I am limited in experience with other distributions. The one example I do know about, which I have mentioned before, is WindRiver. See line 831-835 in https://github.com/WindRiver-Labs/wrlinux-x/blob/WRLINUX_10_21_BASE/bin/setup.py for an example of where they do basically the same as what we do. > > 2. .templateconf is now officially retired as a way to set the > template location. The patch you've sent is completely contrary to > that: rather it makes the incorrect impression that it is still > supported, and the implication is that the core upstream would still > have to accommodate and think of those who are using it going forward. Which is not entirely true, is it? Support for .templateconf I still there. I am only extending what is already there to work without errors for everyone, not just OE and Poky. > Yes it's mildly annoying to delete the build directory, but can you > just do it and move on? And yes, make a plan for replacing it with > TEMPLATECONF, which is what I've been trying to squeeze out of you for > I don't know how many times. > > Alex I do not understand what the problem with .templateconf is? There is a use case for being able to specify a default TEMPLATECONF, and that is what .templateconf satisfies. There are currently three ways to specify where the templates files are (in order of preference): 1) TEMPLATECONF= 2) templateconf.cfg 3) .templateconf In our case, the templates are always in the same location, thus we want to specify this as a default to give the developers a better user experience. With the above three methods available, this is easy as we just set the default in the .templateconf file. Now, if we remove support for .templateconf, how are we going to do it instead? I can of course change our wrapper around oe-build-init-env to do TEMPLATECONF=${TEMPLATECONF:-path/to/our/templates} and this might seem fine at first. However, this means that the wrapper will always set TEMPLATECONF and thus templateconf.cfg looses its meaning, which to the user means that he must now always specify TEMPLATECONF= if he wants to set it to something other than the default where before he only had to specify it the first time he initializes a new build environment. Or I can duplicate the calculations done in the OE scripts to figure out where BUILDDIR will end up to see if there is a templateconf.cfg file available, and then use that to set TEMPLATECONF if the user hasn't specified it. All of this duplicating upstream code, which may change with any future release. (Did I mention that I hate duplicating code?) Now I understand that you have some idea that the user should always specify TEMPLATECONF=, in which case a default TEMPLATECONF has no value. But since we have a method for fetching exactly the layers that are to be used in a given build, to me it does not make sense to also have to specify a TEMPLATECONF to match what has already been fetched. It is just duplicating information, which IMHO is bad. What _I_ would prefer to see, is a way to specify that the build shall use the available layers. I.e., rather than a huge number of static templates configurations, I would like to see the initialization script take the layers as arguments, with the default being anything that matches meta* (or maybe more correctly, anything that has a conf/layers.conf file). As for the local.conf file, in my world, this would be made up from fragment files in the layers. So for any layer that ends up in the bblayers.conf file, the layer would be searched for any local.conf.XX files that would be concatenated together in the order of XX. This is (obviously) very similar to what we do today, so I am of course biased, but I think this model is a lot more appealing than static templates files. YMMV. //Peter
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 27 september 2022 19:05 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core > <openembedded-core@lists.openembedded.org> > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode > invalid paths for templates > > On Tue, 27 Sept 2022 at 18:06, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > I (and the small team I am part of) is the proxy between them and Yocto. > > Thus I try to address their needs by making sure new versions of Yocto > > continues to work for them without having 500+ developers facing the > > same problems. I.e., it is my job to work with upstream, which is what > > I am trying to do here. > > Excuse me, but your job is not to 'work with upstream'. Your job is to > make damn sure 'upstream' continues to exist. Which means: go to your > management, describe the situation we're in (you have the numbers), Which is exactly why I have worked on and finally convinced my management that it is important for us to become a Yocto member, which we now are. > and ask that they allocate at least one developer to work in upstream > full time. That is harder. We have a hard enough time to recruit people to our tools team (which includes the build team that I am part of) as it is. For some reason, working with tools and build systems apparently is not as sexy as working with web development and AI. > Not drive-by patches to fix your specific product problems, I'm sorry to hear that you think all I do is drive-by patches. I thought my contributions were a little bit more appreciated than that. :( > but upstream proper: something that makes the project sustainable, > e.g. fixing medium+/high bugs, working on new features, upgrading > recipes, sending pending patches upstream. This works never ends, and > no one wants to do it. And I can assure you that everyone appreciate the ungrateful work you and others do to keep things running. We really do. > If you can do this, I promise to be more sympathetic to your > particular issues, but as things stand, it's rather difficult. > > Alex //Peter
On Tue, 27 Sept 2022 at 20:53, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > Since we obviously use our own distribution, I am limited in > experience with other distributions. The one example I do know about, > which I have mentioned before, is WindRiver. See line 831-835 in > https://github.com/WindRiver-Labs/wrlinux-x/blob/WRLINUX_10_21_BASE/bin/setup.py > for an example of where they do basically the same as what we do. Yes, they have to fix this too. And not by replacing .templateconf with TEMPLATECONF, but by writing a config directly, particularly using available tooling to compose bblayers.conf. Templates are intended for layers and version control, not as an extra intermediate step when setting up builds - the goal is to standardize build configuration management, and static templates is a starting point. > I do not understand what the problem with .templateconf is? There is > a use case for being able to specify a default TEMPLATECONF, and that > is what .templateconf satisfies. I'd say it does not satisfy the use case, as it relies on being next to oe-init-build-env, which in turn requires custom scripting. Show how to do this using standard tools in core/poky, write a patch for the official documentation, and then you have a point. > There are currently three ways to specify where the templates files > are (in order of preference): > > 1) TEMPLATECONF= > 2) templateconf.cfg > 3) .templateconf > > In our case, the templates are always in the same location, thus we > want to specify this as a default to give the developers a better > user experience. I get this, it is useful to make a pointer to a 'default' template when you have a set of layers, but the tooling for managing a modified .templateconf is absent. As things are, it is useful only as a fallback to the default core/poky template. And if you propose such tooling, it somehow has to avoid modifying any of the layers after they have been fetched. 'repo' allows you to do this, this doesn't mean it would be an acceptable thing to do in general. I did think about it, and I don't have good or even workable ideas. > With the above three methods available, this is easy as we just set > the default in the .templateconf file. > > Now, if we remove support for .templateconf, how are we going to do it > instead? I can of course change our wrapper around oe-build-init-env > to do TEMPLATECONF=${TEMPLATECONF:-path/to/our/templates} and this > might seem fine at first. However, this means that the wrapper will > always set TEMPLATECONF and thus templateconf.cfg looses its meaning, > which to the user means that he must now always specify TEMPLATECONF= > if he wants to set it to something other than the default where before > he only had to specify it the first time he initializes a new build > environment. I'm not sure I understand this. If the build environment has been initialized before from templates, values of TEMPLATECONF or templateconf.cfg become irrelevant, as the files in build/conf/ are already in place, and scripts will skip copying them over from templates. So you can continue to *not* have to specify TEMPLATECONF when you want to reuse an existing build directory without changing anything in it. First time: TEMPLATECONF=path/to/template . oe-init-build-env (will copy from template) Second+ time: . oe-init-build-env (will detect that no copying is needed, regardless of where the template is set to) Can you describe the sequence of commands where the problem arises? If there is a problem here I absolutely want to see it. > What _I_ would prefer to see, is a way to specify that the build > shall use the available layers. I.e., rather than a huge number of > static templates configurations, I would like to see the > initialization script take the layers as arguments, with the default > being anything that matches meta* (or maybe more correctly, anything > that has a conf/layers.conf file). As for the local.conf file, in > my world, this would be made up from fragment files in the layers. > So for any layer that ends up in the bblayers.conf file, the layer > would be searched for any local.conf.XX files that would be > concatenated together in the order of XX. This is (obviously) very > similar to what we do today, so I am of course biased, but I think > this model is a lot more appealing than static templates files. > YMMV. Yes, config fragments are the next step. Whether they be automatic like you prefer, or manual ('oe-setup-build add systemd') or both is to be determined. Alex
On Tue, 27 Sept 2022 at 21:22, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > > Not drive-by patches to fix your specific product problems, > > I'm sorry to hear that you think all I do is drive-by patches. > I thought my contributions were a little bit more appreciated > than that. :( They are appreciated like any contribution is, but let me clarify: anything that is not a part of a long term commitment to maintaining some piece is a drive-by patch. If you're not sure what needs a maintainer, let me remind you of that as well: https://git.yoctoproject.org/poky/tree/MAINTAINERS.md > > but upstream proper: something that makes the project sustainable, > > e.g. fixing medium+/high bugs, working on new features, upgrading > > recipes, sending pending patches upstream. This works never ends, and > > no one wants to do it. > > And I can assure you that everyone appreciate the ungrateful > work you and others do to keep things running. We really do. You should understand the gravity of my situation as an independent consultant, Peter. I constantly have to explain to my clients why fixing their custom internal crap (that benefits only them) is not always my top priority. Last week, I had to essentially blackmail a client that pays real money, saying that I will not do any further work for them, until they allow me to spend time upstream, and allow me to bill them for it. I am still waiting for an answer. The timing was poor as well, on the same day my friends and relatives in my home country started getting draft notices to what is likely to be useless deaths. Alex
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 27 september 2022 21:58 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core > <openembedded-core@lists.openembedded.org> > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode > invalid paths for templates > > On Tue, 27 Sept 2022 at 20:53, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > Since we obviously use our own distribution, I am limited in > > experience with other distributions. The one example I do know about, > > which I have mentioned before, is WindRiver. See line 831-835 in > > https://github.com/WindRiver-Labs/wrlinux- > x/blob/WRLINUX_10_21_BASE/bin/setup.py > > for an example of where they do basically the same as what we do. > > Yes, they have to fix this too. And not by replacing .templateconf > with TEMPLATECONF, but by writing a config directly, particularly > using available tooling to compose bblayers.conf. I am not too fond of the "writing the config directly" idea. The reason for that is that things change upstream (as you are well aware of, that's after all the whole reason for the discussion). This is why I prefer to go through the templates and basically only provide the list of layers to add. That way when things change upstream in the template files, we automatically follow. (And if it has not been obvious before, our wrapper uses the bblayers.conf.sample file from meta-poky and just replaces the list of layers with our list of layers. Which of course sounds like then we could just write the final config file directly, but we also want to continue to rely on oe-init-build-env for determining _where_ to write the final files.) > Templates are > intended for layers and version control, not as an extra intermediate > step when setting up builds - the goal is to standardize build > configuration management, and static templates is a starting point. > > > I do not understand what the problem with .templateconf is? There is > > a use case for being able to specify a default TEMPLATECONF, and that > > is what .templateconf satisfies. > > I'd say it does not satisfy the use case, as it relies on being next > to oe-init-build-env, which in turn requires custom scripting. Show > how to do this using standard tools in core/poky, write a patch for > the official documentation, and then you have a point. Well, up until now there have been no tools in OE/poky to do anything related to setting up the layers, so I will go by the tools that I know are actually being used. With repo and git submodules it is simple to setup the .templateconf file to match the distribution. I cannot speak for kas, as I have never used it, but I believe it uses its own way of doing things. I have tried whisk, though it was a while ago, but if I remember correctly, it also wrote the files itself. And, as of yet, I have not tried your new tools, but I plan on doing that as soon as I have the time, though I am pretty sure they do not rely on .templateconf. ;) > > > There are currently three ways to specify where the templates files > > are (in order of preference): > > > > 1) TEMPLATECONF= > > 2) templateconf.cfg > > 3) .templateconf > > > > In our case, the templates are always in the same location, thus we > > want to specify this as a default to give the developers a better > > user experience. > > I get this, it is useful to make a pointer to a 'default' template > when you have a set of layers, but the tooling for managing a modified > .templateconf is absent. As things are, it is useful only as a > fallback to the default core/poky template. Or anyone who sets up their distribution to mimic OE/Poky... > And if you propose such tooling, it somehow has to avoid modifying any > of the layers after they have been fetched. 'repo' allows you to do > this, this doesn't mean it would be an acceptable thing to do in > general. I did think about it, and I don't have good or even workable > ideas. > > > With the above three methods available, this is easy as we just set > > the default in the .templateconf file. > > > > Now, if we remove support for .templateconf, how are we going to do it > > instead? I can of course change our wrapper around oe-build-init-env > > to do TEMPLATECONF=${TEMPLATECONF:-path/to/our/templates} and this > > might seem fine at first. However, this means that the wrapper will > > always set TEMPLATECONF and thus templateconf.cfg looses its meaning, > > which to the user means that he must now always specify TEMPLATECONF= > > if he wants to set it to something other than the default where before > > he only had to specify it the first time he initializes a new build > > environment. > > I'm not sure I understand this. If the build environment has been > initialized before from templates, values of TEMPLATECONF or > templateconf.cfg become irrelevant, as the files in build/conf/ are > already in place, and scripts will skip copying them over from > templates. So you can continue to *not* have to specify TEMPLATECONF > when you want to reuse an existing build directory without changing > anything in it. > > First time: > TEMPLATECONF=path/to/template . oe-init-build-env (will copy from > template) > > Second+ time: > . oe-init-build-env (will detect that no copying is needed, regardless > of where the template is set to) > > Can you describe the sequence of commands where the problem arises? If > there is a problem here I absolutely want to see it. Well, the real problem is that since the path _is_ stored in templateconf.cfg and this path takes precedence over the path in .templateconf, it causes problems if it is not updated, due to the new requirements on the stored path, whether it is actually used or not. :( This is what my patch tries to address. However, having the path stored in templateconf.cfg is only really useful for anything if you delete one or more of the generated files so that they need to be regenerated. I assume that is why it was done, but I really do not know. We use this feature in a help script we have, where you can have it update the config files by giving it an --update option, in which case it just deletes the old files in the config directory and then relies on oe-init-build-env to regenerate them. We could of course have solved this some other way, but since the feature is there, why not make use of it? That said, having a simple way like that to update the config files is also something I would not mind seeing upstream. What we do in the local.conf is that we add a marker line at the end. Then when the user uses the --update option, we keep anything after that line but regenerate anything before it. So typically you add, e.g., MACHINE= at the end and then it is simple to update the file after an update to a new version of OE. This may not be the way local.conf is intended to be used, but I find it hard to use it if you actually uncomment any of the existing lines and then want to update the file to match a new version of OE. Thus I only use the existing lines as examples and do any modifications at the end. But as I said, this is a convenience script that some of our users use and not part of the wrapper we have for oe-init-build-env. > > What _I_ would prefer to see, is a way to specify that the build > > shall use the available layers. I.e., rather than a huge number of > > static templates configurations, I would like to see the > > initialization script take the layers as arguments, with the default > > being anything that matches meta* (or maybe more correctly, anything > > that has a conf/layers.conf file). As for the local.conf file, in > > my world, this would be made up from fragment files in the layers. > > So for any layer that ends up in the bblayers.conf file, the layer > > would be searched for any local.conf.XX files that would be > > concatenated together in the order of XX. This is (obviously) very > > similar to what we do today, so I am of course biased, but I think > > this model is a lot more appealing than static templates files. > > YMMV. > > Yes, config fragments are the next step. Whether they be automatic > like you prefer, or manual ('oe-setup-build add systemd') or both is > to be determined. > > Alex //Peter
On Tue, 27 Sept 2022 at 23:31, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > I am not too fond of the "writing the config directly" idea. The reason > for that is that things change upstream (as you are well aware of, that's > after all the whole reason for the discussion). This is why I prefer to > go through the templates and basically only provide the list of layers > to add. That way when things change upstream in the template files, we > automatically follow. (And if it has not been obvious before, our wrapper > uses the bblayers.conf.sample file from meta-poky and just replaces the > list of layers with our list of layers. Which of course sounds like then > we could just write the final config file directly, but we also want to > continue to rely on oe-init-build-env for determining _where_ to write > the final files.) Wait, wait, wait. If so, you can initialize using the template from poky, then you get BBPATH or BUILDDIR placed in the environment (by oe-init-build-env) telling where the config files are. So you can write to them directly (e.g. local.conf), or use 'bitbake-layers add-layer' (or 'remove-layer') to tweak the list of layers using supported tooling - after the initialization, but before any builds. No? Alex
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 27 september 2022 23:50 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core > <openembedded-core@lists.openembedded.org> > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode > invalid paths for templates > > On Tue, 27 Sept 2022 at 23:31, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > I am not too fond of the "writing the config directly" idea. The reason > > for that is that things change upstream (as you are well aware of, that's > > after all the whole reason for the discussion). This is why I prefer to > > go through the templates and basically only provide the list of layers > > to add. That way when things change upstream in the template files, we > > automatically follow. (And if it has not been obvious before, our wrapper > > uses the bblayers.conf.sample file from meta-poky and just replaces the > > list of layers with our list of layers. Which of course sounds like then > > we could just write the final config file directly, but we also want to > > continue to rely on oe-init-build-env for determining _where_ to write > > the final files.) > > Wait, wait, wait. If so, you can initialize using the template from > poky, then you get BBPATH or BUILDDIR placed in the environment (by > oe-init-build-env) telling where the config files are. So you can > write to them directly (e.g. local.conf), or use 'bitbake-layers > add-layer' (or 'remove-layer') to tweak the list of layers using > supported tooling - after the initialization, but before any builds. > No? > > Alex I am not sure if you are suggesting that the user shall run `bitbake-layers add-layer` after sourcing oe-init-build-env, or if you mean that our wrapper should do it. If it is the former, then that is obviously not how we want things to work. The layers that are fetched shall by default be part of the build. That is an absolute requirement. There is no expectancy (or want) that the user shall have to add/remove the layers manually. If it is the latter, it wouldn't work (unless I yet again duplicate the code to figure out the build directory before sourcing the real oe-init-build-env) since before sourcing the real oe-init-build-env I cannot tell if there already is a bblayers.conf file in which case the wrapper should not add/remove any layers, and after sourcing the real oe-init-build-env it is too late as I then cannot distinguish between this being the first time oe-init-build-env is sourced or just a reinitialization of an already existing environment. Our philosophy when we originally set this up was to keep it simple and as close to how OE/Poky works as possible. That way, if the developers have any questions and look at the Yocto documentation, the commands suggested there should work the same for us. And that is what we get from using the templates and letting the init scripts provided by OE do the work. The fact that we use the bblayers.conf.sample file from meta-poky rather than providing our own shouldn't change that. It is just a way to avoid having to manually keep our sample file in sync with what happens in OE Core. //Peter
On Wed, 28 Sept 2022 at 11:54, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > If it is the latter, it wouldn't work (unless I yet again duplicate the > code to figure out the build directory before sourcing the real > oe-init-build-env) since before sourcing the real oe-init-build-env I > cannot tell if there already is a bblayers.conf file in which case the > wrapper should not add/remove any layers, and after sourcing the real > oe-init-build-env it is too late as I then cannot distinguish between > this being the first time oe-init-build-env is sourced or just a > reinitialization of an already existing environment. Yes this is the scenario I was thinking of: what should happen inside the wrapper: . oe-init-build-env (this takes the original poky template) bitbake-layers add-layer meta-a bitbake-layers add-layer meta-b ... Why would you want to distinguish between whether this is against a newly made dir or an older one? I just checked: adding a layer that is already in bblayers.conf does nothing. I'm also unsure why are you so adamantly against checking if the build dir already exists before the fact. It's not that difficult (either it's relative to `pwd` or an absolute path), and not likely to change upstream. Alex
On Wed, 28 Sept 2022 at 13:00, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > I'm also unsure why are you so adamantly against checking if the build > dir already exists before the fact. It's not that difficult (either > it's relative to `pwd` or an absolute path), and not likely to change > upstream. Which is a trivial one-liner in python: os.path.exists(os.path.abspath(build_dir)) Alex
> -----Original Message----- > From: Alexander Kanavin <alex.kanavin@gmail.com> > Sent: den 28 september 2022 13:00 > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core > <openembedded-core@lists.openembedded.org> > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode > invalid paths for templates > > On Wed, 28 Sept 2022 at 11:54, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > If it is the latter, it wouldn't work (unless I yet again duplicate the > > code to figure out the build directory before sourcing the real > > oe-init-build-env) since before sourcing the real oe-init-build-env I > > cannot tell if there already is a bblayers.conf file in which case the > > wrapper should not add/remove any layers, and after sourcing the real > > oe-init-build-env it is too late as I then cannot distinguish between > > this being the first time oe-init-build-env is sourced or just a > > reinitialization of an already existing environment. > > Yes this is the scenario I was thinking of: what should happen inside > the wrapper: > > . oe-init-build-env (this takes the original poky template) > bitbake-layers add-layer meta-a > bitbake-layers add-layer meta-b > ... > > Why would you want to distinguish between whether this is against a > newly made dir or an older one? I just checked: adding a layer that > is already in bblayers.conf does nothing. Because, while we want the layers fetched to be present in bblayers.conf by default, it is still expected that developers during development can add and remove layers. And if a developer has, e.g., removed a layer, it is not expected that sourcing oe-init-build-env again should restore the removed layer. > > I'm also unsure why are you so adamantly against checking if the build > dir already exists before the fact. It's not that difficult (either > it's relative to `pwd` or an absolute path), and not likely to change > upstream. Since our wrapper is expected to behave as oe-init-build-env it must also calculate the build directory as oe-init-build-env would. And no, it is not hard to do, but it means code duplication and that we are susceptible to upstream changes. It is not a major problem and it is the way I will solve this if I cannot get a change accepted upstream, but every such small addition adds to the maintenance burden. > Alex //Peter
On Wed, 28 Sept 2022 at 14:45, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > Because, while we want the layers fetched to be present in bblayers.conf > by default, it is still expected that developers during development can > add and remove layers. And if a developer has, e.g., removed a layer, it > is not expected that sourcing oe-init-build-env again should restore the > removed layer. I wonder if we can somehow make oe-init-build-env indicate (in a way that is programmatically robust, e.g. perhaps not a marker in stdout) that it has initialized a new build directory. Ideas? > Since our wrapper is expected to behave as oe-init-build-env it must > also calculate the build directory as oe-init-build-env would. And no, > it is not hard to do, but it means code duplication and that we are > susceptible to upstream changes. It is not a major problem and it is > the way I will solve this if I cannot get a change accepted upstream, > but every such small addition adds to the maintenance burden. All you need is: os.path.exists(os.path.abspath(build_dir)) (probably abspath is unneeded either, it's just there to make clear what the full path is in case it does not exist) And no, that logic is not going to change upstream - it's far too basic and pervasive. Tweaking the list of layers by directly manipulating the config file content when there are tools that do this for you is, I think, a worse example of duplication. Alex
diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir index 225919be92..c1148daf4f 100755 --- a/scripts/oe-setup-builddir +++ b/scripts/oe-setup-builddir @@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!" -if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then - TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg") - # The following two are no longer valid; unsetting them will automatically get them replaced - # with correct ones. - if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then - unset TEMPLATECONF - rm "$BUILDDIR/conf/templateconf.cfg" - fi +TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg" +if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then + TEMPLATECONF=$(cat "$TEMPLATECONF_CFG") + # Unset TEMPLATECONF if it is set to a known invalid value to have it + # automatically replaced with a correct one. + for dir in $INVALID_TEMPLATECONFS; do + if [ "$TEMPLATECONF" = "$dir" ]; then + echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'" + unset TEMPLATECONF + rm "$TEMPLATECONF_CFG" + break; + fi + done fi . "$OEROOT/.templateconf"
Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded as invalid paths for templates. However, this is suboptimal for other distros that are setup similarly to Poky. Instead, add support for a new variable INVALID_TEMPLATECONFS, which takes a list of invalid paths. It is expected that this variable is set in the .templateconf file together with the default value for TEMPLATECONF, typically at the same time that TEMPLATECONF is updated to match the new requirements. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> --- scripts/oe-setup-builddir | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)