| Message ID | 20230130152000.705123-1-jose.quaresma@foundries.io |
|---|---|
| State | Rejected |
| Delegated to: | Ryan Eatmon |
| Headers | show |
| Series | [meta-ti,master/kirkstone] bsp: multiconfig: k3r5: allow customization by donwstream layers | expand |
On 1/30/2023 9:20, Jose Quaresma wrote: > This patch adds the possibility to change some bitbake variable > that cannot be changed anywhere else, one of then is the TMPDIR. > > Using the same TMPDIR it is very sensitive and prone to several errors > when used in more complex situations. > This configuration forces that all native packages have to be the same between all > machines and this requirement is very easy to break. > Suppose you use a macinhe override somewhere on a native recipe > and this requirement is no longer met. > Many of these anomalies can be verified with the use of the rm_work and > create-spdx bitbake classes. > > A previous attempt [1] had already been made but was refused > but this way it can be done without side effects for other users. > > [1] https://lists.yoctoproject.org/g/meta-ti/message/14767 So the difference between this patch and the previous one, is that you are now included a file that is not there normally so that you can put the contents of the previous patch in the new include file to get around the issue in the archiver. I'm not saying I'm opposed to this approach (yet), I'm just trying to understand the entire story. Did you find anything in looking at fixing the archiver to better support the multi-config issue? > Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> > --- > meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf b/meta-ti-bsp/conf/multiconfig/k3r5.conf > index deb07210..b69c5d6d 100644 > --- a/meta-ti-bsp/conf/multiconfig/k3r5.conf > +++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf > @@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}" > DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}" > > MACHINE:append = "-k3r5" > + > +include conf/multiconfig/k3r5.inc > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#15699): https://lists.yoctoproject.org/g/meta-ti/message/15699 > Mute This Topic: https://lists.yoctoproject.org/mt/96629864/6551054 > Group Owner: meta-ti+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/unsub [reatmon@ti.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Mon, Jan 30, 2023 at 11:02:25AM -0600, Ryan Eatmon via lists.yoctoproject.org wrote: > > > On 1/30/2023 9:20, Jose Quaresma wrote: > >This patch adds the possibility to change some bitbake variable > >that cannot be changed anywhere else, one of then is the TMPDIR. > > > >Using the same TMPDIR it is very sensitive and prone to several errors > >when used in more complex situations. > >This configuration forces that all native packages have to be the same between all > >machines and this requirement is very easy to break. > >Suppose you use a macinhe override somewhere on a native recipe Even w/o multiconfig, this will break for multi-machine builds. Native packages are for the build host and have no knowledge about the target. You should probably look into cross packages instead of native ones for this use case. Altering native packages between different targets will cause them to rebuild, which they shouldn't. Moreover, it will mess up your sstate and lead to weird issues down the road when trying to re-use your sstate mirror/cache. > >and this requirement is no longer met. > >Many of these anomalies can be verified with the use of the rm_work and > >create-spdx bitbake classes. > > > >A previous attempt [1] had already been made but was refused > >but this way it can be done without side effects for other users. > > > >[1] https://lists.yoctoproject.org/g/meta-ti/message/14767 > > > So the difference between this patch and the previous one, is that > you are now included a file that is not there normally so that you > can put the contents of the previous patch in the new include file > to get around the issue in the archiver. > > I'm not saying I'm opposed to this approach (yet), I'm just trying > to understand the entire story. > > Did you find anything in looking at fixing the archiver to better > support the multi-config issue? > > > > >Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> > >--- > > meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++ > > 1 file changed, 2 insertions(+) > > > >diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf b/meta-ti-bsp/conf/multiconfig/k3r5.conf > >index deb07210..b69c5d6d 100644 > >--- a/meta-ti-bsp/conf/multiconfig/k3r5.conf > >+++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf > >@@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}" > > DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}" > > MACHINE:append = "-k3r5" > >+ > >+include conf/multiconfig/k3r5.inc
Denys Dmytriyenko <denis@denix.org> escreveu no dia segunda, 30/01/2023 à(s) 17:18: > On Mon, Jan 30, 2023 at 11:02:25AM -0600, Ryan Eatmon via > lists.yoctoproject.org wrote: > > > > > > On 1/30/2023 9:20, Jose Quaresma wrote: > > >This patch adds the possibility to change some bitbake variable > > >that cannot be changed anywhere else, one of then is the TMPDIR. > > > > > >Using the same TMPDIR it is very sensitive and prone to several errors > > >when used in more complex situations. > > >This configuration forces that all native packages have to be the same > between all > > >machines and this requirement is very easy to break. > > >Suppose you use a macinhe override somewhere on a native recipe > > Even w/o multiconfig, this will break for multi-machine builds. Native > packages are for the build host and have no knowledge about the target. > You should probably look into cross packages instead of native ones for > this use case. > > Altering native packages between different targets will cause them to > rebuild, which they shouldn't. Moreover, it will mess up your sstate > and lead to weird issues down the road when trying to re-use your > sstate mirror/cache. > Is that the issue of rebuilding some native packages that I am trying to avoid. But it is very hard to fix this in OE-core and everything else layers what happens. For your information I have some other issues with multiconfig and native packages that have the source files provided on the layer. this two recipes don't work with the rm_work bbclass because the in one of the machines texinfo-dummy-native gettext-minimal-native Another issue is that sometimes one of the machines uses the SOURCE_DATE_EPOCH_FALLBACK and the other uses SOURCE_DATE_EPOCH and this also triggers a rebuild of one of the machines. Recently a new issue with the rm_work and the spdx comes in https://lists.openembedded.org/g/openembedded-core/message/174276 where I have the steps to reproduce. All of these issues can be solved easily with TMDIR for each machine, but because I understand it can be invasive I submit this patch that doesn't change anything in meta-ti but can add the possibility to use a different TMPDIR for each machine. A different TMPDIR for each machine is also referenced in the official documentation "Minimally, each configuration file must define the machine and the temporary directory BitBake uses for the build. Suggested practice dictates that you do not overlap the temporary directories used during the builds." https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build > > > >and this requirement is no longer met. > > >Many of these anomalies can be verified with the use of the rm_work and > > >create-spdx bitbake classes. > > > > > >A previous attempt [1] had already been made but was refused > > >but this way it can be done without side effects for other users. > > > > > >[1] https://lists.yoctoproject.org/g/meta-ti/message/14767 > > > > > > So the difference between this patch and the previous one, is that > > you are now included a file that is not there normally so that you > > can put the contents of the previous patch in the new include file > > to get around the issue in the archiver. > The archiver issue is fixed but it will be very easy to break again. Myself recently broke it again so I added a test to help there https://git.yoctoproject.org/poky/commit/?id=b5baa7dc8bd1c0d2d78c532d97a44120346b5edd > > > > I'm not saying I'm opposed to this approach (yet), I'm just trying > > to understand the entire story. > > > > Did you find anything in looking at fixing the archiver to better > > support the multi-config issue? > The issue now is different and is related with spdx and rm_work https://lists.openembedded.org/g/openembedded-core/message/174276 With this patch I can set a new TMPDIR folder for each machine and can build successfully. > > > > > > > > >Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> > > >--- > > > meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++ > > > 1 file changed, 2 insertions(+) > > > > > >diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf > b/meta-ti-bsp/conf/multiconfig/k3r5.conf > > >index deb07210..b69c5d6d 100644 > > >--- a/meta-ti-bsp/conf/multiconfig/k3r5.conf > > >+++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf > > >@@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}" > > > DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}" > > > MACHINE:append = "-k3r5" > > >+ > > >+include conf/multiconfig/k3r5.inc > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#15702): > https://lists.yoctoproject.org/g/meta-ti/message/15702 > Mute This Topic: https://lists.yoctoproject.org/mt/96629864/5052612 > Group Owner: meta-ti+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Mon, Jan 30, 2023 at 2:57 PM Jose Quaresma <quaresma.jose@gmail.com> wrote: > Denys Dmytriyenko <denis@denix.org> escreveu no dia segunda, 30/01/2023 à(s) 17:18: >> >> On Mon, Jan 30, 2023 at 11:02:25AM -0600, Ryan Eatmon via lists.yoctoproject.org wrote: >> > >> > >> > On 1/30/2023 9:20, Jose Quaresma wrote: >> > >This patch adds the possibility to change some bitbake variable >> > >that cannot be changed anywhere else, one of then is the TMPDIR. >> > > >> > >Using the same TMPDIR it is very sensitive and prone to several errors >> > >when used in more complex situations. >> > >This configuration forces that all native packages have to be the same between all >> > >machines and this requirement is very easy to break. >> > >Suppose you use a macinhe override somewhere on a native recipe >> >> Even w/o multiconfig, this will break for multi-machine builds. Native >> packages are for the build host and have no knowledge about the target. >> You should probably look into cross packages instead of native ones for >> this use case. >> >> Altering native packages between different targets will cause them to >> rebuild, which they shouldn't. Moreover, it will mess up your sstate >> and lead to weird issues down the road when trying to re-use your >> sstate mirror/cache. > > Is that the issue of rebuilding some native packages that I am trying to avoid. > But it is very hard to fix this in OE-core and everything else layers what happens. > > For your information I have some other issues with multiconfig and native packages > that have the source files provided on the layer. this two recipes don't work with the > rm_work bbclass because the in one of the machines > texinfo-dummy-native gettext-minimal-native > > Another issue is that sometimes one of the machines uses the SOURCE_DATE_EPOCH_FALLBACK > and the other uses SOURCE_DATE_EPOCH and this also triggers a rebuild of one of the machines. > > Recently a new issue with the rm_work and the spdx > comes in https://lists.openembedded.org/g/openembedded-core/message/174276 > where I have the steps to reproduce. > > All of these issues can be solved easily with TMDIR for each machine, > but because I understand it can be invasive I submit this patch that doesn't change anything > in meta-ti but can add the possibility to use a different TMPDIR for each machine. > > A different TMPDIR for each machine is also referenced in the official documentation > "Minimally, each configuration file must define the machine and the temporary directory BitBake uses for the build. Suggested practice dictates that you do not overlap the temporary directories used during the builds." > https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build > >> > >and this requirement is no longer met. >> > >Many of these anomalies can be verified with the use of the rm_work and >> > >create-spdx bitbake classes. >> > > >> > >A previous attempt [1] had already been made but was refused >> > >but this way it can be done without side effects for other users. >> > > >> > >[1] https://lists.yoctoproject.org/g/meta-ti/message/14767 >> > >> > >> > So the difference between this patch and the previous one, is that >> > you are now included a file that is not there normally so that you >> > can put the contents of the previous patch in the new include file >> > to get around the issue in the archiver. > > The archiver issue is fixed but it will be very easy to break again. > Myself recently broke it again so I added a test to help there > https://git.yoctoproject.org/poky/commit/?id=b5baa7dc8bd1c0d2d78c532d97a44120346b5edd > >> > >> > I'm not saying I'm opposed to this approach (yet), I'm just trying >> > to understand the entire story. >> > >> > Did you find anything in looking at fixing the archiver to better >> > support the multi-config issue? > > The issue now is different and is related with spdx and rm_work > https://lists.openembedded.org/g/openembedded-core/message/174276 > > With this patch I can set a new TMPDIR folder for each machine and > can build successfully. If you don't have spdx enabled then these sort of issues happen less often, but after the above patch was merged in oe-core (and also backported to kirkstone), we cannot build multiconfig targets from meta-ti anymore with spdx enabled. This is special to spdx because of the way the task dependencies are done, which requires it to be executed before rm_work. This is OK when a machine specific TMPDIR is used and explodes quite quickly (due races) when sharing TMPDIR because of the parallel rm_work tasks that end up removing stuff that is still required by other parallel tasks. Cheers,
Just for your information there are more complaints about that issue [1] of some bad side effects of sharing the TMP folder in multi configs. [1] https://lists.openembedded.org/g/bitbake-devel/message/14860 Jose Ricardo Salveti <ricardo@foundries.io> escreveu no dia segunda, 30/01/2023 à(s) 18:27: > On Mon, Jan 30, 2023 at 2:57 PM Jose Quaresma <quaresma.jose@gmail.com> > wrote: > > Denys Dmytriyenko <denis@denix.org> escreveu no dia segunda, 30/01/2023 > à(s) 17:18: > >> > >> On Mon, Jan 30, 2023 at 11:02:25AM -0600, Ryan Eatmon via > lists.yoctoproject.org wrote: > >> > > >> > > >> > On 1/30/2023 9:20, Jose Quaresma wrote: > >> > >This patch adds the possibility to change some bitbake variable > >> > >that cannot be changed anywhere else, one of then is the TMPDIR. > >> > > > >> > >Using the same TMPDIR it is very sensitive and prone to several > errors > >> > >when used in more complex situations. > >> > >This configuration forces that all native packages have to be the > same between all > >> > >machines and this requirement is very easy to break. > >> > >Suppose you use a macinhe override somewhere on a native recipe > >> > >> Even w/o multiconfig, this will break for multi-machine builds. Native > >> packages are for the build host and have no knowledge about the target. > >> You should probably look into cross packages instead of native ones for > >> this use case. > >> > >> Altering native packages between different targets will cause them to > >> rebuild, which they shouldn't. Moreover, it will mess up your sstate > >> and lead to weird issues down the road when trying to re-use your > >> sstate mirror/cache. > > > > Is that the issue of rebuilding some native packages that I am trying to > avoid. > > But it is very hard to fix this in OE-core and everything else layers > what happens. > > > > For your information I have some other issues with multiconfig and > native packages > > that have the source files provided on the layer. this two recipes don't > work with the > > rm_work bbclass because the in one of the machines > > texinfo-dummy-native gettext-minimal-native > > > > Another issue is that sometimes one of the machines uses the > SOURCE_DATE_EPOCH_FALLBACK > > and the other uses SOURCE_DATE_EPOCH and this also triggers a rebuild > of one of the machines. > > > > Recently a new issue with the rm_work and the spdx > > comes in > https://lists.openembedded.org/g/openembedded-core/message/174276 > > where I have the steps to reproduce. > > > > All of these issues can be solved easily with TMDIR for each machine, > > but because I understand it can be invasive I submit this patch that > doesn't change anything > > in meta-ti but can add the possibility to use a different TMPDIR for > each machine. > > > > A different TMPDIR for each machine is also referenced in the official > documentation > > "Minimally, each configuration file must define the machine and the > temporary directory BitBake uses for the build. Suggested practice dictates > that you do not overlap the temporary directories used during the builds." > > > https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build > > > >> > >and this requirement is no longer met. > >> > >Many of these anomalies can be verified with the use of the rm_work > and > >> > >create-spdx bitbake classes. > >> > > > >> > >A previous attempt [1] had already been made but was refused > >> > >but this way it can be done without side effects for other users. > >> > > > >> > >[1] https://lists.yoctoproject.org/g/meta-ti/message/14767 > >> > > >> > > >> > So the difference between this patch and the previous one, is that > >> > you are now included a file that is not there normally so that you > >> > can put the contents of the previous patch in the new include file > >> > to get around the issue in the archiver. > > > > The archiver issue is fixed but it will be very easy to break again. > > Myself recently broke it again so I added a test to help there > > > https://git.yoctoproject.org/poky/commit/?id=b5baa7dc8bd1c0d2d78c532d97a44120346b5edd > > > >> > > >> > I'm not saying I'm opposed to this approach (yet), I'm just trying > >> > to understand the entire story. > >> > > >> > Did you find anything in looking at fixing the archiver to better > >> > support the multi-config issue? > > > > The issue now is different and is related with spdx and rm_work > > https://lists.openembedded.org/g/openembedded-core/message/174276 > > > > With this patch I can set a new TMPDIR folder for each machine and > > can build successfully. > > If you don't have spdx enabled then these sort of issues happen less > often, but after the above patch was merged in oe-core (and also > backported to kirkstone), we cannot build multiconfig targets from > meta-ti anymore with spdx enabled. > > This is special to spdx because of the way the task dependencies are > done, which requires it to be executed before rm_work. This is OK when > a machine specific TMPDIR is used and explodes quite quickly (due > races) when sharing TMPDIR because of the parallel rm_work tasks that > end up removing stuff that is still required by other parallel tasks. > > Cheers, > -- > Ricardo Salveti >
As you noted yourself in that thread, your patches were workarounds and just masking the actual issue w/o clearly articulating it. The actual race issue has been under investigation for quite some time and I've discussed it with Richard before. Luckily he was finally able to root-cause and fix it! BTW, I'm working on a change that I'm about to submit here, which indirectly gives you separate TMPDIR for k3r5 - stay tuned. On Wed, Jul 05, 2023 at 09:17:44AM +0100, Jose Quaresma wrote: > Just for your information there are more complaints about that issue [1] of > some bad side effects of sharing the TMP folder in multi configs. > > [1] https://lists.openembedded.org/g/bitbake-devel/message/14860 > > Jose > > Ricardo Salveti <ricardo@foundries.io> escreveu no dia segunda, 30/01/2023 > à(s) 18:27: > > > On Mon, Jan 30, 2023 at 2:57 PM Jose Quaresma <quaresma.jose@gmail.com> > > wrote: > > > Denys Dmytriyenko <denis@denix.org> escreveu no dia segunda, 30/01/2023 > > à(s) 17:18: > > >> > > >> On Mon, Jan 30, 2023 at 11:02:25AM -0600, Ryan Eatmon via > > lists.yoctoproject.org wrote: > > >> > > > >> > > > >> > On 1/30/2023 9:20, Jose Quaresma wrote: > > >> > >This patch adds the possibility to change some bitbake variable > > >> > >that cannot be changed anywhere else, one of then is the TMPDIR. > > >> > > > > >> > >Using the same TMPDIR it is very sensitive and prone to several > > errors > > >> > >when used in more complex situations. > > >> > >This configuration forces that all native packages have to be the > > same between all > > >> > >machines and this requirement is very easy to break. > > >> > >Suppose you use a macinhe override somewhere on a native recipe > > >> > > >> Even w/o multiconfig, this will break for multi-machine builds. Native > > >> packages are for the build host and have no knowledge about the target. > > >> You should probably look into cross packages instead of native ones for > > >> this use case. > > >> > > >> Altering native packages between different targets will cause them to > > >> rebuild, which they shouldn't. Moreover, it will mess up your sstate > > >> and lead to weird issues down the road when trying to re-use your > > >> sstate mirror/cache. > > > > > > Is that the issue of rebuilding some native packages that I am trying to > > avoid. > > > But it is very hard to fix this in OE-core and everything else layers > > what happens. > > > > > > For your information I have some other issues with multiconfig and > > native packages > > > that have the source files provided on the layer. this two recipes don't > > work with the > > > rm_work bbclass because the in one of the machines > > > texinfo-dummy-native gettext-minimal-native > > > > > > Another issue is that sometimes one of the machines uses the > > SOURCE_DATE_EPOCH_FALLBACK > > > and the other uses SOURCE_DATE_EPOCH and this also triggers a rebuild > > of one of the machines. > > > > > > Recently a new issue with the rm_work and the spdx > > > comes in > > https://lists.openembedded.org/g/openembedded-core/message/174276 > > > where I have the steps to reproduce. > > > > > > All of these issues can be solved easily with TMDIR for each machine, > > > but because I understand it can be invasive I submit this patch that > > doesn't change anything > > > in meta-ti but can add the possibility to use a different TMPDIR for > > each machine. > > > > > > A different TMPDIR for each machine is also referenced in the official > > documentation > > > "Minimally, each configuration file must define the machine and the > > temporary directory BitBake uses for the build. Suggested practice dictates > > that you do not overlap the temporary directories used during the builds." > > > > > https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build > > > > > >> > >and this requirement is no longer met. > > >> > >Many of these anomalies can be verified with the use of the rm_work > > and > > >> > >create-spdx bitbake classes. > > >> > > > > >> > >A previous attempt [1] had already been made but was refused > > >> > >but this way it can be done without side effects for other users. > > >> > > > > >> > >[1] https://lists.yoctoproject.org/g/meta-ti/message/14767 > > >> > > > >> > > > >> > So the difference between this patch and the previous one, is that > > >> > you are now included a file that is not there normally so that you > > >> > can put the contents of the previous patch in the new include file > > >> > to get around the issue in the archiver. > > > > > > The archiver issue is fixed but it will be very easy to break again. > > > Myself recently broke it again so I added a test to help there > > > > > https://git.yoctoproject.org/poky/commit/?id=b5baa7dc8bd1c0d2d78c532d97a44120346b5edd > > > > > >> > > > >> > I'm not saying I'm opposed to this approach (yet), I'm just trying > > >> > to understand the entire story. > > >> > > > >> > Did you find anything in looking at fixing the archiver to better > > >> > support the multi-config issue? > > > > > > The issue now is different and is related with spdx and rm_work > > > https://lists.openembedded.org/g/openembedded-core/message/174276 > > > > > > With this patch I can set a new TMPDIR folder for each machine and > > > can build successfully. > > > > If you don't have spdx enabled then these sort of issues happen less > > often, but after the above patch was merged in oe-core (and also > > backported to kirkstone), we cannot build multiconfig targets from > > meta-ti anymore with spdx enabled. > > > > This is special to spdx because of the way the task dependencies are > > done, which requires it to be executed before rm_work. This is OK when > > a machine specific TMPDIR is used and explodes quite quickly (due > > races) when sharing TMPDIR because of the parallel rm_work tasks that > > end up removing stuff that is still required by other parallel tasks.
Denys Dmytriyenko <denis@denix.org> escreveu no dia quinta, 6/07/2023 à(s) 19:39: > As you noted yourself in that thread, your patches were workarounds and > just > masking the actual issue w/o clearly articulating it. The actual race > issue > has been under investigation for quite some time and I've discussed it > with > Richard before. Luckily he was finally able to root-cause and fix it! > I have reverted my hack to unshare the tmp directory in my distro when using the oe-core master branch after the recent spdx changes [1] and everything seems to be okay on my tests. The Richard bitbake multiconfig patch to fix the licence collision is for sure another help to use the shared tmp folder. However on kisstone I still use the tmp unshared between machines in multiconfig because the problems were many and I still don't feel safe to revert to the method used on meta-ti kirkstone. [1] https://git.yoctoproject.org/poky/commit/meta/classes/create-spdx-2.2.bbclass?id=42071227f6f54055d8ac44126ab1d95f83f5b264 > BTW, I'm working on a change that I'm about to submit here, which > indirectly > gives you separate TMPDIR for k3r5 - stay tuned. > I'm looking forward to it because the hack I'm using is really ugly and I'd like to drop it also on the kirkstone branch. Cheers, Jose > > > On Wed, Jul 05, 2023 at 09:17:44AM +0100, Jose Quaresma wrote: > > Just for your information there are more complaints about that issue [1] > of > > some bad side effects of sharing the TMP folder in multi configs. > > > > [1] https://lists.openembedded.org/g/bitbake-devel/message/14860 > > > > Jose > > > > Ricardo Salveti <ricardo@foundries.io> escreveu no dia segunda, > 30/01/2023 > > à(s) 18:27: > > > > > On Mon, Jan 30, 2023 at 2:57 PM Jose Quaresma <quaresma.jose@gmail.com > > > > > wrote: > > > > Denys Dmytriyenko <denis@denix.org> escreveu no dia segunda, > 30/01/2023 > > > à(s) 17:18: > > > >> > > > >> On Mon, Jan 30, 2023 at 11:02:25AM -0600, Ryan Eatmon via > > > lists.yoctoproject.org wrote: > > > >> > > > > >> > > > > >> > On 1/30/2023 9:20, Jose Quaresma wrote: > > > >> > >This patch adds the possibility to change some bitbake variable > > > >> > >that cannot be changed anywhere else, one of then is the TMPDIR. > > > >> > > > > > >> > >Using the same TMPDIR it is very sensitive and prone to several > > > errors > > > >> > >when used in more complex situations. > > > >> > >This configuration forces that all native packages have to be the > > > same between all > > > >> > >machines and this requirement is very easy to break. > > > >> > >Suppose you use a macinhe override somewhere on a native recipe > > > >> > > > >> Even w/o multiconfig, this will break for multi-machine builds. > Native > > > >> packages are for the build host and have no knowledge about the > target. > > > >> You should probably look into cross packages instead of native ones > for > > > >> this use case. > > > >> > > > >> Altering native packages between different targets will cause them > to > > > >> rebuild, which they shouldn't. Moreover, it will mess up your sstate > > > >> and lead to weird issues down the road when trying to re-use your > > > >> sstate mirror/cache. > > > > > > > > Is that the issue of rebuilding some native packages that I am > trying to > > > avoid. > > > > But it is very hard to fix this in OE-core and everything else layers > > > what happens. > > > > > > > > For your information I have some other issues with multiconfig and > > > native packages > > > > that have the source files provided on the layer. this two recipes > don't > > > work with the > > > > rm_work bbclass because the in one of the machines > > > > texinfo-dummy-native gettext-minimal-native > > > > > > > > Another issue is that sometimes one of the machines uses the > > > SOURCE_DATE_EPOCH_FALLBACK > > > > and the other uses SOURCE_DATE_EPOCH and this also triggers a > rebuild > > > of one of the machines. > > > > > > > > Recently a new issue with the rm_work and the spdx > > > > comes in > > > https://lists.openembedded.org/g/openembedded-core/message/174276 > > > > where I have the steps to reproduce. > > > > > > > > All of these issues can be solved easily with TMDIR for each machine, > > > > but because I understand it can be invasive I submit this patch that > > > doesn't change anything > > > > in meta-ti but can add the possibility to use a different TMPDIR for > > > each machine. > > > > > > > > A different TMPDIR for each machine is also referenced in the > official > > > documentation > > > > "Minimally, each configuration file must define the machine and the > > > temporary directory BitBake uses for the build. Suggested practice > dictates > > > that you do not overlap the temporary directories used during the > builds." > > > > > > > > https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build > > > > > > > >> > >and this requirement is no longer met. > > > >> > >Many of these anomalies can be verified with the use of the > rm_work > > > and > > > >> > >create-spdx bitbake classes. > > > >> > > > > > >> > >A previous attempt [1] had already been made but was refused > > > >> > >but this way it can be done without side effects for other users. > > > >> > > > > > >> > >[1] https://lists.yoctoproject.org/g/meta-ti/message/14767 > > > >> > > > > >> > > > > >> > So the difference between this patch and the previous one, is that > > > >> > you are now included a file that is not there normally so that you > > > >> > can put the contents of the previous patch in the new include file > > > >> > to get around the issue in the archiver. > > > > > > > > The archiver issue is fixed but it will be very easy to break again. > > > > Myself recently broke it again so I added a test to help there > > > > > > > > https://git.yoctoproject.org/poky/commit/?id=b5baa7dc8bd1c0d2d78c532d97a44120346b5edd > > > > > > > >> > > > > >> > I'm not saying I'm opposed to this approach (yet), I'm just trying > > > >> > to understand the entire story. > > > >> > > > > >> > Did you find anything in looking at fixing the archiver to better > > > >> > support the multi-config issue? > > > > > > > > The issue now is different and is related with spdx and rm_work > > > > https://lists.openembedded.org/g/openembedded-core/message/174276 > > > > > > > > With this patch I can set a new TMPDIR folder for each machine and > > > > can build successfully. > > > > > > If you don't have spdx enabled then these sort of issues happen less > > > often, but after the above patch was merged in oe-core (and also > > > backported to kirkstone), we cannot build multiconfig targets from > > > meta-ti anymore with spdx enabled. > > > > > > This is special to spdx because of the way the task dependencies are > > > done, which requires it to be executed before rm_work. This is OK when > > > a machine specific TMPDIR is used and explodes quite quickly (due > > > races) when sharing TMPDIR because of the parallel rm_work tasks that > > > end up removing stuff that is still required by other parallel tasks. >
On Fri, Jul 07, 2023 at 10:14:54AM +0100, Jose Quaresma wrote: > Denys Dmytriyenko <denis@denix.org> escreveu no dia quinta, 6/07/2023 à(s) > 19:39: > > > As you noted yourself in that thread, your patches were workarounds and > > just > > masking the actual issue w/o clearly articulating it. The actual race > > issue > > has been under investigation for quite some time and I've discussed it > > with > > Richard before. Luckily he was finally able to root-cause and fix it! > > > > I have reverted my hack to unshare the tmp directory in my distro when > using the oe-core master branch > after the recent spdx changes [1] and everything seems to be okay on my > tests. > The Richard bitbake multiconfig patch to fix the licence collision is for Couple comments: 1. This was not a mere "license collision", it could affect any deferred task and there were tons of them when building lots of multiconfigs - you'd see a lot of corresponding Notices scrolling by[0]. I've hit it few times in the past and mentioned it to Richard. [0] https://lists.openembedded.org/g/bitbake-devel/topic/99974646 2. We are not using SPDX class here. But from the description of the commit that you've linked I see a common generic issue of overlapping files causing manifest errors. But normally you'd see them when trying to re-use temp dir and/or deploy dir between the builds for different machines... > sure another help to use the shared tmp folder. > However on kisstone I still use the tmp unshared between machines in > multiconfig because the problems were many and > I still don't feel safe to revert to the method used on meta-ti kirkstone. > > [1] > https://git.yoctoproject.org/poky/commit/meta/classes/create-spdx-2.2.bbclass?id=42071227f6f54055d8ac44126ab1d95f83f5b264 > > > > BTW, I'm working on a change that I'm about to submit here, which > > indirectly > > gives you separate TMPDIR for k3r5 - stay tuned. > > > > I'm looking forward to it because the hack I'm using is really ugly and I'd > like to drop it also on the kirkstone branch. > > Cheers, > Jose > > > > > > > > On Wed, Jul 05, 2023 at 09:17:44AM +0100, Jose Quaresma wrote: > > > Just for your information there are more complaints about that issue [1] > > of > > > some bad side effects of sharing the TMP folder in multi configs. > > > > > > [1] https://lists.openembedded.org/g/bitbake-devel/message/14860 > > > > > > Jose > > > > > > Ricardo Salveti <ricardo@foundries.io> escreveu no dia segunda, > > 30/01/2023 > > > à(s) 18:27: > > > > > > > On Mon, Jan 30, 2023 at 2:57 PM Jose Quaresma <quaresma.jose@gmail.com > > > > > > > wrote: > > > > > Denys Dmytriyenko <denis@denix.org> escreveu no dia segunda, > > 30/01/2023 > > > > à(s) 17:18: > > > > >> > > > > >> On Mon, Jan 30, 2023 at 11:02:25AM -0600, Ryan Eatmon via > > > > lists.yoctoproject.org wrote: > > > > >> > > > > > >> > > > > > >> > On 1/30/2023 9:20, Jose Quaresma wrote: > > > > >> > >This patch adds the possibility to change some bitbake variable > > > > >> > >that cannot be changed anywhere else, one of then is the TMPDIR. > > > > >> > > > > > > >> > >Using the same TMPDIR it is very sensitive and prone to several > > > > errors > > > > >> > >when used in more complex situations. > > > > >> > >This configuration forces that all native packages have to be the > > > > same between all > > > > >> > >machines and this requirement is very easy to break. > > > > >> > >Suppose you use a macinhe override somewhere on a native recipe > > > > >> > > > > >> Even w/o multiconfig, this will break for multi-machine builds. > > Native > > > > >> packages are for the build host and have no knowledge about the > > target. > > > > >> You should probably look into cross packages instead of native ones > > for > > > > >> this use case. > > > > >> > > > > >> Altering native packages between different targets will cause them > > to > > > > >> rebuild, which they shouldn't. Moreover, it will mess up your sstate > > > > >> and lead to weird issues down the road when trying to re-use your > > > > >> sstate mirror/cache. > > > > > > > > > > Is that the issue of rebuilding some native packages that I am > > trying to > > > > avoid. > > > > > But it is very hard to fix this in OE-core and everything else layers > > > > what happens. > > > > > > > > > > For your information I have some other issues with multiconfig and > > > > native packages > > > > > that have the source files provided on the layer. this two recipes > > don't > > > > work with the > > > > > rm_work bbclass because the in one of the machines > > > > > texinfo-dummy-native gettext-minimal-native > > > > > > > > > > Another issue is that sometimes one of the machines uses the > > > > SOURCE_DATE_EPOCH_FALLBACK > > > > > and the other uses SOURCE_DATE_EPOCH and this also triggers a > > rebuild > > > > of one of the machines. > > > > > > > > > > Recently a new issue with the rm_work and the spdx > > > > > comes in > > > > https://lists.openembedded.org/g/openembedded-core/message/174276 > > > > > where I have the steps to reproduce. > > > > > > > > > > All of these issues can be solved easily with TMDIR for each machine, > > > > > but because I understand it can be invasive I submit this patch that > > > > doesn't change anything > > > > > in meta-ti but can add the possibility to use a different TMPDIR for > > > > each machine. > > > > > > > > > > A different TMPDIR for each machine is also referenced in the > > official > > > > documentation > > > > > "Minimally, each configuration file must define the machine and the > > > > temporary directory BitBake uses for the build. Suggested practice > > dictates > > > > that you do not overlap the temporary directories used during the > > builds." > > > > > > > > > > > https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build > > > > > > > > > >> > >and this requirement is no longer met. > > > > >> > >Many of these anomalies can be verified with the use of the > > rm_work > > > > and > > > > >> > >create-spdx bitbake classes. > > > > >> > > > > > > >> > >A previous attempt [1] had already been made but was refused > > > > >> > >but this way it can be done without side effects for other users. > > > > >> > > > > > > >> > >[1] https://lists.yoctoproject.org/g/meta-ti/message/14767 > > > > >> > > > > > >> > > > > > >> > So the difference between this patch and the previous one, is that > > > > >> > you are now included a file that is not there normally so that you > > > > >> > can put the contents of the previous patch in the new include file > > > > >> > to get around the issue in the archiver. > > > > > > > > > > The archiver issue is fixed but it will be very easy to break again. > > > > > Myself recently broke it again so I added a test to help there > > > > > > > > > > > https://git.yoctoproject.org/poky/commit/?id=b5baa7dc8bd1c0d2d78c532d97a44120346b5edd > > > > > > > > > >> > > > > > >> > I'm not saying I'm opposed to this approach (yet), I'm just trying > > > > >> > to understand the entire story. > > > > >> > > > > > >> > Did you find anything in looking at fixing the archiver to better > > > > >> > support the multi-config issue? > > > > > > > > > > The issue now is different and is related with spdx and rm_work > > > > > https://lists.openembedded.org/g/openembedded-core/message/174276 > > > > > > > > > > With this patch I can set a new TMPDIR folder for each machine and > > > > > can build successfully. > > > > > > > > If you don't have spdx enabled then these sort of issues happen less > > > > often, but after the above patch was merged in oe-core (and also > > > > backported to kirkstone), we cannot build multiconfig targets from > > > > meta-ti anymore with spdx enabled. > > > > > > > > This is special to spdx because of the way the task dependencies are > > > > done, which requires it to be executed before rm_work. This is OK when > > > > a machine specific TMPDIR is used and explodes quite quickly (due > > > > races) when sharing TMPDIR because of the parallel rm_work tasks that > > > > end up removing stuff that is still required by other parallel tasks.
diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf b/meta-ti-bsp/conf/multiconfig/k3r5.conf index deb07210..b69c5d6d 100644 --- a/meta-ti-bsp/conf/multiconfig/k3r5.conf +++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf @@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}" DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}" MACHINE:append = "-k3r5" + +include conf/multiconfig/k3r5.inc
This patch adds the possibility to change some bitbake variable that cannot be changed anywhere else, one of then is the TMPDIR. Using the same TMPDIR it is very sensitive and prone to several errors when used in more complex situations. This configuration forces that all native packages have to be the same between all machines and this requirement is very easy to break. Suppose you use a macinhe override somewhere on a native recipe and this requirement is no longer met. Many of these anomalies can be verified with the use of the rm_work and create-spdx bitbake classes. A previous attempt [1] had already been made but was refused but this way it can be done without side effects for other users. [1] https://lists.yoctoproject.org/g/meta-ti/message/14767 Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> --- meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++ 1 file changed, 2 insertions(+)