| Message ID | 20240806125955.160835-1-pedro.silva.ferreira@criticaltechworks.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v2,1/2] buildhistory: fix package output folder creation | expand |
Hello, Because of DMARC, you really need to add your From: here Also, the patch is still mangled and doesn't apply. On 06/08/2024 13:59:54+0100, Pedro Ferreira via lists.openembedded.org wrote: > This fix garantees that output package folder exists on > buildhistory folder to avoid missing files-in-package.txt > creation during task `package` execution. > > This issue happens because the output folder is being > created on task `packagedata` before generating `latest` file. > > Also it ensures that in case of `find` fails we leave with > a hard error instead of hidding the error on the for loop. > > Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com> > --- > v2 changes: > > Fixed Signed-off-by to the correct email > --- > meta/classes/buildhistory.bbclass | 8 +++----- > 1 file changed, 3 insertions(+), 5 deletions(-) > > diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass > index c3d049aea3..ad151092c9 100644 > --- a/meta/classes/buildhistory.bbclass > +++ b/meta/classes/buildhistory.bbclass > @@ -599,15 +599,13 @@ buildhistory_list_files_no_owners() { > > buildhistory_list_pkg_files() { > # Create individual files-in-package for each recipe's package > - for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do > + pkgdirlist=$(find ${PKGDEST}/* -maxdepth 0 -type d) > + for pkgdir in $pkgdirlist; do > pkgname=$(basename $pkgdir) > outfolder="${BUILDHISTORY_DIR_PACKAGE}/$pkgname" > outfile="$outfolder/files-in-package.txt" > # Make sure the output folder exists so we can create the file > - if [ ! -d $outfolder ] ; then > - bbdebug 2 "Folder $outfolder does not exist, file $outfile not created" > - continue > - fi > + mkdir -p $outfolder > buildhistory_list_files $pkgdir $outfile fakeroot > done > } > -- > 2.34.1 > > The information in this communication may contain confidential or legally privileged information. It is intended solely for the use of the individual or entity it addresses and others authorized to receive it. If you are not an intended recipient, you are hereby notified that any disclosure, copying, distribution or action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication by error, please notify us immediately by responding to this e-mail and then delete it from your system. Critical TechWorks is not liable for the proper and complete transmission of the information in this communication nor for any delay in its receipt > > This e-mail is environmentally friendly, just like Critical TechWorks, which lives in a paper-free atmosphere. Therefore, please consider the environment before printing it! > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#203019): https://lists.openembedded.org/g/openembedded-core/message/203019 > Mute This Topic: https://lists.openembedded.org/mt/107750091/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Wed, 2024-08-07 at 17:55 +0200, Alexandre Belloni via lists.openembedded.org wrote: > Hello, > > Because of DMARC, you really need to add your From: here > > Also, the patch is still mangled and doesn't apply. I manually fixed it up when I tested it and I did merge it earlier as the initial test run was fone. The patch has since caused breakage unfortunately: https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/6389 https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/6441 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/8203 https://autobuilder.yoctoproject.org/typhoon/#/builders/120/builds/4576 and more. They're all glib-initial breaking as it doesn't have packages. I've sent a patch to fix that recipe, I'm not sure how many more recipes will be affected overall though. Cheers, Richard
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded- > core@lists.openembedded.org> On Behalf Of Richard Purdie > Sent: den 7 augusti 2024 18:44 > To: alexandre.belloni@bootlin.com; > pedro.silva.ferreira@criticaltechworks.com > Cc: openembedded-core@lists.openembedded.org > Subject: Re: [OE-core] [PATCH v2 1/2] buildhistory: fix package output > folder creation > > On Wed, 2024-08-07 at 17:55 +0200, Alexandre Belloni via > lists.openembedded.org wrote: > > Hello, > > > > Because of DMARC, you really need to add your From: here > > > > Also, the patch is still mangled and doesn't apply. > > I manually fixed it up when I tested it and I did merge it earlier as > the initial test run was fone. > > The patch has since caused breakage unfortunately: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/6389 > https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/6441 > https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/8203 > https://autobuilder.yoctoproject.org/typhoon/#/builders/120/builds/4576 > > and more. They're all glib-initial breaking as it doesn't have > packages. > > I've sent a patch to fix that recipe, I'm not sure how many more > recipes will be affected overall though. > > Cheers, > > Richard I just sent a corresponding patch for libdevmapper in meta-oe. Would it make sense to make the buildhistory code actually handle recipes that do not produce packages instead? Or at least produce a meaningful error in this case. //Peter
Is it possible/feasible to backport this change to LTS branches Thank you, Pedro Ferreira
On Tue, Aug 13, 2024 at 1:05 AM Pedro Ferreira via lists.openembedded.org <pedro.silva.ferreira=criticaltechworks.com@lists.openembedded.org> wrote: > > Is it possible/feasible to backport this change to LTS branches Please send patches for kirkstone and scarthgap and I will add them to my test queue. I see that this resulted in some breakage in the master branch, so you should also include patches for any oe-core breakage. Thanks! Steve > Thank you, > > Pedro Ferreira > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#203258): https://lists.openembedded.org/g/openembedded-core/message/203258 > Mute This Topic: https://lists.openembedded.org/mt/107750091/3620601 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [steve@sakoman.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Steve and Richard, I tried to backport locally patches [4] and [5] to kirkstone and scarthgap but i ran into some issues related to sstate.bbclass. After some investigation i saw that i was missing a patch done by Richard [3] in the context of this buildhistory change but after applying the patch [3], due to the `interceptfuncs` logic i ran into an undefined variable (SSTATE_INSTDIR) while executing `bb.utils.rename` helper function. Applying patch [1] and [2] in combination with [3] , [4] and [5] seems to do the job but im not really sure if patch [1] and [2] can be backported to lts branches since they remove some features/class that might be used by other components that I am not aware of. I tested this with ncurses recipe and with core-image-minimal. 1: siteconfig: Drop siteconfig class/code/support f3766dc038f7ba9780ddaf5eb8d27385ea31d7d0 2: sstate: Drop intercept functions support cfbfd0b2e89eb71783c55a1be4a7e63e6cd82c2f 3: sstate/buildhistory: Fix plaindirs handling to occur before SSTATEPOSTINSTFUNCS 62ee349cf18532dac8736488752c00e89de78fcd 4: buildhistory: Restoring files from preserve list 9f68a45aa238ae5fcdfaca71ba0e7015e9cb720e 5: buildhistory: Fix intermittent package file list creation 8de9b8c1e199896b9a7bc5ed64967c6bfbf84bea My best regards, Pedro -----Original Message----- From: Steve Sakoman <steve@sakoman.com> Sent: Tuesday, August 13, 2024 1:09 PM To: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com> Cc: openembedded-core@lists.openembedded.org Subject: Re: [OE-core] [PATCH v2 1/2] buildhistory: fix package output folder creation Notice: This e-mail has originated from an external email service, so do not click on any links, nor open any attachments unless you know who the sender is and are sure the content is secure. On Tue, Aug 13, 2024 at 1:05 AM Pedro Ferreira via lists.openembedded.org <pedro.silva.ferreira=criticaltechworks.com@lists.openembedded.org> wrote: > > Is it possible/feasible to backport this change to LTS branches Please send patches for kirkstone and scarthgap and I will add them to my test queue. I see that this resulted in some breakage in the master branch, so you should also include patches for any oe-core breakage. Thanks! Steve > Thank you, > > Pedro Ferreira > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#203258): > https://lists.openembedded.org/g/openembedded-core/message/203258 > Mute This Topic: https://lists.openembedded.org/mt/107750091/3620601 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub > [steve@sakoman.com] > -=-=-=-=-=-=-=-=-=-=-=- > The information in this communication may contain confidential or legally privileged information. It is intended solely for the use of the individual or entity it addresses and others authorized to receive it. If you are not an intended recipient, you are hereby notified that any disclosure, copying, distribution or action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication by error, please notify us immediately by responding to this e-mail and then delete it from your system. Critical TechWorks is not liable for the proper and complete transmission of the information in this communication nor for any delay in its receipt This e-mail is environmentally friendly, just like Critical TechWorks, which lives in a paper-free atmosphere. Therefore, please consider the environment before printing it!
On Tue, 2024-08-27 at 12:50 +0000, Pedro Silva Ferreira wrote: > Hi Steve and Richard, > > I tried to backport locally patches [4] and [5] to kirkstone and > scarthgap but i ran into some issues related to sstate.bbclass. After > some investigation i saw that i was missing a patch done by Richard > [3] in the context of this buildhistory change but after applying the > patch [3], due to the `interceptfuncs` logic i ran into an undefined > variable (SSTATE_INSTDIR) while executing `bb.utils.rename` helper > function. Applying patch [1] and [2] in combination with [3] , [4] > and [5] seems to do the job but im not really sure if patch [1] and > [2] can be backported to lts branches since they remove some > features/class that might be used by other components that I am not > aware of. I tested this with ncurses recipe and with core-image- > minimal. > > 1: siteconfig: Drop siteconfig class/code/support > f3766dc038f7ba9780ddaf5eb8d27385ea31d7d0 > 2: sstate: Drop intercept functions support > cfbfd0b2e89eb71783c55a1be4a7e63e6cd82c2f We definitely can't backport those. > 3: sstate/buildhistory: Fix plaindirs handling to occur before > SSTATEPOSTINSTFUNCS > 62ee349cf18532dac8736488752c00e89de78fcd We'd probably need to find a way to make that change and continue to work with the intercept functions :/. Cheers, Richard
Hi Richard, I tried to pickup your patch and make it fit but unsuccessful. Would this be an alternative only for LTS branches? Thanks, Pedro From 3ab48da7a94b80bb67ebef5a06d92ace134e0bbf Mon Sep 17 00:00:00 2001 From: Pedro Ferreira <pedro.silva.ferreira@criticaltechworks.com> Date: Thu, 29 Aug 2024 11:37:19 +0100 Subject: [PATCH] sstate: making sstate_install run after plaindirs creation Based on oe-core rev:62ee349cf18532dac8736488752c00e89de78fcd Plaindirs are handled after SSTATEPOSTINSTFUNCS (buildhistory_emit_pkghistory) and causes buildhistory class fail due to missing dirs. sstate_install call moved after plaindirs creation loop. Signed-off-by: Pedro Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com> --- meta/classes/sstate.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index dd6cf12920..ed6bf59e42 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -415,7 +415,6 @@ def sstate_installpkgdir(ss, d): for state in ss['dirs']: prepdir(state[1]) bb.utils.rename(sstateinst + state[0], state[1]) - sstate_install(ss, d) for plain in ss['plaindirs']: workdir = d.getVar('WORKDIR') @@ -427,6 +426,7 @@ def sstate_installpkgdir(ss, d): bb.utils.mkdirhier(src) prepdir(dest) bb.utils.rename(src, dest) + sstate_install(ss, d) return True -- 2.34.1 -----Original Message----- From: Richard Purdie <richard.purdie@linuxfoundation.org> Sent: Tuesday, August 27, 2024 3:11 PM To: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>; Steve Sakoman <steve@sakoman.com> Cc: openembedded-core@lists.openembedded.org Subject: Re: [OE-core] [PATCH v2 1/2] buildhistory: fix package output folder creation Notice: This e-mail has originated from an external email service, so do not click on any links, nor open any attachments unless you know who the sender is and are sure the content is secure. On Tue, 2024-08-27 at 12:50 +0000, Pedro Silva Ferreira wrote: > Hi Steve and Richard, > > I tried to backport locally patches [4] and [5] to kirkstone and > scarthgap but i ran into some issues related to sstate.bbclass. After > some investigation i saw that i was missing a patch done by Richard > [3] in the context of this buildhistory change but after applying the > patch [3], due to the `interceptfuncs` logic i ran into an undefined > variable (SSTATE_INSTDIR) while executing `bb.utils.rename` helper > function. Applying patch [1] and [2] in combination with [3] , [4] and > [5] seems to do the job but im not really sure if patch [1] and [2] > can be backported to lts branches since they remove some > features/class that might be used by other components that I am not > aware of. I tested this with ncurses recipe and with core-image- > minimal. > > 1: siteconfig: Drop siteconfig class/code/support > f3766dc038f7ba9780ddaf5eb8d27385ea31d7d0 > 2: sstate: Drop intercept functions support > cfbfd0b2e89eb71783c55a1be4a7e63e6cd82c2f We definitely can't backport those. > 3: sstate/buildhistory: Fix plaindirs handling to occur before > SSTATEPOSTINSTFUNCS 62ee349cf18532dac8736488752c00e89de78fcd We'd probably need to find a way to make that change and continue to work with the intercept functions :/. Cheers, Richard The information in this communication may contain confidential or legally privileged information. It is intended solely for the use of the individual or entity it addresses and others authorized to receive it. If you are not an intended recipient, you are hereby notified that any disclosure, copying, distribution or action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication by error, please notify us immediately by responding to this e-mail and then delete it from your system. Critical TechWorks is not liable for the proper and complete transmission of the information in this communication nor for any delay in its receipt This e-mail is environmentally friendly, just like Critical TechWorks, which lives in a paper-free atmosphere. Therefore, please consider the environment before printing it!
On Thu, 2024-08-29 at 10:55 +0000, Pedro Silva Ferreira wrote: > I tried to pickup your patch and make it fit but unsuccessful. Would > this be an alternative only for LTS branches? No, since there are multiple call sites into sstate_install and this therefore only partially fixes the problem. Cheers, Richard
You'd have to test it but I've a patch I've been working on which drops the use of SSTATEPOSTINSTFUNCS from buildhistory entirely. I've just sent that out for review. That might be enough to fix this on the release branches for buildhistory even if it doesn't fix the sstate bug. Cheers, Richard
Hi Richard,
Applying your patches plus the buildhistory patches, files-in-package.txt generated successfully but I noticed that latest file now is missing (from a fresh build).
As far as I could see, it exits buildhistory_emit_pkghistory() execution from a silent exception when it tries to open a file that doesn’t exist yet to pickup the list of packages.
I changed from buildhistory.bbclass:59 - 61, from do_packagedata to do_package_qa just for timing testing purpose and all latest files were generated.
Thanks,
Pedro
-----Original Message-----
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Sent: Thursday, August 29, 2024 1:01 PM
To: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH v2 1/2] buildhistory: fix package output folder creation
Notice: This e-mail has originated from an external email service, so do not click on any links, nor open any attachments unless you know who the sender is and are sure the content is secure.
You'd have to test it but I've a patch I've been working on which drops the use of SSTATEPOSTINSTFUNCS from buildhistory entirely. I've just sent that out for review.
That might be enough to fix this on the release branches for buildhistory even if it doesn't fix the sstate bug.
Cheers,
Richard
The information in this communication may contain confidential or legally privileged information. It is intended solely for the use of the individual or entity it addresses and others authorized to receive it. If you are not an intended recipient, you are hereby notified that any disclosure, copying, distribution or action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication by error, please notify us immediately by responding to this e-mail and then delete it from your system. Critical TechWorks is not liable for the proper and complete transmission of the information in this communication nor for any delay in its receipt
This e-mail is environmentally friendly, just like Critical TechWorks, which lives in a paper-free atmosphere. Therefore, please consider the environment before printing it!
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass index c3d049aea3..ad151092c9 100644 --- a/meta/classes/buildhistory.bbclass +++ b/meta/classes/buildhistory.bbclass @@ -599,15 +599,13 @@ buildhistory_list_files_no_owners() { buildhistory_list_pkg_files() { # Create individual files-in-package for each recipe's package - for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do + pkgdirlist=$(find ${PKGDEST}/* -maxdepth 0 -type d) + for pkgdir in $pkgdirlist; do pkgname=$(basename $pkgdir) outfolder="${BUILDHISTORY_DIR_PACKAGE}/$pkgname" outfile="$outfolder/files-in-package.txt" # Make sure the output folder exists so we can create the file - if [ ! -d $outfolder ] ; then - bbdebug 2 "Folder $outfolder does not exist, file $outfile not created" - continue - fi + mkdir -p $outfolder buildhistory_list_files $pkgdir $outfile fakeroot done }
This fix garantees that output package folder exists on buildhistory folder to avoid missing files-in-package.txt creation during task `package` execution. This issue happens because the output folder is being created on task `packagedata` before generating `latest` file. Also it ensures that in case of `find` fails we leave with a hard error instead of hidding the error on the for loop. Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com> --- v2 changes: Fixed Signed-off-by to the correct email --- meta/classes/buildhistory.bbclass | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) -- 2.34.1 The information in this communication may contain confidential or legally privileged information. It is intended solely for the use of the individual or entity it addresses and others authorized to receive it. If you are not an intended recipient, you are hereby notified that any disclosure, copying, distribution or action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication by error, please notify us immediately by responding to this e-mail and then delete it from your system. Critical TechWorks is not liable for the proper and complete transmission of the information in this communication nor for any delay in its receipt This e-mail is environmentally friendly, just like Critical TechWorks, which lives in a paper-free atmosphere. Therefore, please consider the environment before printing it!