diff mbox series

buildhistory.bbclass: restore BUILDHISTORY_PRESERVE files

Message ID 20250115153149.1827119-1-pmi183@gmail.com
State New
Headers show
Series buildhistory.bbclass: restore BUILDHISTORY_PRESERVE files | expand

Commit Message

Pedro Ferreira Jan. 15, 2025, 3:31 p.m. UTC 
From: Pedro Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>

On each build using sstate-cache, buildhistory will move
content to a temporary folder named `old`.
When buildhistory looks for the main dir, it wont find it
and ends up creating it.
As a consequence how code is structured wont restore any
preserved file.

Code block moved to ensure if old dir exists, it will
attempt to restore those files marked to preserve.

Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
---
 meta/classes/buildhistory.bbclass | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

Comments

Ross Burton Jan. 22, 2025, 1:42 p.m. UTC | #1 
Hi Pedro,

> On 15 Jan 2025, at 15:31, Pedro Ferreira via lists.openembedded.org <pmi183=gmail.com@lists.openembedded.org> wrote:
> 
> From: Pedro Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
> 
> On each build using sstate-cache, buildhistory will move
> content to a temporary folder named `old`.
> When buildhistory looks for the main dir, it wont find it
> and ends up creating it.
> As a consequence how code is structured wont restore any
> preserved file.
> 
> Code block moved to ensure if old dir exists, it will
> attempt to restore those files marked to preserve.

Do you have a reproducer for this behaviour?  As you have seen the buildhistory file management isn’t trivial and we’d like to understand the failure better.

Cheers,
Ross
Pedro Ferreira Jan. 23, 2025, 2:37 p.m. UTC | #2 
Hi Ross,

Here's a simple test do reproduce it.

*Pre-conditions:*
INHERIT += "buildhistory"
BUILDHISTORY_RESET = "1"
BUILDHISTORY_PRESERVE:append = " files-in-package.txt"

*- Starting without sstate cache*
*Run:*
bitbake base-passwd
*Check:*
cat PATH_TO_BUILD_dir/buildhistory/packages/core2-64-poky-linux/base-passwd/base-passwd-dbg/files-in-package.txt
(File exists and content ok)
*Run:*
bitbake base-passwd (it will use sstate cache)
*Check:*
cat PATH_TO_BUILD_dir/buildhistory/packages/core2-64-poky-linux/base-passwd/base-passwd-dbg/files-in-package.txt
(File wont exist)

*- Starting with sstate cache*
*Run:*
bitbake base-passwd --no-setscene
*Check:*
cat PATH_TO_BUILD_dir/buildhistory/packages/core2-64-poky-linux/base-passwd/base-passwd-dbg/files-in-package.txt
(File exists and content ok)
*Run:*
bitbake base-passwd (it will use sstate cache)
*Check:*
cat PATH_TO_BUILD_dir/buildhistory/packages/core2-64-poky-linux/base-passwd/base-passwd-dbg/files-in-package.txt
(File wont exist)

Cheers,
Pedro
Pedro Ferreira March 6, 2025, 2:26 p.m. UTC | #3 
Simple scenario to test it:

*Pre-conditions:*
INHERIT += "buildhistory"
BUILDHISTORY_RESET = "1"
BUILDHISTORY_PRESERVE:append = " files-in-package.txt"

*Run:*
bitbake -c cleansstate base-passwd && bitbake base-passwd
*Check:*
cat PATH_TO_BUILD_dir/buildhistory/packages/core2-64-poky-linux/base-passwd/base-passwd-dbg/files-in-package.txt
(File exists and content ok)
*Run:*
rm -rf tmp && bitbake base-passwd (it will use sstate cache)
*Check:*
cat PATH_TO_BUILD_dir/buildhistory/packages/core2-64-poky-linux/base-passwd/base-passwd-dbg/files-in-package.txt
(File wont exist)

Cheers,
Pedro
Ross Burton March 12, 2025, 1:40 p.m. UTC | #4 
On 6 Mar 2025, at 14:26, Pedro Ferreira via lists.openembedded.org <pedro.silva.ferreira=criticaltechworks.com@lists.openembedded.org> wrote:
> 
> Simple scenario to test it:
>   Pre-conditions:
> INHERIT += "buildhistory"
> BUILDHISTORY_RESET = "1"
> BUILDHISTORY_PRESERVE:append = " files-in-package.txt"
> 
> Run:
> bitbake -c cleansstate base-passwd && bitbake base-passwd
> Check:
> cat PATH_TO_BUILD_dir/buildhistory/packages/core2-64-poky-linux/base-passwd/base-passwd-dbg/files-in-package.txt
> (File exists and content ok)
> Run:
> rm -rf tmp && bitbake base-passwd (it will use sstate cache)
> Check:
> cat PATH_TO_BUILD_dir/buildhistory/packages/core2-64-poky-linux/base-passwd/base-passwd-dbg/files-in-package.txt
> (File wont exist)
>   Cheers,

There’s an even bigger problem with BUILDHISTORY_RESET, where recipes from sstate are just not entering the buildhistory at all…

From a clean buildhistory and build tree, if an entire image can be populated from sstate then there’s just nothing in buildhistory for the packages, as it’s only populated on do_package which won’t actually be re-ran.

Adding files-in-package.txt to preserve feels like a workaround for that problem, am I right? Apart from the fact that it doesn’t actually work when the buildhistory repository is empty from the beginning.

Ross
diff mbox series

Patch

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index d735dd5fb5..b0f395f05e 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -260,14 +260,6 @@  python buildhistory_emit_pkghistory() {
     if not os.path.exists(pkghistdir):
         bb.utils.mkdirhier(pkghistdir)
     else:
-        # We need to make sure that all files kept in
-        # buildhistory/old are restored successfully
-        # otherwise next block of code wont have files to
-        # check and purge
-        if d.getVar("BUILDHISTORY_RESET"):
-            for pkg in packagelist:
-                preservebuildhistoryfiles(pkg, preserve)
-
         # Remove files for packages that no longer exist
         for item in os.listdir(pkghistdir):
             if item not in preserve:
@@ -280,6 +272,13 @@  python buildhistory_emit_pkghistory() {
                     else:
                         os.unlink(itempath)
 
+    if os.path.exists(oldpkghistdir):
+        # We need to make sure that all files in preserve
+        # are restored from buildhistory/old successfully
+        if d.getVar("BUILDHISTORY_RESET"):
+            for pkg in packagelist:
+                preservebuildhistoryfiles(pkg, preserve)
+
     rcpinfo = RecipeInfo(pn)
     rcpinfo.pe = pe
     rcpinfo.pv = pv