diff mbox series

[1/7] cooker: fix bitbake -b silently ignoring bbappends

Message ID 20260815134722.497586-2-adrian.freihofer@siemens.com
State New
Headers show
Series cooker/tinfoil: fix -b bbappend handling and add single-task prepared-task API | expand

Commit Message

Freihofer, Adrian Aug. 15, 2026, 1:46 p.m. UTC
From: Adrian Freihofer <adrian.freihofer@siemens.com>

"bitbake -b <recipe.bb>" builds the recipe without applying any of its
.bbappend files.

buildFileInternal() resolves appends via
self.collections[mc].get_file_appends(fn), but self.collections[mc] is
only ever filled in by collect_bbfiles(), called from updateCache() -
a path -b deliberately skips. matchFiles(), the one -b-path function
that does call collect_bbfiles(), built a fresh CookerCollectFiles into
a throwaway local instead of self.collections[mc], so the append list
stayed empty (or, on a memory-resident server, stale from the last
full parse - e.g. missing a devtool/externalsrc workspace .bbappend
added since). Nothing warns that the built metadata differs from disk.

Make matchFiles() refresh self.collections[mc] itself so the later
append lookup for the same fn sees the same fresh collection.

AI-Generated: Uses GitHub Copilot

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 lib/bb/cooker.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

Comments

Richard Purdie Aug. 15, 2026, 9:39 p.m. UTC | #1
On Sat, 2026-08-15 at 15:46 +0200, Adrian Freihofer via lists.openembedded.org wrote:
> From: Adrian Freihofer <adrian.freihofer@siemens.com>
> 
> "bitbake -b <recipe.bb>" builds the recipe without applying any of its
> .bbappend files.
> 
> buildFileInternal() resolves appends via
> self.collections[mc].get_file_appends(fn), but self.collections[mc] is
> only ever filled in by collect_bbfiles(), called from updateCache() -
> a path -b deliberately skips. matchFiles(), the one -b-path function
> that does call collect_bbfiles(), built a fresh CookerCollectFiles into
> a throwaway local instead of self.collections[mc], so the append list
> stayed empty (or, on a memory-resident server, stale from the last
> full parse - e.g. missing a devtool/externalsrc workspace .bbappend
> added since). Nothing warns that the built metadata differs from disk.
> 
> Make matchFiles() refresh self.collections[mc] itself so the later
> append lookup for the same fn sees the same fresh collection.
> 
> AI-Generated: Uses GitHub Copilot
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> ---
>  lib/bb/cooker.py | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)

I've not checked but doesn't this only add limited collections data so
whilst it fixes -b, it potentially corrupts the cache for the non -b
cases?

This is definitely something we should fix, I just want to make sure
this doesn't corrupt something else. I suspect it deliberately doesn't
write to self...

Cheers,

Richard
Adrian Freihofer Aug. 16, 2026, 10:18 p.m. UTC | #2
On Sat, 2026-08-15 at 22:39 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Sat, 2026-08-15 at 15:46 +0200, Adrian Freihofer via
> lists.openembedded.org wrote:
> > From: Adrian Freihofer <adrian.freihofer@siemens.com>
> > 
> > "bitbake -b <recipe.bb>" builds the recipe without applying any of
> > its
> > .bbappend files.
> > 
> > buildFileInternal() resolves appends via
> > self.collections[mc].get_file_appends(fn), but self.collections[mc]
> > is
> > only ever filled in by collect_bbfiles(), called from updateCache()
> > -
> > a path -b deliberately skips. matchFiles(), the one -b-path
> > function
> > that does call collect_bbfiles(), built a fresh CookerCollectFiles
> > into
> > a throwaway local instead of self.collections[mc], so the append
> > list
> > stayed empty (or, on a memory-resident server, stale from the last
> > full parse - e.g. missing a devtool/externalsrc workspace .bbappend
> > added since). Nothing warns that the built metadata differs from
> > disk.
> > 
> > Make matchFiles() refresh self.collections[mc] itself so the later
> > append lookup for the same fn sees the same fresh collection.
> > 
> > AI-Generated: Uses GitHub Copilot
> > 
> > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> > ---
> >  lib/bb/cooker.py | 6 ++++--
> >  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> I've not checked but doesn't this only add limited collections data
> so
> whilst it fixes -b, it potentially corrupts the cache for the non -b
> cases?

I tried to add a test case
  test_run_prepared_task_recipecache_self_heals
  see https://lists.openembedded.org/g/bitbake-devel/message/19965
which proves that this works as expected with v2:
https://lists.openembedded.org/g/bitbake-devel/message/19964

Regards,
Adrian

> 
> This is definitely something we should fix, I just want to make sure
> this doesn't corrupt something else. I suspect it deliberately
> doesn't
> write to self...
> 
> Cheers,
> 
> Richard
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#19957):
> https://lists.openembedded.org/g/bitbake-devel/message/19957
> Mute This Topic: https://lists.openembedded.org/mt/120763373/4454582
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe:
> https://lists.openembedded.org/g/bitbake-devel/unsub [adrian.freihofer@gmail.com
> ]
> -=-=-=-=-=-=-=-=-=-=-=-
diff mbox series

Patch

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 4b6ba3196..108551a60 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1322,8 +1322,10 @@  You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si
         if bf.startswith("/") or bf.startswith("../"):
             bf = os.path.abspath(bf)
 
-        collections = {mc: CookerCollectFiles(self.bbfile_config_priorities, mc)}
-        filelist, masked, searchdirs = collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
+        # The only place the "bitbake -b" path fills in the bbappends which
+        # buildFileInternal() then reads back from self.collections[mc].
+        self.collections[mc] = CookerCollectFiles(self.bbfile_config_priorities, mc)
+        filelist, masked, searchdirs = self.collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
         try:
             os.stat(bf)
             bf = os.path.abspath(bf)