| Message ID | 20251119090045.107642-1-Qi.Chen@windriver.com |
|---|---|
| State | New |
| Headers | show |
| Series | [bitbake-devel,V3,1/2] bitbake: add SKIP_FRAGMENTS CookerFeature | expand |
On Wed, 2025-11-19 at 17:00 +0800, Qi.Chen@windriver.com wrote: > From: Chen Qi <Qi.Chen@windriver.com> > > This patch add a new CookerFeature, SKIP_FRAGMENTS, which when enabled > will skip fragments parsing. > > This feature is mainly used by bitbake-config-build. It's a tool to > manage fragements. So we want to skip fragment parsing to avoid > unnecessary errors. > > Fixes [YOCTO #16060] > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > --- > bin/bitbake-layers | 7 ++++++- > lib/bb/cooker.py | 8 +++++++- > lib/bb/cookerdata.py | 4 ++++ > lib/bb/parse/ast.py | 3 +++ > 4 files changed, 20 insertions(+), 2 deletions(-) The patch does look like the right thing to do and I like it a lot, thanks for putting it together. Unfortunately, testing on the autobuilder is showing an issue where fragments are disappearing. It is when memory resident bitbake is in use, i.e. BB_SERVER_TIMEOUT=60 For example: https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/2760 where MACHINE seems to be unset in some builds. If the timeout happens, the server does properly reset and the builds work so it is timing sensitive. I've not managed to get to the bottom of it yet so I wanted to at least share the issue before I sleep! Cheers, Richard
Hi Richard, Thanks for information. I can now reproduce the issue with: 1. export BB_SERVER_TIMEOUT=60 2. bitbake-config-build --help && bitbake-getvar MACHINE It looks like that the bitbake server is not doing a re-parsing when re-connected with a removed featureset. I'll check more to see how to fix this. Regards, Qi -----Original Message----- From: Richard Purdie <richard.purdie@linuxfoundation.org> Sent: Thursday, November 20, 2025 8:22 AM To: Chen, Qi <Qi.Chen@windriver.com>; bitbake-devel@lists.openembedded.org Cc: alex.kanavin@gmail.com Subject: Re: [bitbake-devel][PATCH V3 1/2] bitbake: add SKIP_FRAGMENTS CookerFeature On Wed, 2025-11-19 at 17:00 +0800, Qi.Chen@windriver.com wrote: > From: Chen Qi <Qi.Chen@windriver.com> > > This patch add a new CookerFeature, SKIP_FRAGMENTS, which when enabled > will skip fragments parsing. > > This feature is mainly used by bitbake-config-build. It's a tool to > manage fragements. So we want to skip fragment parsing to avoid > unnecessary errors. > > Fixes [YOCTO #16060] > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > --- > bin/bitbake-layers | 7 ++++++- > lib/bb/cooker.py | 8 +++++++- > lib/bb/cookerdata.py | 4 ++++ > lib/bb/parse/ast.py | 3 +++ > 4 files changed, 20 insertions(+), 2 deletions(-) The patch does look like the right thing to do and I like it a lot, thanks for putting it together. Unfortunately, testing on the autobuilder is showing an issue where fragments are disappearing. It is when memory resident bitbake is in use, i.e. BB_SERVER_TIMEOUT=60 For example: https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/2760 where MACHINE seems to be unset in some builds. If the timeout happens, the server does properly reset and the builds work so it is timing sensitive. I've not managed to get to the bottom of it yet so I wanted to at least share the issue before I sleep! Cheers, Richard
diff --git a/bin/bitbake-layers b/bin/bitbake-layers index 341ecbcd9..7d929567a 100755 --- a/bin/bitbake-layers +++ b/bin/bitbake-layers @@ -23,6 +23,7 @@ topdir = os.path.dirname(bindir) sys.path[0:0] = [os.path.join(topdir, 'lib')] import bb.tinfoil +import bb.cooker import bb.msg logger = bb.msg.logger_create(toolname, sys.stdout) @@ -64,7 +65,11 @@ def main(): if global_args.force > 1: bbpaths = [] else: - tinfoil.prepare(True) + if toolname == "bitbake-config-build": + extra_features = [bb.cooker.CookerFeatures.SKIP_FRAGMENTS] + else: + extra_features = None + tinfoil.prepare(True, extra_features=extra_features) bbpaths = tinfoil.config_data.getVar('BBPATH').split(':') for path in ([topdir] + bbpaths): diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 03f262ac1..32669ed96 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -79,7 +79,7 @@ class SkippedPackage: class CookerFeatures(object): - _feature_list = [HOB_EXTRA_CACHES, BASEDATASTORE_TRACKING, SEND_SANITYEVENTS, RECIPE_SIGGEN_INFO] = list(range(4)) + _feature_list = [HOB_EXTRA_CACHES, BASEDATASTORE_TRACKING, SEND_SANITYEVENTS, RECIPE_SIGGEN_INFO, SKIP_FRAGMENTS] = list(range(5)) def __init__(self): self._features=set() @@ -278,6 +278,9 @@ class BBCooker: logger.critical("Unable to import extra RecipeInfo '%s' from '%s': %s" % (cache_name, module_name, exc)) raise bb.BBHandledException() + if CookerFeatures.SKIP_FRAGMENTS in self.featureset: + self.skipFragments() + self.databuilder = bb.cookerdata.CookerDataBuilder(self.configuration, False) self.databuilder.parseBaseConfiguration() self.data = self.databuilder.data @@ -355,6 +358,9 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si if hasattr(self, "data"): self.data.disableTracking() + def skipFragments(self): + self.configuration.skip_fragments = True + def revalidateCaches(self): bb.parse.clear_cache() diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py index 65c153a5b..b6dca7af7 100644 --- a/lib/bb/cookerdata.py +++ b/lib/bb/cookerdata.py @@ -137,6 +137,7 @@ class CookerConfiguration(object): self.build_verbose_stdout = False self.dry_run = False self.tracking = False + self.skip_fragments = False self.writeeventlog = False self.limited_deps = False self.runall = [] @@ -226,12 +227,15 @@ class CookerDataBuilder(object): self.prefiles = cookercfg.prefile self.postfiles = cookercfg.postfile self.tracking = cookercfg.tracking + self.skip_fragments = cookercfg.skip_fragments bb.utils.set_context(bb.utils.clean_context()) bb.event.set_class_handlers(bb.event.clean_class_handlers()) self.basedata = bb.data.init() if self.tracking: self.basedata.enableTracking() + if self.skip_fragments: + self.basedata.setVar("_BB_SKIP_FRAGMENTS", "1") # Keep a datastore of the initial environment variables and their # values from when BitBake was launched to enable child processes diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py index e6ff1ff76..bf9bd4f55 100644 --- a/lib/bb/parse/ast.py +++ b/lib/bb/parse/ast.py @@ -351,6 +351,9 @@ class AddFragmentsNode(AstNode): self.builtin_fragments_variable = builtin_fragments_variable def eval(self, data): + if data.getVar('_BB_SKIP_FRAGMENTS') == '1': + return + # No need to use mark_dependency since we would only match a fragment # from a specific layer and there can only be a single layer with a # given namespace.