diff mbox series

[RFC] bitbake: add support for 'built-in' fragments

Message ID 20250616105552.2939169-1-alex.kanavin@gmail.com
State New
Headers show
Series [RFC] bitbake: add support for 'built-in' fragments | expand

Commit Message

Alexander Kanavin June 16, 2025, 10:55 a.m. UTC
From: Alexander Kanavin <alex@linutronix.de>

Note: this deliberately puts bitbake and core changes into one commit
to make RFC review easier.

When reviewing proposed fragments to add settings for DISTRO and MACHINE,
RP noted that such fragments only add clutter and overhead, and there's
no need to maintain them as separate files.

Rather when bitbake sees 'fragmentvar/fragmentvalue' it can expand that into
FRAGMENTVAR = "fragmentvalue".

This is a RFC proposal to add such 'built-in' fragments; I had to pick a
name for the concept, and open to alternative suggestions.

The centerpiece is the

OE_FRAGMENTS_BUILTIN ?= "machine:MACHINE distro:DISTRO"

setting in bitbake.conf, which acts as both a definition and an allowlist
for such fragments.

With that in place, it's possible to set

OE_FRAGMENTS += "distro/poky machine/qemuarm"

and bitbake will interpret that as

DISTRO = "poky"
MACHINE = "qemuarm"

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 .../bitbake-user-manual-metadata.rst          | 19 ++++++++++++++--
 bitbake/lib/bb/parse/ast.py                   | 16 ++++++++++++--
 bitbake/lib/bb/parse/parse_py/ConfHandler.py  |  2 +-
 meta/conf/bitbake.conf                        |  3 ++-
 meta/lib/bbconfigbuild/configfragments.py     | 22 ++++++++++++++++++-
 5 files changed, 55 insertions(+), 7 deletions(-)

Comments

patchtest@automation.yoctoproject.org June 16, 2025, 11 a.m. UTC | #1
Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/RFC-bitbake-add-support-for-built-in-fragments.patch

FAIL: test target mailing list: Series sent to the wrong mailing list or some patches from the series correspond to different mailing lists (test_mbox.TestMbox.test_target_mailing_list)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!
Richard Purdie June 16, 2025, 11:42 a.m. UTC | #2
On Mon, 2025-06-16 at 12:55 +0200, Alexander Kanavin via lists.openembedded.org wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> Note: this deliberately puts bitbake and core changes into one commit
> to make RFC review easier.
> 
> When reviewing proposed fragments to add settings for DISTRO and MACHINE,
> RP noted that such fragments only add clutter and overhead, and there's
> no need to maintain them as separate files.
> 
> Rather when bitbake sees 'fragmentvar/fragmentvalue' it can expand that into
> FRAGMENTVAR = "fragmentvalue".
> 
> This is a RFC proposal to add such 'built-in' fragments; I had to pick a
> name for the concept, and open to alternative suggestions.
> 
> The centerpiece is the
> 
> OE_FRAGMENTS_BUILTIN ?= "machine:MACHINE distro:DISTRO"
> 
> setting in bitbake.conf, which acts as both a definition and an allowlist
> for such fragments.
> 
> With that in place, it's possible to set
> 
> OE_FRAGMENTS += "distro/poky machine/qemuarm"
> 
> and bitbake will interpret that as
> 
> DISTRO = "poky"
> MACHINE = "qemuarm"
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  .../bitbake-user-manual-metadata.rst          | 19 ++++++++++++++--
>  bitbake/lib/bb/parse/ast.py                   | 16 ++++++++++++--
>  bitbake/lib/bb/parse/parse_py/ConfHandler.py  |  2 +-
>  meta/conf/bitbake.conf                        |  3 ++-
>  meta/lib/bbconfigbuild/configfragments.py     | 22 ++++++++++++++++++-
>  5 files changed, 55 insertions(+), 7 deletions(-)

FWIW at a first glance I like this a lot. I need to have a think about
the details but in general I'm very happy to see this.

Cheers,

Richard
diff mbox series

Patch

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
index a27b7758d97..f60a9d83123 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
@@ -998,9 +998,9 @@  This directive allows fine-tuning local configurations with configuration
 snippets contained in layers in a structured, controlled way. Typically it would
 go into ``bitbake.conf``, for example::
 
-   addfragments conf/fragments OE_FRAGMENTS OE_FRAGMENTS_METADATA_VARS
+   addfragments conf/fragments OE_FRAGMENTS OE_FRAGMENTS_METADATA_VARS OE_BUILTIN_FRAGMENTS
 
-``addfragments`` takes three parameters:
+``addfragments`` takes four parameters:
 
 -  path prefix for fragment files inside the layer file tree that bitbake
    uses to construct full paths to the fragment files
@@ -1011,6 +1011,8 @@  go into ``bitbake.conf``, for example::
 -  name of variable that contains a list of variable names containing
    fragment-specific metadata (such as descriptions)
 
+-  name of variable that contains definitions for built-in fragments
+
 This allows listing enabled configuration fragments in ``OE_FRAGMENTS``
 variable like this::
 
@@ -1035,6 +1037,19 @@  The implementation will add a flag containing the fragment name to each of those
 when parsing fragments, so that the variables are namespaced by fragment name, and do not override
 each other when several fragments are enabled.
 
+The variable containing a built-in fragment definitions could look like this::
+
+   OE_BUILTIN_FRAGMENTS = "someprefix:SOMEVARIABLE anotherprefix:ANOTHERVARIABLE"
+
+and then if 'someprefix/somevalue' is added to the variable that holds the list
+of enabled fragments:
+
+  OE_FRAGMENTS = "... someprefix/somevalue"
+
+bitbake will treat that as direct value assignment in its configuration::
+
+  SOMEVARIABLE = "somevalue"
+
 Functions
 =========
 
diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
index 290ed45048c..f9133c4e60b 100644
--- a/bitbake/lib/bb/parse/ast.py
+++ b/bitbake/lib/bb/parse/ast.py
@@ -345,11 +345,12 @@  class InheritDeferredNode(AstNode):
         data.setVar('__BBDEFINHERITS', inherits)
 
 class AddFragmentsNode(AstNode):
-    def __init__(self, filename, lineno, fragments_path_prefix, fragments_variable, flagged_variables_list_variable):
+    def __init__(self, filename, lineno, fragments_path_prefix, fragments_variable, flagged_variables_list_variable, builtin_fragments_variable):
         AstNode.__init__(self, filename, lineno)
         self.fragments_path_prefix = fragments_path_prefix
         self.fragments_variable = fragments_variable
         self.flagged_variables_list_variable = flagged_variables_list_variable
+        self.builtin_fragments_variable = builtin_fragments_variable
 
     def eval(self, data):
         # No need to use mark_dependency since we would only match a fragment
@@ -362,13 +363,23 @@  class AddFragmentsNode(AstNode):
                    return candidate_fragment_path
            return None
 
+        def check_and_set_builtin_fragment(fragment, data, builtin_fragments):
+            prefix, value = fragment.split('/', 1)
+            if prefix in builtin_fragments.keys():
+                data.setVar(builtin_fragments[prefix], value)
+                return True
+            return False
+
         fragments = data.getVar(self.fragments_variable)
         layers = data.getVar('BBLAYERS')
         flagged_variables = data.getVar(self.flagged_variables_list_variable).split()
+        builtin_fragments = {f[0]:f[1] for f in [f.split(':') for f in data.getVar(self.builtin_fragments_variable).split()] }
 
         if not fragments:
             return
         for f in fragments.split():
+            if check_and_set_builtin_fragment(f, data, builtin_fragments):
+                continue
             layerid, fragment_name = f.split('/', 1)
             full_fragment_name = data.expand("{}/{}.conf".format(self.fragments_path_prefix, fragment_name))
             fragment_path = find_fragment(layers, layerid, full_fragment_name)
@@ -432,7 +443,8 @@  def handleAddFragments(statements, filename, lineno, m):
     fragments_path_prefix = m.group(1)
     fragments_variable = m.group(2)
     flagged_variables_list_variable = m.group(3)
-    statements.append(AddFragmentsNode(filename, lineno, fragments_path_prefix, fragments_variable, flagged_variables_list_variable))
+    builtin_fragments_variable = m.group(4)
+    statements.append(AddFragmentsNode(filename, lineno, fragments_path_prefix, fragments_variable, flagged_variables_list_variable, builtin_fragments_variable))
 
 def runAnonFuncs(d):
     code = []
diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
index 675838d8452..9ddbae123dc 100644
--- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
@@ -48,7 +48,7 @@  __export_regexp__ = re.compile( r"export\s+([a-zA-Z0-9\-_+.${}/~]+)$" )
 __unset_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/~]+)$" )
 __unset_flag_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/~]+)\[([a-zA-Z0-9\-_+.][a-zA-Z0-9\-_+.@]+)\]$" )
 __addpylib_regexp__      = re.compile(r"addpylib\s+(.+)\s+(.+)" )
-__addfragments_regexp__  = re.compile(r"addfragments\s+(.+)\s+(.+)\s+(.+)" )
+__addfragments_regexp__  = re.compile(r"addfragments\s+(.+)\s+(.+)\s+(.+)\s+(.+)" )
 
 def init(data):
     return
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index e600d9d774a..b2414ec2a7f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -821,7 +821,8 @@  include conf/local.conf
 
 OE_FRAGMENTS_PREFIX ?= "conf/fragments"
 OE_FRAGMENTS_METADATA_VARS ?= "BB_CONF_FRAGMENT_SUMMARY BB_CONF_FRAGMENT_DESCRIPTION"
-addfragments ${OE_FRAGMENTS_PREFIX} OE_FRAGMENTS OE_FRAGMENTS_METADATA_VARS
+OE_FRAGMENTS_BUILTIN ?= "machine:MACHINE distro:DISTRO"
+addfragments ${OE_FRAGMENTS_PREFIX} OE_FRAGMENTS OE_FRAGMENTS_METADATA_VARS OE_FRAGMENTS_BUILTIN
 
 require ${@"conf/multiconfig/${BB_CURRENT_MC}.conf" if "${BB_CURRENT_MC}" != "" else ""}
 include conf/machine/${MACHINE}.conf
diff --git a/meta/lib/bbconfigbuild/configfragments.py b/meta/lib/bbconfigbuild/configfragments.py
index c1dddc3e4cb..61c33ac3161 100644
--- a/meta/lib/bbconfigbuild/configfragments.py
+++ b/meta/lib/bbconfigbuild/configfragments.py
@@ -62,7 +62,22 @@  class ConfigFragmentsPlugin(LayerPlugin):
             else:
                 print('Name: {}\nPath: {}\nEnabled: {}\nSummary: {}\nDescription:\n{}\n'.format(f['name'], f['path'], 'yes' if is_enabled else 'no', f['summary'],''.join(f['description'])))
 
+        def print_builtin_fragments(builtin, enabled):
+            print('Available built-in fragments:')
+            builtin_dict = {i[0]:i[1] for i in [f.split(':') for f in builtin]}
+            for prefix,var in builtin_dict.items():
+                print('{}/...\tSets {} = ...'.format(prefix, var))
+            print('')
+            enabled_builtin_fragments = [f for f in enabled if self.builtin_fragment_exists(f)]
+            print('Enabled built-in fragments:')
+            for f in enabled_builtin_fragments:
+                 prefix, value = f.split('/', 1)
+                 print('{}\tSets {} = "{}"'.format(f, builtin_dict[prefix], value))
+            print('')
+
         all_enabled_fragments = (self.tinfoil.config_data.getVar('OE_FRAGMENTS') or "").split()
+        all_builtin_fragments = (self.tinfoil.config_data.getVar('OE_FRAGMENTS_BUILTIN') or "").split()
+        print_builtin_fragments(all_builtin_fragments, all_enabled_fragments)
 
         for layername, layerdata in self.discover_fragments().items():
             layerdir = layerdata['layerdir']
@@ -89,6 +104,11 @@  class ConfigFragmentsPlugin(LayerPlugin):
                   return True
         return False
 
+    def builtin_fragment_exists(self, fragmentname):
+        fragment_prefix = fragmentname.split("/",1)[0]
+        fragment_prefix_defs = set([f.split(':')[0] for f in self.tinfoil.config_data.getVar('OE_FRAGMENTS_BUILTIN').split()])
+        return fragment_prefix in fragment_prefix_defs
+
     def create_conf(self, confpath):
         if not os.path.exists(confpath):
             with open(confpath, 'w') as f:
@@ -112,7 +132,7 @@  class ConfigFragmentsPlugin(LayerPlugin):
             return " ".join(enabled_fragments), None, 0, True
 
         for f in args.fragmentname:
-            if not self.fragment_exists(f):
+            if not self.fragment_exists(f) and not self.builtin_fragment_exists(f):
                 raise Exception("Fragment {} does not exist; use 'list-fragments' to see the full list.".format(f))
 
         self.create_conf(args.confpath)