diff mbox series

[2/2] bitbake-config-build: add a plugin for config fragments

Message ID 20241114111100.2624737-2-alex.kanavin@gmail.com
State New
Headers show
Series [1/2] bitbake.conf: add an addfragments directive for oe-core and dependent layers | expand

Commit Message

Alexander Kanavin Nov. 14, 2024, 11:11 a.m. UTC
From: Alexander Kanavin <alex@linutronix.de>

This allows fine-tuning local configurations with pre-frabricated
configuration snippets in a structured, controlled way. It's also
an important building block for bitbake-setup.

There are three (and a half) operations (list/enable/disable/disable all), and here's the 'list' output:

alex@Zen2:/srv/storage/alex/yocto/build-64$ bitbake-config-build list-fragments
NOTE: Starting bitbake server...
Available fragments in selftest layer located in /srv/work/alex/poky/meta-selftest:

selftest/test-another-fragment	This is a second configuration fragment intended for testing in oe-selftest context
selftest/test-fragment	This is a configuration fragment intended for testing in oe-selftest context

The tool requires that each fragment contains a one-line summary, and one or more
lines of description, as BB_CONF_FRAGMENT_SUMMARY[fragmentname] style metadata.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 .../selftest/test-another-fragment.conf       |   3 +
 .../fragments/selftest/test-fragment.conf     |   3 +
 meta/lib/bbconfigbuild/configfragments.py     | 138 ++++++++++++++++++
 meta/lib/oeqa/selftest/cases/bblayers.py      |  31 ++++
 4 files changed, 175 insertions(+)
 create mode 100644 meta-selftest/conf/fragments/selftest/test-another-fragment.conf
 create mode 100644 meta-selftest/conf/fragments/selftest/test-fragment.conf
 create mode 100644 meta/lib/bbconfigbuild/configfragments.py

Comments

patchtest@automation.yoctoproject.org Nov. 14, 2024, 11:16 a.m. UTC | #1
Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/2-2-bitbake-config-build-add-a-plugin-for-config-fragments.patch

FAIL: test max line length: Patch line too long (current length 213, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!
Mathieu Dubois-Briand Nov. 14, 2024, 1:20 p.m. UTC | #2
On Thu, Nov 14, 2024 at 12:11:00PM +0100, Alexander Kanavin via lists.openembedded.org wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> This allows fine-tuning local configurations with pre-frabricated
> configuration snippets in a structured, controlled way. It's also
> an important building block for bitbake-setup.
> 
> There are three (and a half) operations (list/enable/disable/disable all), and here's the 'list' output:
> 
> alex@Zen2:/srv/storage/alex/yocto/build-64$ bitbake-config-build list-fragments
> NOTE: Starting bitbake server...
> Available fragments in selftest layer located in /srv/work/alex/poky/meta-selftest:
> 
> selftest/test-another-fragment	This is a second configuration fragment intended for testing in oe-selftest context
> selftest/test-fragment	This is a configuration fragment intended for testing in oe-selftest context
> 
> The tool requires that each fragment contains a one-line summary, and one or more
> lines of description, as BB_CONF_FRAGMENT_SUMMARY[fragmentname] style metadata.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>

Hi,

I did not manage to apply this patch on master as I get conflicts on
meta/lib/oeqa/selftest/cases/bblayers.py.

I might be missing something, but can do you confirm this patch is to be
applied on master without any dependency on another patch ?
Alexander Kanavin Nov. 14, 2024, 1:33 p.m. UTC | #3
On Thu, 14 Nov 2024 at 14:20, Mathieu Dubois-Briand
<mathieu.dubois-briand@bootlin.com> wrote:

> I did not manage to apply this patch on master as I get conflicts on
> meta/lib/oeqa/selftest/cases/bblayers.py.
>
> I might be missing something, but can do you confirm this patch is to be
> applied on master without any dependency on another patch ?

A mistake on my side: I sent it from a branch that had additional
unrelated commits before these. I removed that, and just sent a v2.

Alex
diff mbox series

Patch

diff --git a/meta-selftest/conf/fragments/selftest/test-another-fragment.conf b/meta-selftest/conf/fragments/selftest/test-another-fragment.conf
new file mode 100644
index 00000000000..4b4bf537964
--- /dev/null
+++ b/meta-selftest/conf/fragments/selftest/test-another-fragment.conf
@@ -0,0 +1,3 @@ 
+BB_CONF_FRAGMENT_SUMMARY[selftest/test-another-fragment] = "This is a second configuration fragment intended for testing in oe-selftest context"
+BB_CONF_FRAGMENT_DESCRIPTION[selftest/test-another-fragment] = "It defines another variable that can be checked inside the test."
+SELFTEST_FRAGMENT_ANOTHER_VARIABLE = "someothervalue"
diff --git a/meta-selftest/conf/fragments/selftest/test-fragment.conf b/meta-selftest/conf/fragments/selftest/test-fragment.conf
new file mode 100644
index 00000000000..63ebc1fca68
--- /dev/null
+++ b/meta-selftest/conf/fragments/selftest/test-fragment.conf
@@ -0,0 +1,3 @@ 
+BB_CONF_FRAGMENT_SUMMARY[selftest/test-fragment] = "This is a configuration fragment intended for testing in oe-selftest context"
+BB_CONF_FRAGMENT_DESCRIPTION[selftest/test-fragment] = "It defines a variable that can be checked inside the test."
+SELFTEST_FRAGMENT_VARIABLE = "somevalue"
diff --git a/meta/lib/bbconfigbuild/configfragments.py b/meta/lib/bbconfigbuild/configfragments.py
new file mode 100644
index 00000000000..f0d6e87b8ff
--- /dev/null
+++ b/meta/lib/bbconfigbuild/configfragments.py
@@ -0,0 +1,138 @@ 
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import logging
+import os
+import sys
+
+import bb.utils
+
+from bblayers.common import LayerPlugin
+
+logger = logging.getLogger('bitbake-config-layers')
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+def plugin_init(plugins):
+    return ConfigFragmentsPlugin()
+
+class ConfigFragmentsPlugin(LayerPlugin):
+    def get_fragment_info(self, path, name):
+        d = bb.data.init()
+        bb.parse.handle(path, d, True)
+        summary = d.getVarFlag('BB_CONF_FRAGMENT_SUMMARY', name)
+        description = d.getVarFlag('BB_CONF_FRAGMENT_DESCRIPTION', name)
+        if not summary:
+            raise Exception('Please add a one-line summary as BB_CONF_FRAGMENT_SUMMARY[{}] = \"...\" variable at the beginning of {}'.format(name, path))
+
+        if not description:
+            raise Exception('Please add a description as BB_CONF_FRAGMENT_DESCRIPTION[{}] = \"...\" variable at the beginning of {}'.format(name, path))
+
+        return summary, description
+
+    def discover_fragments(self):
+        fragments_path_prefix = self.tinfoil.config_data.getVar('OE_FRAGMENTS_PREFIX')
+        allfragments = {}
+        for layername in self.bbfile_collections:
+             layerdir = self.bbfile_collections[layername]
+             fragments = []
+             for topdir, dirs, files in os.walk(os.path.join(layerdir, fragments_path_prefix)):
+                 fragmentdir = topdir.split(fragments_path_prefix+'/')[-1]
+                 for fragmentfile in sorted(files):
+                     fragmentname = "/".join((fragmentdir, fragmentfile.split('.')[0]))
+                     fragmentpath = os.path.join(topdir, fragmentfile)
+                     fragmentsummary, fragmentdesc = self.get_fragment_info(fragmentpath, fragmentname)
+                     fragments.append({'path':fragmentpath, 'name':fragmentname, 'summary':fragmentsummary, 'description':fragmentdesc})
+             if fragments:
+                 allfragments[layername] = {'layerdir':layerdir,'fragments':fragments}
+        return allfragments
+
+    def do_list_fragments(self, args):
+        """ List available configuration fragments """
+        enabled_fragments = (self.tinfoil.config_data.getVar('OE_FRAGMENTS') or "").split()
+
+        for layername, layerdata in self.discover_fragments().items():
+            layerdir = layerdata['layerdir']
+            fragments = layerdata['fragments']
+
+            print('Available fragments in {} layer located in {}:\n'.format(layername, layerdir))
+            for f in fragments:
+                if not args.verbose:
+                    print('{}\t{}\t{}'.format(f['name'], '(enabled)' if f['name'] in enabled_fragments else '(disabled)', f['summary']))
+                else:
+                    print('Name: {}\nPath: {}\nEnabled: {}\nSummary: {}\nDescription:\n{}\n'.format(f['name'], f['path'], 'yes' if f['name'] in enabled_fragments else 'no', f['summary'],''.join(f['description'])))
+            print('')
+
+    def fragment_exists(self, fragmentname):
+        for layername, layerdata in self.discover_fragments().items():
+            for f in layerdata['fragments']:
+              if f['name'] == fragmentname:
+                  return True
+        return False
+
+    def read_conf(self, confpath):
+        try:
+            with open(confpath) as f:
+                lines = f.readlines()
+        except Exception:
+            lines = []
+        return lines
+
+    def do_enable_fragment(self, args):
+        """ Enable a fragment in the local build configuration """
+        if not self.fragment_exists(args.fragmentname):
+            raise Exception("Fragment {} does not exist; use 'list-fragments' to see the full list.".format(args.fragmentname))
+
+        appendline = "OE_FRAGMENTS += \"{}\"\n".format(args.fragmentname)
+
+        lines = self.read_conf(args.confpath)
+        for l in lines:
+            if l == appendline:
+                print("Fragment {} already included in {}".format(args.fragmentname, args.confpath))
+                return
+
+        lines.append(appendline)
+        with open(args.confpath, 'w') as f:
+            f.write(''.join(lines))
+        print("Fragment {} added to {}.".format(args.fragmentname, args.confpath))
+
+    def do_disable_fragment(self, args):
+        """ Disable a fragment in the local build configuration """
+        removeline = "OE_FRAGMENTS += \"{}\"\n".format(args.fragmentname)
+
+        lines = [l for l in self.read_conf(args.confpath) if l != removeline]
+
+        with open(args.confpath, 'w') as f:
+            f.write(''.join(lines))
+        print("Fragment {} removed from {}.".format(args.fragmentname, args.confpath))
+
+    def do_disable_all_fragments(self, args):
+        """ Disable all fragments in the local build configuration """
+        removeline = "OE_FRAGMENTS += "
+
+        lines = [l for l in self.read_conf(args.confpath) if not l.startswith(removeline)]
+
+        with open(args.confpath, 'w') as f:
+            f.write(''.join(lines))
+        print("All fragment removed from {}.".format(args.confpath))
+
+    def register_commands(self, sp):
+        default_confpath = os.path.join(os.environ["BBPATH"], "conf/auto.conf")
+
+        parser_list_fragments = self.add_command(sp, 'list-fragments', self.do_list_fragments, parserecipes=False)
+        parser_list_fragments.add_argument("--confpath", default=default_confpath, help='Configuration file which contains a list of enabled fragments (default is {}).'.format(default_confpath))
+        parser_list_fragments.add_argument('--verbose', '-v', action='store_true', help='Print extended descriptions of the fragments')
+
+        parser_enable_fragment = self.add_command(sp, 'enable-fragment', self.do_enable_fragment, parserecipes=False)
+        parser_enable_fragment.add_argument("--confpath", default=default_confpath, help='Configuration file which contains a list of enabled fragments (default is {}).'.format(default_confpath))
+        parser_enable_fragment.add_argument('fragmentname', help='The name of the fragment (use list-fragments to see them)')
+
+        parser_disable_fragment = self.add_command(sp, 'disable-fragment', self.do_disable_fragment, parserecipes=False)
+        parser_disable_fragment.add_argument("--confpath", default=default_confpath, help='Configuration file which contains a list of enabled fragments (default is {}).'.format(default_confpath))
+        parser_disable_fragment.add_argument('fragmentname', help='The name of the fragment')
+
+        parser_disable_all = self.add_command(sp, 'disable-all-fragments', self.do_disable_all_fragments, parserecipes=False)
+        parser_disable_all.add_argument("--confpath", default=default_confpath, help='Configuration file which contains a list of enabled fragments (default is {}).'.format(default_confpath))
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py
index 8b2bc319d50..945d9a511f7 100644
--- a/meta/lib/oeqa/selftest/cases/bblayers.py
+++ b/meta/lib/oeqa/selftest/cases/bblayers.py
@@ -253,3 +253,34 @@  class BitbakeLayers(OESelftestTestCase):
                 meta_selftest_found = True
         self.assertTrue(oe_core_found, "meta/conf/layer.conf not found in {}".format(testcopydir))
         self.assertTrue(meta_selftest_found, "meta-selftest/conf/layer.conf not found in {}".format(testcopydir))
+
+class BitbakeConfigBuild(OESelftestTestCase):
+    def test_add_remove_fragments(self):
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_VARIABLE'), None)
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_ANOTHER_VARIABLE'), None)
+
+        runCmd('bitbake-config-build enable-fragment selftest/test-fragment')
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_VARIABLE'), 'somevalue')
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_ANOTHER_VARIABLE'), None)
+
+        runCmd('bitbake-config-build enable-fragment selftest/test-another-fragment')
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_VARIABLE'), 'somevalue')
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_ANOTHER_VARIABLE'), 'someothervalue')
+
+        fragment_metadata_command = "bitbake-getvar -f {} --value {}"
+        result = runCmd(fragment_metadata_command.format("selftest/test-fragment", "BB_CONF_FRAGMENT_SUMMARY"))
+        self.assertIn("This is a configuration fragment intended for testing in oe-selftest context", result.output)
+        result = runCmd(fragment_metadata_command.format("selftest/test-fragment", "BB_CONF_FRAGMENT_DESCRIPTION"))
+        self.assertIn("It defines a variable that can be checked inside the test.", result.output)
+        result = runCmd(fragment_metadata_command.format("selftest/test-another-fragment", "BB_CONF_FRAGMENT_SUMMARY"))
+        self.assertIn("This is a second configuration fragment intended for testing in oe-selftest context", result.output)
+        result = runCmd(fragment_metadata_command.format("selftest/test-another-fragment", "BB_CONF_FRAGMENT_DESCRIPTION"))
+        self.assertIn("It defines another variable that can be checked inside the test.", result.output)
+
+        runCmd('bitbake-config-build disable-fragment selftest/test-fragment')
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_VARIABLE'), None)
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_ANOTHER_VARIABLE'), 'someothervalue')
+
+        runCmd('bitbake-config-build disable-fragment selftest/test-another-fragment')
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_VARIABLE'), None)
+        self.assertEqual(get_bb_var('SELFTEST_FRAGMENT_ANOTHER_VARIABLE'), None)