diff mbox series

[RFC] bitbake-setup: add source choices

Message ID 20260616-bb-setup-source-choices-integrated-tests-v1-1-a1e51bdeead5@toradex.com
State New
Headers show
Series [RFC] bitbake-setup: add source choices | expand

Commit Message

Ernest Van Hoecke June 16, 2026, 11:05 a.m. UTC
From: Ernest Van Hoecke <ernest.vanhoecke@toradex.com>

Allow setup configuration files to define named source sets with a
top-level source-choices property.

BitBake configurations can use source-selection to name one source choice
or list the choices the user may select from. Single-choice selections
are used directly, while multi-choice selections can be passed on the
command line or selected interactively.

Store the selected source choice in config-upstream.json so status and
update continue to use the same source set after init.

Signed-off-by: Ernest Van Hoecke <ernest.vanhoecke@toradex.com>
---
In a previously sent series [1], the idea came up to allow defining
multiple named sets of sources in the top level configuration.

A configuration could then select a set of sources. This way, multiple
setups can share configuration even if they use, for example, different
source revisions. Such a change would allow folding the current Poky
master, wrynose and whinlatter setups into one config file:

https://git.openembedded.org/bitbake/tree/default-registry/configurations

This patch adds a top-level "source-choices" that can contain named
sets of sources.

Alex's original idea was to have source choices be user selectable, with
a possible later add-on of an allow list to limit sources to certain
configs. Possibly, the source choices schema would also have an expires
field.

In this proposal, I instead added an optional "source-selection"
property to the configurations node. This would behave as follows:
  * If the selected config has no "source-selection": use top-level
    "sources"
  * If the selected config has "source-selection", but no top-level
    "source-choices" is present: error out
  * If the selected config has "source-selection" and "source-choices"
    is present:
    - When "source-selection" is a string or 1-element array, choice is
      made automatically
    - When "source-selection" is an array with multiple entries, user
      has to make a choice out of the given options

A BitBake configuration opts into source choices by setting
"source-selection". Configurations without "source-selection" continue
to use top-level "sources".

This "source-selection" mechanism has a few benefits to me:
  * Limits each configuration to the source choices that make sense for
    it.
  * Allows defining configs that are tied to a given set of sources
    without requiring more choices
  * Configs already have an "expires" field, with this things can be
    tightly coupled and source-choices do not need to be aware of
    expirations
  * Existing configs do not change behaviour even if "source-choices" is
    added to the top-level.

Drawbacks:
  * Positional arguments can now be a choice for oe-fragments-one-of or
    for source-selection. This follows the existing oe-fragments-one-of
    command line style.

Points for review:
  * Should we allow "sources" to not be present if "source-choices" are
    defined?
  * Should "source-selection" share the existing positional argument
    namespace with "oe-fragments-one-of", or should source choices use a
    separate command line mechanism?

Sending this as an RFC to get some feedback on the shape of this patch
before adding documentation and test coverage.

[1] https://lore.kernel.org/bitbake-devel/CANNYZj9rKRS0CD-LGq=phvB_T1XS+SmR5Uti0p8tu49PtTm-QA@mail.gmail.com/
---
 bin/bitbake-setup                      | 63 ++++++++++++++++++++++++++++++++--
 setup-schema/bitbake-setup.schema.json | 42 +++++++++++++++++++++++
 2 files changed, 102 insertions(+), 3 deletions(-)


---
base-commit: 7e6466f48191c1e4ab9b91705deb237eff2c7f01
change-id: 20260616-bb-setup-source-choices-integrated-tests-5c2d3a31a95e

Best regards,

Comments

Alexander Kanavin June 16, 2026, 12:27 p.m. UTC | #1
On Tue, 16 Jun 2026 at 13:05, Ernest Van Hoecke via
lists.openembedded.org
<ernestvanhoecke=gmail.com@lists.openembedded.org> wrote:
> In this proposal, I instead added an optional "source-selection"
> property to the configurations node. This would behave as follows:
>   * If the selected config has no "source-selection": use top-level
>     "sources"
>   * If the selected config has "source-selection", but no top-level
>     "source-choices" is present: error out
>   * If the selected config has "source-selection" and "source-choices"
>     is present:
>     - When "source-selection" is a string or 1-element array, choice is
>       made automatically
>     - When "source-selection" is an array with multiple entries, user
>       has to make a choice out of the given options

Thanks for pushing this forward!

Keep in mind that I'd like to support 'folding' existing registry
configurations with this into just two files (oe, poky), so any
proposal should include a rework of them that showcases the feature
(e.g. add new json files, but don't touch existing ones).

Allowing both 'sources' and 'source-choices' complicates the logic and
understanding of it. The relation between 'sources' and
'source-choices' when both are present is not obvious. The use case of
choosing from all available options when there's no explicit
'source-selection' is not supported, it just falls back to 'sources'.

So I'm leaning towards making "sources" and "source-choices" mutually
exclusive, you can have one, or the other, but not both. This makes
the logic much simpler: if you have 'sources', then that is always
selected because there is nothing else. If you have 'source-choices',
then the user needs to select from available options (subject to
possible filters), unless there is only one choice, which is then
auto-selected. Making a choice works same way as for configurations:
either an identifier on the command line, or interactive selection.
Each choice should have its own expiration date, because each choice
can be a yocto release.

There's also the question of the order in which the user should be
making selections: what is more natural, sources first, configurations
second, or the opposite? Typically one selects a yocto release (or tip
of master) first, and a particular configuration in that release next,
but this RFC forces the opposite order. I think we should keep the
order.

To keep things manageable, I would suggest that this is split into
several commits, so the parts that are not controversial can be merged
quicker, and the parts that are can be refined and adjusted.
Specifically, can you first add the mutliple source choices support
that can be used with any configuration, and then add 'source-choices'
filter in a separate commit? So that how we do filters can be
discussed separately.

Thanks,
Alex
diff mbox series

Patch

diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index 2829e753e145..4aeb78798f86 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -462,8 +462,14 @@  def merge_overrides_into_sources(sources, overrides):
             layers[k] = v
     return layers
 
+def get_configured_sources(config):
+    selected_source_choice = config.get("selected-source-choice")
+    if selected_source_choice is not None:
+        return config["data"]["source-choices"][selected_source_choice]["sources"]
+    return config["data"]["sources"]
+
 def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt", init_vscode=False, rebase_conflicts_strategy='abort'):
-    layer_config = merge_overrides_into_sources(config["data"]["sources"], config["source-overrides"]["sources"])
+    layer_config = merge_overrides_into_sources(get_configured_sources(config), config["source-overrides"]["sources"])
     sources_fixed_revisions = checkout_layers(layer_config, confdir, layerdir, d, rebase_conflicts_strategy=rebase_conflicts_strategy)
     bitbake_config = config["bitbake-config"]
     thisdir = os.path.dirname(config["path"]) if config["type"] == 'local' else None
@@ -594,6 +600,38 @@  def choose_fragments(possibilities, parameters, non_interactive, skip_selection)
         choices[k] = options_enumerated[option_n][1]["name"]
     return choices
 
+def choose_source_choice(possibilities, selection, parameters, non_interactive):
+    possible_choices = selection if isinstance(selection, list) else [selection]
+    if not possible_choices:
+        raise Exception("Source selection does not contain any source choices.")
+
+    missing_choices = [c for c in possible_choices if c not in possibilities]
+    if missing_choices:
+        raise Exception("Unknown source choice(s): {}".format(missing_choices))
+
+    if len(possible_choices) == 1:
+        only_choice = possible_choices[0]
+        logger.plain("\nSelecting the only available source choice {}".format(only_choice))
+        return only_choice
+
+    cmdline_choices = [c for c in possible_choices if c in parameters]
+    if len(cmdline_choices) > 1:
+        raise Exception("Options specified on command line do not allow a single selection "
+                        f"from source choices {possible_choices}, please remove one or more from {parameters}")
+    if len(cmdline_choices) == 1:
+        return cmdline_choices[0]
+
+    if non_interactive:
+        raise Exception(f"Unable to choose from source choices in non-interactive mode: {possible_choices}")
+
+    logger.plain("")
+    print_configs("Available source choices",
+                  possible_choices,
+                  [possibilities[c]["description"] for c in possible_choices])
+    choice_n = int_input([i[0] for i in list(enumerate(possible_choices, 1))],
+                         "\nPlease select one of the above source choices by its number: ") - 1
+    return possible_choices[choice_n]
+
 def obtain_config(top_dir, registry, args, source_overrides, d):
     if args.config:
         config_id = args.config[0]
@@ -633,8 +671,27 @@  def obtain_config(top_dir, registry, args, source_overrides, d):
         upstream_config = {'type':'registry','registry':registry,'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))}
 
     upstream_config['bitbake-config'] = choose_bitbake_config(upstream_config['data']['bitbake-setup']['configurations'], config_parameters, args.non_interactive)
+
+    source_selection = upstream_config['bitbake-config'].get('source-selection')
+
+    selected_source_choice = None
+    if source_selection is None:
+        if 'sources' not in upstream_config['data']:
+            raise Exception("Configuration template does not define 'sources', and the selected BitBake configuration does not select a source choice.")
+    else:
+        source_choices = upstream_config['data'].get('source-choices')
+        if source_choices is None:
+            raise Exception("BitBake configuration uses 'source-selection' but the configuration template does not define 'source-choices'.")
+        selected_source_choice = choose_source_choice(source_choices, source_selection, config_parameters[1:], args.non_interactive)
+        upstream_config['selected-source-choice'] = selected_source_choice
+
     upstream_config['bitbake-config']['oe-fragment-choices'] = choose_fragments(upstream_config['bitbake-config'].get('oe-fragments-one-of',{}), config_parameters[1:], args.non_interactive, args.skip_selection)
-    upstream_config['non-interactive-cmdline-options'] = [config_id, upstream_config['bitbake-config']['name']] + sorted(upstream_config['bitbake-config']['oe-fragment-choices'].values())
+
+    non_interactive_cmdline_options = [config_id, upstream_config['bitbake-config']['name']]
+    if selected_source_choice is not None:
+        non_interactive_cmdline_options.append(selected_source_choice)
+    non_interactive_cmdline_options += sorted(upstream_config['bitbake-config']['oe-fragment-choices'].values())
+    upstream_config['non-interactive-cmdline-options'] = non_interactive_cmdline_options
     upstream_config['source-overrides'] = source_overrides
     upstream_config['skip-selection'] = args.skip_selection
     return upstream_config
@@ -952,7 +1009,7 @@  def build_status(top_dir, settings, args, d, update=False):
             bb.process.run(["git", "-C", confdir, "restore", "config-upstream.json"])
         return
 
-    layer_config = merge_overrides_into_sources(current_upstream_config["data"]["sources"], current_upstream_config["source-overrides"]["sources"])
+    layer_config = merge_overrides_into_sources(get_configured_sources(current_upstream_config), current_upstream_config["source-overrides"]["sources"])
     if are_layers_changed(layer_config, layerdir, d):
         if update:
             update_build(current_upstream_config, confdir, setupdir, layerdir,
diff --git a/setup-schema/bitbake-setup.schema.json b/setup-schema/bitbake-setup.schema.json
index 99f47f73d0a6..149f1ebab558 100644
--- a/setup-schema/bitbake-setup.schema.json
+++ b/setup-schema/bitbake-setup.schema.json
@@ -15,6 +15,30 @@ 
         "sources": {
             "$ref": "layers.schema.json#/properties/sources"
         },
+        "source-choices": {
+            "type": "object",
+            "description": "Named choices of sources that configurations can select from with source-selection",
+            "patternProperties": {
+                ".*": {
+                    "type": "object",
+                    "required": [
+                        "description",
+                        "sources"
+                    ],
+                    "properties": {
+                        "description": {
+                            "type": "string",
+                            "description": "Human-readable description of the source choice"
+                        },
+                        "sources": {
+                            "$ref": "layers.schema.json#/properties/sources"
+                        }
+                    },
+                    "additionalProperties": false
+                }
+            },
+            "additionalProperties": false
+        },
         "expires": {
             "type": "string",
             "description": "End of life date for this configuration, in ISO 8601 format (YYYY-MM-DD)"
@@ -73,6 +97,24 @@ 
                                 "type": "string",
                                 "description": "OE-template configuration"
                             },
+                            "source-selection": {
+                                "description": "Source choice, or list of source choices, that can be used with this configuration",
+                                "oneOf": [
+                                    {
+                                        "type": "string",
+                                        "description": "Named source choice to use for this configuration"
+                                    },
+                                    {
+                                        "type": "array",
+                                        "description": "List of named source choices that can be chosen from for this configuration",
+                                        "minItems": 1,
+                                        "items": {
+                                            "type": "string",
+                                            "description": "Named source choice"
+                                        }
+                                    }
+                                ]
+                            },
                             "oe-fragments": {
                                 "$anchor": "oe-fragments",
                                 "type": "array",