[1/2] bitbake-setup: support multiple source override files

From: Ernest Van Hoecke <ernest.vanhoecke@toradex.com>

The bitbake-setup documentation currently states that:
"The --source-overrides option can be passed multiple times, in which
case the overrides are applied in the order specified in the
command-line."

However, this is not currently true and only the last --source-overrides
argument is actually used.

Allow --source-overrides to be specified more than once and follow the
behaviour set out in the documentation, i.e.: merge override files in
command-line order.

Extend the setup test to exercise two override files through init,
checking that both are applied and that they are applied in order.

Signed-off-by: Ernest Van Hoecke <ernest.vanhoecke@toradex.com>
---
 bin/bitbake-setup     | 17 +++++++-------
 lib/bb/tests/setup.py | 61 ++++++++++++++++++++++++++++++++++-----------------
 2 files changed, 50 insertions(+), 28 deletions(-)

On Fri, 29 May 2026 at 16:52, Ernest Van Hoecke via
lists.openembedded.org
<ernestvanhoecke=gmail.com@lists.openembedded.org> wrote:
> The bitbake-setup documentation currently states that:
> "The --source-overrides option can be passed multiple times, in which
> case the overrides are applied in the order specified in the
> command-line."
>
> However, this is not currently true and only the last --source-overrides
> argument is actually used.

To be honest, I'd rather fix the documentation, and avoid increasing
code complexity in the lack of clear use case. It's always possible to
merge multiple jsons into one with a simple python script. This would
also keep things consistent with the 'multiple sets of sources in
json' idea, where you'd only pick one of them.

Alex

On Fri, May 29, 2026 at 06:46:32PM +0200, Alexander Kanavin wrote:
> On Fri, 29 May 2026 at 16:52, Ernest Van Hoecke via
> lists.openembedded.org
> <ernestvanhoecke=gmail.com@lists.openembedded.org> wrote:
> > The bitbake-setup documentation currently states that:
> > "The --source-overrides option can be passed multiple times, in which
> > case the overrides are applied in the order specified in the
> > command-line."
> >
> > However, this is not currently true and only the last --source-overrides
> > argument is actually used.
> 
> To be honest, I'd rather fix the documentation, and avoid increasing
> code complexity in the lack of clear use case. It's always possible to
> merge multiple jsons into one with a simple python script. This would
> also keep things consistent with the 'multiple sets of sources in
> json' idea, where you'd only pick one of them.
> 
> Alex
>

I arrived at this from a real use case. Admittedly, that might now be
semi-artifical if we don't want the other default-source-overrides
patch.

My setup idea was to have all sources move, then use a fixed revisions
override to create a "release" setup at known pins, and then apply
another override on top of that one to create an "integration" setup for
CI where most sources are at known pins, but some repos are at latest.

By overriding again, it prevents needing to maintain a "release" and
"integration" sources override files, that are mostly duplication of
each other.

You're right that this can easily be managed with other tools, the small
risk I see is that lots of custom solutions start growing around
bitbake-setup, but with your feedback on the other patch I also see less
value in this one.

In short, the expectation here is that people use their own tooling to
manage the .conf.json and source overrides .json files? If so I'll send
a patch later to remove the multiple --source-overrides option from the
docs.

Kind regards,
Ernest

On Mon, 1 Jun 2026 at 14:43, Ernest Van Hoecke
<ernestvanhoecke@gmail.com> wrote:
> I arrived at this from a real use case. Admittedly, that might now be
> semi-artifical if we don't want the other default-source-overrides
> patch.
>
> My setup idea was to have all sources move, then use a fixed revisions
> override to create a "release" setup at known pins, and then apply
> another override on top of that one to create an "integration" setup for
> CI where most sources are at known pins, but some repos are at latest.
>
> By overriding again, it prevents needing to maintain a "release" and
> "integration" sources override files, that are mostly duplication of
> each other.

But why can't 'release' be a full json config? If that leads to
multiple duplicated configs for each release, then we should pursue
the 'souces-one-of' idea to have multiple source sets in configs.

> You're right that this can easily be managed with other tools, the small
> risk I see is that lots of custom solutions start growing around
> bitbake-setup, but with your feedback on the other patch I also see less
> value in this one.
>
> In short, the expectation here is that people use their own tooling to
> manage the .conf.json and source overrides .json files? If so I'll send
> a patch later to remove the multiple --source-overrides option from the
> docs.

Overiddes are meant as something local and throw-away, and not
something that is ever checked into a public git repo or otherwise
shared. They're not self-describing for example, if you look at the
content, there's no way to tell what is it for. Allowing multiple
overrides that are overlaid on top of each other also makes it more
difficult to tell what would be the final result, it's hard enough to
figure how bitbake variables get their final values, and we don't need
to repeat that :)

So if a certain set of sources is reused between many builds, it
should really be in a config. Bitbake-setup could help with that, e.g.
'take this override and add it to this config' command.

Alex

On Mon, Jun 01, 2026 at 03:50:05PM +0200, Alexander Kanavin wrote:
> On Mon, 1 Jun 2026 at 14:43, Ernest Van Hoecke
> <ernestvanhoecke@gmail.com> wrote:
> > I arrived at this from a real use case. Admittedly, that might now be
> > semi-artifical if we don't want the other default-source-overrides
> > patch.
> >
> > My setup idea was to have all sources move, then use a fixed revisions
> > override to create a "release" setup at known pins, and then apply
> > another override on top of that one to create an "integration" setup for
> > CI where most sources are at known pins, but some repos are at latest.
> >
> > By overriding again, it prevents needing to maintain a "release" and
> > "integration" sources override files, that are mostly duplication of
> > each other.
> 
> But why can't 'release' be a full json config? If that leads to
> multiple duplicated configs for each release, then we should pursue
> the 'souces-one-of' idea to have multiple source sets in configs.
>

Agreed, I'm playing with this idea now. How would you feel about a
top-level definition of:

    "source-override-sets": {
        "release": { "sources": { ... } },
        "dev": { "sources": { ... } }
    }

That can then be selected by a config with

    "source-override-set": "release"

I don't really see the use case of having this be user selectable such
as the "oe-template-one-of" property. Seems to me that it'd just create
another axis of choice for the setup but at least for me, the sources
would always be tied to a certain config. For example, my dev config
also applies some other templates that differentiate it from the release
config, so release+"dev sources" is not enough, and I'm not interested
in creating a dev setup with release sources or vice versa.

I'm struggling a bit with the semantics of this new approach tho. It
makes me think that we really just want multiple sources nodes, not
overrides.
 
> > You're right that this can easily be managed with other tools, the small
> > risk I see is that lots of custom solutions start growing around
> > bitbake-setup, but with your feedback on the other patch I also see less
> > value in this one.
> >
> > In short, the expectation here is that people use their own tooling to
> > manage the .conf.json and source overrides .json files? If so I'll send
> > a patch later to remove the multiple --source-overrides option from the
> > docs.
> 
> Overiddes are meant as something local and throw-away, and not
> something that is ever checked into a public git repo or otherwise
> shared. They're not self-describing for example, if you look at the
> content, there's no way to tell what is it for. Allowing multiple
> overrides that are overlaid on top of each other also makes it more
> difficult to tell what would be the final result, it's hard enough to
> figure how bitbake variables get their final values, and we don't need
> to repeat that :)
> 

That is a very compelling argument :) 

> So if a certain set of sources is reused between many builds, it
> should really be in a config. Bitbake-setup could help with that, e.g.
> 'take this override and add it to this config' command.
> 
> Alex
> 

Agreed and I've experimented with just using jq to overwrite the sources
node, I see this as the way forward. If I find a reason maybe I'll
integrate the behaviour straight  into bitbake-setup once we have an
idea on the multiple source sets/overrides concept. Thanks for your
thinking on this.                     
                                                                         
Ernest

On Mon, 1 Jun 2026 at 21:21, Ernest Van Hoecke
<ernestvanhoecke@gmail.com> wrote:
> Agreed, I'm playing with this idea now. How would you feel about a
> top-level definition of:
>
>     "source-override-sets": {
>         "release": { "sources": { ... } },
>         "dev": { "sources": { ... } }
>     }
>
> That can then be selected by a config with
>
>     "source-override-set": "release"
>
> I don't really see the use case of having this be user selectable such
> as the "oe-template-one-of" property. Seems to me that it'd just create
> another axis of choice for the setup but at least for me, the sources
> would always be tied to a certain config. For example, my dev config
> also applies some other templates that differentiate it from the release
> config, so release+"dev sources" is not enough, and I'm not interested
> in creating a dev setup with release sources or vice versa.
>
> I'm struggling a bit with the semantics of this new approach tho. It
> makes me think that we really just want multiple sources nodes, not
> overrides.

I wouldn't call them 'override sets' as they're actually the starting
point, I'd go with something like 'source-choices'.

The use case for making them user selectable can be seen right here:
https://git.openembedded.org/bitbake/tree/default-registry/configurations

There's a lot of copy-paste in these configs, because each can hold
only one set of sources, and folding different yocto releases (plus
master) into a single config file would be great, so that actual build
configurations need to be written only once. I can imagine downstream
users could use it in a similar way: share configs between different
releases.

Adding restrictions on what config can be used with what set of
sources can be done as a followup, and doesn't need to be decided and
agreed up front. Until then you'd have to maintain that restriction
externally, by ensuring bitbake-setup is called with matching items:

$ bitbake-setup init my-mega-config-file dev-sources dev-config

I'm considering whether the source sets should hold optional allow
lists and deny lists of what configs they can be used with:

    "source-choices": {
         "release": { "sources": { ... }, "configs-allow":
["release"], "description": "...", "expires": "yyyymmdd" },
         "dev": { "sources": { ... }, "configs-allow": ["dev"],
"description": "..." }
     }

This feels more elegant and flexible. But again, it doesn't have to be
implemented up front. The best way to add new features is incremental
and iterative.

Alex

On Tue, Jun 02, 2026 at 10:03:50AM +0200, Alexander Kanavin wrote:
> On Mon, 1 Jun 2026 at 21:21, Ernest Van Hoecke
> <ernestvanhoecke@gmail.com> wrote:
> > Agreed, I'm playing with this idea now. How would you feel about a
> > top-level definition of:
> >
> >     "source-override-sets": {
> >         "release": { "sources": { ... } },
> >         "dev": { "sources": { ... } }
> >     }
> >
> > That can then be selected by a config with
> >
> >     "source-override-set": "release"
> >
> > I don't really see the use case of having this be user selectable such
> > as the "oe-template-one-of" property. Seems to me that it'd just create
> > another axis of choice for the setup but at least for me, the sources
> > would always be tied to a certain config. For example, my dev config
> > also applies some other templates that differentiate it from the release
> > config, so release+"dev sources" is not enough, and I'm not interested
> > in creating a dev setup with release sources or vice versa.
> >
> > I'm struggling a bit with the semantics of this new approach tho. It
> > makes me think that we really just want multiple sources nodes, not
> > overrides.
> 
> I wouldn't call them 'override sets' as they're actually the starting
> point, I'd go with something like 'source-choices'.
> 
> The use case for making them user selectable can be seen right here:
> https://git.openembedded.org/bitbake/tree/default-registry/configurations
> 
> There's a lot of copy-paste in these configs, because each can hold
> only one set of sources, and folding different yocto releases (plus
> master) into a single config file would be great, so that actual build
> configurations need to be written only once. I can imagine downstream
> users could use it in a similar way: share configs between different
> releases.
> 

That makes sense to me, in this case I'd add "source-choices" that
contain multiple "sources" nodes as you proposed. I'd like to start with
just that and the ability for a (leaf) config to select that with a
property "source-choice" so it does not have to be specified by the
user. That would already solve my use case, and also allow for folding
all these OE distros into one.

I can see how you might prefer the choice so you do not even need to 
specify multiple leaf configs, but then indeed we might (later) want
allow lists. I'll send an RFC or PATCH for this and we can see how it
feels.

> Adding restrictions on what config can be used with what set of
> sources can be done as a followup, and doesn't need to be decided and
> agreed up front. Until then you'd have to maintain that restriction
> externally, by ensuring bitbake-setup is called with matching items:
> 
> $ bitbake-setup init my-mega-config-file dev-sources dev-config
> 
> I'm considering whether the source sets should hold optional allow
> lists and deny lists of what configs they can be used with:
> 
>     "source-choices": {
>          "release": { "sources": { ... }, "configs-allow":
> ["release"], "description": "...", "expires": "yyyymmdd" },
>          "dev": { "sources": { ... }, "configs-allow": ["dev"],
> "description": "..." }
>      }
> 
> This feels more elegant and flexible. But again, it doesn't have to be
> implemented up front. The best way to add new features is incremental
> and iterative.
> 
> Alex
>