[1/2] meta-poky/conf/templates/default/bblayers.conf.sample: do not assume integrated poky repo

From: Alexander Kanavin <alex@linutronix.de>

The list of layers was written assuming the template is used with
the 'integrated' poky checkout, where content of meta-yocto is
'overlaid' on content of openembedded-core, and the layers are
all in the same top level poky repository directory.

This doesn't work when oe-core and meta-yocto are cloned separately,
where relative path that starts from oe-core would have to go one
level above oe-core and then go into meta-yocto, assuming the
separate repo clones are in the same top level directory.

This requires a custom tweak to bblayers.conf when it makes its way from
meta-yocto into poky that would strip the ../meta-yocto/ part in
BBLAYERS. As there are already numerous other tweaks of similar
nature, I think this is feasible.

This was prompted by
https://lists.openembedded.org/g/openembedded-architecture/topic/layer_layout_poky_vs/111010762

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta-poky/conf/templates/default/bblayers.conf.sample | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

On Mon, 2025-06-23 at 18:55 +0200, Alexander Kanavin via
lists.yoctoproject.org wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> The list of layers was written assuming the template is used with
> the 'integrated' poky checkout, where content of meta-yocto is
> 'overlaid' on content of openembedded-core, and the layers are
> all in the same top level poky repository directory.
> 
> This doesn't work when oe-core and meta-yocto are cloned separately,
> where relative path that starts from oe-core would have to go one
> level above oe-core and then go into meta-yocto, assuming the
> separate repo clones are in the same top level directory.
> 
> This requires a custom tweak to bblayers.conf when it makes its way
> from meta-yocto into poky that would strip the ../meta-yocto/ part in
> BBLAYERS. As there are already numerous other tweaks of similar
> nature, I think this is feasible.

Which other tweaks of a similar nature are made? I'm not aware of any
file contents changing between repos...

Cheers,

Richard

On Tue, 24 Jun 2025 at 15:11, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > This requires a custom tweak to bblayers.conf when it makes its way
> > from meta-yocto into poky that would strip the ../meta-yocto/ part in
> > BBLAYERS. As there are already numerous other tweaks of similar
> > nature, I think this is feasible.
>
> Which other tweaks of a similar nature are made? I'm not aware of any
> file contents changing between repos...

Right, this refers to the poky integration process that cherry-picks
files and directories from original repos, e.g. the default template
from oe-core (meta/conf/templates/default) does not make it into poky
at all.

The patch follows the suggestion that such a tweak is possible:
https://lists.openembedded.org/g/openembedded-architecture/message/2129

Alex

On Tue, 2025-06-24 at 15:42 +0200, Alexander Kanavin wrote:
> On Tue, 24 Jun 2025 at 15:11, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > > This requires a custom tweak to bblayers.conf when it makes its
> > > way
> > > from meta-yocto into poky that would strip the ../meta-yocto/
> > > part in
> > > BBLAYERS. As there are already numerous other tweaks of similar
> > > nature, I think this is feasible.
> > 
> > Which other tweaks of a similar nature are made? I'm not aware of
> > any
> > file contents changing between repos...
> 
> Right, this refers to the poky integration process that cherry-picks
> files and directories from original repos, e.g. the default template
> from oe-core (meta/conf/templates/default) does not make it into poky
> at all.
> 
> The patch follows the suggestion that such a tweak is possible:
> https://lists.openembedded.org/g/openembedded-architecture/message/2129

Ah, that is a bit different. There aren't tweaks of a similar nature.
I'll need to give this more thought...

Cheers,

Richard

On Tue, 24 Jun 2025 at 15:58, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> >
> > The patch follows the suggestion that such a tweak is possible:
> > https://lists.openembedded.org/g/openembedded-architecture/message/2129
>
> Ah, that is a bit different. There aren't tweaks of a similar nature.
> I'll need to give this more thought...

No problem. I'll adjust the commit message meanwhile.

Alex

On Tue, 24 Jun 2025 at 15:58, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> > Right, this refers to the poky integration process that cherry-picks
> > files and directories from original repos, e.g. the default template
> > from oe-core (meta/conf/templates/default) does not make it into poky
> > at all.
> >
> > The patch follows the suggestion that such a tweak is possible:
> > https://lists.openembedded.org/g/openembedded-architecture/message/2129
>
> Ah, that is a bit different. There aren't tweaks of a similar nature.
> I'll need to give this more thought...

I want to put my thoughts (and a proposal) forward: the real issue
here is that BBLAYERS is a list of full paths, and thus is not
relocatable. And there's lots of awkwardness around changing that list
relative to ##OEROOT## and back in templates (and ##OEROOT##-prefixed
paths are still not relocatable).

I tried to think of a way to specify layer identifiers instead, and
have some kind of 'magic' figuring out where the layers are in the
filesystem. One option is something like this:

BBLAYERS_SEARCH_PATHS = "/path/to/where/layers/are /some/other/path/with/layers"
BBLAYERS_IDS = "core yocto intel"

e.g. bitbake would walk the filesystem starting from those paths (they
would be set by bitbake-setup or other tools performing layer
checkouts), and add any matching layers to the build.

I think that's actually not too horrible. Should I try to prototype
this? One nice consequence is that it makes templates optional for
those who do not like them: one could instead set the build entirely
from just a list of fragments (machine/distro built-ins at a minimum
:) and layer ids. Having to maintain templates for any possible layer
combination doesn't scale, same way as on-disk fragments for
machines/distros wouldn't scale.

Alex

On Tue, 24 Jun 2025 at 18:01, Alexander Kanavin via
lists.yoctoproject.org <alex.kanavin=gmail.com@lists.yoctoproject.org>
wrote:

> I think that's actually not too horrible. Should I try to prototype
> this? One nice consequence is that it makes templates optional for
> those who do not like them: one could instead set the build entirely
> from just a list of fragments (machine/distro built-ins at a minimum
> :) and layer ids. Having to maintain templates for any possible layer
> combination doesn't scale, same way as on-disk fragments for
> machines/distros wouldn't scale.

After some 'background brain processing', I've convinced myself that
the best way out is that bitbake-setup would be able to write out
bblayers.conf itself directly, from its own configuration file. Yes,
like kas does :) I've resisted this for a long time, on the principle
that bitbake-setup has no business directly meddling in build/conf/,
but really, every other option is not as good.

So, a configuration entry could look like this:

            "qemux86-64-poky": {
                "description": "Poky reference distro build for qemux86-64",
                "bb-layers":
["openembedded-core/meta","meta-yocto/meta-poky","meta-yocto/meta-yocto-bsp"],
                "oe-fragments":
["machine/qemux86-64","distro/poky","core/yocto/sstate-mirror-cdn"],
                "targets": ["bitbake core-image-minimal
core-image-full-cmdline core-image-sato-sdk core-image-weston"]
            },

and would not require templates at all.

Bitbake-setup would write out bblayers.conf with BBLAYERS entry
('bitbake-layers add-layer' can't be used at that point yet), and then
enable fragments via bitbake-config-build (this is already
implemented). This should result in a functional build directory, with
a blank (or absent) local.conf.

Then this patch, and the ones adding 'blank' templates become unnecessary.

Alex

On Wed, 25 Jun 2025 at 15:56, Alexander Kanavin via
lists.yoctoproject.org <alex.kanavin=gmail.com@lists.yoctoproject.org>
wrote:

> Bitbake-setup would write out bblayers.conf with BBLAYERS entry
> ('bitbake-layers add-layer' can't be used at that point yet), and then
> enable fragments via bitbake-config-build (this is already
> implemented). This should result in a functional build directory, with
> a blank (or absent) local.conf.
>
> Then this patch, and the ones adding 'blank' templates become unnecessary.

I've now implemented this and resent the bitbake-setup patches. The
sample 'poky-ng' config that doesn't need templates and specifies
layers directly is here:

https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master.conf.json

Alex