diff mbox series

[2/2] bitbake-setup: unpack sources (e.g. layers and bitbake) into 'sources' rather than 'layers'

Message ID 20251107080026.2599821-2-alex.kanavin@gmail.com
State New
Headers show
Series [1/2] bitbake-setup: rename bb-layers -> bb-layers-in-sources, bb-layers-relative -> bb-layers-in-thisdir | expand

Commit Message

Alexander Kanavin Nov. 7, 2025, 8 a.m. UTC 
From: Alexander Kanavin <alex@linutronix.de>

'layers' is not a good directory name, as it contains bitbake, or repositories
that can contain multiple layers or no layers at all (e.g. yocto-docs).

While there's a chance, let's name it right: 'sources'. This aligns nicely
with both the 'sources' property in the json configs, and oe-core's unpack dir
which also 'sources'.

Then the setup directory contains:

- build (bitbake's build directory)
- config (bitbake-setup's configuration data under version control)
- sources (everything that was fetched and unpacked)

A patch to correct everything in yocto-autobuilder-helper will be sent
shortly.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 bin/bitbake-setup     | 8 ++++----
 lib/bb/tests/setup.py | 6 +++---
 2 files changed, 7 insertions(+), 7 deletions(-)

Comments

Richard Purdie Nov. 7, 2025, 9:49 a.m. UTC | #1 
On Fri, 2025-11-07 at 09:00 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> 'layers' is not a good directory name, as it contains bitbake, or repositories
> that can contain multiple layers or no layers at all (e.g. yocto-docs).
> 
> While there's a chance, let's name it right: 'sources'. This aligns nicely
> with both the 'sources' property in the json configs, and oe-core's unpack dir
> which also 'sources'.

I'm afraid I strongly disagree with this change. Whilst "layers" isn't
ideal, to me "sources" is worse since it implies source code and the
general downloads. It is all too easy to think this is going to be a 
downloads cache directory of some kind, or where S might get unpacked,
and so on.

From that standpoint "repositories" would be a more fitting description
but I think "layers" is reasonable. 

> Then the setup directory contains:
> 
> - build (bitbake's build directory)
> - config (bitbake-setup's configuration data under version control)
> - sources (everything that was fetched and unpacked)

so the linux kernel?

I say that tongue in cheek but it illustrates my point, new users will
not understand that.

Cheers,

Richard
Peter Kjellerstedt Nov. 7, 2025, 11:07 a.m. UTC | #2 
> -----Original Message-----
> From: bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org> On Behalf Of Richard Purdie via lists.openembedded.org
> Sent: den 7 november 2025 10:49
> To: alex.kanavin@gmail.com; bitbake-devel@lists.openembedded.org
> Cc: Alexander Kanavin <alex@linutronix.de>
> Subject: Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: unpack sources (e.g. layers and bitbake) into 'sources' rather than 'layers'
> 
> On Fri, 2025-11-07 at 09:00 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> > From: Alexander Kanavin <alex@linutronix.de>
> >
> > 'layers' is not a good directory name, as it contains bitbake, or repositories
> > that can contain multiple layers or no layers at all (e.g. yocto-docs).
> >
> > While there's a chance, let's name it right: 'sources'. This aligns nicely
> > with both the 'sources' property in the json configs, and oe-core's unpack dir
> > which also 'sources'.
> 
> I'm afraid I strongly disagree with this change. Whilst "layers" isn't
> ideal, to me "sources" is worse since it implies source code and the
> general downloads. It is all too easy to think this is going to be a
> downloads cache directory of some kind, or where S might get unpacked,
> and so on.
> 
> From that standpoint "repositories" would be a more fitting description

In our setup, I have called the corresponding directory 'repos'.

> but I think "layers" is reasonable.

Please don't call it 'layers'. To me, a directory called 'layers' should 
be used for individual layers, e.g., meta, meta-oe, meta-virtualization, 
etc, whereas this directory contains full repositories, which may each be 
a single layer, a collection of layers, or may contain something not 
layer-related at all, e.g., documentation. I have some ideas for the 
future, and can see a use case for a 'layers' directory, which would then 
typically contain symbolic links to the individual layer directories 
somewhere within the 'repos' directory.

> 
> > Then the setup directory contains:
> >
> > - build (bitbake's build directory)
> > - config (bitbake-setup's configuration data under version control)
> > - sources (everything that was fetched and unpacked)
> 
> so the linux kernel?
> 
> I say that tongue in cheek but it illustrates my point, new users will
> not understand that.

And I think they will even less understand why a directory called 'layers' 
contains bitbake and the documentation. I believe 'repos' (or the longer 
'repositories') would be the best option.

> 
> Cheers,
> 
> Richard

//Peter
Richard Purdie Nov. 7, 2025, 11:44 a.m. UTC | #3 
On Fri, 2025-11-07 at 11:07 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org> On Behalf Of Richard Purdie via lists.openembedded.org
> > Sent: den 7 november 2025 10:49
> > To: alex.kanavin@gmail.com; bitbake-devel@lists.openembedded.org
> > Cc: Alexander Kanavin <alex@linutronix.de>
> > Subject: Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: unpack sources (e.g. layers and bitbake) into 'sources' rather than 'layers'
> > 
> > On Fri, 2025-11-07 at 09:00 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> > > From: Alexander Kanavin <alex@linutronix.de>
> > > 
> > > 'layers' is not a good directory name, as it contains bitbake, or repositories
> > > that can contain multiple layers or no layers at all (e.g. yocto-docs).
> > > 
> > > While there's a chance, let's name it right: 'sources'. This aligns nicely
> > > with both the 'sources' property in the json configs, and oe-core's unpack dir
> > > which also 'sources'.
> > 
> > I'm afraid I strongly disagree with this change. Whilst "layers" isn't
> > ideal, to me "sources" is worse since it implies source code and the
> > general downloads. It is all too easy to think this is going to be a
> > downloads cache directory of some kind, or where S might get unpacked,
> > and so on.
> > 
> > From that standpoint "repositories" would be a more fitting description
> 
> In our setup, I have called the corresponding directory 'repos'.
> 
> > but I think "layers" is reasonable.
> 
> Please don't call it 'layers'.

It is already called layers and has been in the patches for months.


>  To me, a directory called 'layers' should 
> be used for individual layers, e.g., meta, meta-oe, meta-virtualization, 
> etc, whereas this directory contains full repositories, which may each be 
> a single layer, a collection of layers, or may contain something not 
> layer-related at all, e.g., documentation. I have some ideas for the 
> future, and can see a use case for a 'layers' directory, which would then 
> typically contain symbolic links to the individual layer directories 
> somewhere within the 'repos' directory.

At this point that ship has sailed.

> > > Then the setup directory contains:
> > > 
> > > - build (bitbake's build directory)
> > > - config (bitbake-setup's configuration data under version control)
> > > - sources (everything that was fetched and unpacked)
> > 
> > so the linux kernel?
> > 
> > I say that tongue in cheek but it illustrates my point, new users will
> > not understand that.
> 
> And I think they will even less understand why a directory called 'layers' 
> contains bitbake and the documentation. I believe 'repos' (or the longer 
> 'repositories') would be the best option.

I disagree, quite strongly and I don't really want to open this for
discussion. That should have happened months ago.

Cheers,

Richard
Peter Kjellerstedt Nov. 7, 2025, 2:36 p.m. UTC | #4 
> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 7 november 2025 12:44
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; alex.kanavin@gmail.com; bitbake-devel@lists.openembedded.org
> Cc: Alexander Kanavin <alex@linutronix.de>
> Subject: Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: unpack sources (e.g. layers and bitbake) into 'sources' rather than 'layers'
> 
> On Fri, 2025-11-07 at 11:07 +0000, Peter Kjellerstedt wrote:
> > > -----Original Message-----
> > > From: bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org> On Behalf Of Richard Purdie via lists.openembedded.org
> > > Sent: den 7 november 2025 10:49
> > > To: alex.kanavin@gmail.com; bitbake-devel@lists.openembedded.org
> > > Cc: Alexander Kanavin <alex@linutronix.de>
> > > Subject: Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: unpack sources (e.g. layers and bitbake) into 'sources' rather than 'layers'
> > >
> > > On Fri, 2025-11-07 at 09:00 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> > > > From: Alexander Kanavin <alex@linutronix.de>
> > > >
> > > > 'layers' is not a good directory name, as it contains bitbake, or repositories
> > > > that can contain multiple layers or no layers at all (e.g. yocto-docs).
> > > >
> > > > While there's a chance, let's name it right: 'sources'. This aligns nicely
> > > > with both the 'sources' property in the json configs, and oe-core's unpack dir
> > > > which also 'sources'.
> > >
> > > I'm afraid I strongly disagree with this change. Whilst "layers" isn't
> > > ideal, to me "sources" is worse since it implies source code and the
> > > general downloads. It is all too easy to think this is going to be a
> > > downloads cache directory of some kind, or where S might get unpacked,
> > > and so on.
> > >
> > > From that standpoint "repositories" would be a more fitting description
> >
> > In our setup, I have called the corresponding directory 'repos'.
> >
> > > but I think "layers" is reasonable.
> >
> > Please don't call it 'layers'.
> 
> It is already called layers and has been in the patches for months.
> 
> 
> >  To me, a directory called 'layers' should
> > be used for individual layers, e.g., meta, meta-oe, meta-virtualization,
> > etc, whereas this directory contains full repositories, which may each be
> > a single layer, a collection of layers, or may contain something not
> > layer-related at all, e.g., documentation. I have some ideas for the
> > future, and can see a use case for a 'layers' directory, which would then
> > typically contain symbolic links to the individual layer directories
> > somewhere within the 'repos' directory.
> 
> At this point that ship has sailed.
> 
> > > > Then the setup directory contains:
> > > >
> > > > - build (bitbake's build directory)
> > > > - config (bitbake-setup's configuration data under version control)
> > > > - sources (everything that was fetched and unpacked)
> > >
> > > so the linux kernel?
> > >
> > > I say that tongue in cheek but it illustrates my point, new users will
> > > not understand that.
> >
> > And I think they will even less understand why a directory called 'layers'
> > contains bitbake and the documentation. I believe 'repos' (or the longer
> > 'repositories') would be the best option.
> 
> I disagree, quite strongly and I don't really want to open this for
> discussion. That should have happened months ago.
> 
> Cheers,
> 
> Richard

Tbh, I was not surprised by your response. My reply was primarily in 
the context of _if_ we were going to rename the 'layers' directory now, 
as suggested by Alex, then I would much rather see it named 'repos' (or 
'repositories') than 'sources'.

That said, unless I am mistaken, the 'layers' directory is not part 
of any API. It is created and maintained by bitbake-setup, and if it 
is renamed, then that should only affect bitbake-setup. It is not 
really any different than renaming sources-unpack to sources. Ok, 
that may be a slight exaggeration as it is more visible to the user, 
but at least I don't think the current name is something that should 
be seen as set in stone.

And while your are quite right that it would have been better if this 
discussion had happened months ago, there is still room for improving 
things, after the Whinlatter release.

//Peter
Richard Purdie Nov. 7, 2025, 3:28 p.m. UTC | #5 
On Fri, 2025-11-07 at 14:36 +0000, Peter Kjellerstedt via
lists.openembedded.org wrote:
> > -----Original Message-----
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Sent: den 7 november 2025 12:44
> > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>;
> > alex.kanavin@gmail.com; bitbake-devel@lists.openembedded.org
> > Cc: Alexander Kanavin <alex@linutronix.de>
> > Subject: Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: unpack
> > sources (e.g. layers and bitbake) into 'sources' rather than
> > 'layers'
> > 
> > On Fri, 2025-11-07 at 11:07 +0000, Peter Kjellerstedt wrote:
> > > > -----Original Message-----
> > > > From:
> > > > bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org
> > > > > On Behalf Of Richard Purdie via lists.openembedded.org
> > > > Sent: den 7 november 2025 10:49
> > > > To: alex.kanavin@gmail.com;
> > > > bitbake-devel@lists.openembedded.org
> > > > Cc: Alexander Kanavin <alex@linutronix.de>
> > > > Subject: Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: unpack
> > > > sources (e.g. layers and bitbake) into 'sources' rather than
> > > > 'layers'
> > > > 
> > > > On Fri, 2025-11-07 at 09:00 +0100, Alexander Kanavin via
> > > > lists.openembedded.org wrote:
> > > > > From: Alexander Kanavin <alex@linutronix.de>
> > > > > 
> > > > > 'layers' is not a good directory name, as it contains
> > > > > bitbake, or repositories
> > > > > that can contain multiple layers or no layers at all (e.g.
> > > > > yocto-docs).
> > > > > 
> > > > > While there's a chance, let's name it right: 'sources'. This
> > > > > aligns nicely
> > > > > with both the 'sources' property in the json configs, and oe-
> > > > > core's unpack dir
> > > > > which also 'sources'.
> > > > 
> > > > I'm afraid I strongly disagree with this change. Whilst
> > > > "layers" isn't
> > > > ideal, to me "sources" is worse since it implies source code
> > > > and the
> > > > general downloads. It is all too easy to think this is going to
> > > > be a
> > > > downloads cache directory of some kind, or where S might get
> > > > unpacked,
> > > > and so on.
> > > > 
> > > > From that standpoint "repositories" would be a more fitting
> > > > description
> > > 
> > > In our setup, I have called the corresponding directory 'repos'.
> > > 
> > > > but I think "layers" is reasonable.
> > > 
> > > Please don't call it 'layers'.
> > 
> > It is already called layers and has been in the patches for months.
> > 
> > 
> > >  To me, a directory called 'layers' should
> > > be used for individual layers, e.g., meta, meta-oe, meta-
> > > virtualization,
> > > etc, whereas this directory contains full repositories, which may
> > > each be
> > > a single layer, a collection of layers, or may contain something
> > > not
> > > layer-related at all, e.g., documentation. I have some ideas for
> > > the
> > > future, and can see a use case for a 'layers' directory, which
> > > would then
> > > typically contain symbolic links to the individual layer
> > > directories
> > > somewhere within the 'repos' directory.
> > 
> > At this point that ship has sailed.
> > 
> > > > > Then the setup directory contains:
> > > > > 
> > > > > - build (bitbake's build directory)
> > > > > - config (bitbake-setup's configuration data under version
> > > > > control)
> > > > > - sources (everything that was fetched and unpacked)
> > > > 
> > > > so the linux kernel?
> > > > 
> > > > I say that tongue in cheek but it illustrates my point, new
> > > > users will
> > > > not understand that.
> > > 
> > > And I think they will even less understand why a directory called
> > > 'layers'
> > > contains bitbake and the documentation. I believe 'repos' (or the
> > > longer
> > > 'repositories') would be the best option.
> > 
> > I disagree, quite strongly and I don't really want to open this for
> > discussion. That should have happened months ago.
> > 
> > Cheers,
> > 
> > Richard
> 
> Tbh, I was not surprised by your response. My reply was primarily in 
> the context of _if_ we were going to rename the 'layers' directory
> now, 
> as suggested by Alex, then I would much rather see it named 'repos'
> (or 
> 'repositories') than 'sources'.
> 
> That said, unless I am mistaken, the 'layers' directory is not part 
> of any API. It is created and maintained by bitbake-setup, and if it 
> is renamed, then that should only affect bitbake-setup. It is not 
> really any different than renaming sources-unpack to sources. Ok, 
> that may be a slight exaggeration as it is more visible to the user, 
> but at least I don't think the current name is something that should 
> be seen as set in stone.
> 
> And while your are quite right that it would have been better if this
> discussion had happened months ago, there is still room for improving
> things, after the Whinlatter release.

Perhaps. The bigger picture of these changes are testing the limits of
my sanity as things are though so I don't really want to do this now as
I don't think there is a big issue there.

Alex's change to autobuilder-helper also showed just how invasive that
change was. Yes, it is 'only' CI, so fewer people notice/care and it
only breaks people maintaining CI but it did mean a lot of change
there.

We have bigger problems right now.

Cheers,

Richard
diff mbox series

Patch

diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index 50b7f1488..a9674a527 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -461,7 +461,7 @@  def init_config(top_dir, settings, args, d):
     print("\nRun 'bitbake-setup init --non-interactive {}' to select this configuration non-interactively.\n".format(" ".join(upstream_config['non-interactive-cmdline-options'])))
 
     setupdir = os.path.join(os.path.abspath(top_dir), args.setup_dir_name or "{}-{}".format(upstream_config['name']," ".join(upstream_config['non-interactive-cmdline-options'][1:]).replace(" ","-").replace("/","_")))
-    if os.path.exists(os.path.join(setupdir, "layers")):
+    if os.path.exists(os.path.join(setupdir, "sources")):
         print(f"Setup already initialized in:\n    {setupdir}\nUse 'bitbake-setup status' to check if it needs to be updated, or 'bitbake-setup update' to perform the update.\nIf you would like to start over and re-initialize in this directory, remove it, and run 'bitbake-setup init' again.")
         return
 
@@ -475,7 +475,7 @@  def init_config(top_dir, settings, args, d):
     os.makedirs(setupdir, exist_ok=True)
 
     confdir = os.path.join(setupdir, "config")
-    layerdir = os.path.join(setupdir, "layers")
+    layerdir = os.path.join(setupdir, "sources")
 
     os.makedirs(confdir)
     os.makedirs(layerdir)
@@ -535,7 +535,7 @@  def build_status(top_dir, settings, args, d, update=False):
     setupdir = args.setup_dir
 
     confdir = os.path.join(setupdir, "config")
-    layerdir = os.path.join(setupdir, "layers")
+    layerdir = os.path.join(setupdir, "sources")
 
     current_upstream_config = json.load(open(os.path.join(confdir, "config-upstream.json")))
 
@@ -648,7 +648,7 @@  def install_buildtools(top_dir, settings, args, d):
             return
         shutil.rmtree(buildtools_install_dir)
 
-    install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
+    install_buildtools = os.path.join(args.setup_dir, 'sources/oe-scripts/install-buildtools')
     buildtools_download_dir = os.path.join(args.setup_dir, 'buildtools-downloads/{}'.format(time.strftime("%Y%m%d%H%M%S")))
     print("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir))
     subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True)
diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
index b8a5a03ad..9eebd709e 100644
--- a/lib/bb/tests/setup.py
+++ b/lib/bb/tests/setup.py
@@ -181,7 +181,7 @@  print("BBPATH is {{}}".format(os.environ["BBPATH"]))
     def check_setupdir_files(self, setuppath, test_file_content):
         with open(os.path.join(setuppath, 'config', "config-upstream.json")) as f:
             config_upstream = json.load(f)
-        with open(os.path.join(setuppath, 'layers', 'test-repo', 'test-file')) as f:
+        with open(os.path.join(setuppath, 'sources', 'test-repo', 'test-file')) as f:
             self.assertEqual(f.read(), test_file_content)
         bitbake_config = config_upstream["bitbake-config"]
         bb_build_path = os.path.join(setuppath, 'build')
@@ -205,7 +205,7 @@  print("BBPATH is {{}}".format(os.environ["BBPATH"]))
             with open(os.path.join(bb_conf_path, 'bblayers.conf')) as f:
                 bblayers = f.read()
                 for l in bitbake_config["bb-layers-in-sources"]:
-                    self.assertIn(os.path.join(setuppath, "layers", l), bblayers)
+                    self.assertIn(os.path.join(setuppath, "sources", l), bblayers)
                 for l in bitbake_config.get("bb-layers-in-this-dir") or []:
                     thisdir_layer = os.path.join(
                             os.path.dirname(config_upstream["path"]),
@@ -328,7 +328,7 @@  print("BBPATH is {{}}".format(os.environ["BBPATH"]))
             setuppath = os.path.join(self.tempdir, 'bitbake-builds', 'test-config-1-{}'.format(c))
             os.environ['BBPATH'] = os.path.join(setuppath, 'build')
             out = self.runbbsetup("status")
-            self.assertIn("Layer repository file://{} checked out into {}/layers/test-repo updated revision master from".format(self.testrepopath, setuppath), out[0])
+            self.assertIn("Layer repository file://{} checked out into {}/sources/test-repo updated revision master from".format(self.testrepopath, setuppath), out[0])
             out = self.runbbsetup("update")
             if c in ('gadget', 'gizmo'):
                 self.assertIn("Existing bitbake configuration directory renamed to {}/build/conf-backup.".format(setuppath), out[0])