diff mbox series

bitbake-setup: pass install-buildtools options from config and CLI

Message ID 20260615121856.96250-1-jaipaul.cheernam@est.tech
State New
Headers show
Series bitbake-setup: pass install-buildtools options from config and CLI | expand

Commit Message

Jaipaul Cheernam June 15, 2026, 12:18 p.m. UTC
Allow distro-specific buildtools to be configured with a custom URL,
filename, and SDK name via an install-buildtools section in
config-upstream.json. Expose the same options as CLI arguments: --url,
--filename, --sdk-name, --no-check. CLI arguments take precedence over
config values.

This requires the --sdk-name option to be available in
oe-scripts/install-buildtools from openembedded-core.

AI-Generated: Kiro with Claude Opus 4.6

Signed-off-by: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
---
 bin/bitbake-setup                             | 49 +++++++++++-
 .../bitbake-user-manual-environment-setup.rst | 33 ++++++++
 lib/bb/tests/setup.py                         | 80 ++++++++++++++++++-
 setup-schema/bitbake-setup.schema.json        | 30 +++++++
 4 files changed, 188 insertions(+), 4 deletions(-)

Comments

Alexander Kanavin June 15, 2026, 1:31 p.m. UTC | #1
On Mon, 15 Jun 2026 at 14:19, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> Allow distro-specific buildtools to be configured with a custom URL,
> filename, and SDK name via an install-buildtools section in
> config-upstream.json. Expose the same options as CLI arguments: --url,
> --filename, --sdk-name, --no-check. CLI arguments take precedence over
> config values.
>
> This requires the --sdk-name option to be available in
> oe-scripts/install-buildtools from openembedded-core.

Can you please describe the use case? Presumably this is useful in
your environment, I'd like to understand the specifics of what you put
in the custom buildtools archive and why it has to be there and not
built via recipes.

Alex
Jaipaul Cheernam June 15, 2026, 2:59 p.m. UTC | #2
Hi Alex,

Any downstream distro that builds its own extended buildtools tarball under a non-Poky distro name hits a FileNotFoundError because install-buildtools
hardcodes pokysdk when looking for environment-setup-<arch>-pokysdk-linux. The --sdk-name option fixes that hardcoded assumption.

The archive itself contains the same tools as upstream extended buildtools — it can't be provided via recipes because buildtools exist specifically to
bootstrap the host environment before BitBake can run.

The --url and --filename options are needed for projects that host their buildtools on an internal server rather than yoctoproject.org, and want to pin a
specific version across the team without requiring manual arguments on every invocation.

The workaround today is to pass arguments manually or modify wrapper scripts, which is probably why this hasn't surfaced as a bug report before.

The --sdk-name option itself is introduced in a companion patch to oe-scripts/install-buildtools in openembedded-core:
https://lists.openembedded.org/g/openembedded-core/topic/patch_install_buildtools/119814299

Thanks,
Jaipaul Cheernam


From: Alexander Kanavin <alex.kanavin@gmail.com>
Date: Monday, 15 June 2026 at 15:32
To: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
Cc: bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>
Subject: Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI

On Mon, 15 Jun 2026 at 14:19, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> Allow distro-specific buildtools to be configured with a custom URL,
> filename, and SDK name via an install-buildtools section in
> config-upstream.json. Expose the same options as CLI arguments: --url,
> --filename, --sdk-name, --no-check. CLI arguments take precedence over
> config values.
>
> This requires the --sdk-name option to be available in
> oe-scripts/install-buildtools from openembedded-core.

Can you please describe the use case? Presumably this is useful in
your environment, I'd like to understand the specifics of what you put
in the custom buildtools archive and why it has to be there and not
built via recipes.

Alex
Alexander Kanavin June 15, 2026, 3:38 p.m. UTC | #3
On Mon, 15 Jun 2026 at 16:59, Jaipaul Cheernam
<jaipaul.cheernam@est.tech> wrote:
> The archive itself contains the same tools as upstream extended buildtools — it can't be provided via recipes because buildtools exist specifically to
> bootstrap the host environment before BitBake can run.
>
> The --url and --filename options are needed for projects that host their buildtools on an internal server rather than yoctoproject.org, and want to pin a
> specific version across the team without requiring manual arguments on every invocation.
>
> The workaround today is to pass arguments manually or modify wrapper scripts, which is probably why this hasn't surfaced as a bug report before.
>
> The --sdk-name option itself is introduced in a companion patch to oe-scripts/install-buildtools in openembedded-core:
> https://lists.openembedded.org/g/openembedded-core/topic/patch_install_buildtools/119814299

Thanks. I'm broadly in favor of this, but I don't know what the
bitbake patch reviewers will say, and especially RP :) Buildtools can
be used to do isolated, reproducible 'container builds' without actual
containers, and I like that.

I wonder if the archive should be downloaded by bitbake-setup itself
using bitbake fetchers (similar to how layers are obtained), and then
given to install-buildtools as a local file, or even executed
directly. There are benefits to this: it will be preserved in the
download cache together with the layers and upstream components (which
helps if upstream server isn't available or altogether disappears),
and checksummed against sha256sum provided in the config (which adds a
reproducibility check, and protection against supply chain attacks,
similar to tarball recipes). What you think?

Alex
Richard Purdie June 16, 2026, 10:06 a.m. UTC | #4
On Mon, 2026-06-15 at 14:18 +0200, Jaipaul Cheernam via lists.openembedded.org wrote:
> Allow distro-specific buildtools to be configured with a custom URL,
> filename, and SDK name via an install-buildtools section in
> config-upstream.json. Expose the same options as CLI arguments: --url,
> --filename, --sdk-name, --no-check. CLI arguments take precedence over
> config values.
> 
> This requires the --sdk-name option to be available in
> oe-scripts/install-buildtools from openembedded-core.

I replied to the oe-core patch, lets just remove the need to have --
sdk-name, I don't think we need that. I also don't understand why you
want/need the --no-check option, we should be able to tell if we should
be checking the checksum or not? If that means fixing the install
script, so be it.

Cheers,

Richard
Jaipaul Cheernam June 16, 2026, 2:18 p.m. UTC | #5
Hi Richard,

Agreed on removing --sdk-name  and I have sent Patch v2 for review in oe-core.

For --no-check — I'm fine removing it and having the install-buildtools auto-detect, but we need to decide on the failure mode: if the checksum file can't be fetched
(404 or network error), should we:

(a) warn and continue the install without verification, or
(b) hard fail and refuse to install?

If (a), we're effectively auto---no-check on failure, which may silently skip validation on transient network issues. If (b), users with custom/local
URLs that don't host checksum files would have no way to install without us re-adding an override flag or having them to add checksum file.

I checked the history why "--no-check” was added and it was part of bf902a810f98f55dd9e8cb9e6c6b0903f9902157

What's your preference?

Based on review verdict on oe-core patch, I will send patch v2 here.

Regards,
Jaipaul Cheernam

From: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Tuesday, 16 June 2026 at 12:07
To: Jaipaul Cheernam <jaipaul.cheernam@est.tech>; bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>
Subject: Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI

On Mon, 2026-06-15 at 14:18 +0200, Jaipaul Cheernam via lists.openembedded.org wrote:
> Allow distro-specific buildtools to be configured with a custom URL,
> filename, and SDK name via an install-buildtools section in
> config-upstream.json. Expose the same options as CLI arguments: --url,
> --filename, --sdk-name, --no-check. CLI arguments take precedence over
> config values.
>
> This requires the --sdk-name option to be available in
> oe-scripts/install-buildtools from openembedded-core.

I replied to the oe-core patch, lets just remove the need to have --
sdk-name, I don't think we need that. I also don't understand why you
want/need the --no-check option, we should be able to tell if we should
be checking the checksum or not? If that means fixing the install
script, so be it.

Cheers,

Richard
Alexander Kanavin June 16, 2026, 2:26 p.m. UTC | #6
On Tue, 16 Jun 2026 at 16:18, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> Agreed on removing --sdk-name  and I have sent Patch v2 for review in oe-core.
>
> For --no-check — I'm fine removing it and having the install-buildtools auto-detect, but we need to decide on the failure mode: if the checksum file can't be fetched
> (404 or network error), should we:
>
> (a) warn and continue the install without verification, or
> (b) hard fail and refuse to install?
>
> If (a), we're effectively auto---no-check on failure, which may silently skip validation on transient network issues. If (b), users with custom/local
> URLs that don't host checksum files would have no way to install without us re-adding an override flag or having them to add checksum file.
>
> I checked the history why "--no-check” was added and it was part of bf902a810f98f55dd9e8cb9e6c6b0903f9902157
>
> What's your preference?

Please note my comment in this thread. If we use bitbake fetcher with
checksum in a bitbake-setup config file, then the checksum would be
enforced in a way that is secure against tampering or accidental
replacement.

Alex
diff mbox series

Patch

diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index fe3b6b0e6..343d428e8 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -1052,10 +1052,49 @@  def install_buildtools(top_dir, settings, args, d):
             return
         shutil.rmtree(buildtools_install_dir)
 
-    install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
+    install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
     buildtools_download_dir = add_unique_timestamp_to_path(os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
+
+    buildtools_config = {}
+    upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json')
+    try:
+        with open(upstream_config_path) as f:
+            upstream_config = json.load(f)
+        buildtools_config = upstream_config.get('data', {}).get('bitbake-setup', {}).get('install-buildtools', {})
+    except FileNotFoundError:
+        pass
+    except json.JSONDecodeError as e:
+        logger.error("Failed to parse %s: %s" % (upstream_config_path, e))
+        sys.exit(1)
+
+    url = args.url if args.url is not None else buildtools_config.get('url')
+    filename = args.filename if args.filename is not None else buildtools_config.get('filename')
+    sdk_name = args.sdk_name if args.sdk_name is not None else buildtools_config.get('sdk-name')
+
+    for name, value in (("url", url), ("filename", filename), ("sdk-name", sdk_name)):
+        if value == "":
+            logger.error("%s is empty" % name)
+            sys.exit(1)
+    no_check = args.no_check if args.no_check is not None else buildtools_config.get('no-check', False)
+
+    if (url is None) != (filename is None):
+        logger.error("url and filename must both be set (from CLI or config) or both omitted")
+        sys.exit(1)
+
+    cmd = [
+        install_buildtools_script,
+        '-d', buildtools_install_dir,
+        '--downloads-directory', buildtools_download_dir,
+    ]
+    if url and filename:
+        cmd += ['--url', url, '--filename', filename]
+    if sdk_name:
+        cmd += ['--sdk-name', sdk_name]
+    if no_check:
+        cmd.append('--no-check')
+
     logger.plain("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir))
-    subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True)
+    subprocess.check_call(cmd)
 
 def create_siteconf(top_dir, non_interactive, settings):
     siteconfpath = os.path.join(top_dir, 'site.conf')
@@ -1285,6 +1324,12 @@  def main():
     parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine')
     add_setup_dir_arg(parser_install_buildtools)
     parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.')
+    parser_install_buildtools.add_argument('--url', action='store', help='URL from where to fetch the buildtools SDK installer, not including filename. Requires --filename. Overrides config value.')
+    parser_install_buildtools.add_argument('--filename', action='store', help='Filename for the buildtools SDK installer. Requires --url. Overrides config value.')
+    parser_install_buildtools.add_argument('--sdk-name', action='store', default=None, help='SDK name used in the environment setup script (e.g. customsdk). Overrides config value.')
+    check_group = parser_install_buildtools.add_mutually_exclusive_group()
+    check_group.add_argument('--no-check', action='store_const', const=True, default=None, dest='no_check', help='Disable checksum validation. Overrides config value.')
+    check_group.add_argument('--check', action='store_const', const=False, dest='no_check', help='Enable checksum validation. Overrides config no-check: true.')
     parser_install_buildtools.set_defaults(func=install_buildtools)
 
     parser_settings_arg_global = argparse.ArgumentParser(add_help=False)
diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
index aa546c30b..931fda798 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
@@ -660,6 +660,39 @@  In addition, the command can take the following arguments:
 -  ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
    required if :term:`BBPATH` is already configured.
 
+-  ``--url``: URL from where to fetch the buildtools SDK installer, not
+   including filename. Requires ``--filename``. Overrides the value from the
+   configuration file.
+
+-  ``--filename``: filename of the buildtools installer. Requires ``--url``.
+   Overrides the value from the configuration file.
+
+-  ``--sdk-name``: SDK name used in the environment setup script installed
+   by the buildtools tarball (e.g. ``customsdk`` for
+   ``environment-setup-<arch>-customsdk-linux``). Defaults to ``pokysdk``.
+   Overrides the value from the configuration file.
+
+-  ``--no-check``: disable checksum validation when the server does not
+   provide a ``.sha256sum`` file alongside the installer. Checksum
+   validation is enabled by default.
+
+The ``url``, ``filename``, ``sdk-name`` and ``no-check`` options can also be
+set in the :term:`Configuration File` under the ``bitbake-setup`` section::
+
+   "bitbake-setup": {
+       "install-buildtools": {
+           "url": "https://example.com/buildtools",
+           "filename": "x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
+           "sdk-name": "customsdk",
+           "no-check": true
+       },
+       "configurations": [ ... ]
+   }
+
+CLI arguments take precedence over values in the configuration file.
+Note that setting ``no-check`` to ``false`` in the configuration file has no
+effect; omit the key to use the default (enabled).
+
 .. _ref-bbsetup-command-settings:
 
 ``bitbake-setup settings``
diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
index 5592e8196..f14da5a60 100644
--- a/lib/bb/tests/setup.py
+++ b/lib/bb/tests/setup.py
@@ -11,6 +11,7 @@  import glob
 import hashlib
 import json
 import os
+import shutil
 import stat
 from bb.tests.support.httpserver import HTTPService
 
@@ -61,13 +62,22 @@  import getopt
 import sys
 import os
 
-opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="])
+opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory=", "url=", "filename=", "sdk-name=", "no-check"])
+installdir = None
 for option, value in opts:
     if option == '-d':
         installdir = value
+    elif option == '--url':
+        print("install-buildtools url={}".format(value))
+    elif option == '--filename':
+        print("install-buildtools filename={}".format(value))
+    elif option == '--sdk-name':
+        print("install-buildtools sdk-name={}".format(value))
+    elif option == '--no-check':
+        print("install-buildtools no-check")
 
 print("Buildtools installed into {}".format(installdir))
-os.makedirs(installdir)
+os.makedirs(installdir, exist_ok=True)
 """
         self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True)
 
@@ -644,6 +654,72 @@  print("BBPATH is {{}}".format(os.environ["BBPATH"]))
     def _count_layer_backups(self, layers_path):
         return len([f for f in os.listdir(layers_path) if 'backup' in f])
 
+    def test_install_buildtools_options(self):
+        """Test that url, filename, sdk-name and no-check are passed through to install-buildtools"""
+        if 'BBPATH' in os.environ:
+            del os.environ['BBPATH']
+        os.chdir(self.tempdir)
+
+        self.runbbsetup("settings set default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath))
+        self.add_file_to_testrepo('test-file', 'initial\n')
+        self.add_json_config_to_registry('test-config-bt.conf.json', 'master', 'master')
+        self.runbbsetup("init --non-interactive test-config-bt gadget")
+        setuppath = self.get_setup_path('test-config-bt', 'gadget')
+
+        # test CLI options are passed through
+        out = self.runbbsetup("install-buildtools --setup-dir {} "
+                              "--url https://example.com/buildtools "
+                              "--filename x86_64-buildtools.sh "
+                              "--sdk-name customsdk "
+                              "--no-check".format(setuppath))
+        self.assertIn("install-buildtools url=https://example.com/buildtools", out[0])
+        self.assertIn("install-buildtools filename=x86_64-buildtools.sh", out[0])
+        self.assertIn("install-buildtools sdk-name=customsdk", out[0])
+        self.assertIn("install-buildtools no-check", out[0])
+
+        # test config options are passed through
+        shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+        with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
+            config_upstream = json.load(f)
+        config_upstream['data']['bitbake-setup']['install-buildtools'] = {
+                'url': 'https://example.com/buildtools',
+                'filename': 'x86_64-buildtools-from-config.sh',
+                'sdk-name': 'customsdk',
+                'no-check': True
+            }
+        with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
+            json.dump(config_upstream, f)
+        out = self.runbbsetup("install-buildtools --setup-dir {}".format(setuppath))
+        self.assertIn("install-buildtools url=https://example.com/buildtools", out[0])
+        self.assertIn("install-buildtools filename=x86_64-buildtools-from-config.sh", out[0])
+        self.assertIn("install-buildtools sdk-name=customsdk", out[0])
+        self.assertIn("install-buildtools no-check", out[0])
+
+        # test CLI overrides config — pass --url/--filename on CLI to avoid implicit dependency on case 2 config state
+        shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+        out = self.runbbsetup("install-buildtools --setup-dir {} "
+                              "--url https://example.com/buildtools "
+                              "--filename x86_64-buildtools.sh "
+                              "--sdk-name overridden".format(setuppath))
+        self.assertIn("install-buildtools sdk-name=overridden", out[0])
+        self.assertIn("install-buildtools url=https://example.com/buildtools", out[0])
+        self.assertIn("install-buildtools filename=x86_64-buildtools.sh", out[0])
+
+        # test --check overrides config no-check: true
+        shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+        out = self.runbbsetup("install-buildtools --setup-dir {} --check".format(setuppath))
+        self.assertNotIn("install-buildtools no-check", out[0])
+
+        # test --url without --filename is rejected
+        with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
+            config_upstream = json.load(f)
+        config_upstream['data']['bitbake-setup'].pop('install-buildtools', None)
+        with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
+            json.dump(config_upstream, f)
+        # --force used instead of shutil.rmtree since buildtools dir exists from prior subcase
+        with self.assertRaises(bb.process.ExecutionError):
+            self.runbbsetup("install-buildtools --setup-dir {} --force --url https://example.com/buildtools".format(setuppath))
+
     def test_update_rebase_conflicts_strategy(self):
         """Test the --rebase-conflicts-strategy option for the update command.
 
diff --git a/setup-schema/bitbake-setup.schema.json b/setup-schema/bitbake-setup.schema.json
index 99f47f73d..2a967c96e 100644
--- a/setup-schema/bitbake-setup.schema.json
+++ b/setup-schema/bitbake-setup.schema.json
@@ -141,6 +141,36 @@ 
                         },
                         "additionalProperties": false
                     }
+                },
+                "install-buildtools": {
+                    "type": "object",
+                    "description": "Optional settings passed through to oe-scripts/install-buildtools",
+                    "properties": {
+                        "url": {
+                            "type": "string",
+                            "minLength": 1,
+                            "description": "URL from where to fetch the buildtools SDK installer, not including filename. Requires 'filename'."
+                        },
+                        "filename": {
+                            "type": "string",
+                            "minLength": 1,
+                            "description": "Filename for the buildtools SDK installer. Requires 'url'."
+                        },
+                        "sdk-name": {
+                            "type": "string",
+                            "minLength": 1,
+                            "description": "SDK name in the environment setup script (e.g. 'customsdk'). Defaults to 'pokysdk'."
+                        },
+                        "no-check": {
+                            "type": "boolean",
+                            "description": "Set to true to disable checksum validation when the server does not provide a .sha256sum file. no-check: false is equivalent to omitting the key."
+                        }
+                    },
+                    "dependencies": {
+                        "url": ["filename"],
+                        "filename": ["url"]
+                    },
+                    "additionalProperties": false
                 }
             },
             "additionalProperties": false