[v4] bitbake-setup: use bb.fetch for buildtools installer download

When url and sha256sum are configured in the selected configuration,
download the buildtools installer via bb.fetch with checksum
enforcement. The file is unpacked to the standard buildtools download
location and passed to install-buildtools via --local-file.

If no config is present or --default is passed, fall back to
calling install-buildtools directly with its built-in defaults.

CLI overrides available via --url and --sha256.

Depends: [oe-core] install-buildtools: add --local-file option
AI-Generated: Kiro with Claude Opus 4.6
Signed-off-by: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
---

Changes since v3:
- install-buildtools config is per-configuration only (no top-level)
- Use buildtools_download_dir for unpack (consistent with default path)
- Updated schema, docs, and tests to match

Changes since v2:
- Added --default to bypass config and use script defaults
- Use fetcher.unpack() instead of reading from DL_DIR directly
- Removed explicit sha256 verification (fetcher handles it for https)

Changes since v1:
- Use bb.fetch with mandatory sha256sum when configured
- Fall back to install-buildtools defaults when no config present
- Combined url+filename into single url field
- Removed --sdk-name, --filename, --no-check/--check
- Added --url and --sha256 CLI overrides
- Updated schema, docs, and tests
 bin/bitbake-setup                             | 89 ++++++++++++++++++-
 .../bitbake-user-manual-environment-setup.rst | 24 +++++
 lib/bb/tests/setup.py                         | 77 +++++++++++++++-
 setup-schema/bitbake-setup.schema.json        | 21 +++++
 4 files changed, 203 insertions(+), 8 deletions(-)

Thanks, I think this is good overall. Perhaps there's no need to set
buildtools_download_dir twice in the same way in two different code
paths, but that's a minor issue. The patch should go to general review
and CI now.

Alex

On Mon, 29 Jun 2026 at 22:34, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
>
> When url and sha256sum are configured in the selected configuration,
> download the buildtools installer via bb.fetch with checksum
> enforcement. The file is unpacked to the standard buildtools download
> location and passed to install-buildtools via --local-file.
>
> If no config is present or --default is passed, fall back to
> calling install-buildtools directly with its built-in defaults.
>
> CLI overrides available via --url and --sha256.
>
> Depends: [oe-core] install-buildtools: add --local-file option
> AI-Generated: Kiro with Claude Opus 4.6
> Signed-off-by: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
> ---
>
> Changes since v3:
> - install-buildtools config is per-configuration only (no top-level)
> - Use buildtools_download_dir for unpack (consistent with default path)
> - Updated schema, docs, and tests to match
>
> Changes since v2:
> - Added --default to bypass config and use script defaults
> - Use fetcher.unpack() instead of reading from DL_DIR directly
> - Removed explicit sha256 verification (fetcher handles it for https)
>
> Changes since v1:
> - Use bb.fetch with mandatory sha256sum when configured
> - Fall back to install-buildtools defaults when no config present
> - Combined url+filename into single url field
> - Removed --sdk-name, --filename, --no-check/--check
> - Added --url and --sha256 CLI overrides
> - Updated schema, docs, and tests
>  bin/bitbake-setup                             | 89 ++++++++++++++++++-
>  .../bitbake-user-manual-environment-setup.rst | 24 +++++
>  lib/bb/tests/setup.py                         | 77 +++++++++++++++-
>  setup-schema/bitbake-setup.schema.json        | 21 +++++
>  4 files changed, 203 insertions(+), 8 deletions(-)
>
> diff --git a/bin/bitbake-setup b/bin/bitbake-setup
> index 97ea08d11..9a3507c1e 100755
> --- a/bin/bitbake-setup
> +++ b/bin/bitbake-setup
> @@ -1053,10 +1053,88 @@ def install_buildtools(top_dir, settings, args, d):
>              return
>          shutil.rmtree(buildtools_install_dir)
>
> -    install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
> -    buildtools_download_dir = add_unique_timestamp_to_path(os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
> -    logger.plain("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir))
> -    subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True)
> +    install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
> +
> +    buildtools_config = {}
> +    upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json')
> +    try:
> +        with open(upstream_config_path) as f:
> +            upstream_config = json.load(f)
> +        buildtools_config = upstream_config.get('bitbake-config', {}).get('install-buildtools', {})
> +    except FileNotFoundError:
> +        pass
> +    except json.JSONDecodeError as e:
> +        logger.error("Failed to parse %s: %s" % (upstream_config_path, e))
> +        sys.exit(1)
> +
> +    url = args.url if args.url is not None else buildtools_config.get('url')
> +    sha256 = args.sha256 if args.sha256 is not None else buildtools_config.get('sha256sum')
> +
> +    if args.default or (not url and not sha256):
> +        # No config or --default: fall back to install-buildtools defaults
> +        logger.plain("Using install-buildtools script defaults")
> +        buildtools_download_dir = add_unique_timestamp_to_path(
> +            os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
> +        subprocess.check_call([install_buildtools_script,
> +                               '-d', buildtools_install_dir,
> +                               '--downloads-directory', buildtools_download_dir])
> +        return
> +
> +    if not url:
> +        logger.error("No buildtools URL configured. "
> +                     "Set 'url' in the install-buildtools config section or pass --url.")
> +        sys.exit(1)
> +
> +    if not sha256:
> +        logger.error("No sha256sum configured for buildtools installer. "
> +                     "Add 'sha256sum' to the install-buildtools config section or pass --sha256.\n"
> +                     "You can obtain it with: sha256sum <installer-file>")
> +        sys.exit(1)
> +
> +    # Download using bb.fetch with sha256 validation
> +    src_uri = "{};sha256sum={}".format(url, sha256)
> +
> +    logger.plain("Fetching buildtools installer using bitbake fetcher")
> +    logger.plain("    {}".format(url))
> +
> +    fetcher = bb.fetch.Fetch([src_uri], d)
> +    try:
> +        fetcher.download()
> +    except bb.fetch2.ChecksumError as e:
> +        logger.error("Checksum mismatch for buildtools installer. "
> +                     "Verify the URL and downloaded file.\n%s" % str(e))
> +        sys.exit(1)
> +    except bb.fetch2.FetchError as e:
> +        logger.error("Failed to download buildtools installer: %s" % str(e))
> +        sys.exit(1)
> +
> +    # Unpack into the standard buildtools download location
> +    buildtools_download_dir = add_unique_timestamp_to_path(
> +        os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
> +    os.makedirs(buildtools_download_dir, exist_ok=True)
> +    fetcher.unpack(buildtools_download_dir)
> +
> +    filename = os.path.basename(url.rstrip('/'))
> +    matches = glob.glob(os.path.join(buildtools_download_dir, '**', filename), recursive=True)
> +    if not matches:
> +        logger.error("Unpacked file not found: %s" % filename)
> +        sys.exit(1)
> +    local_path = matches[0]
> +
> +    logger.plain("Buildtools installer will be installed into {}".format(
> +        buildtools_install_dir))
> +
> +    cmd = [
> +        install_buildtools_script,
> +        '--local-file', local_path,
> +        '-d', buildtools_install_dir,
> +    ]
> +
> +    try:
> +        subprocess.check_call(cmd)
> +    except subprocess.CalledProcessError as e:
> +        logger.error("install-buildtools failed with exit code %d" % e.returncode)
> +        sys.exit(1)
>
>  def create_siteconf(top_dir, non_interactive, settings):
>      siteconfpath = os.path.join(top_dir, 'site.conf')
> @@ -1286,6 +1364,9 @@ def main():
>      parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine')
>      add_setup_dir_arg(parser_install_buildtools)
>      parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.')
> +    parser_install_buildtools.add_argument('--default', action='store_true', help='Use install-buildtools script defaults, ignoring any config.')
> +    parser_install_buildtools.add_argument('--url', help='Full URL to the buildtools SDK installer. Overrides config value.')
> +    parser_install_buildtools.add_argument('--sha256', help='SHA256 checksum of the buildtools installer. Overrides config value.')
>      parser_install_buildtools.set_defaults(func=install_buildtools)
>
>      parser_settings_arg_global = argparse.ArgumentParser(add_help=False)
> diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
> index aa546c30b..e1701555e 100644
> --- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
> +++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
> @@ -660,6 +660,30 @@ In addition, the command can take the following arguments:
>  -  ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
>     required if :term:`BBPATH` is already configured.
>
> +-  ``--url``: full URL to the buildtools SDK installer. Overrides the value
> +   from the configuration file.
> +
> +-  ``--sha256``: SHA256 checksum of the buildtools installer. Overrides the
> +   value from the configuration file.
> +
> +When ``url`` and ``sha256sum`` are set in the :term:`Configuration File`,
> +the installer is downloaded via ``bb.fetch`` (cached in ``DL_DIR``) and its
> +checksum is enforced. If no configuration is present, the script falls back
> +to its built-in defaults::
> +
> +   "bitbake-setup": {
> +       "configurations": [
> +           {
> +               "name": "my-config",
> +               "install-buildtools": {
> +                   "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
> +                   "sha256sum": "af76648b..."
> +               },
> +               ...
> +           }
> +       ]
> +   }
> +
>  .. _ref-bbsetup-command-settings:
>
>  ``bitbake-setup settings``
> diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
> index 5592e8196..8ca64cb50 100644
> --- a/lib/bb/tests/setup.py
> +++ b/lib/bb/tests/setup.py
> @@ -61,16 +61,31 @@ import getopt
>  import sys
>  import os
>
> -opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="])
> +opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory=", "local-file="])
> +installdir = None
> +local_file = None
>  for option, value in opts:
>      if option == '-d':
>          installdir = value
> +    elif option == '--local-file':
> +        local_file = value
> +        print("install-buildtools local-file={}".format(value))
>
>  print("Buildtools installed into {}".format(installdir))
> -os.makedirs(installdir)
> +os.makedirs(installdir, exist_ok=True)
>  """
>          self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True)
>
> +        # Dummy buildtools installer for bb.fetch testing
> +        self.buildtools_dir = os.path.join(self.tempdir, "buildtools-dl")
> +        os.makedirs(self.buildtools_dir)
> +        self.buildtools_filename = "x86_64-buildtools-nativesdk-standalone-test.sh"
> +        buildtools_filepath = os.path.join(self.buildtools_dir, self.buildtools_filename)
> +        with open(buildtools_filepath, 'w') as f:
> +            f.write("#!/bin/sh\necho dummy\n")
> +        with open(buildtools_filepath, 'rb') as f:
> +            self.buildtools_sha256 = hashlib.sha256(f.read()).hexdigest()
> +
>          bitbakeconfigbuild = """#!/usr/bin/env python3
>  import os
>  import sys
> @@ -108,7 +123,11 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
>                  "name": "gadget",
>                  "description": "Gadget configuration",
>                  "oe-template": "test-configuration-gadget",
> -                "oe-fragments": ["test-fragment-1"]
> +                "oe-fragments": ["test-fragment-1"],
> +                "install-buildtools": {
> +                    "url": "file://%s/%s",
> +                    "sha256sum": "%s"
> +                }
>              },
>              {
>                  "name": "gizmo",
> @@ -176,7 +195,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
>      },
>      "version": "1.0"
>  }
> -""" % (sources)
> +""" % (sources, self.buildtools_dir, self.buildtools_filename, self.buildtools_sha256)
>          os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True)
>          with open(os.path.join(self.registrypath, name), 'w') as f:
>              f.write(config)
> @@ -644,6 +663,56 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
>      def _count_layer_backups(self, layers_path):
>          return len([f for f in os.listdir(layers_path) if 'backup' in f])
>
> +    def test_install_buildtools_fetch(self):
> +        """Test that install-buildtools uses bb.fetch with sha256 and passes --local-file"""
> +        import shutil
> +
> +        if 'BBPATH' in os.environ:
> +            del os.environ['BBPATH']
> +        os.chdir(self.tempdir)
> +
> +        registry_uri = "git://{};protocol=file;branch=master;rev=master".format(
> +            self.registrypath)
> +        self.runbbsetup(["settings", "set", "default", "registry", registry_uri])
> +        self.add_file_to_testrepo('test-file', 'initial\n')
> +        self.add_json_config_to_registry('test-config-bt.conf.json', 'master', 'master')
> +        self.runbbsetup(["init", "--non-interactive", "test-config-bt", "gadget"])
> +        setuppath = self.get_setup_path('test-config-bt', 'gadget')
> +
> +        # test config-driven install (url + sha256sum from config)
> +        out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath])
> +        self.assertIn("Buildtools installed into", out[0])
> +        self.assertIn("install-buildtools local-file=", out[0])
> +        self.assertTrue(os.path.exists(os.path.join(setuppath, 'buildtools')))
> +
> +        # test CLI overrides (use a different file to prove precedence)
> +        shutil.rmtree(os.path.join(setuppath, 'buildtools'))
> +        alt_filename = "alt-buildtools-test.sh"
> +        alt_filepath = os.path.join(self.buildtools_dir, alt_filename)
> +        with open(alt_filepath, 'w') as f:
> +            f.write("#!/bin/sh\necho alt\n")
> +        with open(alt_filepath, 'rb') as f:
> +            alt_sha256 = hashlib.sha256(f.read()).hexdigest()
> +        alt_url = "file://{}/{}".format(self.buildtools_dir, alt_filename)
> +        out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath,
> +                               "--url", alt_url,
> +                               "--sha256", alt_sha256])
> +        self.assertIn("Buildtools installed into", out[0])
> +        self.assertIn("install-buildtools local-file=", out[0])
> +        self.assertIn(alt_filename, out[0])
> +
> +        # test missing sha256sum is a hard error
> +        shutil.rmtree(os.path.join(setuppath, 'buildtools'), ignore_errors=True)
> +        with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
> +            config_upstream = json.load(f)
> +        config_upstream['bitbake-config']['install-buildtools'] = {
> +            'url': 'file://{}/{}'.format(self.buildtools_dir, self.buildtools_filename)
> +        }
> +        with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
> +            json.dump(config_upstream, f)
> +        with self.assertRaises(bb.process.ExecutionError):
> +            self.runbbsetup(["install-buildtools", "--setup-dir", setuppath, "--force"])
> +
>      def test_update_rebase_conflicts_strategy(self):
>          """Test the --rebase-conflicts-strategy option for the update command.
>
> diff --git a/setup-schema/bitbake-setup.schema.json b/setup-schema/bitbake-setup.schema.json
> index 99f47f73d..797d45252 100644
> --- a/setup-schema/bitbake-setup.schema.json
> +++ b/setup-schema/bitbake-setup.schema.json
> @@ -137,6 +137,27 @@
>                              "setup-dir-name": {
>                                  "type": "string",
>                                  "description": "A suggestion for the setup directory name, $-prefixed keys from oe-fragments-one-of will be substituted with user selections."
> +                            },
> +                            "install-buildtools": {
> +                                "type": "object",
> +                                "description": "Settings for downloading buildtools via bb.fetch",
> +                                "properties": {
> +                                    "url": {
> +                                        "type": "string",
> +                                        "minLength": 1,
> +                                        "description": "Full URL to the buildtools SDK installer."
> +                                    },
> +                                    "sha256sum": {
> +                                        "type": "string",
> +                                        "minLength": 1,
> +                                        "description": "SHA256 checksum of the buildtools installer."
> +                                    }
> +                                },
> +                                "required": [
> +                                    "url",
> +                                    "sha256sum"
> +                                ],
> +                                "additionalProperties": false
>                              }
>                          },
>                          "additionalProperties": false
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#19808): https://lists.openembedded.org/g/bitbake-devel/message/19808
> Mute This Topic: https://lists.openembedded.org/mt/120037327/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>