From patchwork Mon Sep 1 13:58:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 69344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72BC4CA1002 for ; Mon, 1 Sep 2025 13:58:50 +0000 (UTC) Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by mx.groups.io with SMTP id smtpd.web11.52233.1756735125279287184 for ; Mon, 01 Sep 2025 06:58:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=XmeMhTyn; spf=pass (domain: gmail.com, ip: 209.85.218.43, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-b043da5a55fso74593666b.0 for ; Mon, 01 Sep 2025 06:58:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756735123; x=1757339923; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Gmvcl1hFUFrhZb52iRcKGU9mThHx+5Wv4e+r32SHRfg=; b=XmeMhTynVd11547AFUmhZ7rYfQvzDpy7tSvLiumLnT01iL3ld4qIW+C1Cf/3RWKkfl KurNUwxBxHrcoxS/JquuoI+d9JRUfA3kmDbBGAIisX305ANNcD2V+1eHPUV53+PlYYxi qnOAyY+LI3MooBPA8l9GBT5+iT2gv7KhjTO0xF6Fee8ROy0/mLQqv735DYHPCcDQvZ4q D3yWwdysHlxZX0SmezaZf1K6T+pFHEhky0ecqnT2dnxSci2HlZ/MQ8vwOsTil11HfR2f ALMGJYI+esQAqixZNevFz9XRjxVwdAieCIQk1dcDAeluBRxco+oTIZiNlRgc2AR81vo0 F51Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756735123; x=1757339923; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Gmvcl1hFUFrhZb52iRcKGU9mThHx+5Wv4e+r32SHRfg=; b=HNc6Ty/1qLGADnILmnz48tmPhjauvgL4F2IgTm6ha1zpOAi9FFaluoo0oRImR5CWFC ut088sI9WVejuu1TSqP+lGZXI6jv5qNV8ZQogqb+ctfUGtmRC0dKpkT3TgGUw2tEXG7n +QAdkUTafiKuCqIr9XdzdcR05uMicA8RXzfbjuqLhbQ0gomTQcbFUmXetwOzr4Slm3u4 o62PcRCRSz6+CM06E7Nx9ck4uq2HUd8nnKGRhAg+YJpJhE8fYDC8xgCATMiJHssmnwIm MbcWOJl9PuCERYEH4BxbN9NsWY7Y9PVQrbekkuixP0nYLfXGPFM1SvsH/FWXg30d1Man cJVQ== X-Gm-Message-State: AOJu0YyMDiPed5BYDwHSZeYWkr/2gDvuGArEWVVqkugS9/fypkexLYsm kcc91Q/lDZf9g+VQwFKbN/EecGP/U9gMiHBKOa6NATnZjCZTzSlQiXp1aaSWcw== X-Gm-Gg: ASbGncuoO7eB5Avj+bnuOKsvs5Qo/jYrjMs3C/4CR/5zJKEkSj4V56/99QjRGCbCJZB mVqysSTv7Jz1oIuEZy9c9vWhIqGS19W6G0xGUkjF88Qg1/AMpIQt7Jq9KXLhU7SosDbICnpPV35 uX9uxwi8D8Sm261VsUE9Zvxlv8ju16hh62m8Qg51L6yHYK2dkQ4S6aRbuchL4zwt9TXx1KCJCwD lTIU6Bg/jsqZluxCgqtcn8HjdiXIw646sF9w9paPWN37EHYR02A+45YYfiSqJO2HGtM7QZPm4WC Aup1HqMHO/duJedzOhY6Dc6A4UQquK/xj7C+mZQgW2c4pwY/fVpB48Vx8AhDL1DxB2e1uPMfGX+ nfqw/otJBExEuBaCRyhkPfrb3F0dbdURa5/UGf8W3lRHlKUj0p8/0zKAIOiqmeLZ5l8ddhm1Re0 eaHVA0e/4s0zPBT3+XxT4ROEXGK9Mox8Quytp//W0s0C3eWqWyif3DQrdJ3A== X-Google-Smtp-Source: AGHT+IHreCFsoh2vC0Sw5QIA9mBlsMhdPY3ChP9wzMtQ8joDGqrxU0Cj3dEIuA1CdHQlbh+VuTuchA== X-Received: by 2002:a17:907:868b:b0:b04:31c6:a43a with SMTP id a640c23a62f3a-b0431c6a90fmr297021966b.41.1756735122893; Mon, 01 Sep 2025 06:58:42 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-afefcc1c6fdsm873503166b.75.2025.09.01.06.58.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 06:58:42 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 1/6] bitbake-setup: add the initial implementation Date: Mon, 1 Sep 2025 15:58:31 +0200 Message-Id: <20250901135836.2927686-1-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Sep 2025 13:58:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17913 From: Alexander Kanavin Preamble ======== The latest iteration of this patchset is available at https://github.com/kanavin/bitbake I recommend taking the patches from there to ensure that you are not trying out outdated code. For the rationale and design guidelines please see this message: https://lists.openembedded.org/g/openembedded-architecture/message/1913 Left out for now but will be done later: - official configuration repository - documentation Amble *scratch* HOWTO ===================== 1. If you don't know where to start, run 'bitbake-setup init'. Bitbake-setup will ask a few questions about available configuration choices and set up a build. Note: 'init' sub-command can also take a path or a URL with a configuration file directly. You can see how those files look like here: https://github.com/kanavin/bitbake-setup-configurations 2. You can then source the bitbake environment and run bitbake to perform builds as usual: $ . /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env Also, subsequent status/update commands will not require a separate --build-dir argument telling bitbake-setup where the build is. 3. To check if the build configuration needs to be updated, run: === $ bin/bitbake-setup status ... Configuration in /home/alex/bitbake-builds/poky-alex/ has not changed. === If the configuration has changed, you will see the difference as a diff. ... - "rev": "akanavin/sstate-for-all" + "rev": "akanavin/bitbake-setup-testing" ... If the configuration has not changed, but layer revisions referred to it have (for example if the configuration specifies a tip of a branch), you will see that too: === ... Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to aeb73e29379fe6007a8adc8d94c1ac18a93e68de === 4. If the configuration has changed, you can bring it in sync with: $ bin/bitbake-setup update Note that it will also rename/preserve the existing build/conf directory, and print changes in bitbake configuration (diff of content of build/conf/) if that has changed. I can't at the moment think of anything more clever that is also not much more brittle or complex to implement, but open to suggestions. Terminology =========== - 'top directory' means the place under which bitbake-setup reads and writes everything. bitbake-setup makes a promise to not touch anything outside of that, unless otherwise directed to by entries in settings (currently there is one such setting for fetcher downloads for layers and config registries). Top directory can be selected by an environment variable, a command line option, or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment (e.g. we are in a bitbake environment), then the top directory is deduced from that and doesn't need to be specified by hand. - 'settings' means bitbake-setup operational parameters that are global to all builds under a top directory. E.g. the location of configuration registry, or where the bitbake fetcher should place the downloads (DL_DIR setting). Settings are stored in a .conf file in ini format just under the top directory. - 'build' means a tree structure set up by 'bitbake-setup init', consisting of, at least, a layers checkout, and a bitbake build. It maps 1:1 to the json data it was constructed from, which is called 'build configuration'. Build configurations are constructed from generic configurations that may involve making one or more choices about available options in them. Generic configurations are files, URLs or are obtained from git repositories called 'config registries', in which case they can be listed with 'bitbake-setup list'. There can be multiple 'builds' under a top directory. Here are two example generic configurations that showcase this: https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-options.conf.json https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-nested-configs.conf.json - 'bitbake-setup status' will tell if a build is in sync with the generic configuration it was made from. 'bitbake-setup update' will bring a build in sync with a configuration if needed. - 'bitbake build' means a particular sub-tree inside a build that bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR by oe-init-build-env. conf/* in that tree is 'bitbake configuration'. Bitbake configurations are constructed from templates and fragments, with existing mechanisms provided by oe-core. The configuration file format is specified such that other mechanisms to set up a bitbake build can be added; there was a mention of ability to specify local.conf content and a set of layers directly in a configuration. I think that scales poorly compared to templates and fragments, but I made sure alternative ways to configure a bitbake build are possible to add in the future :) - 'source override' is a json file that can be used to modify revisions and origins of layers that need to be checkout into a build (e.g. when master branches need to be changed to master-next for purposes of testing). Such a file is specified with a command-line option to 'init' and an example can be seen here: https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-next.override.json This commit includes a fix by Ryan Eatmon : https://github.com/kanavin/bitbake/pull/1 Signed-off-by: Alexander Kanavin --- bin/bitbake-setup | 711 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 711 insertions(+) create mode 100755 bin/bitbake-setup diff --git a/bin/bitbake-setup b/bin/bitbake-setup new file mode 100755 index 000000000..a05433206 --- /dev/null +++ b/bin/bitbake-setup @@ -0,0 +1,711 @@ +#!/usr/bin/env python3 + +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import sys +import argparse +import warnings +import json +import shutil +import time +import stat +import tempfile +import configparser +import datetime + +default_registry = 'git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main' + +bindir = os.path.abspath(os.path.dirname(__file__)) +sys.path[0:0] = [os.path.join(os.path.dirname(bindir), 'lib')] + +import bb.msg +import bb.process + +logger = bb.msg.logger_create('bitbake-setup', sys.stdout) + +def cache_dir(top_dir): + return os.path.join(top_dir, '.bitbake-setup-cache') + +def init_bb_cache(settings, args): + dldir = settings["default"]["dl-dir"] + bb_cachedir = os.path.join(cache_dir(args.top_dir), 'bitbake-cache') + + d = bb.data.init() + d.setVar("DL_DIR", dldir) + d.setVar("BB_CACHEDIR", bb_cachedir) + d.setVar("__BBSRCREV_SEEN", "1") + if args.no_network: + d.setVar("BB_SRCREV_POLICY", "cache") + bb.fetch.fetcher_init(d) + return d + +def save_bb_cache(): + bb.fetch2.fetcher_parse_save() + bb.fetch2.fetcher_parse_done() + +def get_config_name(config): + return os.path.basename(config).split('.')[0] + +def write_config(config, config_dir): + with open(os.path.join(config_dir, "config-upstream.json"),'w') as s: + json.dump(config, s, sort_keys=True, indent=4) + +def commit_config(config_dir): + bb.process.run("git -C {} add .".format(config_dir)) + bb.process.run("git -C {} commit -a -m 'Configuration at {}'".format(config_dir, time.asctime())) + +def _write_layer_list(dest, repodirs): + layers = [] + for r in repodirs: + for root, dirs, files in os.walk(os.path.join(dest,r)): + if os.path.basename(root) == 'conf' and 'layer.conf' in files: + layers.append(os.path.relpath(os.path.dirname(root), dest)) + layers_f = os.path.join(dest, ".oe-layers.json") + with open(layers_f, 'w') as f: + json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4) + +def checkout_layers(layers, layerdir, d): + repodirs = [] + oesetupbuild = None + print("Fetching layer/tool repositories into {}".format(layerdir)) + for r_name in layers: + r_data = layers[r_name] + repodir = r_data["path"] + repodirs.append(repodir) + + r_remote = r_data['git-remote'] + rev = r_remote['rev'] + remotes = r_remote['remotes'] + + for remote in remotes: + type,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + print(" {}".format(r_name)) + fetcher = bb.fetch.Fetch(["{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir)], d) + do_fetch(fetcher, layerdir) + + if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')): + oesetupbuild = os.path.join(layerdir, repodir, 'scripts/oe-setup-build') + oeinitbuildenv = os.path.join(layerdir, repodir, 'oe-init-build-env') + + print(" ") + _write_layer_list(layerdir, repodirs) + + if oesetupbuild: + links = {'setup-build': oesetupbuild, 'oe-scripts': os.path.dirname(oesetupbuild), 'init-build-env': oeinitbuildenv} + for l,t in links.items(): + symlink = os.path.join(layerdir, l) + if os.path.lexists(symlink): + os.remove(symlink) + os.symlink(os.path.relpath(t,layerdir),symlink) + +def setup_bitbake_build(bitbake_config, layerdir, builddir): + def _setup_build_conf(layers, build_conf_dir): + os.makedirs(build_conf_dir) + layers_s = "\n".join([" {} \\".format(os.path.join(layerdir,l)) for l in layers]) + bblayers_conf = """BBLAYERS ?= " \ +{} + " +""".format(layers_s) + with open(os.path.join(build_conf_dir, "bblayers.conf"), 'w') as f: + f.write(bblayers_conf) + + local_conf = """# +# This file is intended for local configuration tweaks. +# +# If you would like to publish and share changes made to this file, +# it is recommended to put them into a distro config, or to create +# layer fragments from changes made here. +# +""" + with open(os.path.join(build_conf_dir, "local.conf"), 'w') as f: + f.write(local_conf) + + with open(os.path.join(build_conf_dir, "templateconf.cfg"), 'w') as f: + f.write("") + + with open(os.path.join(build_conf_dir, "conf-summary.txt"), 'w') as f: + f.write(bitbake_config["description"] + "\n") + + with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as f: + f.write("") + + def _make_init_build_env(builddir, initbuildenv): + cmd = ". {} {}".format(initbuildenv, builddir) + initbuild_in_builddir = os.path.join(builddir, 'init-build-env') + with open(initbuild_in_builddir, 'w') as f: + f.write(cmd) + + bitbake_builddir = os.path.join(builddir, "build") + print("Setting up bitbake configuration in\n {}\n".format(bitbake_builddir)) + + template = bitbake_config.get("oe-template") + layers = bitbake_config.get("bb-layers") + if not template and not layers: + print("Bitbake configuration does not contain a reference to an OpenEmbedded build template via 'oe-template' or a list of layers via 'bb-layers'; please use oe-setup-build, oe-init-build-env or another mechanism manually to complete the setup.") + return + oesetupbuild = os.path.join(layerdir, 'setup-build') + if template and not os.path.exists(oesetupbuild): + raise Exception("Cannot complete setting up a bitbake build directory from OpenEmbedded template '{}' as oe-setup-build was not found in any layers; please use oe-init-build-env manually.".format(template)) + + bitbake_confdir = os.path.join(bitbake_builddir, 'conf') + backup_bitbake_confdir = bitbake_confdir + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + if os.path.exists(bitbake_confdir): + os.rename(bitbake_confdir, backup_bitbake_confdir) + + if layers: + _setup_build_conf(layers, bitbake_confdir) + + if template: + bb.process.run("{} setup -c {} -b {} --no-shell".format(oesetupbuild, template, bitbake_builddir)) + else: + initbuildenv = os.path.join(layerdir, 'init-build-env') + if not os.path.exists(initbuildenv): + print("Could not find oe-init-build-env in any of the layers; please use another mechanism to initialize the bitbake environment") + return + _make_init_build_env(bitbake_builddir, os.path.realpath(initbuildenv)) + + siteconf_symlink = os.path.join(bitbake_confdir, "site.conf") + siteconf = os.path.normpath(os.path.join(builddir, '..', "site.conf")) + if os.path.lexists(siteconf_symlink): + os.remove(symlink) + os.symlink(os.path.relpath(siteconf, bitbake_confdir) ,siteconf_symlink) + + + init_script = os.path.join(bitbake_builddir, "init-build-env") + shell = os.path.basename(os.environ.get("SHELL","bash")) + fragments = bitbake_config.get("oe-fragments", []) + sorted(bitbake_config.get("oe-fragment-choices",{}).values()) + if fragments: + bb.process.run("{} -c '. {} && bitbake-config-build enable-fragment {}'".format(shell, init_script, " ".join(fragments))) + + if os.path.exists(backup_bitbake_confdir): + bitbake_config_diff = get_diff(backup_bitbake_confdir, bitbake_confdir) + if bitbake_config_diff: + print("Existing bitbake configuration directory renamed to {}".format(backup_bitbake_confdir)) + print("The bitbake configuration has changed:") + print(bitbake_config_diff) + else: + shutil.rmtree(backup_bitbake_confdir) + + print("This bitbake configuration provides:\n {}\n".format(bitbake_config["description"])) + + readme = """{}\n\nAdditional information is in {} and {}\n +Source the environment using '. {}' to run builds from the command line. +The bitbake configuration files (local.conf, bblayers.conf and more) can be found in {}/conf +""".format( + bitbake_config["description"], + os.path.join(bitbake_builddir,'conf/conf-summary.txt'), + os.path.join(bitbake_builddir,'conf/conf-notes.txt'), + init_script, + bitbake_builddir + ) + readme_file = os.path.join(bitbake_builddir, "README") + with open(readme_file, 'w') as f: + f.write(readme) + print("Usage instructions and additional information are in\n {}\n".format(readme_file)) + print("The bitbake configuration files (local.conf, bblayers.conf and more) can be found in\n {}/conf\n".format(bitbake_builddir)) + print("To run builds, source the environment using\n source {}".format(init_script)) + +def get_registry_config(registry_path, id): + for root, dirs, files in os.walk(registry_path): + for f in files: + if f.endswith('.conf.json') and id == get_config_name(f): + return os.path.join(root, f) + raise Exception("Unable to find {} in available configurations; use 'list' sub-command to see what is available".format(id)) + +def update_build(config, confdir, builddir, layerdir, d): + layer_config = config["data"]["sources"] + layer_overrides = config["source-overrides"]["sources"] + for k,v in layer_overrides.items(): + if k in layer_config: + layer_config[k]["git-remote"] = v["git-remote"] + checkout_layers(layer_config, layerdir, d) + bitbake_config = config["bitbake-config"] + setup_bitbake_build(bitbake_config, layerdir, builddir) + +def flatten_bitbake_configs(configs): + def merge_configs(c1,c2): + c_merged = {} + for k,v in c2.items(): + if k not in c1.keys(): + c_merged[k] = v + for k,v in c1.items(): + if k not in c2.keys(): + c_merged[k] = v + else: + c_merged[k] = c1[k] + c2[k] + del c_merged['configurations'] + return c_merged + + flattened_configs = [] + for c in configs: + if 'configurations' not in c: + flattened_configs.append(c) + else: + for sub_c in flatten_bitbake_configs(c['configurations']): + flattened_configs.append(merge_configs(c, sub_c)) + return flattened_configs + +def choose_bitbake_config(configs, parameters, non_interactive): + flattened_configs = flatten_bitbake_configs(configs) + configs_dict = {i["name"]:i for i in flattened_configs} + + if parameters: + config_id = parameters[0] + if config_id not in configs_dict: + raise Exception("Bitbake configuration {} not found; replace with one of {}".format(config_id, configs_dict)) + return configs_dict[config_id] + + enumerated_configs = list(enumerate(flattened_configs)) + if len(enumerated_configs) == 1: + only_config = flattened_configs[0] + print("\nSelecting the only available bitbake configuration {}".format(only_config["name"])) + return only_config + + if non_interactive: + raise Exception("Unable to choose from bitbake configurations in non-interactive mode: {}".format(configs_dict)) + + print("\nAvailable bitbake configurations:") + for n, config_data in enumerated_configs: + print("{}. {}\t{}".format(n, config_data["name"], config_data["description"])) + print("\nPlease select one of the above bitbake configurations by its number:") + config_n = int(input()) + return flattened_configs[config_n] + +def choose_config(configs, non_interactive): + not_expired_configs = [k for k in configs.keys() if not has_expired(configs[k].get("expires", None))] + config_list = list(enumerate(not_expired_configs)) + if len(config_list) == 1: + only_config = config_list[0][1] + print("\nSelecting the only available configuration {}\n".format(only_config)) + return only_config + + if non_interactive: + raise Exception("Unable to choose from configurations in non-interactive mode: {}".format(not_expired_configs)) + + print("\nAvailable configurations:") + for n, config_name in config_list: + config_data = configs[config_name] + expiry_date = config_data.get("expires", None) + config_desc = config_data["description"] + if expiry_date: + print("{}. {}\t{} (supported until {})".format(n, config_name, config_desc, expiry_date)) + else: + print("{}. {}\t{}".format(n, config_name, config_desc)) + print("\nPlease select one of the above configurations by its number:") + config_n = int(input()) + return config_list[config_n][1] + +def choose_fragments(possibilities, parameters, non_interactive): + choices = {} + for k,v in possibilities.items(): + choice = [o for o in v["options"] if o in parameters] + if len(choice) > 1: + raise Exception("Options specified on command line do not allow a single selection from possibilities {}, please remove one or more from {}".format(v["options"], parameters)) + if len(choice) == 1: + choices[k] = choice[0] + continue + + if non_interactive: + raise Exception("Unable to choose from options in non-interactive mode: {}".format(v["options"])) + + print("\n" + v["description"] + ":") + options_enumerated = list(enumerate(v["options"])) + for n,o in options_enumerated: + print("{}. {}".format(n, o)) + print("\nPlease select one of the above options by its number:") + option_n = int(input()) + choices[k] = options_enumerated[option_n][1] + return choices + +def obtain_config(settings, args, source_overrides, d): + if args.config: + config_id = args.config[0] + config_parameters = args.config[1:] + if os.path.exists(config_id): + print("Reading configuration from local file\n {}".format(config_id)) + upstream_config = {'type':'local','path':os.path.abspath(args.config),'name':get_config_name(config_id),'data':json.load(open(config_path))} + elif config_id.startswith("http://") or config_id.startswith("https://"): + print("Reading configuration from network URI\n {}".format(config_id)) + import urllib.request + with urllib.request.urlopen(config_id) as f: + upstream_config = {'type':'network','uri':config_id,'name':get_config_name(config_id),'data':json.load(f)} + else: + print("Looking up config {} in configuration registry".format(config_id)) + registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + if config_id not in registry_configs: + raise Exception("Config {} not found in configuration registry, re-run 'init' without parameters to choose from available configurations.".format(config_id)) + upstream_config = {'type':'registry','registry':settings["default"]["registry"],'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + expiry_date = upstream_config['data'].get("expires", None) + if has_expired(expiry_date): + print("This configuration is no longer supported after {}. Please consider changing to a supported configuration.".format(expiry_date)) + else: + registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + config_id = choose_config(registry_configs, args.non_interactive) + config_parameters = [] + upstream_config = {'type':'registry','registry':settings["default"]["registry"],'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + + upstream_config['bitbake-config'] = choose_bitbake_config(upstream_config['data']['bitbake-setup']['configurations'], config_parameters, args.non_interactive) + upstream_config['bitbake-config']['oe-fragment-choices'] = choose_fragments(upstream_config['bitbake-config'].get('oe-fragments-one-of',{}), config_parameters[1:], args.non_interactive) + upstream_config['non-interactive-cmdline-options'] = " ".join([upstream_config['name'], upstream_config['bitbake-config']['name']] + sorted(upstream_config['bitbake-config']['oe-fragment-choices'].values())) + upstream_config['source-overrides'] = source_overrides + return upstream_config + +def init_config(settings, args, d): + stdout = sys.stdout + def handle_task_progress(event, d): + rate = event.rate if event.rate else '' + progress = event.progress if event.progress > 0 else 0 + print("{}% {} ".format(progress, rate), file=stdout, end='\r') + + source_overrides = json.load(open(args.source_overrides)) if args.source_overrides else {'sources':{}} + upstream_config = obtain_config(settings, args, source_overrides, d) + print("\nRun 'bitbake-setup --non-interactive init {}' to select this configuration non-interactively.\n".format(upstream_config['non-interactive-cmdline-options'])) + + builddir = os.path.join(os.path.abspath(args.top_dir), upstream_config['non-interactive-cmdline-options'].replace(" ","-").replace("/","_")) + if os.path.exists(builddir): + print("Build already initialized in {}\nUse 'bitbake-setup status' to check if it needs to be updated or 'bitbake-setup update' to perform the update.".format(builddir)) + return + + print("Initializing a build in\n {}".format(builddir)) + if not args.non_interactive: + y_or_n = input('Continue? y/n: ') + if y_or_n != 'y': + exit() + print() + + + os.makedirs(builddir) + + confdir = os.path.join(builddir, "config") + layerdir = os.path.join(builddir, "layers") + + os.makedirs(confdir) + os.makedirs(layerdir) + + bb.process.run("git -C {} init -b main".format(confdir)) + bb.process.run("git -C {} commit --allow-empty -m 'Initial commit'".format(confdir)) + + bb.event.register("bb.build.TaskProgress", handle_task_progress, data=d) + + write_config(upstream_config, confdir) + commit_config(confdir) + update_build(upstream_config, confdir, builddir, layerdir, d) + + bb.event.remove("bb.build.TaskProgress", None) + +def get_diff(file1, file2): + try: + bb.process.run('diff -uNr {} {}'.format(file1, file2)) + except bb.process.ExecutionError as e: + if e.exitcode == 1: + return e.stdout + else: + raise e + return None + +def are_layers_changed(layers, layerdir, d): + changed = False + for r_name in layers: + r_data = layers[r_name] + repodir = r_data["path"] + + r_remote = r_data['git-remote'] + rev = r_remote['rev'] + remotes = r_remote['remotes'] + + for remote in remotes: + type,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + fetcher = bb.fetch.FetchData("{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir), d) + upstream_revision = fetcher.method.latest_revision(fetcher, d, 'default') + rev_parse_result = bb.process.run('git -C {} rev-parse HEAD'.format(os.path.join(layerdir, repodir))) + local_revision = rev_parse_result[0].strip() + if upstream_revision != local_revision: + changed = True + print('Layer repository {} checked out into {} updated revision {} from {} to {}'.format(remotes[remote]["uri"], os.path.join(layerdir, repodir), rev, local_revision, upstream_revision)) + + return changed + +def build_status(settings, args, d, update=False): + builddir = args.build_dir + + confdir = os.path.join(builddir, "config") + layerdir = os.path.join(builddir, "layers") + + current_upstream_config = json.load(open(os.path.join(confdir, "config-upstream.json"))) + + args.config = current_upstream_config['non-interactive-cmdline-options'].split() + args.non_interactive = True + source_overrides = current_upstream_config["source-overrides"] + new_upstream_config = obtain_config(settings, args, source_overrides, d) + + write_config(new_upstream_config, confdir) + config_diff = bb.process.run('git -C {} diff'.format(confdir))[0] + + if config_diff: + print('\nConfiguration in {} has changed:\n{}'.format(builddir, config_diff)) + if update: + commit_config(confdir) + update_build(new_upstream_config, confdir, builddir, layerdir, d) + else: + bb.process.run('git -C {} restore config-upstream.json'.format(confdir)) + return + + if are_layers_changed(current_upstream_config["data"]["sources"], layerdir, d): + if update: + update_build(current_upstream_config, confdir, builddir, layerdir, d) + return + + print("\nConfiguration in {} has not changed.".format(builddir)) + +def build_update(settings, args, d): + build_status(settings, args, d, update=True) + +def do_fetch(fetcher, dir): + # git fetcher simply dumps git output to stdout; in bitbake context that is redirected to temp/log.do_fetch + # and we need to set up smth similar here + fetchlogdir = os.path.join(dir, 'logs') + os.makedirs(fetchlogdir, exist_ok=True) + fetchlog = os.path.join(fetchlogdir, 'fetch_log.{}'.format(datetime.datetime.now().strftime("%Y%m%d%H%M%S"))) + with open(fetchlog, 'a') as f: + oldstdout = sys.stdout + sys.stdout = f + fetcher.download() + fetcher.unpack(dir) + sys.stdout = oldstdout + +def update_registry(registry, cachedir, d): + registrydir = 'configurations' + full_registrydir = os.path.join(cachedir, registrydir) + print("Fetching configuration registry\n {}\ninto\n {}".format(registry, full_registrydir)) + fetcher = bb.fetch.Fetch(["{};destsuffix={}".format(registry, registrydir)], d) + do_fetch(fetcher, cachedir) + return full_registrydir + +def has_expired(expiry_date): + if expiry_date: + return datetime.datetime.now() > datetime.datetime.fromisoformat(expiry_date) + return False + +def list_registry(registry_path, with_expired): + json_data = {} + + for root, dirs, files in os.walk(registry_path): + for f in files: + if f.endswith('.conf.json'): + config_name = get_config_name(f) + config_data = json.load(open(os.path.join(root, f))) + config_desc = config_data["description"] + expiry_date = config_data.get("expires", None) + if expiry_date: + if with_expired or not has_expired(expiry_date): + json_data[config_name] = {"description": config_desc, "expires": expiry_date} + else: + json_data[config_name] = {"description": config_desc} + return json_data + +def list_configs(settings, args, d): + registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + json_data = list_registry(registry_path, args.with_expired) + print("\nAvailable configurations:") + for config_name, config_data in json_data.items(): + expiry_date = config_data.get("expires", None) + config_desc = config_data["description"] + if expiry_date: + if args.with_expired or not has_expired(expiry_date): + print("{}\t{} (supported until {})".format(config_name, config_desc, expiry_date)) + else: + print("{}\t{}".format(config_name, config_desc)) + print("\nRun 'init' with one of the above configuration identifiers to set up a build.") + + if args.write_json: + with open(args.write_json, 'w') as f: + json.dump(json_data, f, sort_keys=True, indent=4) + print("Available configurations written into {}".format(args.write_json)) + +def default_settings_path(top_dir): + return os.path.join(top_dir, 'bitbake-setup.conf') + +def write_settings(top_dir, force_replace, non_interactive=True): + settings_path = default_settings_path(top_dir) + if not os.path.exists(settings_path) or force_replace: + + settings = configparser.ConfigParser() + settings['default'] = { + 'registry':default_registry, + 'dl-dir':os.path.join(top_dir, '.bitbake-setup-downloads'), + } + os.makedirs(os.path.dirname(settings_path), exist_ok=True) + + siteconfpath = os.path.join(top_dir, 'site.conf') + print('Configuration registry set to\n {}\n'.format(settings['default']['registry'])) + print('Bitbake-setup download cache (DL_DIR) set to\n {}\n'.format(settings['default']['dl-dir'])) + print('A new settings file will be created in\n {}\n'.format(settings_path)) + print('A common site.conf file will be created, please edit or replace before running builds\n {}\n'.format(siteconfpath)) + if not non_interactive: + y_or_n = input('Bitbake-setup will be configured in {}, y/n: '.format(top_dir)) + if y_or_n != 'y': + print("\nYou can run 'bitbake-setup write-settings' to edit them before setting up builds") + exit() + print() + + if os.path.exists(settings_path): + backup_conf = settings_path + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + os.rename(settings_path, backup_conf) + print("Previous settings are in {}".format(backup_conf)) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + + if os.path.exists(siteconfpath): + backup_siteconf = siteconfpath + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + os.rename(siteconfpath, backup_siteconf) + print("Previous settings are in {}".format(backup_siteconf)) + with open(siteconfpath, 'w') as siteconffile: + siteconffile.write('# This file is intended for build host-specific bitbake settings\n') + +def load_settings(top_dir, non_interactive): + # This creates a new settings file if it does not yet exist + write_settings(top_dir, force_replace=False, non_interactive=non_interactive) + + settings_path = default_settings_path(top_dir) + settings = configparser.ConfigParser() + print('Loading settings from\n {}\n'.format(settings_path)) + settings.read([settings_path]) + return settings + +def change_settings(top_dir, new_settings): + # This creates a new settings file if it does not yet exist + write_settings(top_dir, force_replace=False) + + settings = load_settings(top_dir, non_interactive=True) + for section, section_settings in new_settings.items(): + for setting, value in section_settings.items(): + settings[section][setting] = value + print("Setting '{}' in section '{}' is changed to '{}'".format(setting, section, value)) + + settings_path = default_settings_path(top_dir) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + print("New settings written to {}".format(settings_path)) + return settings + +def get_build_dir_via_bbpath(): + bbpath = os.environ.get('BBPATH') + if bbpath: + bitbake_dir = os.path.normpath(bbpath.split(':')[0]) + if os.path.exists(os.path.join(bitbake_dir,'init-build-env')): + build_dir = os.path.dirname(bitbake_dir) + return build_dir + return None + +def get_default_top_dir(): + build_dir_via_bbpath = get_build_dir_via_bbpath() + if build_dir_via_bbpath: + top_dir = os.path.dirname(build_dir_via_bbpath) + if os.path.exists(default_settings_path(top_dir)): + return top_dir + return os.environ.get('BITBAKE_SETUP_TOP_DIR') or os.path.join(os.path.expanduser('~'), 'bitbake-builds') + +def main(): + def add_top_dir_arg(parser): + parser.add_argument('--top-dir', default=get_default_top_dir(), help='Top level directory where builds are set up and downloaded configurations and layers are cached for reproducibility and offline builds, default is %(default)s, can be overriden via BITBAKE_SETUP_TOP_DIR environment variable') + + def add_build_dir_arg(parser): + build_dir = get_build_dir_via_bbpath() + if build_dir: + parser.add_argument('--build-dir', default=build_dir, help="Path to the build, default is %(default)s via BBPATH") + else: + parser.add_argument('--build-dir', required=True, help="Path to the build") + + parser = argparse.ArgumentParser( + description="BitBake setup utility. Run with 'init' argument to get started.", + epilog="Use %(prog)s --help to get help on a specific command" + ) + parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true') + parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true') + parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR') + parser.add_argument('--no-network', action='store_true', help='Do not check whether configuration repositories and layer repositories have been updated; use only the local cache.') + + subparsers = parser.add_subparsers() + + parser_list = subparsers.add_parser('list', help='List available configurations') + add_top_dir_arg(parser_list) + parser_list.add_argument('--with-expired', action='store_true', help='List also configurations that are no longer supported due to reaching their end-of-life dates.') + parser_list.add_argument('--write-json', action='store', help='Write available configurations into a json file so they can be programmatically processed.') + parser_list.set_defaults(func=list_configs) + + parser_init = subparsers.add_parser('init', help='Select a configuration and initialize a build from it') + add_top_dir_arg(parser_init) + parser_init.add_argument('config', nargs='*', help="path/URL/id to a configuration file (use 'list' command to get available ids), followed by configuration options. Bitbake-setup will ask to choose from available choices if command line doesn't completely specify them.") + parser_init.add_argument('--non-interactive', action='store_true', help='Do not ask to interactively choose from available options; if bitbake-setup cannot make a decision it will stop with a failure.') + parser_init.add_argument('--source-overrides', action='store', help='Override sources information (repositories/revisions) with values from a local json file.') + parser_init.set_defaults(func=init_config) + + parser_status = subparsers.add_parser('status', help='Check if the build needs to be synchronized with configuration') + add_build_dir_arg(parser_status) + parser_status.set_defaults(func=build_status) + + parser_update = subparsers.add_parser('update', help='Update a build to be in sync with configuration') + add_build_dir_arg(parser_update) + parser_update.set_defaults(func=build_update) + + parser_install_settings = subparsers.add_parser('install-settings', help='Write a settings file with default values into the top level directory (contains the location of build configuration registry, downloads directory and other global settings)') + add_top_dir_arg(parser_install_settings) + parser_install_settings.set_defaults(func=write_settings) + + parser_change_setting = subparsers.add_parser('change-setting', help='Change a setting in the settings file') + add_top_dir_arg(parser_change_setting) + parser_change_setting.add_argument('section', help="Section in a settings file, typically 'default'") + parser_change_setting.add_argument('key', help="Name of the setting") + parser_change_setting.add_argument('value', help="Value of the setting") + parser_change_setting.set_defaults(func=change_settings) + + args = parser.parse_args() + + logging.basicConfig(stream=sys.stdout) + if args.debug: + logger.setLevel(logging.DEBUG) + elif args.quiet: + logger.setLevel(logging.ERROR) + + # Need to re-run logger_create with color argument + # (will be the same logger since it has the same name) + bb.msg.logger_create('bitbake-setup', output=sys.stdout, + color=args.color, + level=logger.getEffectiveLevel()) + + + if hasattr(args, 'build_dir'): + if not os.path.exists(os.path.join(args.build_dir,'build', 'init-build-env')): + print("Not a valid build directory: build/init-build-env does not exist in {}".format(args.build_dir)) + return + # commands without --top-dir argument (status, update) still need to know where + # the top dir is, but it should be auto-deduced as parent of args.build_dir + if not hasattr(args, 'top_dir'): + args.top_dir = os.path.dirname(os.path.normpath(args.build_dir)) + + if 'func' in args: + print('Bitbake-setup is using {} as top directory (can be changed with --top-dir argument or BITBAKE_SETUP_TOP_DIR variable).\n'.format(args.top_dir)) + if args.func == write_settings: + write_settings(args.top_dir, force_replace=True) + elif args.func == change_settings: + change_settings(args.top_dir, {args.section:{args.key:args.value}}) + else: + if not hasattr(args, 'non_interactive'): + args.non_interactive = True + settings = load_settings(args.top_dir, args.non_interactive) + d = init_bb_cache(settings, args) + args.func(settings, args, d) + save_bb_cache() + else: + from argparse import Namespace + parser.print_help() + +main() From patchwork Mon Sep 1 13:58:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 69342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 842DFCA1006 for ; Mon, 1 Sep 2025 13:58:50 +0000 (UTC) Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by mx.groups.io with SMTP id smtpd.web10.52453.1756735125510590384 for ; Mon, 01 Sep 2025 06:58:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=IT4u6Tjc; spf=pass (domain: gmail.com, ip: 209.85.218.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-b0415e03e25so180195666b.0 for ; Mon, 01 Sep 2025 06:58:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756735124; x=1757339924; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nm781LfaLfIxwhkNwszFvYlERkd11CO7+FWdmA3zOLo=; b=IT4u6Tjc6gH1WnPitX3ziYnMUAFD4VdUzWl+BdSYFcRX/GAT/nJJQI5r2g/n9tBN5s 9L4nU3e/3VhqdMCCpvWcZE0JIdsNm61BZS8FexhAjFIKs5sjtEWHhg5u+M5RCnwPqckn jpl0ORjJn8o+YPaJT3YhIgc46f7HMghbv3T0Yow5EeIXQPKwaZe2qUaHVNCZH3XFIfV3 S+ciuGQ4kBQWwZbpshYhPFdAhRXZ4M4+siZVBVV5x1RmpuOa5K5jyJ1KbSgKTXad3zqT zh12bHdqV1FsTbSRHOXuFFk4qXKolHr6zDpNU+bN68/pGx6LcIBOCvVdCoqmOWN3UQUU qBwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756735124; x=1757339924; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nm781LfaLfIxwhkNwszFvYlERkd11CO7+FWdmA3zOLo=; b=H4ZoGiZu/71w5C/kNSklXQ2WtVjBzo29S1/XFS3p6tmRhLPQpBrriUBE9VfmDcrSW3 XpbGi+2ZWZzt6FeWqcd3VUnwi9N24nUD57UhXArAWCjz51s2d5I80uGsDKarmX+f/5h/ cz9N6bESpK5MQ0lKljOrXTgW/q8RmwoL/SPlSLlCoKRnEHNclHruxNv5EJuOVQn4HCNN OJm2hOQ0l/NSkS8bxjwoyg0D/Ky5oXQV2a0sVey7yyFJtug6LYvLtmyzd+k9hJZ+lctL QWNPwG4yUjLzQ3Y/IhCUnEdYjjMl3sEr8lgh/KqAj3/7fqBvwhI6df2fJbM+T5a0fQQg A/Qw== X-Gm-Message-State: AOJu0YxwiaTEAaAZB08RspdJ9ZUIFsOyK/PclkPuW5T1BU1P3fXavYC4 TYsaGErmOY8cT+Vw8ImB0L8BrlfFH3eP1IMV68VqR/7q7SHO3BlfAC6Eu2/bJQ== X-Gm-Gg: ASbGncsMDaT/pjJq0b6r86P0laAWTcv5mjiv9tw+LI71TzGbs8g4/csSbZvia9rVCQp UXaU9A1pP0mDa21qWxSubfWcaTmYUHawNafaNLIdERXWTdw5y2H5KF3rMQ50TQdJYJv1BwlgomM MV1vINi5qS6Ep33u2DByXv9TyA0RSyBZn5c00zEVaTeQ85iJpBrGoOTbptr5KZaxDMCmb8rs/st jOkmhGtTycuNrMQyPlP1ambOOjUZefo6Baelvrgwryab77zyV9fck04kh3DDa7R4fXDWLhXmZ4R Eh/RSwsJzwbgWJoxYc4uP9oK1Ggh44WPEu1LHg7Phs/TTs/cJ3k28CvON9AWAr6XCpl4hp9ZLtx OBnbvtko37pB4W4vUedVdSbrLJEUgE6LtkYhuu1vOJ1XOHWfclSQsTC4wzDF19IYXmwWGUdneEX xX2Ly2LgwV9xEhQCV2RebjlhQBHSRAOE/1Sj/zIqEGyGnZI7VfK8PEzm7xuWWhX/ifAyII X-Google-Smtp-Source: AGHT+IGi39cMsXVLgNz/6Yxcq+nZpzxgI0n9a9tbKEJ7LnYMYWiGQuRdlSZRCEKhcZ5+lSDsgwRAwA== X-Received: by 2002:a17:907:3f10:b0:af6:2f1d:a73f with SMTP id a640c23a62f3a-b01d976e782mr811674966b.53.1756735123604; Mon, 01 Sep 2025 06:58:43 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-afefcc1c6fdsm873503166b.75.2025.09.01.06.58.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 06:58:43 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 2/6] bitbake-setup: add tests to bitbake-selftest Date: Mon, 1 Sep 2025 15:58:32 +0200 Message-Id: <20250901135836.2927686-2-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250901135836.2927686-1-alex.kanavin@gmail.com> References: <20250901135836.2927686-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Sep 2025 13:58:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17914 From: Alexander Kanavin Run like this: alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-selftest -v bb.tests.setup test_setup (bb.tests.setup.BitbakeSetupTest.test_setup) ... ok ---------------------------------------------------------------------- Ran 1 test in 9.223s OK The test does a basic run-through of init, then status/update on an unchanged configuration, then status/update on a configuration changed via new commits to the test layer, then status/update on configuration changed via the top level json config file. Note that nothing whatsoever is fetched from the network; the test relies entirely on synthetic data contained inside itself, including minimal stubs for oe-setup-build and bitbake-config-build. This data is used to create temporary git repositories then clone them via local filesystem URIs. Later on this can be supplemented by an oe-selftest that tests bitbake-setup against real config files in the official configuration repository and real layers, templates and fragments. Signed-off-by: Alexander Kanavin --- bin/bitbake-selftest | 1 + lib/bb/tests/setup.py | 257 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 258 insertions(+) create mode 100644 lib/bb/tests/setup.py diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest index d45c2d406..fb7c57dd8 100755 --- a/bin/bitbake-selftest +++ b/bin/bitbake-selftest @@ -29,6 +29,7 @@ tests = ["bb.tests.codeparser", "bb.tests.fetch", "bb.tests.parse", "bb.tests.runqueue", + "bb.tests.setup", "bb.tests.siggen", "bb.tests.utils", "bb.tests.compression", diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py new file mode 100644 index 000000000..370a1badb --- /dev/null +++ b/lib/bb/tests/setup.py @@ -0,0 +1,257 @@ +# +# Copyright BitBake Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# + +from bb.tests.fetch import FetcherTest +import json + +class BitbakeSetupTest(FetcherTest): + def setUp(self): + super(BitbakeSetupTest, self).setUp() + + self.topdir = os.path.join(self.tempdir, "bitbake-builds") + + self.registrypath = os.path.join(self.tempdir, "bitbake-setup-configurations") + + os.makedirs(self.registrypath) + self.git_init(cwd=self.registrypath) + self.git('commit --allow-empty -m "Initial commit"', cwd=self.registrypath) + + self.testrepopath = os.path.join(self.tempdir, "test-repo") + os.makedirs(self.testrepopath) + self.git_init(cwd=self.testrepopath) + self.git('commit --allow-empty -m "Initial commit"', cwd=self.testrepopath) + + oeinitbuildenv = """BBPATH=$1 +export BBPATH +PATH={}:$PATH +""".format(os.path.join(self.testrepopath, 'scripts')) + self.add_file_to_testrepo('oe-init-build-env',oeinitbuildenv, script=True) + + oesetupbuild = """#!/usr/bin/env python3 +import getopt +import sys +import os +import shutil +opts, args = getopt.getopt(sys.argv[2:], "c:b:", ["no-shell"]) +for option, value in opts: + if option == '-c': + template = value + if option == '-b': + builddir = value +confdir = os.path.join(builddir, 'conf') +os.makedirs(confdir, exist_ok=True) +with open(os.path.join(confdir, 'conf-summary.txt'), 'w') as f: + f.write(template) +shutil.copy(os.path.join(os.path.dirname(__file__), 'test-repo/test-file'), confdir) +with open(os.path.join(builddir, 'init-build-env'), 'w') as f: + f.write("BBPATH={}\\nexport BBPATH\\nPATH={}:$PATH".format(builddir, os.path.join(os.path.dirname(__file__), 'test-repo/scripts'))) +""" + self.add_file_to_testrepo('scripts/oe-setup-build', oesetupbuild, script=True) + + bitbakeconfigbuild = """#!/usr/bin/env python3 +import os +import sys +confdir = os.path.join(os.environ['BBPATH'], 'conf') +fragment = sys.argv[2] +with open(os.path.join(confdir, fragment), 'w') as f: + f.write('') +""" + self.add_file_to_testrepo('scripts/bitbake-config-build', bitbakeconfigbuild, script=True) + + sometargetexecutable_template = """#!/usr/bin/env python3 +import os +print("This is {}") +print("BBPATH is {{}}".format(os.environ["BBPATH"])) +""" + for e_name in ("some-target-executable-1", "some-target-executable-2"): + sometargetexecutable = sometargetexecutable_template.format(e_name) + self.add_file_to_testrepo('scripts/{}'.format(e_name), sometargetexecutable, script=True) + + def runbbsetup(self, cmd): + bbsetup = os.path.abspath(os.path.dirname(__file__) + "/../../../bin/bitbake-setup") + return bb.process.run("{} {}".format(bbsetup, cmd)) + + def add_json_config_to_registry(self, name, rev): + config = """ +{ + "sources": { + "test-repo": { + "git-remote": { + "remotes": { + "origin": { + "uri": "file://%s" + } + }, + "rev": "%s" + }, + "path": "test-repo" + } + }, + "description": "Test configuration", + "bitbake-setup": { + "configurations": [ + { + "name": "gadget", + "description": "Gadget build configuration", + "oe-template": "test-configuration-gadget", + "oe-fragments": ["test-fragment-1"] + }, + { + "name": "gizmo", + "description": "Gizmo build configuration", + "oe-template": "test-configuration-gizmo", + "oe-fragments": ["test-fragment-2"] + }, + { + "name": "gadget-notemplate", + "description": "Gadget notemplate build configuration", + "bb-layers": ["layerA","layerB/meta-layer"], + "oe-fragments": ["test-fragment-1"] + }, + { + "name": "gizmo-notemplate", + "description": "Gizmo notemplate build configuration", + "bb-layers": ["layerC","layerD/meta-layer"], + "oe-fragments": ["test-fragment-2"] + } + ] + }, + "version": "1.0" +} +""" % (self.testrepopath, rev) + os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True) + with open(os.path.join(self.registrypath, name), 'w') as f: + f.write(config) + self.git('add {}'.format(name), cwd=self.registrypath) + self.git('commit -m "Adding {}"'.format(name), cwd=self.registrypath) + return json.loads(config) + + def add_file_to_testrepo(self, name, content, script=False): + fullname = os.path.join(self.testrepopath, name) + os.makedirs(os.path.join(self.testrepopath, os.path.dirname(name)), exist_ok=True) + with open(fullname, 'w') as f: + f.write(content) + if script: + import stat + st = os.stat(fullname) + os.chmod(fullname, st.st_mode | stat.S_IEXEC) + self.git('add {}'.format(name), cwd=self.testrepopath) + self.git('commit -m "Adding {}"'.format(name), cwd=self.testrepopath) + + def check_builddir_files(self, buildpath, test_file_content, json_config): + with open(os.path.join(buildpath, 'layers', 'test-repo', 'test-file')) as f: + self.assertEqual(f.read(), test_file_content) + bitbake_config = json_config["bitbake-config"] + bb_build_path = os.path.join(buildpath, 'build') + bb_conf_path = os.path.join(bb_build_path, 'conf') + self.assertTrue(os.path.exists(os.path.join(bb_build_path, 'init-build-env'))) + + if "oe-template" in bitbake_config: + with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f: + self.assertEqual(f.read(), bitbake_config["oe-template"]) + with open(os.path.join(bb_conf_path, 'test-file')) as f: + self.assertEqual(f.read(), test_file_content) + else: + with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f: + self.assertIn(bitbake_config["description"], f.read()) + with open(os.path.join(bb_conf_path, 'bblayers.conf')) as f: + bblayers = f.read() + for l in bitbake_config["bb-layers"]: + self.assertIn(os.path.join(buildpath, 'layers', l), bblayers) + + for f in bitbake_config["oe-fragments"]: + self.assertTrue(os.path.exists(os.path.join(bb_conf_path, f))) + + + def test_setup(self): + # check that no arguments works + self.runbbsetup("") + + # check that --help works + self.runbbsetup("--help") + + # check that writing settings works and then adjust them to point to + # test registry repo + out = self.runbbsetup("install-settings --top-dir {}".format(self.topdir)) + settings_path = "{}/bitbake-setup.conf".format(self.topdir) + self.assertIn(settings_path, out[0]) + out = self.runbbsetup("change-setting --top-dir {} default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.topdir, self.registrypath)) + self.assertIn("Setting 'registry' in section 'default' is changed to", out[0]) + self.assertIn("New settings written to".format(settings_path), out[0]) + + # check that 'list' produces correct output with no configs, one config and two configs + out = self.runbbsetup("list --top-dir {}".format(self.topdir)) + self.assertNotIn("test-config-1", out[0]) + self.assertNotIn("test-config-2", out[0]) + + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', 'master') + out = self.runbbsetup("list --top-dir {}".format(self.topdir)) + self.assertIn("test-config-1", out[0]) + self.assertNotIn("test-config-2", out[0]) + + json_2 = self.add_json_config_to_registry('config-2/test-config-2.conf.json', 'master') + out = self.runbbsetup("list --top-dir {} --write-json={}".format(self.topdir, os.path.join(self.tempdir, "test-configs.json"))) + self.assertIn("test-config-1", out[0]) + self.assertIn("test-config-2", out[0]) + with open(os.path.join(self.tempdir, "test-configs.json")) as f: + json_configs = json.load(f) + self.assertIn("test-config-1", json_configs) + self.assertIn("test-config-2", json_configs) + + # check that init/status/update work + # (the latter two should do nothing and say that config hasn't changed) + test_file_content = 'initial\n' + self.add_file_to_testrepo('test-file', test_file_content) + for c in ('gadget','gizmo','gadget-notemplate','gizmo-notemplate'): + out = self.runbbsetup("init --non-interactive test-config-1 {} --top-dir {}".format(c, self.topdir)) + buildpath = os.path.join(self.topdir, 'test-config-1-{}'.format(c)) + with open(os.path.join(buildpath, 'config', "config-upstream.json")) as f: + config_upstream = json.load(f) + self.check_builddir_files(buildpath, test_file_content, config_upstream) + os.environ['BBPATH'] = os.path.join(buildpath, 'build') + out = self.runbbsetup("status") + self.assertIn("Configuration in {} has not changed".format(buildpath), out[0]) + out = self.runbbsetup("update") + self.assertIn("Configuration in {} has not changed".format(buildpath), out[0]) + + # change a file in the test layer repo, make a new commit and + # test that status/update correctly report the change and update the config + prev_test_file_content = test_file_content + test_file_content = 'modified\n' + self.add_file_to_testrepo('test-file', test_file_content) + for c in ('gadget','gizmo'): + buildpath = os.path.join(self.topdir, 'test-config-1-{}'.format(c)) + os.environ['BBPATH'] = os.path.join(buildpath, 'build') + out = self.runbbsetup("status") + self.assertIn("Layer repository file://{} checked out into {}/layers/test-repo updated revision master from".format(self.testrepopath, buildpath), out[0]) + out = self.runbbsetup("update") + self.assertIn("Existing bitbake configuration directory renamed to {}/build/conf-backup.".format(buildpath), out[0]) + self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0]) + with open(os.path.join(buildpath, 'config', "config-upstream.json")) as f: + config_upstream = json.load(f) + self.check_builddir_files(buildpath, test_file_content, config_upstream) + + # make a new branch in the test layer repo, change a file on that branch, + # make a new commit, update the top level json config to refer to that branch, + # and test that status/update correctly report the change and update the config + prev_test_file_content = test_file_content + test_file_content = 'modified-in-branch\n' + branch = "another-branch" + self.git('checkout -b {}'.format(branch), cwd=self.testrepopath) + self.add_file_to_testrepo('test-file', test_file_content) + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch) + for c in ('gadget','gizmo'): + buildpath = os.path.join(self.topdir, 'test-config-1-{}'.format(c)) + os.environ['BBPATH'] = os.path.join(buildpath, 'build') + out = self.runbbsetup("status") + self.assertIn("Configuration in {} has changed:".format(buildpath), out[0]) + self.assertIn('- "rev": "master"\n+ "rev": "another-branch"', out[0]) + out = self.runbbsetup("update") + self.assertIn("Existing bitbake configuration directory renamed to {}/build/conf-backup.".format(buildpath), out[0]) + self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0]) + with open(os.path.join(buildpath, 'config', "config-upstream.json")) as f: + config_upstream = json.load(f) + self.check_builddir_files(buildpath, test_file_content, config_upstream) From patchwork Mon Sep 1 13:58:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 69343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 926BACA100B for ; Mon, 1 Sep 2025 13:58:50 +0000 (UTC) Received: from mail-ej1-f47.google.com (mail-ej1-f47.google.com [209.85.218.47]) by mx.groups.io with SMTP id smtpd.web10.52454.1756735126448323100 for ; Mon, 01 Sep 2025 06:58:46 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=MtwyyMbt; spf=pass (domain: gmail.com, ip: 209.85.218.47, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f47.google.com with SMTP id a640c23a62f3a-affc2eb83c5so358152666b.2 for ; Mon, 01 Sep 2025 06:58:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756735125; x=1757339925; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s+ZnrXJF6+k7F5TENY/yTbtIRHQyZzBFbxoX/AcwhC0=; b=MtwyyMbteC6Mbgszm07Pu7ITbEX3Yg/vuAcewyVJH4BE7ww689hkbwhv8KRrBA8N4a 6UXRZRhGxtotunoqoDmJR/BRgz2qpC1YSIoah+wKBVGtjfaYOBP2Z59HWMJhl3QyUpWz ddp8OAt/EFuvPzJTkMOSk1GYnEpEROFzTwn8gkbR59MTOLKVLkvjEY1HY0FRNATpGfe7 lSlwmAYD2hG73wGpY6VLJrzss/Z7AdIShRNLh772zYvqB6pheC/NQ8sgF9w3C3tjH1J2 d1lthMhYZHEwCTsHmjESa3bkCJVQz25rXfuV7V7ao2vrEqBBOmMHSLzFCua4thF3qz4/ YEyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756735125; x=1757339925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s+ZnrXJF6+k7F5TENY/yTbtIRHQyZzBFbxoX/AcwhC0=; b=VaJ5u6OQ/YehvZPpv+kSwNjMzFXIzzzSE0pEI2yoRNTcVAmjlfETVB8McUlWOllQiP TiHhX13zu++Vpd16KjPiGVfe0OkfWOawnCJan4QzwkszBdnJwDJDQA9lbywO8CyYi8fe 5N1SluQYgBv0GE/GaUvu6E44FJL+dA+l8CAZaSwlNCMDSyhw/4HTK5CvNWLZoUSrQ1L6 pddBVz45csYok2QOR2NJIaqxlkpNgEQG3PO0WctuvOHElEMMOnrAYARz9X+HNkkNZWgv iPe+9O5HeAFkLnRBF3Un90vWwfkEli3JGI2c3j/W7BlUyAy5VKc7nprg5imDRO2MsJq2 vCmQ== X-Gm-Message-State: AOJu0Yy9TCtTQPd7ITEJ4vG7q3OA1dlDrNGNH7ulqdNti6tfB9yII3yR lEMe/5JkQx9FfFQaG2xeZqaG9zge6g9zBmLii5+YgDmIBH61UrnXe5Pdk3qZGg== X-Gm-Gg: ASbGnctK1QrYg5TpyZTYu0hKM0p3phgZqxbkjk4uzsvsh29PkWmT6JKzkJRs4vt5ydG xOP8tNHZmDock2aVGXAmuzhC2GCokTDn0wpq7ybdE3bQwvKOh5qI4q8sEdjylsHFyoAHh2blMAC /BXBmTsiSw8IaWVvMLjHMBDHmUoswWazViTXeGKS88xi/mvTT0mVv01ypo4hwsB9ajZKVNNgUKU 8ZQA4vObMVvYuPW1OJic/gl35tqrcnqT5qk4SPeGZ6jmC3c/IwFP6NpTvoOBCZd7r8YgY6/33nI Qi/+DXLOc39p1Fdrz0brQupJEUKMZ8L/PKOWXbxqd3ThrHN0rnBgU4uBbXgJEAvLr0tlzyIZUN8 4NCBloRRSXupa+4gZwmw9/802giYOxNITIu3Nb+rWJqtuNWhgYz74FYWeks+9/AuZTxT/6Shoyx Ysu0ev/HSO88t96jel2YlzHDhamOxSTFnnBGF9QUKDVbV6h9zY1tKqvCDeSm7zUR1UfG4p X-Google-Smtp-Source: AGHT+IEhpbgAa9peT2XbERfP4TqdlwXyyLKJ+Fjwn6CTaauruyyKFT4p4CXqF2DdVmTfel2Uz1dHuQ== X-Received: by 2002:a17:907:724c:b0:afe:c1bd:b6d6 with SMTP id a640c23a62f3a-b01d8a327damr672820566b.5.1756735124571; Mon, 01 Sep 2025 06:58:44 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-afefcc1c6fdsm873503166b.75.2025.09.01.06.58.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 06:58:44 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 3/6] bitbake-setup: add 'install-buildtools' command Date: Mon, 1 Sep 2025 15:58:33 +0200 Message-Id: <20250901135836.2927686-3-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250901135836.2927686-1-alex.kanavin@gmail.com> References: <20250901135836.2927686-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Sep 2025 13:58:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17915 From: Alexander Kanavin This basically calls install-buildtools from oe-core/poky, but it ensures via command line parameters that the installation location is stable and the downloads are preserved for reproducibility: $ bin/bitbake-setup install-buildtools Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf ====== Buildtools archive is downloaded into /home/alex/bitbake-builds/yocto-master-testing/buildtools-downloads/20250319141333 and its content installed into /home/alex/bitbake-builds/yocto-master-testing/buildtools ... (output from install-buildtools script) ====== It also detects when buildtools are already installed, and will direct users what to do: ====== alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup install-buildtools Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf Buildtools are already installed in /home/alex/bitbake-builds/yocto-master-testing/buildtools. If you wish to use them, you need to source the the environment setup script e.g. $ . /home/alex/bitbake-builds/yocto-master-testing/buildtools/environment-setup-x86_64-pokysdk-linux You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation ====== Signed-off-by: Alexander Kanavin --- bin/bitbake-setup | 26 ++++++++++++++++++++++++++ lib/bb/tests/setup.py | 20 ++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index a05433206..596c9885a 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -16,6 +16,8 @@ import stat import tempfile import configparser import datetime +import glob +import subprocess default_registry = 'git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main' @@ -530,6 +532,25 @@ def list_configs(settings, args, d): json.dump(json_data, f, sort_keys=True, indent=4) print("Available configurations written into {}".format(args.write_json)) +def install_buildtools(settings, args, d): + buildtools_install_dir = os.path.join(args.build_dir, 'buildtools') + if os.path.exists(buildtools_install_dir): + if not args.force: + print("Buildtools are already installed in {}.".format(buildtools_install_dir)) + env_scripts = glob.glob(os.path.join(buildtools_install_dir, 'environment-setup-*')) + if env_scripts: + print("If you wish to use them, you need to source the the environment setup script e.g.") + for s in env_scripts: + print("$ . {}".format(s)) + print("You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation.") + return + shutil.rmtree(buildtools_install_dir) + + install_buildtools = os.path.join(args.build_dir, 'layers/oe-scripts/install-buildtools') + buildtools_download_dir = os.path.join(args.build_dir, 'buildtools-downloads/{}'.format(time.strftime("%Y%m%d%H%M%S"))) + print("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir)) + subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True) + def default_settings_path(top_dir): return os.path.join(top_dir, 'bitbake-setup.conf') @@ -656,6 +677,11 @@ def main(): add_build_dir_arg(parser_update) parser_update.set_defaults(func=build_update) + parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine') + add_build_dir_arg(parser_install_buildtools) + parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.') + parser_install_buildtools.set_defaults(func=install_buildtools) + parser_install_settings = subparsers.add_parser('install-settings', help='Write a settings file with default values into the top level directory (contains the location of build configuration registry, downloads directory and other global settings)') add_top_dir_arg(parser_install_settings) parser_install_settings.set_defaults(func=write_settings) diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index 370a1badb..f360efaca 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -51,6 +51,21 @@ with open(os.path.join(builddir, 'init-build-env'), 'w') as f: """ self.add_file_to_testrepo('scripts/oe-setup-build', oesetupbuild, script=True) + installbuildtools = """#!/usr/bin/env python3 +import getopt +import sys +import os + +opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="]) +for option, value in opts: + if option == '-d': + installdir = value + +print("Buildtools installed into {}".format(installdir)) +os.makedirs(installdir) +""" + self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True) + bitbakeconfigbuild = """#!/usr/bin/env python3 import os import sys @@ -217,6 +232,11 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) out = self.runbbsetup("update") self.assertIn("Configuration in {} has not changed".format(buildpath), out[0]) + # install buildtools + out = self.runbbsetup("install-buildtools") + self.assertIn("Buildtools installed into", out[0]) + self.assertTrue(os.path.exists(os.path.join(buildpath, 'buildtools'))) + # change a file in the test layer repo, make a new commit and # test that status/update correctly report the change and update the config prev_test_file_content = test_file_content From patchwork Mon Sep 1 13:58:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 69341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 909B6CA1009 for ; Mon, 1 Sep 2025 13:58:50 +0000 (UTC) Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) by mx.groups.io with SMTP id smtpd.web11.52234.1756735126996156748 for ; Mon, 01 Sep 2025 06:58:47 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=ah0iZRes; spf=pass (domain: gmail.com, ip: 209.85.218.42, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-b043da5a55fso74596966b.0 for ; Mon, 01 Sep 2025 06:58:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756735125; x=1757339925; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x2yuXcYicNsLfmVmTvV6iIFzMrQnvqDQBWylQv3X+Z4=; b=ah0iZResWUAv1sMe2PC3T4wW4kXXEirJJMMtd+kGX7f5yxjWadTOxDp6Q6/rv1wvLa st0JwOM0x301GgZ5QpsV5hoUc03hUtMyvh9cl9vxsiMUcxO8Am5+SFeKI4YuczQrtLPk 1KLbKiYozybJNljCT5Fwh0PsihOiIk6zjyq7i4ao49a24afv4Wqu1d8XfOqdjPf4afIC TLLwS8kgk8Gt9uNG8Ca5E8AX6XjxmtVuvhe+9/d9omFvcmhTVJunaIT/gud7GYB1fJNQ 8C8lLPFvD/plznuaxzxyrNAjEx3N2CVDNV6YTl8yeUxT0KvNbKRch0wPpXhN49kXp0sD cYrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756735125; x=1757339925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x2yuXcYicNsLfmVmTvV6iIFzMrQnvqDQBWylQv3X+Z4=; b=GLAZR+cgYSgZLZIkfbsI2mwbidZ/RuQM5im6FzqjjphzbXtRtgGzguxaECui2NnGQL dRpeDVY1Y4EFnB0an5BpoSaNYzWgiVeWM4EB+qPrhActTGixIfnKaXyxqUnquwrTQcAi 3htJCci6WKYOfj2anPP6J0EdFi27VAUtR8iIrsQKYEzJTvPEZDjXSMKnrbSEO4QD7XP1 NY2yAVDlLWOpYUOKe+Gs7I6G7CQu5aKAy/sSunghpyLWEHHuASp1aM251gNECc99HXfZ pFiIncwsobTO1LDC8dnFUYTV4cEp9iwgtzDdROOoJqtb1IbeBFVdXxqRaW0YCFANCgP/ qb0Q== X-Gm-Message-State: AOJu0YwGdtK7t02XsBoWz957U40iwNMDeI9iqjdHh86uXdKEueCKIn/9 GfvgNMkoS06a9ODRD15FGU4h4KA6wjVb/HbwRAcB8iCPwSIa1gl2GaAKrRUGFA== X-Gm-Gg: ASbGncuaDwQVKPD/H7XpwvT6g2bktxc60LD1CJY4sGltuEgZFiArXuGUG5WYyLxZ7Hp hn11M4VrMuLLhHwabgSew7Fj4uYvtosfPIE5KzV6U0V4CSOWSCmqNLBsr49BAlqqw/NOP0nRHWV a0TsPKNzLpFX4UNl1TXCwo2s5TOvgiwbgHocgNwxVnD9QetePPJ7L/9gesRUgLF1asuCT58xFlU WS5BtZGd7RXyqAYbdaQSL7ToSVWUVz5L7mdLD24T4DVUbfgRVsjdfbWO1J84gIqdI9pN86NOH1K jtygFi7SRLtUuIhssVXJKIrjkwtJ2MOb/Ry+HL3HuhMDNMzTnB8+QNd6DUFcldQCUeh7Q3/DqJP pYyFFXUV+dgITSYEIsrbP3eNeS0pJlTVVWjuksHa01izvBuAzNJaA9Cl+duDvA1LPclhd+9j34B 3hBabcc6qlGgDFnLQ0LW8xQK2LkuH8Yk+Fs5sc5NNMxDb8faPzB+5K8EC5NRTx+O8NfwT7 X-Google-Smtp-Source: AGHT+IEmPzvNdbCjf8gfkZFwcLcw2CeYEE/LU7N4FqIy6lBKMdU4MgPAj2fMNXwOvacdcKPkgtq/xw== X-Received: by 2002:a17:907:1b10:b0:afe:c099:aeb3 with SMTP id a640c23a62f3a-b01d8c90624mr897008366b.25.1756735125304; Mon, 01 Sep 2025 06:58:45 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-afefcc1c6fdsm873503166b.75.2025.09.01.06.58.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 06:58:44 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Gyorgy Sarvari Subject: [PATCH 4/6] bitbake-setup: fix typo Date: Mon, 1 Sep 2025 15:58:34 +0200 Message-Id: <20250901135836.2927686-4-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250901135836.2927686-1-alex.kanavin@gmail.com> References: <20250901135836.2927686-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Sep 2025 13:58:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17916 From: Gyorgy Sarvari Signed-off-by: Gyorgy Sarvari --- bin/bitbake-setup | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index 596c9885a..9c2922b59 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -539,7 +539,7 @@ def install_buildtools(settings, args, d): print("Buildtools are already installed in {}.".format(buildtools_install_dir)) env_scripts = glob.glob(os.path.join(buildtools_install_dir, 'environment-setup-*')) if env_scripts: - print("If you wish to use them, you need to source the the environment setup script e.g.") + print("If you wish to use them, you need to source the environment setup script e.g.") for s in env_scripts: print("$ . {}".format(s)) print("You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation.") From patchwork Mon Sep 1 13:58:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 69340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71857CA1000 for ; Mon, 1 Sep 2025 13:58:50 +0000 (UTC) Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by mx.groups.io with SMTP id smtpd.web11.52235.1756735127802716406 for ; Mon, 01 Sep 2025 06:58:48 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=F0uXLb9h; spf=pass (domain: gmail.com, ip: 209.85.218.50, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-b042cc3953cso119943966b.2 for ; Mon, 01 Sep 2025 06:58:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756735126; x=1757339926; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1VsXliYwYcT3sgAjyKE0heVKtRXmDppGmlM6vylsstw=; b=F0uXLb9hJhcqCROTTslulpXsROh2p/xLrSyWe6F/FWaQ+tDa5NnWHoh4Rd9w2oyvNt MT23mIrgRT/2vUGnmVjbeOy5oNZ8CLptTJZptu/546SJNhNMKvRpTpdRowF3McEOkEE/ gzCEV8kPCS9LfQcW7HsIgaZG+BLIbc9FYMZqty/o91gnsp4GLAgG/tnJxaYUBPp75Iz9 h8tnU+jCxKpUZ5bR2TXJQhR7GR1resyETrUSAQbZ+N/bVBesS2pXe2Arp/iF24GkWR3s e0o+9b96GFcLjcQhTSsGBGAUr95Oj+0/q/Bb/PVDAksT3QQXEinyVnKh4zZud8vsHSVk zEqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756735126; x=1757339926; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1VsXliYwYcT3sgAjyKE0heVKtRXmDppGmlM6vylsstw=; b=MMwMtM2jB6Vd4s8g5pjcsTxpD6Qczf9FqQZ2IawO1Ozirjv+fImkkDnUK3asVGumue JBnSL4041fHRkTIYl/8HQWm+l+LyOKK1vlBcCFEgN6VX7eL6HSD+bs0j4mqwinwejwvq dwXf/l43IHI2oR30PepCigYJg9ILjc2i29PAhXjqYVWVpN0Yaek1aHg8zN9iW05HPnwh HMN/H/ZKvZdU2EzJOf4MeObmZ8QTSRyCf2N8LbYRXNmfuBU3jY6qPXpjQOWcu09JoLui pG1xBMwc2K9nJfQH81b+qTZdTBeydgGx5kHm8KOaAZFLZJGOPozNPvq86WzqeUkB6Cae 6ZmQ== X-Gm-Message-State: AOJu0YxQEooMu2CRZAUCvLvWds+1VCiVMRtUm5HDamoQOqw/ZSuM/OeS F/FBhhURpha7PJZL4ObiwI4GWYm5KOnbgQHIfi6QtrSU80mHxATVjWnufiUf0A== X-Gm-Gg: ASbGnct7HW0bSGGPgFvla5yRt6AlWs6iTIjaDRrZ0OUospX05ICcfovzazDgg7XmlRX qXfTv5dKqlbY8ex7hj1Y7uhAs34m48z8u6vIEj8W2J+3K5OpsNobphaXiOMhU0fszUDkSyBgy4Y skOBPqfCsAvPjnt6V3zZK1NEONq70vAbMgqzH/jZKt/c6T+9SQWeIMFiUnqgWWTijg5fVhknEO2 +qGlLmgV0TteoH8ZDQiWqLkjRqIRXFXmG5adsStmgH+G48ktrXGOvLopJHbkefwR/cVxhZ+b3Zt 73AGTVKXLFRuqwuW3R+j1dGuUfRv6R90WjvoYwSKtZubkmuIuPIYRZbG8DOgLVvTWnXNd8jUcIB gcfCadlvZ0KZTfs/xb+TPaM0/SKxCuPackcVre0InZzAtZZzkGu9PBKsktieRWxN84NQJYrKnl6 atzt7yRN29Cvy3NwTPT1FM641+YfczPTyLESROS9fYw9SO7DIl1KQHrtw6CGPGprtrujHv X-Google-Smtp-Source: AGHT+IHEJkbP/nAs2VP1jn74NBYP03nfT2BKtTqqQrktxH5/YP2o7RWVUwvrFxEipKOpWiz8i4cncw== X-Received: by 2002:a17:907:3f12:b0:afe:7027:56f9 with SMTP id a640c23a62f3a-b01d8a708d7mr852544566b.17.1756735126145; Mon, 01 Sep 2025 06:58:46 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-afefcc1c6fdsm873503166b.75.2025.09.01.06.58.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 06:58:45 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Gyorgy Sarvari Subject: [PATCH 5/6] bitbake-setup: fix non-interactive init instructions Date: Mon, 1 Sep 2025 15:58:35 +0200 Message-Id: <20250901135836.2927686-5-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250901135836.2927686-1-alex.kanavin@gmail.com> References: <20250901135836.2927686-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Sep 2025 13:58:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17917 From: Gyorgy Sarvari The --non-interactive option is position-sensitive, it needs to stand after the "init" command, otherwise argparser complains: bitbake-setup: error: unrecognized arguments: --non-interactive Signed-off-by: Gyorgy Sarvari --- bin/bitbake-setup | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index 9c2922b59..7382dc8d2 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -368,7 +368,7 @@ def init_config(settings, args, d): source_overrides = json.load(open(args.source_overrides)) if args.source_overrides else {'sources':{}} upstream_config = obtain_config(settings, args, source_overrides, d) - print("\nRun 'bitbake-setup --non-interactive init {}' to select this configuration non-interactively.\n".format(upstream_config['non-interactive-cmdline-options'])) + print("\nRun 'bitbake-setup init --non-interactive {}' to select this configuration non-interactively.\n".format(upstream_config['non-interactive-cmdline-options'])) builddir = os.path.join(os.path.abspath(args.top_dir), upstream_config['non-interactive-cmdline-options'].replace(" ","-").replace("/","_")) if os.path.exists(builddir): From patchwork Mon Sep 1 13:58:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 69339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74004CA0FF0 for ; Mon, 1 Sep 2025 13:58:50 +0000 (UTC) Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by mx.groups.io with SMTP id smtpd.web11.52238.1756735128625777418 for ; Mon, 01 Sep 2025 06:58:48 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=nd8i70if; spf=pass (domain: gmail.com, ip: 209.85.218.49, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-b02c719a117so154709466b.1 for ; Mon, 01 Sep 2025 06:58:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756735127; x=1757339927; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qxCq429xabFqBnHMzNrzLK2ruCvT4eXGuxMXF+pYFxw=; b=nd8i70ifOTLiPgP/WOGajzgNmLN38j9xG3JYLPNBLhflb43A4Twf7hAaGMDIpVg0Tl sEiAWolUq9UFDipagArWoRVEoXACCyf4U9C7oCxYZU5I8GKxjpsAfMMHez4iw/accIsT Gf2MYjMLXCi+x0FNgcRzTkP/9qhjaj3dNNVotqB6+PMd2zlTdVxj2XBq85BK5bX9eBqt 4Yq3vXxOU/1910roLmUFq7b1eE6RL9YHy9RFzvie8y3O+hsh1S1uwgcB+ysGjsV1u1ZR wEd/QyarW4JMqO03fgctFxN0UOM4BqJiY6ODUON0Uz9A7qRe/mYPafx5BNYLGis+lmKv xjMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756735127; x=1757339927; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qxCq429xabFqBnHMzNrzLK2ruCvT4eXGuxMXF+pYFxw=; b=tOkMTtXBnSUDa3ASSe7E3TtZbfLwp82lM2hb6WtJ3o0BHWfwExLdBvramTkC+IbU0O K+VXd3ynslwIV0j6hj0Mz1p4XN0Tra4uqpYmgWeD0pODiM5Wzz/9dL9O4XlJRDMH+kEY BaU0posPYOjWPmmrq/oXNnm9Se4X3jOcmU0a30OxJueJHrpaRx83MBNhnypxVjolKmGk dWAGlSxiIwe/Xu32hrI0dfwUa3xYnKXrqxRJ1Uq0mo+QvKFTGBrJYh9dVmstQ0lYuvwp 9W9Uqn73yEJjpnDn2BSzN29vvhYGXbf/J4CCezb412F0+YFoiZNbXKqnvSplUQP6DdLS oxtw== X-Gm-Message-State: AOJu0YwNPBW1Om8G2903qm4q2Gq8TVv9Egl9hG2EMgz3VOkID859OugR S/tx2MgpBZWk6yCsm4KrJJIeA/HSH7Wrm9Ux/VjVlFg5EJcJGogBTKGxJbbRvQ== X-Gm-Gg: ASbGncv3/QmdzR+7EHttbfsEc4d8VmiXRRQECe6Qc5/zWXipx7KfyLaAQLzFgDIxBgd OZkrokjL+vq7LIh52MiG2e/BB+02hmTcp7tZobBvPHJDqWYdLBj1XyeTDTGnyhu1LHadOt/7vAs EM2bMP61gh0Q9olNKURZRNTj1Omko3P2z1KnYzw8TCAUDVakNJLvuNE69ZvfSb6kUj/IChZiLPl j+tq7GsaTBdnVqF/3CXIv3k12/BPyhBGjW0B6SqCK3UDlFA/J7hSbxQSkFPJ/9EfETc/HlWX4Tm Iiix7f4Lk9tjS7y652TagWTDEKT9uUai5OFZ2jott+zXzLLqt8M/pHqcHgyRu2AdyhCDlZPmOpt bTZ7zkMn3ybQlqN8Iq3YVP3XpB+BiCb8H+z6yHarMot9RWiy9tUvX3R2MH5Idg8YFGpZEZhcKsr +Llx35A8Y2IxZhjMAanU9cor3pgMbhbZgBAiHUo54mQGwIXdzreCXDxcHk3g== X-Google-Smtp-Source: AGHT+IHOdVl4g5zw38pMj3H4nvzDaaLY+w8HDQc0AO4Cdn0IDKhdzLezUX3qrNWFm6gsPO3g4lzpAg== X-Received: by 2002:a17:907:1c1e:b0:b04:26d8:97b7 with SMTP id a640c23a62f3a-b0426d89b05mr419712066b.17.1756735126852; Mon, 01 Sep 2025 06:58:46 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-afefcc1c6fdsm873503166b.75.2025.09.01.06.58.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 06:58:46 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Gyorgy Sarvari Subject: [PATCH 6/6] bitbake-setup: make sure git user is always configured Date: Mon, 1 Sep 2025 15:58:36 +0200 Message-Id: <20250901135836.2927686-6-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250901135836.2927686-1-alex.kanavin@gmail.com> References: <20250901135836.2927686-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Sep 2025 13:58:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17918 From: Gyorgy Sarvari In case a git username and email are not configured, then git errors out when commiting the initial and later the changed configuration of the setup. This can happen if bitbake-setup is running in a container, or for example if the user just doesn't use git for committing. To avoid this failure, configure a dummy user to be used with comitting the config changes. Signed-off-by: Gyorgy Sarvari --- bin/bitbake-setup | 3 +++ 1 file changed, 3 insertions(+) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index 7382dc8d2..67188891f 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -392,6 +392,9 @@ def init_config(settings, args, d): os.makedirs(layerdir) bb.process.run("git -C {} init -b main".format(confdir)) + # Make sure commiting doesn't fail if no default git user is configured on the machine + bb.process.run("git -C {} config user.name bitbake-setup".format(confdir)) + bb.process.run("git -C {} config user.email bitbake-setup@not.set".format(confdir)) bb.process.run("git -C {} commit --allow-empty -m 'Initial commit'".format(confdir)) bb.event.register("bb.build.TaskProgress", handle_task_progress, data=d)