From patchwork Mon Sep 29 12:56:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BCC3CAC5B0 for ; Mon, 29 Sep 2025 12:56:35 +0000 (UTC) Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) by mx.groups.io with SMTP id smtpd.web10.53018.1759150594104910759 for ; Mon, 29 Sep 2025 05:56:34 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Ud7Jg4ju; spf=pass (domain: gmail.com, ip: 209.85.208.42, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-61cc281171cso9689374a12.0 for ; Mon, 29 Sep 2025 05:56:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150592; x=1759755392; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=bdUwhVR/kSshkypPxgtDZGi20hHR5k/335MWcRe3r0s=; b=Ud7Jg4juBwILtiVfE50DRf9t/ylof0HDeniJcmQW7Lr7a6+8mTSj8zb3aWaD4OHZNx Kkgek+gOW2HBoiqibnnzfWyiW2VnI2fYbaVvHjn1ucblDMY+3v1tDvTnYQXQI48EseHx iJ76tZPXC4cy/NFoPjHE7QNDXk03mg21up7x0US3QirRF9qW7sgkYi2Az5fOt7TwAWub 6pj1Rhgxl70sr5PNqYoYFIgL5i7XDeCGdgcD4mMyMyQ2M4u9khE0qAj1/poCVCzdESry el2RSGkkWimQegxhzwCschqYZUqCUUK8x7MQ/J2Vc51dlCt9hUzXZ5M+qJUh0AidOii6 /q8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150592; x=1759755392; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=bdUwhVR/kSshkypPxgtDZGi20hHR5k/335MWcRe3r0s=; b=CY8XTyCcRanvHJrkzXu7bJ7WA53s8pw0V0Pag/mhW6Al4wUxlO251zpeG1hRuu/Xqi 7q1mf/88UkUDTzXxgWE8We7I2Cehijih2z+ReJ6sqBtey5b+Obaw2cyQ7USLR/91+gN5 0hUmcpmn0dXE6LnJUiLC2Qxk5/+91T12Io/hF8J3Q2ojyzkUlPZ53D/KG1+RM7tKExF0 VSyFwiPapo6z5gYRKgxx3jjjan/CcqTK9CBH1YDMxrPjLGpsHLO5GR9+8iVOSBdHKLuy JOEKtyq3ye6sJHiRo8g4kgIORKHZPJI+6tgT96L/vwS7pbZ3J9nZdc6RCtqHqGporS1Q zOGQ== X-Gm-Message-State: AOJu0Yz2U7pSx7LIdyqAPcVjR+UzSL6lV0L6QGxXX2XeRWhdTT4GZGHZ 8vsVP/qUA9haCUJEdqv0oIBv3Po109dQ15Y8wwOi1+GcDYUYcZ1Jb+J8PXfZag== X-Gm-Gg: ASbGnctZjgzd6ZO8XvKP9h4Tn2/nzVREwwbqPx9Br+ksI/UCXh/ZuHaNDxdHVEUwWvG NFhnK2kGxJfb+KqiGZhK88DocK0pAu8Pzt12RTBjuGtesn5jlzV8uzgvA7HFoWkBcf+gBUAG5ef TEZ6b6tM8gkdILl+51hFZJXhGqSYsF7jcandLeOpLW29h0+WM612n777KD4hqJRzPgeR3W32Q06 NA62t+xgjjKe25qQbkbhIK0bfZduxH1QAqoVb2evy+Z+K7+D/Rl1VTij/bmTeb86klsxCMMES6p iOuDKBe3XWUJdAjierJL/W/ggDKTP9nrXQTdf464LCtCg8QGCZfduTTTL4WRBeMplGlfTCf/tA+ LfI0fcS3kkA0HMAd17S/C/+Qg6ea0upLxS6drlUFtUjLiUq0RDrkVPJcChHBNuUfn3vP3A11Elu oRhjW4pBR1q9EsJQRepZmLI6rnB3ynXx1FAdHaaLiR0XFA3cm3Xy/yBK3eVc/QqvjoHeVP X-Google-Smtp-Source: AGHT+IGYzSKjVOqboSn4woGm2ARBCsnDs2aDdV3ojkGs9AZX+5rl6P54rLAZ1I3pnqNN22HXyLVuZQ== X-Received: by 2002:a05:6402:88a:b0:634:66c8:9e6d with SMTP id 4fb4d7f45d1cf-6349fa9e87emr12272841a12.35.1759150591881; Mon, 29 Sep 2025 05:56:31 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:31 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin , Richard Purdie Subject: [PATCH 01/10] bitbake-setup: add the initial implementation Date: Mon, 29 Sep 2025 14:56:07 +0200 Message-Id: <20250929125616.1751116-1-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:35 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18100 From: Alexander Kanavin Preamble ======== The latest iteration of this patchset is available at https://github.com/kanavin/bitbake I recommend taking the patches from there to ensure that you are not trying out outdated code. For the rationale and design guidelines please see this message: https://lists.openembedded.org/g/openembedded-architecture/message/1913 Left out for now but will be done later: - official configuration repository - documentation Amble *scratch* HOWTO ===================== 1. If you don't know where to start, run 'bitbake-setup init'. Bitbake-setup will ask a few questions about available configuration choices and set up a build. Note: 'init' sub-command can also take a path or a URL with a configuration file directly. You can see how those files look like here: https://github.com/kanavin/bitbake-setup-configurations 2. You can then source the bitbake environment and run bitbake to perform builds as usual: $ . /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env Also, subsequent status/update commands will not require a separate --build-dir argument telling bitbake-setup where the build is. 3. To check if the build configuration needs to be updated, run: === $ bin/bitbake-setup status ... Configuration in /home/alex/bitbake-builds/poky-alex/ has not changed. === If the configuration has changed, you will see the difference as a diff. ... - "rev": "akanavin/sstate-for-all" + "rev": "akanavin/bitbake-setup-testing" ... If the configuration has not changed, but layer revisions referred to it have (for example if the configuration specifies a tip of a branch), you will see that too: === ... Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to aeb73e29379fe6007a8adc8d94c1ac18a93e68de === 4. If the configuration has changed, you can bring it in sync with: $ bin/bitbake-setup update Note that it will also rename/preserve the existing build/conf directory, and print changes in bitbake configuration (diff of content of build/conf/) if that has changed. I can't at the moment think of anything more clever that is also not much more brittle or complex to implement, but open to suggestions. Terminology =========== - 'top directory' means the place under which bitbake-setup reads and writes everything. bitbake-setup makes a promise to not touch anything outside of that, unless otherwise directed to by entries in settings (currently there is one such setting for fetcher downloads for layers and config registries). Top directory can be selected by an environment variable, a command line option, or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment (e.g. we are in a bitbake environment), then the top directory is deduced from that and doesn't need to be specified by hand. - 'settings' means bitbake-setup operational parameters that are global to all builds under a top directory. E.g. the location of configuration registry, or where the bitbake fetcher should place the downloads (DL_DIR setting). Settings are stored in a .conf file in ini format just under the top directory. - 'build' means a tree structure set up by 'bitbake-setup init', consisting of, at least, a layers checkout, and a bitbake build. It maps 1:1 to the json data it was constructed from, which is called 'build configuration'. Build configurations are constructed from generic configurations that may involve making one or more choices about available options in them. Generic configurations are files, URLs or are obtained from git repositories called 'config registries', in which case they can be listed with 'bitbake-setup list'. There can be multiple 'builds' under a top directory. Here are two example generic configurations that showcase this: https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-options.conf.json https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-nested-configs.conf.json - 'bitbake-setup status' will tell if a build is in sync with the generic configuration it was made from. 'bitbake-setup update' will bring a build in sync with a configuration if needed. - 'bitbake build' means a particular sub-tree inside a build that bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR by oe-init-build-env. conf/* in that tree is 'bitbake configuration'. Bitbake configurations are constructed from templates and fragments, with existing mechanisms provided by oe-core. The configuration file format is specified such that other mechanisms to set up a bitbake build can be added; there was a mention of ability to specify local.conf content and a set of layers directly in a configuration. I think that scales poorly compared to templates and fragments, but I made sure alternative ways to configure a bitbake build are possible to add in the future :) - 'source override' is a json file that can be used to modify revisions and origins of layers that need to be checkout into a build (e.g. when master branches need to be changed to master-next for purposes of testing). Such a file is specified with a command-line option to 'init' and an example can be seen here: https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-next.override.json This commit includes fixes by Ryan Eatmon https://github.com/kanavin/bitbake/pull/1 Gyorgy Sarvari https://github.com/kanavin/bitbake/pull/2 Johannes Schneider https://github.com/kanavin/bitbake/pull/3 https://github.com/kanavin/bitbake/pull/5 Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie --- bin/bitbake-setup | 809 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 809 insertions(+) create mode 100755 bin/bitbake-setup diff --git a/bin/bitbake-setup b/bin/bitbake-setup new file mode 100755 index 000000000..739474003 --- /dev/null +++ b/bin/bitbake-setup @@ -0,0 +1,809 @@ +#!/usr/bin/env python3 + +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import sys +import argparse +import warnings +import json +import shutil +import time +import stat +import tempfile +import configparser +import datetime + +default_registry = 'git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main' + +bindir = os.path.abspath(os.path.dirname(__file__)) +sys.path[0:0] = [os.path.join(os.path.dirname(bindir), 'lib')] + +import bb.msg +import bb.process + +logger = bb.msg.logger_create('bitbake-setup', sys.stdout) + +def cache_dir(top_dir): + return os.path.join(top_dir, '.bitbake-setup-cache') + +def init_bb_cache(settings, args): + dldir = settings["default"]["dl-dir"] + bb_cachedir = os.path.join(cache_dir(args.top_dir), 'bitbake-cache') + + d = bb.data.init() + d.setVar("DL_DIR", dldir) + d.setVar("BB_CACHEDIR", bb_cachedir) + d.setVar("__BBSRCREV_SEEN", "1") + if args.no_network: + d.setVar("BB_SRCREV_POLICY", "cache") + bb.fetch.fetcher_init(d) + return d + +def save_bb_cache(): + bb.fetch2.fetcher_parse_save() + bb.fetch2.fetcher_parse_done() + +def get_config_name(config): + suffix = '.conf.json' + config_file = os.path.basename(config) + if config_file.endswith(suffix): + return config_file[:-len(suffix)] + else: + raise Exception("Config file {} does not end with {}, please rename the file.".format(config, suffix)) + +def write_config(config, config_dir): + with open(os.path.join(config_dir, "config-upstream.json"),'w') as s: + json.dump(config, s, sort_keys=True, indent=4) + +def commit_config(config_dir): + bb.process.run("git -C {} add .".format(config_dir)) + bb.process.run("git -C {} commit --no-verify -a -m 'Configuration at {}'".format(config_dir, time.asctime())) + +def _write_layer_list(dest, repodirs): + layers = [] + for r in repodirs: + for root, dirs, files in os.walk(os.path.join(dest,r)): + if os.path.basename(root) == 'conf' and 'layer.conf' in files: + layers.append(os.path.relpath(os.path.dirname(root), dest)) + layers_f = os.path.join(dest, ".oe-layers.json") + with open(layers_f, 'w') as f: + json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4) + +def checkout_layers(layers, layerdir, d): + repodirs = [] + oesetupbuild = None + print("Fetching layer/tool repositories into {}".format(layerdir)) + for r_name in layers: + r_data = layers[r_name] + repodir = r_data["path"] + repodirs.append(repodir) + + r_remote = r_data['git-remote'] + rev = r_remote['rev'] + remotes = r_remote['remotes'] + + for remote in remotes: + type,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + print(" {}".format(r_name)) + fetcher = bb.fetch.Fetch(["{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir)], d) + do_fetch(fetcher, layerdir) + + if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')): + oesetupbuild = os.path.join(layerdir, repodir, 'scripts/oe-setup-build') + oeinitbuildenv = os.path.join(layerdir, repodir, 'oe-init-build-env') + + print(" ") + _write_layer_list(layerdir, repodirs) + + if oesetupbuild: + links = {'setup-build': oesetupbuild, 'oe-scripts': os.path.dirname(oesetupbuild), 'init-build-env': oeinitbuildenv} + for l,t in links.items(): + symlink = os.path.join(layerdir, l) + if os.path.lexists(symlink): + os.remove(symlink) + os.symlink(os.path.relpath(t,layerdir),symlink) + +def setup_bitbake_build(bitbake_config, layerdir, builddir): + def _setup_build_conf(layers, build_conf_dir): + os.makedirs(build_conf_dir) + layers_s = "\n".join([" {} \\".format(os.path.join(layerdir,l)) for l in layers]) + bblayers_conf = """BBLAYERS ?= " \\ +{} + " +""".format(layers_s) + with open(os.path.join(build_conf_dir, "bblayers.conf"), 'w') as f: + f.write(bblayers_conf) + + local_conf = """# +# This file is intended for local configuration tweaks. +# +# If you would like to publish and share changes made to this file, +# it is recommended to put them into a distro config, or to create +# layer fragments from changes made here. +# +""" + with open(os.path.join(build_conf_dir, "local.conf"), 'w') as f: + f.write(local_conf) + + with open(os.path.join(build_conf_dir, "templateconf.cfg"), 'w') as f: + f.write("") + + with open(os.path.join(build_conf_dir, "conf-summary.txt"), 'w') as f: + f.write(bitbake_config["description"] + "\n") + + with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as f: + f.write("") + + def _make_init_build_env(builddir, initbuildenv): + cmd = ". {} {}".format(initbuildenv, builddir) + initbuild_in_builddir = os.path.join(builddir, 'init-build-env') + with open(initbuild_in_builddir, 'w') as f: + f.write(cmd) + + bitbake_builddir = os.path.join(builddir, "build") + print("Setting up bitbake configuration in\n {}\n".format(bitbake_builddir)) + + template = bitbake_config.get("oe-template") + layers = bitbake_config.get("bb-layers") + if not template and not layers: + print("Bitbake configuration does not contain a reference to an OpenEmbedded build template via 'oe-template' or a list of layers via 'bb-layers'; please use oe-setup-build, oe-init-build-env or another mechanism manually to complete the setup.") + return + oesetupbuild = os.path.join(layerdir, 'setup-build') + if template and not os.path.exists(oesetupbuild): + raise Exception("Cannot complete setting up a bitbake build directory from OpenEmbedded template '{}' as oe-setup-build was not found in any layers; please use oe-init-build-env manually.".format(template)) + + bitbake_confdir = os.path.join(bitbake_builddir, 'conf') + backup_bitbake_confdir = bitbake_confdir + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + if os.path.exists(bitbake_confdir): + os.rename(bitbake_confdir, backup_bitbake_confdir) + + if layers: + _setup_build_conf(layers, bitbake_confdir) + + if template: + bb.process.run("{} setup -c {} -b {} --no-shell".format(oesetupbuild, template, bitbake_builddir)) + else: + initbuildenv = os.path.join(layerdir, 'init-build-env') + if not os.path.exists(initbuildenv): + print("Could not find oe-init-build-env in any of the layers; please use another mechanism to initialize the bitbake environment") + return + _make_init_build_env(bitbake_builddir, os.path.realpath(initbuildenv)) + + siteconf_symlink = os.path.join(bitbake_confdir, "site.conf") + siteconf = os.path.normpath(os.path.join(builddir, '..', "site.conf")) + if os.path.lexists(siteconf_symlink): + os.remove(symlink) + os.symlink(os.path.relpath(siteconf, bitbake_confdir) ,siteconf_symlink) + + + init_script = os.path.join(bitbake_builddir, "init-build-env") + shell = "bash" + fragments = bitbake_config.get("oe-fragments", []) + sorted(bitbake_config.get("oe-fragment-choices",{}).values()) + if fragments: + bb.process.run("{} -c '. {} && bitbake-config-build enable-fragment {}'".format(shell, init_script, " ".join(fragments))) + + if os.path.exists(backup_bitbake_confdir): + bitbake_config_diff = get_diff(backup_bitbake_confdir, bitbake_confdir) + if bitbake_config_diff: + print("Existing bitbake configuration directory renamed to {}".format(backup_bitbake_confdir)) + print("The bitbake configuration has changed:") + print(bitbake_config_diff) + else: + shutil.rmtree(backup_bitbake_confdir) + + print("This bitbake configuration provides:\n {}\n".format(bitbake_config["description"])) + + readme = """{}\n\nAdditional information is in {} and {}\n +Source the environment using '. {}' to run builds from the command line. +The bitbake configuration files (local.conf, bblayers.conf and more) can be found in {}/conf +""".format( + bitbake_config["description"], + os.path.join(bitbake_builddir,'conf/conf-summary.txt'), + os.path.join(bitbake_builddir,'conf/conf-notes.txt'), + init_script, + bitbake_builddir + ) + readme_file = os.path.join(bitbake_builddir, "README") + with open(readme_file, 'w') as f: + f.write(readme) + print("Usage instructions and additional information are in\n {}\n".format(readme_file)) + print("The bitbake configuration files (local.conf, bblayers.conf and more) can be found in\n {}/conf\n".format(bitbake_builddir)) + print("To run builds, source the environment using\n source {}".format(init_script)) + +def get_registry_config(registry_path, id): + for root, dirs, files in os.walk(registry_path): + for f in files: + if f.endswith('.conf.json') and id == get_config_name(f): + return os.path.join(root, f) + raise Exception("Unable to find {} in available configurations; use 'list' sub-command to see what is available".format(id)) + +def update_build(config, confdir, builddir, layerdir, d): + layer_config = config["data"]["sources"] + layer_overrides = config["source-overrides"]["sources"] + for k,v in layer_overrides.items(): + if k in layer_config: + layer_config[k]["git-remote"] = v["git-remote"] + checkout_layers(layer_config, layerdir, d) + bitbake_config = config["bitbake-config"] + setup_bitbake_build(bitbake_config, layerdir, builddir) + +def int_input(allowed_values): + n = None + while n is None: + try: + n = int(input()) + except ValueError: + print('Not a valid number, please try again:') + continue + if n not in allowed_values: + print('Number {} not one of {}, please try again:'.format(n, allowed_values)) + n = None + return n + +def flatten_bitbake_configs(configs): + def merge_configs(c1,c2): + c_merged = {} + for k,v in c2.items(): + if k not in c1.keys(): + c_merged[k] = v + for k,v in c1.items(): + if k not in c2.keys(): + c_merged[k] = v + else: + c_merged[k] = c1[k] + c2[k] + del c_merged['configurations'] + return c_merged + + flattened_configs = [] + for c in configs: + if 'configurations' not in c: + flattened_configs.append(c) + else: + for sub_c in flatten_bitbake_configs(c['configurations']): + flattened_configs.append(merge_configs(c, sub_c)) + return flattened_configs + +def choose_bitbake_config(configs, parameters, non_interactive): + flattened_configs = flatten_bitbake_configs(configs) + configs_dict = {i["name"]:i for i in flattened_configs} + + if parameters: + config_id = parameters[0] + if config_id not in configs_dict: + raise Exception("Bitbake configuration {} not found; replace with one of {}".format(config_id, configs_dict)) + return configs_dict[config_id] + + enumerated_configs = list(enumerate(flattened_configs)) + if len(enumerated_configs) == 1: + only_config = flattened_configs[0] + print("\nSelecting the only available bitbake configuration {}".format(only_config["name"])) + return only_config + + if non_interactive: + raise Exception("Unable to choose from bitbake configurations in non-interactive mode: {}".format(configs_dict)) + + print("\nAvailable bitbake configurations:") + for n, config_data in enumerated_configs: + print("{}. {}\t{}".format(n, config_data["name"], config_data["description"])) + print("\nPlease select one of the above bitbake configurations by its number:") + config_n = int_input([i[0] for i in enumerated_configs]) + return flattened_configs[config_n] + +def choose_config(configs, non_interactive): + not_expired_configs = [k for k in configs.keys() if not has_expired(configs[k].get("expires", None))] + config_list = list(enumerate(not_expired_configs)) + if len(config_list) == 1: + only_config = config_list[0][1] + print("\nSelecting the only available configuration {}\n".format(only_config)) + return only_config + + if non_interactive: + raise Exception("Unable to choose from configurations in non-interactive mode: {}".format(not_expired_configs)) + + print("\nAvailable configurations:") + for n, config_name in config_list: + config_data = configs[config_name] + expiry_date = config_data.get("expires", None) + config_desc = config_data["description"] + if expiry_date: + print("{}. {}\t{} (supported until {})".format(n, config_name, config_desc, expiry_date)) + else: + print("{}. {}\t{}".format(n, config_name, config_desc)) + print("\nPlease select one of the above configurations by its number:") + config_n = int_input([i[0] for i in config_list]) + return config_list[config_n][1] + +def choose_fragments(possibilities, parameters, non_interactive): + choices = {} + for k,v in possibilities.items(): + choice = [o for o in v["options"] if o in parameters] + if len(choice) > 1: + raise Exception("Options specified on command line do not allow a single selection from possibilities {}, please remove one or more from {}".format(v["options"], parameters)) + if len(choice) == 1: + choices[k] = choice[0] + continue + + if non_interactive: + raise Exception("Unable to choose from options in non-interactive mode: {}".format(v["options"])) + + print("\n" + v["description"] + ":") + options_enumerated = list(enumerate(v["options"])) + for n,o in options_enumerated: + print("{}. {}".format(n, o)) + print("\nPlease select one of the above options by its number:") + option_n = int_input([i[0] for i in options_enumerated]) + choices[k] = options_enumerated[option_n][1] + return choices + +def obtain_config(settings, args, source_overrides, d): + if args.config: + config_id = args.config[0] + config_parameters = args.config[1:] + if os.path.exists(config_id): + print("Reading configuration from local file\n {}".format(config_id)) + upstream_config = {'type':'local', + 'path':os.path.abspath(config_id), + 'name':get_config_name(config_id), + 'data':json.load(open(config_id)) + } + elif config_id.startswith("http://") or config_id.startswith("https://"): + print("Reading configuration from network URI\n {}".format(config_id)) + import urllib.request + with urllib.request.urlopen(config_id) as f: + upstream_config = {'type':'network','uri':config_id,'name':get_config_name(config_id),'data':json.load(f)} + else: + print("Looking up config {} in configuration registry".format(config_id)) + registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + if config_id not in registry_configs: + raise Exception("Config {} not found in configuration registry, re-run 'init' without parameters to choose from available configurations.".format(config_id)) + upstream_config = {'type':'registry','registry':settings["default"]["registry"],'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + expiry_date = upstream_config['data'].get("expires", None) + if has_expired(expiry_date): + print("This configuration is no longer supported after {}. Please consider changing to a supported configuration.".format(expiry_date)) + else: + registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + config_id = choose_config(registry_configs, args.non_interactive) + config_parameters = [] + upstream_config = {'type':'registry','registry':settings["default"]["registry"],'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + + upstream_config['bitbake-config'] = choose_bitbake_config(upstream_config['data']['bitbake-setup']['configurations'], config_parameters, args.non_interactive) + upstream_config['bitbake-config']['oe-fragment-choices'] = choose_fragments(upstream_config['bitbake-config'].get('oe-fragments-one-of',{}), config_parameters[1:], args.non_interactive) + upstream_config['non-interactive-cmdline-options'] = [config_id, upstream_config['bitbake-config']['name']] + sorted(upstream_config['bitbake-config']['oe-fragment-choices'].values()) + upstream_config['source-overrides'] = source_overrides + return upstream_config + +def init_config(settings, args, d): + stdout = sys.stdout + def handle_task_progress(event, d): + rate = event.rate if event.rate else '' + progress = event.progress if event.progress > 0 else 0 + print("{}% {} ".format(progress, rate), file=stdout, end='\r') + + source_overrides = json.load(open(args.source_overrides)) if args.source_overrides else {'sources':{}} + upstream_config = obtain_config(settings, args, source_overrides, d) + print("\nRun 'bitbake-setup init --non-interactive {}' to select this configuration non-interactively.\n".format(" ".join(upstream_config['non-interactive-cmdline-options']))) + + builddir = os.path.join(os.path.abspath(args.top_dir), args.build_dir_name or "{}-{}".format(upstream_config['name']," ".join(upstream_config['non-interactive-cmdline-options'][1:]).replace(" ","-").replace("/","_"))) + if os.path.exists(builddir): + print("Build already initialized in {}\nUse 'bitbake-setup status' to check if it needs to be updated or 'bitbake-setup update' to perform the update.".format(builddir)) + return + + print("Initializing a build in\n {}".format(builddir)) + if not args.non_interactive: + y_or_n = input('Continue? y/n: ') + if y_or_n != 'y': + exit() + print() + + os.makedirs(builddir) + + confdir = os.path.join(builddir, "config") + layerdir = os.path.join(builddir, "layers") + + os.makedirs(confdir) + os.makedirs(layerdir) + + bb.process.run("git -C {} init -b main".format(confdir)) + # Make sure commiting doesn't fail if no default git user is configured on the machine + bb.process.run("git -C {} config user.name bitbake-setup".format(confdir)) + bb.process.run("git -C {} config user.email bitbake-setup@not.set".format(confdir)) + bb.process.run("git -C {} commit --no-verify --allow-empty -m 'Initial commit'".format(confdir)) + + bb.event.register("bb.build.TaskProgress", handle_task_progress, data=d) + + write_config(upstream_config, confdir) + commit_config(confdir) + update_build(upstream_config, confdir, builddir, layerdir, d) + + bb.event.remove("bb.build.TaskProgress", None) + +def get_diff(file1, file2): + try: + bb.process.run('diff -uNr {} {}'.format(file1, file2)) + except bb.process.ExecutionError as e: + if e.exitcode == 1: + return e.stdout + else: + raise e + return None + +def are_layers_changed(layers, layerdir, d): + changed = False + for r_name in layers: + r_data = layers[r_name] + repodir = r_data["path"] + + r_remote = r_data['git-remote'] + rev = r_remote['rev'] + remotes = r_remote['remotes'] + + for remote in remotes: + type,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + fetcher = bb.fetch.FetchData("{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir), d) + upstream_revision = fetcher.method.latest_revision(fetcher, d, 'default') + rev_parse_result = bb.process.run('git -C {} rev-parse HEAD'.format(os.path.join(layerdir, repodir))) + local_revision = rev_parse_result[0].strip() + if upstream_revision != local_revision: + changed = True + print('Layer repository {} checked out into {} updated revision {} from {} to {}'.format(remotes[remote]["uri"], os.path.join(layerdir, repodir), rev, local_revision, upstream_revision)) + + return changed + +def build_status(settings, args, d, update=False): + builddir = args.build_dir + + confdir = os.path.join(builddir, "config") + layerdir = os.path.join(builddir, "layers") + + current_upstream_config = json.load(open(os.path.join(confdir, "config-upstream.json"))) + + args.config = current_upstream_config['non-interactive-cmdline-options'] + args.non_interactive = True + source_overrides = current_upstream_config["source-overrides"] + new_upstream_config = obtain_config(settings, args, source_overrides, d) + + write_config(new_upstream_config, confdir) + config_diff = bb.process.run('git -C {} diff'.format(confdir))[0] + + if config_diff: + print('\nConfiguration in {} has changed:\n{}'.format(builddir, config_diff)) + if update: + commit_config(confdir) + update_build(new_upstream_config, confdir, builddir, layerdir, d) + else: + bb.process.run('git -C {} restore config-upstream.json'.format(confdir)) + return + + if are_layers_changed(current_upstream_config["data"]["sources"], layerdir, d): + if update: + update_build(current_upstream_config, confdir, builddir, layerdir, d) + return + + print("\nConfiguration in {} has not changed.".format(builddir)) + +def build_update(settings, args, d): + build_status(settings, args, d, update=True) + +def do_fetch(fetcher, dir): + # git fetcher simply dumps git output to stdout; in bitbake context that is redirected to temp/log.do_fetch + # and we need to set up smth similar here + fetchlogdir = os.path.join(dir, 'logs') + os.makedirs(fetchlogdir, exist_ok=True) + fetchlog = os.path.join(fetchlogdir, 'fetch_log.{}'.format(datetime.datetime.now().strftime("%Y%m%d%H%M%S"))) + with open(fetchlog, 'a') as f: + oldstdout = sys.stdout + sys.stdout = f + fetcher.download() + fetcher.unpack(dir) + sys.stdout = oldstdout + +def update_registry(registry, cachedir, d): + registrydir = 'configurations' + full_registrydir = os.path.join(cachedir, registrydir) + print("Fetching configuration registry\n {}\ninto\n {}".format(registry, full_registrydir)) + fetcher = bb.fetch.Fetch(["{};destsuffix={}".format(registry, registrydir)], d) + do_fetch(fetcher, cachedir) + return full_registrydir + +def has_expired(expiry_date): + if expiry_date: + return datetime.datetime.now() > datetime.datetime.fromisoformat(expiry_date) + return False + +def list_registry(registry_path, with_expired): + json_data = {} + + for root, dirs, files in os.walk(registry_path): + for f in files: + if f.endswith('.conf.json'): + config_name = get_config_name(f) + config_data = json.load(open(os.path.join(root, f))) + config_desc = config_data["description"] + expiry_date = config_data.get("expires", None) + if expiry_date: + if with_expired or not has_expired(expiry_date): + json_data[config_name] = {"description": config_desc, "expires": expiry_date} + else: + json_data[config_name] = {"description": config_desc} + return json_data + +def list_configs(settings, args, d): + registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + json_data = list_registry(registry_path, args.with_expired) + print("\nAvailable configurations:") + for config_name, config_data in json_data.items(): + expiry_date = config_data.get("expires", None) + config_desc = config_data["description"] + if expiry_date: + if args.with_expired or not has_expired(expiry_date): + print("{}\t{} (supported until {})".format(config_name, config_desc, expiry_date)) + else: + print("{}\t{}".format(config_name, config_desc)) + print("\nRun 'init' with one of the above configuration identifiers to set up a build.") + + if args.write_json: + with open(args.write_json, 'w') as f: + json.dump(json_data, f, sort_keys=True, indent=4) + print("Available configurations written into {}".format(args.write_json)) + +def default_settings_path(top_dir): + return os.path.join(top_dir, 'bitbake-setup.conf') + +def write_settings(top_dir, force_replace, non_interactive=True): + settings_path = default_settings_path(top_dir) + if not os.path.exists(settings_path) or force_replace: + + settings = configparser.ConfigParser() + settings['default'] = { + 'registry':default_registry, + 'dl-dir':os.path.join(top_dir, '.bitbake-setup-downloads'), + } + os.makedirs(os.path.dirname(settings_path), exist_ok=True) + + siteconfpath = os.path.join(top_dir, 'site.conf') + print('Configuration registry set to\n {}\n'.format(settings['default']['registry'])) + print('Bitbake-setup download cache (DL_DIR) set to\n {}\n'.format(settings['default']['dl-dir'])) + print('A new settings file will be created in\n {}\n'.format(settings_path)) + print('A common site.conf file will be created, please edit or replace before running builds\n {}\n'.format(siteconfpath)) + if not non_interactive: + y_or_n = input('Bitbake-setup will be configured with the above settings in {}, y/n: '.format(top_dir)) + if y_or_n != 'y': + print("\nYou can run 'bitbake-setup install-settings' to edit them before setting up builds") + exit() + print() + + if os.path.exists(settings_path): + backup_conf = settings_path + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + os.rename(settings_path, backup_conf) + print("Previous settings are in {}".format(backup_conf)) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + + if os.path.exists(siteconfpath): + backup_siteconf = siteconfpath + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + os.rename(siteconfpath, backup_siteconf) + print("Previous settings are in {}".format(backup_siteconf)) + with open(siteconfpath, 'w') as siteconffile: + siteconffile.write('# This file is intended for build host-specific bitbake settings\n') + +def load_settings(top_dir, non_interactive): + # This creates a new settings file if it does not yet exist + write_settings(top_dir, force_replace=False, non_interactive=non_interactive) + + settings_path = default_settings_path(top_dir) + settings = configparser.ConfigParser() + print('Loading settings from\n {}\n'.format(settings_path)) + settings.read([settings_path]) + return settings + +def global_settings_path(args): + return args.global_settings if args.global_settings else os.path.join(os.path.expanduser('~'), '.config', 'bitbake-setup', 'config') + +def write_global_settings(settings_path, force_replace, non_interactive=True): + if not os.path.exists(settings_path) or force_replace: + + settings = configparser.ConfigParser() + settings['default'] = { + 'top-dir-prefix':os.path.expanduser('~'), + 'top-dir-name':'bitbake-builds' + } + os.makedirs(os.path.dirname(settings_path), exist_ok=True) + print('Configuring global settings in\n {}\n'.format(settings_path)) + print('Top directory prefix (where all top level directories are created) set to\n {}\n'.format(settings['default']['top-dir-prefix'])) + print('Top directory name (this is added to the top directory prefix to form a top directory where builds are set up) set to\n {}\n'.format(settings['default']['top-dir-name'])) + if not non_interactive: + y_or_n = input('Write out the global settings as specified above (y/n)? ') + if y_or_n != 'y': + print("\nYou can run 'bitbake-setup install-global-settings' to edit them before setting up builds") + exit() + print() + + if os.path.exists(settings_path): + backup_conf = settings_path + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + os.rename(settings_path, backup_conf) + print("Previous global settings are in {}".format(backup_conf)) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + +def load_global_settings(settings_path, non_interactive): + # This creates a new settings file if it does not yet exist + write_global_settings(settings_path, force_replace=False, non_interactive=non_interactive) + + settings = configparser.ConfigParser() + print('Loading global settings from\n {}\n'.format(settings_path)) + settings.read([settings_path]) + return settings + +def change_settings(top_dir, new_settings): + settings = load_settings(top_dir, non_interactive=True) + for section, section_settings in new_settings.items(): + for setting, value in section_settings.items(): + settings[section][setting] = value + print("Setting '{}' in section '{}' is changed to '{}'".format(setting, section, value)) + + settings_path = default_settings_path(top_dir) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + print("New settings written to {}".format(settings_path)) + return settings + +def change_global_settings(settings_path, new_settings): + settings = load_global_settings(settings_path, non_interactive=True) + for section, section_settings in new_settings.items(): + for setting, value in section_settings.items(): + settings[section][setting] = value + print("Setting '{}' in section '{}' is changed to '{}'".format(setting, section, value)) + + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + print("New global settings written to {}".format(settings_path)) + return settings + +def get_build_dir_via_bbpath(): + bbpath = os.environ.get('BBPATH') + if bbpath: + bitbake_dir = os.path.normpath(bbpath.split(':')[0]) + if os.path.exists(os.path.join(bitbake_dir,'init-build-env')): + build_dir = os.path.dirname(bitbake_dir) + return build_dir + return None + +def get_top_dir(args, global_settings): + build_dir_via_bbpath = get_build_dir_via_bbpath() + if build_dir_via_bbpath: + top_dir = os.path.dirname(build_dir_via_bbpath) + if os.path.exists(default_settings_path(top_dir)): + return top_dir + + if hasattr(args, 'build_dir'): + # commands without --top-dir-prefix/name arguments (status, update) still need to know where + # the top dir is, but it should be auto-deduced as parent of args.build_dir + top_dir = os.path.dirname(os.path.normpath(args.build_dir)) + return top_dir + + top_dir_prefix = args.top_dir_prefix if args.top_dir_prefix else global_settings['default']['top-dir-prefix'] + top_dir_name = args.top_dir_name if args.top_dir_name else global_settings['default']['top-dir-name'] + return os.path.join(top_dir_prefix, top_dir_name) + +def main(): + def add_top_dir_arg(parser): + parser.add_argument('--top-dir-prefix', help='Top level directory prefix. This is where all top level directories are created.') + parser.add_argument('--top-dir-name', help='Top level directory name. Together with the top directory prefix this forms a top directory where builds are set up and downloaded configurations and layers are cached for reproducibility and offline builds.') + + def add_build_dir_arg(parser): + build_dir = get_build_dir_via_bbpath() + if build_dir: + parser.add_argument('--build-dir', default=build_dir, help="Path to the build, default is %(default)s via BBPATH") + else: + parser.add_argument('--build-dir', required=True, help="Path to the build") + + parser = argparse.ArgumentParser( + description="BitBake setup utility. Run with 'init' argument to get started.", + epilog="Use %(prog)s --help to get help on a specific command" + ) + parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true') + parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true') + parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR') + parser.add_argument('--no-network', action='store_true', help='Do not check whether configuration repositories and layer repositories have been updated; use only the local cache.') + parser.add_argument('--global-settings', action='store', help='Path to the global settings file where defaults for top directory prefix and name can be specified') + + subparsers = parser.add_subparsers() + + parser_list = subparsers.add_parser('list', help='List available configurations') + add_top_dir_arg(parser_list) + parser_list.add_argument('--with-expired', action='store_true', help='List also configurations that are no longer supported due to reaching their end-of-life dates.') + parser_list.add_argument('--write-json', action='store', help='Write available configurations into a json file so they can be programmatically processed.') + parser_list.set_defaults(func=list_configs) + + parser_init = subparsers.add_parser('init', help='Select a configuration and initialize a build from it') + add_top_dir_arg(parser_init) + parser_init.add_argument('config', nargs='*', help="path/URL/id to a configuration file (use 'list' command to get available ids), followed by configuration options. Bitbake-setup will ask to choose from available choices if command line doesn't completely specify them.") + parser_init.add_argument('--non-interactive', action='store_true', help='Do not ask to interactively choose from available options; if bitbake-setup cannot make a decision it will stop with a failure.') + parser_init.add_argument('--source-overrides', action='store', help='Override sources information (repositories/revisions) with values from a local json file.') + parser_init.add_argument('--build-dir-name', action='store', help='A custom build directory name under the top directory.') + parser_init.set_defaults(func=init_config) + + parser_status = subparsers.add_parser('status', help='Check if the build needs to be synchronized with configuration') + add_build_dir_arg(parser_status) + parser_status.set_defaults(func=build_status) + + parser_update = subparsers.add_parser('update', help='Update a build to be in sync with configuration') + add_build_dir_arg(parser_update) + parser_update.set_defaults(func=build_update) + + parser_install_settings = subparsers.add_parser('install-settings', help='Write a settings file with default values into the top level directory (contains the location of build configuration registry, downloads directory and other settings specific to a top directory)') + add_top_dir_arg(parser_install_settings) + parser_install_settings.set_defaults(func=write_settings) + + parser_install_global_settings = subparsers.add_parser('install-global-settings', help='Write a global settings file with default values (contains the default prefix and name of the top directory)') + parser_install_global_settings.set_defaults(func=write_global_settings) + + parser_change_setting = subparsers.add_parser('change-setting', help='Change a setting in the settings file') + add_top_dir_arg(parser_change_setting) + parser_change_setting.add_argument('section', help="Section in a settings file, typically 'default'") + parser_change_setting.add_argument('key', help="Name of the setting") + parser_change_setting.add_argument('value', help="Value of the setting") + parser_change_setting.set_defaults(func=change_settings) + + parser_change_global_setting = subparsers.add_parser('change-global-setting', help='Change a setting in the global settings file') + parser_change_global_setting.add_argument('section', help="Section in a global settings file, typically 'default'") + parser_change_global_setting.add_argument('key', help="Name of the setting") + parser_change_global_setting.add_argument('value', help="Value of the setting") + parser_change_global_setting.set_defaults(func=change_global_settings) + + args = parser.parse_args() + + logging.basicConfig(stream=sys.stdout) + if args.debug: + logger.setLevel(logging.DEBUG) + elif args.quiet: + logger.setLevel(logging.ERROR) + + # Need to re-run logger_create with color argument + # (will be the same logger since it has the same name) + bb.msg.logger_create('bitbake-setup', output=sys.stdout, + color=args.color, + level=logger.getEffectiveLevel()) + + if 'func' in args: + if args.func == write_global_settings: + write_global_settings(global_settings_path(args), force_replace=True) + return + elif args.func == change_global_settings: + change_global_settings(global_settings_path(args), {args.section:{args.key:args.value}}) + return + + if hasattr(args, 'build_dir'): + if not os.path.exists(os.path.join(args.build_dir,'build', 'init-build-env')): + print("Not a valid build directory: build/init-build-env does not exist in {}".format(args.build_dir)) + return + + if not hasattr(args, 'non_interactive'): + args.non_interactive = True + + global_settings = load_global_settings(global_settings_path(args), args.non_interactive) + args.top_dir = get_top_dir(args, global_settings) + + print('Bitbake-setup is using {} as top directory (can be changed with --top-dir-prefix/name arguments or by setting them in {}).\n'.format(args.top_dir, global_settings_path(args))) + if args.func == write_settings: + write_settings(args.top_dir, force_replace=True) + elif args.func == change_settings: + change_settings(args.top_dir, {args.section:{args.key:args.value}}) + else: + settings = load_settings(args.top_dir, args.non_interactive) + d = init_bb_cache(settings, args) + args.func(settings, args, d) + save_bb_cache() + else: + from argparse import Namespace + parser.print_help() + +main() From patchwork Mon Sep 29 12:56:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CF42CAC5BE for ; Mon, 29 Sep 2025 12:56:35 +0000 (UTC) Received: from mail-ed1-f47.google.com (mail-ed1-f47.google.com [209.85.208.47]) by mx.groups.io with SMTP id smtpd.web11.52694.1759150594712346084 for ; Mon, 29 Sep 2025 05:56:35 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=PYqW6din; spf=pass (domain: gmail.com, ip: 209.85.208.47, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f47.google.com with SMTP id 4fb4d7f45d1cf-6228de280a4so9025262a12.2 for ; Mon, 29 Sep 2025 05:56:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150593; x=1759755393; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hijAlmrGt7+w9c0+XbEDr/gNbZwmt3SUp0OMAmOQSss=; b=PYqW6dinUubSKDdB9/oyB4LiAEJHyw+sQOXePl8zvhmnJCL+wUBFOh27v7CxDTPB4W Oi/91QDT0RiPGx3xIJCviiA3XbLBqdv9wnURIhQ9Qz9hUxBfjiA3pC0lPKRGq/cTHjBP NO9qVNxzY2xAMDO54UOKLNzole49LyMbCsv0JW4ADpMSaFaXqCghTjJz2pQcBuP7gFp9 dmW8oJImSEXtiwUw9JfR6KrZqPkRdhXqjnS9ft4caq/sUnRp2omnrd+sC3+XmpuWxGfm j/3noGKEJA5t5NhuOzxsEj1nQA9CMqq1g7KAAX/fIHmV5l8O+5A1CY6ofRoldH/XT0xr oY1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150593; x=1759755393; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hijAlmrGt7+w9c0+XbEDr/gNbZwmt3SUp0OMAmOQSss=; b=pIKxIcLpSFAiaVSt5dH8bUXr1fhbZ1DWadlWFN8KBherAlphAyAefiZ71lf+M+i/C2 mdzPWnvOPQgPtW3TQnfeV26xlN5ROiigMEEaBjLxYWGUBZDOabix6CGjGgokDnMAVK2t ACmKgBVT2BEwvMUT8dVfSalIpLfmNiSk6GxtRmIUHTEQ6iCSCxRZc++jtYzXnkLB9VhI CuakSLbdFhi/J186uZEp/h682sVp77tnvU4Rd/uyYoVFa/yYgs0QJe9tlTLu/VS3pohK AH6FtjpI597gIPd/7+y5hVBd6cpaVuSnJssdBLpgg5ydFHpYKCCgN+74QrOZlSJv6Qi6 wFfA== X-Gm-Message-State: AOJu0YzR5DOOEAZwnYPdJ84L8jT79MXYViGf+Czqv6nItr4q7bGuN3Jd uYGDJKQ1lDI4qaBPqRkGyuIn0bZFYbCNfWIP7BVLiwUjF/oKC2vxOYjg+YGhRQ== X-Gm-Gg: ASbGnctq3+nKy3f0lx+h5pSBATAFdaZR3OvYCS3uk+85VoBUM5F6VpxANlOqC2r43SJ bfvYn9OWIXR9l8ftktsK0XT4n5pPae3Y1NZDqmhVrXhgKMFjuVdwyqKJ5PfToyNxodJQzLa61ho pZbQwEb0Ftv/2gBGFUVCtEIG+fCwR5fP27oIwe/tIJoCQ8R1l214nQC9k26u0htMOQwPg2eh/79 fD+ky9hLEfQGLpNguTwj+8J2tM1P2MXaIt64K+JNhHlrViF7AZDBqZ6GVGChe+qh05ZJ9qQrwde gFgxQZctsJY5FMglvmlpQV5UR6MaoHQu7AewdM/0YKhhZS+4NnHb3+R3s1YTiQcCBlGn2UyLjIi 8o2nXbdbak1BXU/VdIntORZ6Q7pK6pkxbuFJJKyJMCKPHiTUa5sUtpbatdYurlGD06M3q+LOHCY 4JkZWBUhRj3SwSfuinDXy9bmDytM103yYtWzH2rlq2J5jUP7Uz/dCKSM8WeGMaDrfezuKA X-Google-Smtp-Source: AGHT+IEpYHRg7zBEupjwCK8yB4q9rfZhnSfuHoKTsxQzQL9+h7uZSxvkQ0KSeiLQPy+8dXL17j8oWg== X-Received: by 2002:a05:6402:5191:b0:631:d76b:4c3 with SMTP id 4fb4d7f45d1cf-6349fa7e694mr15599456a12.19.1759150592706; Mon, 29 Sep 2025 05:56:32 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:32 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin , Richard Purdie Subject: [PATCH 02/10] bitbake-setup: add tests to bitbake-selftest Date: Mon, 29 Sep 2025 14:56:08 +0200 Message-Id: <20250929125616.1751116-2-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250929125616.1751116-1-alex.kanavin@gmail.com> References: <20250929125616.1751116-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:35 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18101 From: Alexander Kanavin Run like this: alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-selftest -v bb.tests.setup test_setup (bb.tests.setup.BitbakeSetupTest.test_setup) ... ok ---------------------------------------------------------------------- Ran 1 test in 9.223s OK The test does a basic run-through of init, then status/update on an unchanged configuration, then status/update on a configuration changed via new commits to the test layer, then status/update on configuration changed via the top level json config file. Note that nothing whatsoever is fetched from the network; the test relies entirely on synthetic data contained inside itself, including minimal stubs for oe-setup-build and bitbake-config-build. This data is used to create temporary git repositories then clone them via local filesystem URIs. Later on this can be supplemented by an oe-selftest that tests bitbake-setup against real config files in the official configuration repository and real layers, templates and fragments. Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie --- bin/bitbake-selftest | 1 + lib/bb/tests/setup.py | 266 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 267 insertions(+) create mode 100644 lib/bb/tests/setup.py diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest index d45c2d406..fb7c57dd8 100755 --- a/bin/bitbake-selftest +++ b/bin/bitbake-selftest @@ -29,6 +29,7 @@ tests = ["bb.tests.codeparser", "bb.tests.fetch", "bb.tests.parse", "bb.tests.runqueue", + "bb.tests.setup", "bb.tests.siggen", "bb.tests.utils", "bb.tests.compression", diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py new file mode 100644 index 000000000..ab4dbf762 --- /dev/null +++ b/lib/bb/tests/setup.py @@ -0,0 +1,266 @@ +# +# Copyright BitBake Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# + +from bb.tests.fetch import FetcherTest +import json + +class BitbakeSetupTest(FetcherTest): + def setUp(self): + super(BitbakeSetupTest, self).setUp() + + self.registrypath = os.path.join(self.tempdir, "bitbake-setup-configurations") + + os.makedirs(self.registrypath) + self.git_init(cwd=self.registrypath) + self.git('commit --allow-empty -m "Initial commit"', cwd=self.registrypath) + + self.testrepopath = os.path.join(self.tempdir, "test-repo") + os.makedirs(self.testrepopath) + self.git_init(cwd=self.testrepopath) + self.git('commit --allow-empty -m "Initial commit"', cwd=self.testrepopath) + + oeinitbuildenv = """BBPATH=$1 +export BBPATH +PATH={}:$PATH +""".format(os.path.join(self.testrepopath, 'scripts')) + self.add_file_to_testrepo('oe-init-build-env',oeinitbuildenv, script=True) + + oesetupbuild = """#!/usr/bin/env python3 +import getopt +import sys +import os +import shutil +opts, args = getopt.getopt(sys.argv[2:], "c:b:", ["no-shell"]) +for option, value in opts: + if option == '-c': + template = value + if option == '-b': + builddir = value +confdir = os.path.join(builddir, 'conf') +os.makedirs(confdir, exist_ok=True) +with open(os.path.join(confdir, 'conf-summary.txt'), 'w') as f: + f.write(template) +shutil.copy(os.path.join(os.path.dirname(__file__), 'test-repo/test-file'), confdir) +with open(os.path.join(builddir, 'init-build-env'), 'w') as f: + f.write("BBPATH={}\\nexport BBPATH\\nPATH={}:$PATH".format(builddir, os.path.join(os.path.dirname(__file__), 'test-repo/scripts'))) +""" + self.add_file_to_testrepo('scripts/oe-setup-build', oesetupbuild, script=True) + + bitbakeconfigbuild = """#!/usr/bin/env python3 +import os +import sys +confdir = os.path.join(os.environ['BBPATH'], 'conf') +fragment = sys.argv[2] +with open(os.path.join(confdir, fragment), 'w') as f: + f.write('') +""" + self.add_file_to_testrepo('scripts/bitbake-config-build', bitbakeconfigbuild, script=True) + + sometargetexecutable_template = """#!/usr/bin/env python3 +import os +print("This is {}") +print("BBPATH is {{}}".format(os.environ["BBPATH"])) +""" + for e_name in ("some-target-executable-1", "some-target-executable-2"): + sometargetexecutable = sometargetexecutable_template.format(e_name) + self.add_file_to_testrepo('scripts/{}'.format(e_name), sometargetexecutable, script=True) + + def runbbsetup(self, cmd): + bbsetup = os.path.abspath(os.path.dirname(__file__) + "/../../../bin/bitbake-setup") + return bb.process.run("{} --global-settings {} {}".format(bbsetup, os.path.join(self.tempdir, 'global-config'), cmd)) + + def add_json_config_to_registry(self, name, rev): + config = """ +{ + "sources": { + "test-repo": { + "git-remote": { + "remotes": { + "origin": { + "uri": "file://%s" + } + }, + "rev": "%s" + }, + "path": "test-repo" + } + }, + "description": "Test configuration", + "bitbake-setup": { + "configurations": [ + { + "name": "gadget", + "description": "Gadget build configuration", + "oe-template": "test-configuration-gadget", + "oe-fragments": ["test-fragment-1"] + }, + { + "name": "gizmo", + "description": "Gizmo build configuration", + "oe-template": "test-configuration-gizmo", + "oe-fragments": ["test-fragment-2"] + }, + { + "name": "gadget-notemplate", + "description": "Gadget notemplate build configuration", + "bb-layers": ["layerA","layerB/meta-layer"], + "oe-fragments": ["test-fragment-1"] + }, + { + "name": "gizmo-notemplate", + "description": "Gizmo notemplate build configuration", + "bb-layers": ["layerC","layerD/meta-layer"], + "oe-fragments": ["test-fragment-2"] + } + ] + }, + "version": "1.0" +} +""" % (self.testrepopath, rev) + os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True) + with open(os.path.join(self.registrypath, name), 'w') as f: + f.write(config) + self.git('add {}'.format(name), cwd=self.registrypath) + self.git('commit -m "Adding {}"'.format(name), cwd=self.registrypath) + return json.loads(config) + + def add_file_to_testrepo(self, name, content, script=False): + fullname = os.path.join(self.testrepopath, name) + os.makedirs(os.path.join(self.testrepopath, os.path.dirname(name)), exist_ok=True) + with open(fullname, 'w') as f: + f.write(content) + if script: + import stat + st = os.stat(fullname) + os.chmod(fullname, st.st_mode | stat.S_IEXEC) + self.git('add {}'.format(name), cwd=self.testrepopath) + self.git('commit -m "Adding {}"'.format(name), cwd=self.testrepopath) + + def check_builddir_files(self, buildpath, test_file_content, json_config): + with open(os.path.join(buildpath, 'layers', 'test-repo', 'test-file')) as f: + self.assertEqual(f.read(), test_file_content) + bitbake_config = json_config["bitbake-config"] + bb_build_path = os.path.join(buildpath, 'build') + bb_conf_path = os.path.join(bb_build_path, 'conf') + self.assertTrue(os.path.exists(os.path.join(bb_build_path, 'init-build-env'))) + + if "oe-template" in bitbake_config: + with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f: + self.assertEqual(f.read(), bitbake_config["oe-template"]) + with open(os.path.join(bb_conf_path, 'test-file')) as f: + self.assertEqual(f.read(), test_file_content) + else: + with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f: + self.assertIn(bitbake_config["description"], f.read()) + with open(os.path.join(bb_conf_path, 'bblayers.conf')) as f: + bblayers = f.read() + for l in bitbake_config["bb-layers"]: + self.assertIn(os.path.join(buildpath, 'layers', l), bblayers) + + for f in bitbake_config["oe-fragments"]: + self.assertTrue(os.path.exists(os.path.join(bb_conf_path, f))) + + + def test_setup(self): + # check that no arguments works + self.runbbsetup("") + + # check that --help works + self.runbbsetup("--help") + + # set up global location for top-dir-prefix + out = self.runbbsetup("install-global-settings") + settings_path = "{}/global-config".format(self.tempdir) + self.assertIn(settings_path, out[0]) + out = self.runbbsetup("change-global-setting default top-dir-prefix {}".format(self.tempdir)) + self.assertIn("Setting 'top-dir-prefix' in section 'default' is changed to", out[0]) + self.assertIn("New global settings written to".format(settings_path), out[0]) + + # check that writing settings works and then adjust them to point to + # test registry repo + out = self.runbbsetup("install-settings") + settings_path = "{}/bitbake-builds/bitbake-setup.conf".format(self.tempdir) + self.assertIn(settings_path, out[0]) + out = self.runbbsetup("change-setting default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)) + self.assertIn("Setting 'registry' in section 'default' is changed to", out[0]) + self.assertIn("New settings written to".format(settings_path), out[0]) + + # check that 'list' produces correct output with no configs, one config and two configs + out = self.runbbsetup("list") + self.assertNotIn("test-config-1", out[0]) + self.assertNotIn("test-config-2", out[0]) + + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', 'master') + out = self.runbbsetup("list") + self.assertIn("test-config-1", out[0]) + self.assertNotIn("test-config-2", out[0]) + + json_2 = self.add_json_config_to_registry('config-2/test-config-2.conf.json', 'master') + out = self.runbbsetup("list --write-json={}".format(os.path.join(self.tempdir, "test-configs.json"))) + self.assertIn("test-config-1", out[0]) + self.assertIn("test-config-2", out[0]) + with open(os.path.join(self.tempdir, "test-configs.json")) as f: + json_configs = json.load(f) + self.assertIn("test-config-1", json_configs) + self.assertIn("test-config-2", json_configs) + + # check that init/status/update work + # (the latter two should do nothing and say that config hasn't changed) + test_file_content = 'initial\n' + self.add_file_to_testrepo('test-file', test_file_content) + for cf in ('test-config-1', 'test-config-2'): + for c in ('gadget','gizmo','gadget-notemplate','gizmo-notemplate'): + out = self.runbbsetup("init --non-interactive {} {}".format(os.path.join(self.registrypath,'config-2/test-config-2.conf.json') if cf == 'test-config-2' else cf, c)) + buildpath = os.path.join(self.tempdir, 'bitbake-builds', '{}-{}'.format(cf, c)) + with open(os.path.join(buildpath, 'config', "config-upstream.json")) as f: + config_upstream = json.load(f) + self.check_builddir_files(buildpath, test_file_content, config_upstream) + os.environ['BBPATH'] = os.path.join(buildpath, 'build') + out = self.runbbsetup("status") + self.assertIn("Configuration in {} has not changed".format(buildpath), out[0]) + out = self.runbbsetup("update") + self.assertIn("Configuration in {} has not changed".format(buildpath), out[0]) + + # change a file in the test layer repo, make a new commit and + # test that status/update correctly report the change and update the config + prev_test_file_content = test_file_content + test_file_content = 'modified\n' + self.add_file_to_testrepo('test-file', test_file_content) + for c in ('gadget','gizmo','gadget-notemplate','gizmo-notemplate'): + buildpath = os.path.join(self.tempdir, 'bitbake-builds', 'test-config-1-{}'.format(c)) + os.environ['BBPATH'] = os.path.join(buildpath, 'build') + out = self.runbbsetup("status") + self.assertIn("Layer repository file://{} checked out into {}/layers/test-repo updated revision master from".format(self.testrepopath, buildpath), out[0]) + out = self.runbbsetup("update") + if c in ('gadget','gizmo'): + self.assertIn("Existing bitbake configuration directory renamed to {}/build/conf-backup.".format(buildpath), out[0]) + self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0]) + with open(os.path.join(buildpath, 'config', "config-upstream.json")) as f: + config_upstream = json.load(f) + self.check_builddir_files(buildpath, test_file_content, config_upstream) + + # make a new branch in the test layer repo, change a file on that branch, + # make a new commit, update the top level json config to refer to that branch, + # and test that status/update correctly report the change and update the config + prev_test_file_content = test_file_content + test_file_content = 'modified-in-branch\n' + branch = "another-branch" + self.git('checkout -b {}'.format(branch), cwd=self.testrepopath) + self.add_file_to_testrepo('test-file', test_file_content) + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch) + for c in ('gadget','gizmo','gadget-notemplate','gizmo-notemplate'): + buildpath = os.path.join(self.tempdir, 'bitbake-builds', 'test-config-1-{}'.format(c)) + os.environ['BBPATH'] = os.path.join(buildpath, 'build') + out = self.runbbsetup("status") + self.assertIn("Configuration in {} has changed:".format(buildpath), out[0]) + self.assertIn('- "rev": "master"\n+ "rev": "another-branch"', out[0]) + out = self.runbbsetup("update") + if c in ('gadget','gizmo'): + self.assertIn("Existing bitbake configuration directory renamed to {}/build/conf-backup.".format(buildpath), out[0]) + self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0]) + with open(os.path.join(buildpath, 'config', "config-upstream.json")) as f: + config_upstream = json.load(f) + self.check_builddir_files(buildpath, test_file_content, config_upstream) From patchwork Mon Sep 29 12:56:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71220 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39B93CAC5B5 for ; Mon, 29 Sep 2025 12:56:45 +0000 (UTC) Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) by mx.groups.io with SMTP id smtpd.web11.52696.1759150595382387511 for ; Mon, 29 Sep 2025 05:56:35 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=mK62DeKV; spf=pass (domain: gmail.com, ip: 209.85.208.44, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-628f29d68ecso9554865a12.3 for ; Mon, 29 Sep 2025 05:56:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150594; x=1759755394; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7hHrQGbXOAOF6NziVlc4Sok8BrCa5xqiNxmYZDr/QWE=; b=mK62DeKVzJxA7xDjidCm/CnwxHZYLHBGK6d1KCti+lysZws4FVt/bxzL+srKOv8I1F 38+AIDEydPxbhhstj6ajvbs0zUDuUpNjrpQmKoqHnaZrpyuxzMJMQ5OU4OTUceYIMrvK 6YiQmFaEe59X6BS5csllSTfdXK4KBVqapxlz+BGHO/S9KJ+/Esco6WULKKqeOAGTQWyC jZjUWe86yvxmXer492T8/VJSwtK5qzxpkSe6rM1p9ENgLqm84+iwob5V8lSeQhcwaf4R v4bPkd8bgPPy5mw22yGvUfL8GxBOQj5egl2Wgi4lVX6sxRbBiTEhf8zjJTDJu6r0O0JP u9/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150594; x=1759755394; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7hHrQGbXOAOF6NziVlc4Sok8BrCa5xqiNxmYZDr/QWE=; b=LAMv/2yyt9AJrzEAl+Pm8Zovshhl4oQeEzugq+xQnXgznFfpP/WdZzVRrRnfj4cZ2P vKaAvFsoGMg1+TG8YB3711DmLLoOmN5vRGiA0nSxgjR2vudjolCYSLxqhl4khNlGkkz9 dqpnH90guxga+Vy0XQAFMzDeghmZvTwmPtWf2/+LBx5RnlV/K+WMdrrZVMoATBkQZ8/x MGENgfD7zy0hqQUBblrPYKW3yErSOiHjrdP2ATrTWhp5Lx6UjfVoXzr6J2XLMlF8JF8p D65xiRK2Kmvc0eBKbUzGgWy9g9NMvJOFNxBJ5afWOvFKPjF6Od+OFqJAmJDzeAVSiUTz A85w== X-Gm-Message-State: AOJu0Yze3eGelgD+zYUabvMSBQ4VZDQFQojcaGFPTD4jJ1Ir7s3WH7hD zAT+v7EpiygJa4qW6C3pym2B1on4ckH9pFEJLvIbtwUnoQYAvGl/7E/jY3F5uw== X-Gm-Gg: ASbGncth1j4l9gTkgsXs1p/Y0gPohiwxpWdkQYh+cJ4l9fYKMo8R0IXuvs6hnVrV46Y IUtEzfZe8iYG9tIOj1SnTOGBBrL2n+VyLtNaJGKaSQVOJ1FMjGs7CU1tV2YAVjt0URiFvCuFJmt kCfkCR2qRRBGP5RfyUFvRSmquY0CsvEBwPiGfRLP28aeBavvN6YqiI8EMTXUbJmMxSMZtsh8FjC i6PbOTd50mQmUk9m2CfZnPn3N7BT5jRtbqUyWDZN2elsMeW4HAR7hbeTmVq4At3bz3OJCgUOK2d r2wzWOQXEOcm63dLAzHv0hwduMlD8uCnVxu6ydHIBsTVLsTr3stnGjNRv0fHRyFBE7pKod08uHQ hvM5Fr4/7DsKJMEEuTswRMySNnrec0bEWR2CVfftXlSrgbS3mKAuLSX88OUaFMekcwCdP22WC9z mbfwUzey5+NzI6Sb2WwJ3k7OCSiAppRww9FqawXCu/YTJfDVQkwolfqeEwszGuxGtkJtHQ X-Google-Smtp-Source: AGHT+IEAeJrLR2ZE8CYZBEt3JQU5e2EVnEVTsYUrP2BHr1ea7wRra/KFCST19fUwoNj49j8qsv79/Q== X-Received: by 2002:a05:6402:40c2:b0:634:cb54:8115 with SMTP id 4fb4d7f45d1cf-634cb548583mr9512708a12.7.1759150593527; Mon, 29 Sep 2025 05:56:33 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:33 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin , Richard Purdie Subject: [PATCH 03/10] bitbake-setup: add 'install-buildtools' command Date: Mon, 29 Sep 2025 14:56:09 +0200 Message-Id: <20250929125616.1751116-3-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250929125616.1751116-1-alex.kanavin@gmail.com> References: <20250929125616.1751116-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18102 From: Alexander Kanavin This basically calls install-buildtools from oe-core/poky, but it ensures via command line parameters that the installation location is stable and the downloads are preserved for reproducibility: $ bin/bitbake-setup install-buildtools Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf ====== Buildtools archive is downloaded into /home/alex/bitbake-builds/yocto-master-testing/buildtools-downloads/20250319141333 and its content installed into /home/alex/bitbake-builds/yocto-master-testing/buildtools ... (output from install-buildtools script) ====== It also detects when buildtools are already installed, and will direct users what to do: ====== alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup install-buildtools Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf Buildtools are already installed in /home/alex/bitbake-builds/yocto-master-testing/buildtools. If you wish to use them, you need to source the the environment setup script e.g. $ . /home/alex/bitbake-builds/yocto-master-testing/buildtools/environment-setup-x86_64-pokysdk-linux You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation ====== This commits includes fixes by Gyorgy Sarvari https://github.com/kanavin/bitbake/pull/2 Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie --- bin/bitbake-setup | 26 ++++++++++++++++++++++++++ lib/bb/tests/setup.py | 20 ++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index 739474003..d6509500d 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -16,6 +16,8 @@ import stat import tempfile import configparser import datetime +import glob +import subprocess default_registry = 'git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main' @@ -554,6 +556,25 @@ def list_configs(settings, args, d): json.dump(json_data, f, sort_keys=True, indent=4) print("Available configurations written into {}".format(args.write_json)) +def install_buildtools(settings, args, d): + buildtools_install_dir = os.path.join(args.build_dir, 'buildtools') + if os.path.exists(buildtools_install_dir): + if not args.force: + print("Buildtools are already installed in {}.".format(buildtools_install_dir)) + env_scripts = glob.glob(os.path.join(buildtools_install_dir, 'environment-setup-*')) + if env_scripts: + print("If you wish to use them, you need to source the environment setup script e.g.") + for s in env_scripts: + print("$ . {}".format(s)) + print("You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation.") + return + shutil.rmtree(buildtools_install_dir) + + install_buildtools = os.path.join(args.build_dir, 'layers/oe-scripts/install-buildtools') + buildtools_download_dir = os.path.join(args.build_dir, 'buildtools-downloads/{}'.format(time.strftime("%Y%m%d%H%M%S"))) + print("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir)) + subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True) + def default_settings_path(top_dir): return os.path.join(top_dir, 'bitbake-setup.conf') @@ -739,6 +760,11 @@ def main(): add_build_dir_arg(parser_update) parser_update.set_defaults(func=build_update) + parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine') + add_build_dir_arg(parser_install_buildtools) + parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.') + parser_install_buildtools.set_defaults(func=install_buildtools) + parser_install_settings = subparsers.add_parser('install-settings', help='Write a settings file with default values into the top level directory (contains the location of build configuration registry, downloads directory and other settings specific to a top directory)') add_top_dir_arg(parser_install_settings) parser_install_settings.set_defaults(func=write_settings) diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index ab4dbf762..fb2c15f54 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -49,6 +49,21 @@ with open(os.path.join(builddir, 'init-build-env'), 'w') as f: """ self.add_file_to_testrepo('scripts/oe-setup-build', oesetupbuild, script=True) + installbuildtools = """#!/usr/bin/env python3 +import getopt +import sys +import os + +opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="]) +for option, value in opts: + if option == '-d': + installdir = value + +print("Buildtools installed into {}".format(installdir)) +os.makedirs(installdir) +""" + self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True) + bitbakeconfigbuild = """#!/usr/bin/env python3 import os import sys @@ -224,6 +239,11 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) out = self.runbbsetup("update") self.assertIn("Configuration in {} has not changed".format(buildpath), out[0]) + # install buildtools + out = self.runbbsetup("install-buildtools") + self.assertIn("Buildtools installed into", out[0]) + self.assertTrue(os.path.exists(os.path.join(buildpath, 'buildtools'))) + # change a file in the test layer repo, make a new commit and # test that status/update correctly report the change and update the config prev_test_file_content = test_file_content From patchwork Mon Sep 29 12:56:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 468D8CCA468 for ; Mon, 29 Sep 2025 12:56:45 +0000 (UTC) Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) by mx.groups.io with SMTP id smtpd.web10.53021.1759150596091611063 for ; Mon, 29 Sep 2025 05:56:36 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=FX7wK211; spf=pass (domain: gmail.com, ip: 209.85.208.45, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f45.google.com with SMTP id 4fb4d7f45d1cf-62fb48315ddso8382593a12.2 for ; Mon, 29 Sep 2025 05:56:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150594; x=1759755394; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ch6knsTbgDGE9TNf53GLPhss2+wLFV2nM9m1CBJiGL4=; b=FX7wK211pOTveT0mCYoDfi6HAfxQYZcGCLWRhPNUMPTLth8X2RuGwR0yS3FFxMjlLJ FmkuK6GpCnKH+xSXsKIJ+Tt8lp/ylkG7540ubwTuDLAmyeqFuCW8q7rmyqA9A8pO3mPH pHH4alQ8YwySUHVvzcsJDPP4YVRb+ao1nQygz3pa4TNKsJ4hvV0VWkm42rP41Z+ViE/v J7/VSoHZAngipEt6KFetbNab1y4b9S2n6iuZNevRdL9dURGQzfYOqRMcmT9V1Cz673eL e0ToiE+EMcUtXactBeqpjuxIHd+HjoiQ/fewJfCoGSelHnIsTx9T+4AUHAtnNJIyIX+O SJlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150594; x=1759755394; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ch6knsTbgDGE9TNf53GLPhss2+wLFV2nM9m1CBJiGL4=; b=obwFHFLGG9aCOEuzbErosVPqe7C313A+xQFOcN2Jo+ZtmR27kfgklPdSxSRpO5cLoB pFbkO1DDmPeUt4UQWjPFds+9aS6063Gs7T46HLasZolJVIItKNxvt0eM5MM6an+dk7fi jxHqZiDXugnf22n3bkGIY92LBhcLpmwvY7aIfe7zJTdId2fxXOvZfEpYORm/d3wGWs3G WYhf6KG/elulH5U7V+FhPoqXRsIO67oMTArIOoXYQ/53pe/u1Ja139ik0lyo0BXtdzOj pSCa834AtyOdAoNMNS/b3Fu7wxb9wh0m5+kJZ074Aodytdh9tXFzxHIOD70+KlQcQ6qG DGJw== X-Gm-Message-State: AOJu0Yzjlu0ry8FpWO5zjHsA6YLC6YvUO8AOPwecOS6SGfsuIYieqdVw VAxexHY+862t5b9mX429KPWLrwc03JrAszvODHLht75aBzgZ/0USSiUcis4jWQ== X-Gm-Gg: ASbGncvmHwG5KKepv7b/CmMeXFEgffuvktS1tZsgpBK/7eRPx9Jp8WZQxKl9B0rdCW5 tnSoEXuLEc2rFYU8mOhKBw2nhrDO370aL9qArZ6GVSmk4re/EQlFpzqT1VmahBURmvdXZyVWFc+ 0UooAPtQxR4T+qu1KpUX/Wj8WjRA4Z3sqd+7ebFKQ44+FSZZVZw4NKWlFzvxkVHjp3YSpIrqiHq p1Hg/Ls6vEG/4YSA54KGCCuCd/b8C14JlzgBDmu5L5+3qm+xiUAVvhPIIEvq4CJlApyFcnFVzOW 5S5sWmrvlFxhwjxRuiki0onIg6CifehCc44J/M89l3tNk0tFripL7jgfCIvVwGzpLBUhkEpKVMc JXQqj7h+/jQ/Lr4nmHd03Eef8LQ3gh2jIc0b/0aOUfDoDPavz5Qv5nTLlRBo1s+GM4k1DRbTuWk UCc62W9tkMVlsrMkveJHUZnFoW09Yhj7HnELB8HlXwvtrS5IhMCI21BQqcm7bMCfbgIRiC X-Google-Smtp-Source: AGHT+IFqwy4q+cUUVtSVSdNPEhd+C24JC+ra0s7Y/LlSUV0UreP+aMQPJ19JadGcFSJNc+q73Y6I0Q== X-Received: by 2002:aa7:db47:0:b0:62f:50cb:764c with SMTP id 4fb4d7f45d1cf-6349fa97608mr12375102a12.35.1759150594299; Mon, 29 Sep 2025 05:56:34 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:33 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin , Richard Purdie Subject: [PATCH 04/10] cooker: adjust the error message about missing websockets module to mention bitbake-setup Date: Mon, 29 Sep 2025 14:56:10 +0200 Message-Id: <20250929125616.1751116-4-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250929125616.1751116-1-alex.kanavin@gmail.com> References: <20250929125616.1751116-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18103 From: Alexander Kanavin Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index fb87368f1..03f262ac1 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -320,7 +320,10 @@ class BBCooker: except ImportError as e: bb.fatal(""""Unable to use hash equivalence server at '%s' due to missing or incorrect python module: %s -Please install the needed module on the build host, or use an environment containing it (e.g a pip venv or OpenEmbedded's buildtools tarball). +Please install the needed module on the build host, or use an environment containing it: + - if you are using bitbake-setup, run 'bitbake-setup install-buildtools' + - openembedded-core layer contains 'scripts/install-buildtools' that can also be used + - or set up pip venv You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in significantly longer build times as bitbake will be unable to reuse prebuilt sstate artefacts.""" % (upstream, repr(e))) except ConnectionError as e: From patchwork Mon Sep 29 12:56:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56123CCA471 for ; Mon, 29 Sep 2025 12:56:45 +0000 (UTC) Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by mx.groups.io with SMTP id smtpd.web10.53023.1759150596925512677 for ; Mon, 29 Sep 2025 05:56:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=TuByk20f; spf=pass (domain: gmail.com, ip: 209.85.208.41, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-62fc0b7bf62so6848802a12.2 for ; Mon, 29 Sep 2025 05:56:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150595; x=1759755395; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4bLTUF9/N8IcqeBSQC3/AqZ/bOhiF23/H+MqoNOhb0U=; b=TuByk20frCzlqLLgdq3pPt+B+4/W2OyWTZLYQZnYcnIUq585Dx8BFA4j+L0CZSCzbE RWMKgBQcPs2DenZRhVeqwdRXP98jnb7ZDrmg8bfln/eBSbekzYfruxx2Un+xTKTUezXa K0iFabeV/xwy1nxUSdysJDDjU4rny99gnIW/Pbc4pkpWl9Xxwg/aP40gVuyCAES0ccTd FwaTmScHZwZIRimDzTLVsA+7VIvST74jbWz95YehQvneJ2oPKTLEhsSHALfgwE8DBpOW oIksmbhveyhQaabC5Gyvz3uGl9FwcQFiNbq286eeWBEU0NmUvz64hqY4rwfY+aZ0xIk9 LreA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150595; x=1759755395; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4bLTUF9/N8IcqeBSQC3/AqZ/bOhiF23/H+MqoNOhb0U=; b=IibvBMWS4px4I2O33ZbL6fej/3AROrcUJhNHc70e2bhXJoy91SYzlsOvOPUEGtOKIW 7LxwFS8aeL0/xzHUFJR4wws1sRmYD21zkBzYDJJQ/yizxcstjgTG04gNZty9PmHNf0jE +ppYnyR3EfiuRoyMVlbx1nFHnY4Ayi4rvbnOgT0eYSRgpneTzpZiUMs80JWAbFywSIA6 Xck0ygVZof57ZFSYccI5TferwhH8oF6pPVc/vPoXzroLr/t+rixmA+hgoh7mEIyMYVSJ JPrUdU0nhPEi5wjXX1cbaFLn4jq3Ml9SOhQqaE3Qo6d8B9TqOGzSWuvhIUcMEbdncMwY ZA7g== X-Gm-Message-State: AOJu0YxRLeb0+v96VKEmegLeLvBQ9mhhN9GyHl2459qT01LnLKQTRgEV wF6DzULUIKEtkZ680vQ3JZoia2xzDOGJJwQWg5FSOZiCpoyetOgrY+W8dXKvaA== X-Gm-Gg: ASbGnctdmQbQfJ1zkQiTxr1FwkaikVcqaUu+kH1qrVoQBAEC591+bbcaF7PhFrgJIcd gWhf+JOpYlcxAvN0po3/a/sWLpWX9KIYAlSbaii2woVafeI++hax0SVzxbQveMFuIlTDRAfGdlz X7VIJr3QDnuqCxvF5ItwkF3O/BRBZfMdjSjQxvczh/h+++j72RRCEftQ2jtYeDSeGwO6Uk1G2ul ZdewYjwruerWupNWt/BIGDrkgBIp7NxlDyBA4AEbMZS5PD+Awb/ZyQ2jfZF9v12s1aPmCmTSr39 xVCnJoU6FdPCa25fv+mGA+B95e3L0ByVbbqVWhc973GokappDuL/m7DJ/uZYgWY9WjPRSCclp3g ACo0riwYVJLKR7WMFk5vWUA/sjNVhX4jwRZKWeUf5SDzkvDaiC6vIAmB+RANDYssoL9s5EiUnDu Xc8xtNcnhnvpGhmmjndlUaso8xDXoUZLXt6oStjUXpzJo1B6eqCAEybvMXCA== X-Google-Smtp-Source: AGHT+IHt6V6JM8ANQzt6dPflrLmTP8VBrwuGgKGzNJ9tgO8TuDZCMnTlKWTC3EMMgCc0r51nB3rR4Q== X-Received: by 2002:aa7:c147:0:b0:617:9bff:be16 with SMTP id 4fb4d7f45d1cf-6349fa79a44mr12695366a12.22.1759150595146; Mon, 29 Sep 2025 05:56:35 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:34 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Richard Purdie Subject: [PATCH 05/10] bitbake-setup: Allow local registry paths Date: Mon, 29 Sep 2025 14:56:11 +0200 Message-Id: <20250929125616.1751116-5-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250929125616.1751116-1-alex.kanavin@gmail.com> References: <20250929125616.1751116-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18104 From: Richard Purdie It is useful for bitbake-setup to support local paths without access through the fetcher so that internal data to the bitbake repository can be used as a default. Signed-off-by: Richard Purdie --- bin/bitbake-setup | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index d6509500d..b07bf2eb7 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -509,10 +509,15 @@ def do_fetch(fetcher, dir): def update_registry(registry, cachedir, d): registrydir = 'configurations' - full_registrydir = os.path.join(cachedir, registrydir) - print("Fetching configuration registry\n {}\ninto\n {}".format(registry, full_registrydir)) - fetcher = bb.fetch.Fetch(["{};destsuffix={}".format(registry, registrydir)], d) - do_fetch(fetcher, cachedir) + if registry.startswith("."): + full_registrydir = os.path.join(os.getcwd(), registry, registrydir) + elif registry.startswith("/"): + full_registrydir = os.path.join(registry, registrydir) + else: + full_registrydir = os.path.join(cachedir, registrydir) + print("Fetching configuration registry\n {}\ninto\n {}".format(registry, full_registrydir)) + fetcher = bb.fetch.Fetch(["{};destsuffix={}".format(registry, registrydir)], d) + do_fetch(fetcher, cachedir) return full_registrydir def has_expired(expiry_date): From patchwork Mon Sep 29 12:56:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71222 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60D9ACCA470 for ; Mon, 29 Sep 2025 12:56:45 +0000 (UTC) Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) by mx.groups.io with SMTP id smtpd.web10.53024.1759150597669412611 for ; Mon, 29 Sep 2025 05:56:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=g4dekBcW; spf=pass (domain: gmail.com, ip: 209.85.208.53, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f53.google.com with SMTP id 4fb4d7f45d1cf-62fa0653cd1so6686652a12.0 for ; Mon, 29 Sep 2025 05:56:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150596; x=1759755396; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NqX9Jd4yw5qApHVpTfwlMm+7dpLTzqZIG/l2k9o0cgA=; b=g4dekBcW3Ibh5Oi1TojeIEME1npiz+To8USQWCm/4C+ByqCZed/npD8mH6YZI3z7fA 2CF9eyBEZWSnpOP8iMJM0yioamLKrr/1BYgnywL9JqY3eNTwHL7DKimST084kbaf6lPP y/v1M5FDVTEixDIbsUd475ifMtkbaof1L0Ht10bSuaUPGuK3+n8UWKyloybSkpGySMnh v4pjYTe1FbB2a5jCzzfHcSbtxHWchY55oZ+oFxa2SgGv8OKgTSE4DdhsluQzskoCHld3 ZV5ozyuOCr3t9Kmx//mim+u8T8C2ZQulSS7QxpUI0U08Md4iRMG+rnjupHSrgxRgN3pc r/wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150596; x=1759755396; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NqX9Jd4yw5qApHVpTfwlMm+7dpLTzqZIG/l2k9o0cgA=; b=qSjUx6OLEP3LgwCwkDKvpjmfaTx0iUJ20dxg7HSPyo8KuChdMs45XQZKPR/tc1ZGII Klusvd85DU2u2hnLsXpzphsagj24VXreaaFiMZLjBsQ3FtRVQt45VffnuI4DIgZLFO0k FL9I3sME93n+ZMUqv+vAqar59D9q66BsVa+yhNRnWFAkWDb7wqDu54zp7Rk5oiKyhs3b QZGbji4QCSsqQ21wA4kqFTM9qHnBa/silIPi6tgEzzwNga8VTP2w4S4LHtot3rgDDCKc BXCiF4FLejoR0kUyXlcG/Je6GspuuzmDUNQgOcWrn3SiCgeciMwUeOY8C2OPhNZVcc18 LqpQ== X-Gm-Message-State: AOJu0Yzmusy6+VgstPmWzP9MfD5Cs7dcn99UjTM5SV+J6liZqf3/ozGI ftldyLg7fAI7+Oo/b6/L4klJEAPL5C+BRVLa0YHbMuJ057AbdFtwnn8xeNB+0A== X-Gm-Gg: ASbGncs5DApsC5mzHau7la/3KQt3SajTnmOQiAtlBQ4SuqMndxmsN5Nkh359bOgNTe/ JOZN+5k5o7dEWn46BwhA/xo3EiiJLrWzmuXvl1T+03yKvgEcNmWf1TebaGNz9XAbVDhKckpunIc TmVUTAoXVqDDWaugobV0+JnyVfMfucGW8Zb11jUZ1pJWGDJxLQUO++YtEBed4YCg0PDdBXVFPMq rpa0fhdwCKnT5TAKgs+hGOEltlFNNhJIxuaVn1OVLMsYb6pqBy/zKuqvDSxXjAARfiOGyutZm19 rell8xSdU9uSEvigI1MeibU7VlXAhLlDcbW7uMTmGGXoZS7do1ogat7eQdAIulu5yM2rTzqm1Bn OUxuYthaPX6Y1jfPDarjsG0L4nNJKKA/R7vTb5WjC83hFIzapMr7vzQ9MV3LqOB0eZcNXDgzMai 1ADB6OJdSS9bL5J/jfTBhJ7yZtI4e9oYxO/hSm7KT47Gkldl0CR2KcuugHkbsiHMtWbhUl X-Google-Smtp-Source: AGHT+IFfY+yod0d7DYkEL1U41llCjg+DWpdxM/LUK4FPn4Fmsl6Sv1xun/UzLWIkwZgTA91B2x27Jw== X-Received: by 2002:aa7:c708:0:b0:633:d65a:af16 with SMTP id 4fb4d7f45d1cf-6349f9ee8e7mr10927898a12.9.1759150595952; Mon, 29 Sep 2025 05:56:35 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:35 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Richard Purdie Subject: [PATCH 06/10] Add default registry config for OpenEmbedded's nodistro and Yocto Project's poky Date: Mon, 29 Sep 2025 14:56:12 +0200 Message-Id: <20250929125616.1751116-6-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250929125616.1751116-1-alex.kanavin@gmail.com> References: <20250929125616.1751116-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18105 From: Richard Purdie Rather than forcing every user to have to access a separate repository for data for some key defaults, add those for nodistro and poky. This gives us some commonly used setups and also something to test against too. We will need to come up with a criteria for adding these, most likely based on community usage/demand with the OE TSC governing that policy. Signed-off-by: Richard Purdie --- .../configurations/oe-nodistro.conf.json | 54 ++++++++++++++ .../configurations/poky-master.conf.json | 70 +++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100644 default-registry/configurations/oe-nodistro.conf.json create mode 100644 default-registry/configurations/poky-master.conf.json diff --git a/default-registry/configurations/oe-nodistro.conf.json b/default-registry/configurations/oe-nodistro.conf.json new file mode 100644 index 000000000..f4a7492f3 --- /dev/null +++ b/default-registry/configurations/oe-nodistro.conf.json @@ -0,0 +1,54 @@ +{ + "description": "OpenEmbedded - 'nodistro' basic configuration", + "sources": { + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "git://git.openembedded.org/bitbake;protocol=https" + } + }, + "rev": "master" + }, + "path": "bitbake" + }, + "openembedded-core": { + "git-remote": { + "remotes": { + "origin": { + "uri": "git://git.openembedded.org/openembedded-core;protocol=https" + } + }, + "rev": "master" + }, + "path": "openembedded-core" + }, + "yocto-docs": { + "git-remote": { + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/yocto-docs;protocol=https" + } + }, + "rev": "master" + }, + "path": "yocto-docs" + } + }, + "bitbake-setup": { + "configurations": [ + { + "name": "nodistro", + "description": "OpenEmbedded 'nodistro'", + "bb-layers": ["openembedded-core/meta"], + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64"] + } + } + } + ] + }, + "version": "1.0" +} diff --git a/default-registry/configurations/poky-master.conf.json b/default-registry/configurations/poky-master.conf.json new file mode 100644 index 000000000..227816983 --- /dev/null +++ b/default-registry/configurations/poky-master.conf.json @@ -0,0 +1,70 @@ +{ + "description": "Poky - The Yocto Project testing distribution configurations and hardware test platforms", + "sources": { + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "git://git.openembedded.org/bitbake;protocol=https" + } + }, + "rev": "master" + }, + "path": "bitbake" + }, + "openembedded-core": { + "git-remote": { + "remotes": { + "origin": { + "uri": "git://git.openembedded.org/openembedded-core;protocol=https" + } + }, + "rev": "master" + }, + "path": "openembedded-core" + }, + "meta-yocto": { + "git-remote": { + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/meta-yocto;protocol=https" + } + }, + "rev": "master" + }, + "path": "meta-yocto" + }, + "yocto-docs": { + "git-remote": { + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/yocto-docs;protocol=https" + } + }, + "rev": "master" + }, + "path": "yocto-docs" + } + }, + "bitbake-setup": { + "configurations": [ + { + "name": "poky", + "description": "Poky - The Yocto Project testing distribution", + "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"], + "oe-fragments": ["core/yocto/sstate-mirror-cdn"], + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64", "machine/genericarm64", "machine/genericx86-64"] + }, + "distro": { + "description": "Distribution configuration variants", + "options" : ["distro/poky", "distro/poky-altcfg", "distro/poky-tiny"] + } + } + } + ] + }, + "version": "1.0" +} From patchwork Mon Sep 29 12:56:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71218 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5315FCCA46F for ; Mon, 29 Sep 2025 12:56:45 +0000 (UTC) Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by mx.groups.io with SMTP id smtpd.web11.52699.1759150598410649393 for ; Mon, 29 Sep 2025 05:56:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=bBJWYHIc; spf=pass (domain: gmail.com, ip: 209.85.208.41, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-62fb48315ddso8382676a12.2 for ; Mon, 29 Sep 2025 05:56:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150597; x=1759755397; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qX9xxXcci50IbKX9GSpABXq+hnqeOEo8lXky4qGze2A=; b=bBJWYHIcbztPbR1G9bHbwhyM+uF2FOAN9DBnx+eBs/xfjPNkUBix98s8YX/K8Fg2bO TPFHVL0lup4FzAP4qByli6juDuXLHI38FQZ1lzJ27sSNQn/8OGMy+UVzFGGLTSCXyQcd 60L1lV4UGhvKQeVDoPKduKsqSaPErG+qUvA/rORWlAMYNGN/+69wJQfZxVEoBbBCRUEP f+mJy2CyJbIiwaRSP15Y7tuJz4MR7IpJSnpdKKOXauQDFTcyY/s3eY686YJ1TS6Vpxxe oAYyWpVa+9c3SWpjpGcR7A0oBZ1KTbs2co/6umKtT+9eEboy9I8LT99ZBHTnK11BPP5A 2eaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150597; x=1759755397; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qX9xxXcci50IbKX9GSpABXq+hnqeOEo8lXky4qGze2A=; b=iGGiYDbes6137RPBvdDq9ubzimr425lYX++zXoR+eKmFLDauQk/47jqGLMRGcCJvEI hB9YMl915g2uIpKWyYn5Bd9DJOurmXaxtJ3P/wC7tKdglbwt3UW8tbd2nySP1mlO+jy4 AVdxhQnH1yxFWQ3KxUMCjcFZPlT6C1f0Ba9vb4DycPkKMZMGpE9I4R/3iliDmw/zWOa2 igawzJO34+2cRo2d/tBeoQVpqjzyNdQXoRZZpSm0ZJoA8JeJmsmpa1hmeVLe9/57Kdms /E5fNJ54/YfIT+XrRjHQ3NVMTpDPTfE/7T4uB+Ff+UfifuHXaQF1ILY+iiwHNu4B87/e BHUg== X-Gm-Message-State: AOJu0YyKAAfF1lHkPejfiVwvCcNsYMUH+j0LkReGANCK0JrH/oLrFF79 bYvGFDRbLYOSohuBTnh4XFQXRM7cFDUgKZSBYHH0mEmxN/iZVvwtwjxMDP2v2Q== X-Gm-Gg: ASbGnctbX/0jhLecY5ron1HdjH8UxFo75jJ7eGGVCuIbzhQ05/qbcr6XjRhSfc5va7H Acm5ay4VWjwQvPn/KOTFryWoB0Q8mxZhxAA+ZgGBohD31EOTpiMZh9lzdoaAxNtqKn+BKf1s0dl otcD57GinDDQzM+kUK/GveisADVaq8+3nbzYIoOo2tIrclqxA9xCbgv7I64KIUJ20YHijSP8PX/ BXzJ8FYpp2yb6nyK8Fr5H60wb7so/EKjAtSKfQeZNggZfT0+b3Mv3uk8IR2xZwDPZR5L4ivLNmp 1GxCjZhc4n2BOHm+K64vLHUbs4gg63vI2ACPAlSSV0a/mBRaAz7paehyFnRNWCHwu7M++hKQbxb bSNjxSq31F7dZoxIU9o0sVYvQiTQW6UoWDoKCSRhPB4SJzcHBWgyOMjHG0Wo4AqOhBFSSMIrtmQ r0yrEBme8NAaJ8qx3E0KkYADqc6DMophBO+Hvi3kLoL8Wjt0MKIm+MjJrAG0PTQL3F6LPw X-Google-Smtp-Source: AGHT+IGXAZ7wgLD+Qcsn+BhqArWra5qCF92tB5Idr5fWq3Q51/YMw0oqcGmtRmIWpxQrHVQCYo5nrA== X-Received: by 2002:a05:6402:42ca:b0:634:b3d1:8f91 with SMTP id 4fb4d7f45d1cf-634b3d19105mr13396369a12.0.1759150596718; Mon, 29 Sep 2025 05:56:36 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:36 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Richard Purdie Subject: [PATCH 07/10] bitbake-setup: Switch to internal default registry files Date: Mon, 29 Sep 2025 14:56:13 +0200 Message-Id: <20250929125616.1751116-7-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250929125616.1751116-1-alex.kanavin@gmail.com> References: <20250929125616.1751116-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18106 From: Richard Purdie Switch the url to be the default internal registry rather than a private repo which was intended for testing. Signed-off-by: Richard Purdie --- bin/bitbake-setup | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index b07bf2eb7..7067cbd4f 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -19,7 +19,7 @@ import datetime import glob import subprocess -default_registry = 'git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main' +default_registry = os.path.normpath(os.path.dirname(__file__) + "/../default-registry") bindir = os.path.abspath(os.path.dirname(__file__)) sys.path[0:0] = [os.path.join(os.path.dirname(bindir), 'lib')] From patchwork Mon Sep 29 12:56:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DC4CCCA472 for ; Mon, 29 Sep 2025 12:56:45 +0000 (UTC) Received: from mail-ed1-f49.google.com (mail-ed1-f49.google.com [209.85.208.49]) by mx.groups.io with SMTP id smtpd.web11.52700.1759150599173280629 for ; Mon, 29 Sep 2025 05:56:39 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Yg/t5KPc; spf=pass (domain: gmail.com, ip: 209.85.208.49, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f49.google.com with SMTP id 4fb4d7f45d1cf-6349e3578adso8731068a12.1 for ; Mon, 29 Sep 2025 05:56:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150597; x=1759755397; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=S/+KkXj+dTvYxpn2pz9NNtOP6iLajjLfsm5Kk4HCxXg=; b=Yg/t5KPcs6zvWKz7/yjY4oQVZbZJdtFIwSL6PwNHaBVzFglZrZtMiuurwjPIQg5kY+ YXU+r0agU7fAXMImD5D1gIMH1O7xlu0AASbca4afn76iP0LFSkC1Dr2mffXuIYUDqJQV KmS3GdL5fkyDvKwvsLEy/KDPTDgIhUVJO58XRT3Mz/c6HaSq1VKUC3xY1AZ3/4CqHuCl ciUOVMNO7lOyRA8PRWUlWOjODh4/6RUh8+gh7ajwFtneYHBWH0rO/tn4Mrsvv2YsOGGM MtbC9eZzbjje/aR3OWbLOaIPNlQNkhPkBcNhul6SogseiYCEiVmud1/x34hMG9naftha x43w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150597; x=1759755397; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S/+KkXj+dTvYxpn2pz9NNtOP6iLajjLfsm5Kk4HCxXg=; b=KeUWS4tRoFLlp9DnhZFFjVJmCL8hZQMbUcr5ZXZpx+TOcwuTsrKS8h6L2f6g4fAXz3 6OhEGRYVRTfiln/yYeZOtEoYQiK2mg8py+zh7CZQOCgrUvDsroLDf4U2GQnu521ll3Hv XWunTm50GX2KWFgqiUBeR2ZBnQAyaWaY9RQpp2e0HwRiYwPlwvCB3EJL6kE3KSIElcnq ZwiV1uxXP4jEEGD+ASFbH/BkNp4Lodia5VFHgYRFA1BtJ72pz1x/kXxFxnnxz3rdqNRf PgM2pBcgqymNltsjKr88JGP1YvGvTPfAmRIw03dskFBtBz+KPf150hZAzs4YJFiWgwqw vmSQ== X-Gm-Message-State: AOJu0YzOGLLqwTytpHNSLCkGdShD1d8yRW2DYhNBUrqvfZaOut2PyFEJ 4xBbKnvh1WGuMLHR0MHPpKzYuVnHt8zskWA8HyYz2qKxb0TGej1/tn0GKL6Rmg== X-Gm-Gg: ASbGnctBSkAuNbTgwM1iN89vrvTO8PLe5IbWntA/fNDKb7j1oLwes4K8y5LJ0uHWLbA T2uD4s4IXlrSNO2Z2LKSQEWSkIinIAbksLw9pdmIAIx7NkjXIBQLGx8FEb9F0a2OxvG/ARg52B7 DAQX4dfR2HmnzfPpyxn7lDuFgnIHcyjyXOfNoWGm0KQIslR1lqKV09oNIu+oUTMgUbNNv88/vAJ J7yPdx7sD4PUKraHHZeeNJj36BUo3q9CuNBsiGhNF9wx+kQjgM75Tmp7cxk2wpchxBkEcXgPmuT a9TtUIfYZhV/iYh5B9JEHcz3t/g7WAH7En3UfyEt6O1sDqw5QGvA6sEF3HlId87zkmX1/Ory2Nc jkHN5+BPdcbI7P/UUSLh1lw+0o9BXmZeJxGtXNPKTcPbPN9Ul1WKD8uSVe2nxljdgqDGhUh/s3l pte4LcEoF13KP1d0ehDoyxS4FQm7RDn416z23csmdWPJqsvB2lHMAxUeawQE7IFnSGz19U X-Google-Smtp-Source: AGHT+IFOQY01HC5a0dxf5KJ5eHwCtGSickIdM1lPGOZI/26i+XriEvJP+dsiYIcWgIW76SSE89Q5Sg== X-Received: by 2002:a05:6402:42ce:b0:62e:e5b3:6388 with SMTP id 4fb4d7f45d1cf-6349fa81093mr16911305a12.19.1759150597481; Mon, 29 Sep 2025 05:56:37 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:37 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Richard Purdie Subject: [PATCH 08/10] bitbake-setup: Improve the already initialized test Date: Mon, 29 Sep 2025 14:56:14 +0200 Message-Id: <20250929125616.1751116-8-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250929125616.1751116-1-alex.kanavin@gmail.com> References: <20250929125616.1751116-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18107 From: Richard Purdie If the directory already exists but hasn't been setup, the current test can fail so improve it. Signed-off-by: Richard Purdie --- bin/bitbake-setup | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index 7067cbd4f..ce5cd5e65 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -393,7 +393,7 @@ def init_config(settings, args, d): print("\nRun 'bitbake-setup init --non-interactive {}' to select this configuration non-interactively.\n".format(" ".join(upstream_config['non-interactive-cmdline-options']))) builddir = os.path.join(os.path.abspath(args.top_dir), args.build_dir_name or "{}-{}".format(upstream_config['name']," ".join(upstream_config['non-interactive-cmdline-options'][1:]).replace(" ","-").replace("/","_"))) - if os.path.exists(builddir): + if os.path.exists(os.path.join(builddir, "layers")): print("Build already initialized in {}\nUse 'bitbake-setup status' to check if it needs to be updated or 'bitbake-setup update' to perform the update.".format(builddir)) return @@ -404,7 +404,7 @@ def init_config(settings, args, d): exit() print() - os.makedirs(builddir) + os.makedirs(builddir, exist_ok=True) confdir = os.path.join(builddir, "config") layerdir = os.path.join(builddir, "layers") From patchwork Mon Sep 29 12:56:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DB0BCAC5B0 for ; Mon, 29 Sep 2025 12:56:45 +0000 (UTC) Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) by mx.groups.io with SMTP id smtpd.web11.52701.1759150599963781742 for ; Mon, 29 Sep 2025 05:56:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Xs7Ed1+Y; spf=pass (domain: gmail.com, ip: 209.85.208.50, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-636255b92c9so3514026a12.3 for ; Mon, 29 Sep 2025 05:56:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150598; x=1759755398; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o3yaHIV3BUBaQPrBuPfnjVaoB0do0RoaArd8zBuUTgA=; b=Xs7Ed1+YVoaveaxuZaJ/joh8Rwq0u0sraU63wZjZawx6iurzQjz61R1UFcEN5O3ylC fOR0aFzWIGDnInuuT8bUTEInOVSCxsoHyNtLmQahvuSorHuxLcwbJ35xQNtKWPlNHEab UH5F8hzijBfi4lEixpWqa3YpVWed0h+XxZWNXwrbYladQ92dj+tzEMzfEqXs9NbXcQtS psiIKEJQ3KO4fco1B93+pESisNvBynJ+Jw7K84vZ+aIxKzcogFa+v3U9awgH/0GmA73M jQ6YaI1M4kvfIeSOpbl5wvt2m9XYiLTE+WZllhjfcCSpunSdjLk+UMWu9hj3OUxzDG6E pXVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150598; x=1759755398; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o3yaHIV3BUBaQPrBuPfnjVaoB0do0RoaArd8zBuUTgA=; b=F6ASWkd33gszTsut11m/IclBAt9i8050tykD9JiIe6XK2kCLvS5RjSGjjC3G7eUJwJ pwNFgiOP+xjjUtfSDDsWoVt2w8DAU2iDA1LIqEf4Pn+J4PQNFzzksiduszHaO0o3ugkD gAcKAWlaSj+q/6KtaseXDiO/ORB479IHwZoBEBsQsAHmxS5w+/FWb0D6O7+al2vmuUx8 +bUOcu+8/T+vEW/mTodX61pMAKHriNjPaoip+PrLlyp9A2rZM9xWZOkFlrbnu805pK+u 30bZvH6I1ETCIJHIu+ZNDC5fbI+SuB41fN9fOAtQkP61SCUWCyHXnL4nNx/OcjMo+0tn vO4A== X-Gm-Message-State: AOJu0YzEh151VSTGolSQMZqz6dvzIrmFYtttH3FI2Dwkdc5kvyCRSThw 2HkvJSxC5lnCvx+wShNlezsneOiPeSmCcm6GsMotz5lhwbenqCLgdnarvt8qig== X-Gm-Gg: ASbGncsAOKiwBwzLfHp4/PHUB5Kr0EZoSQD3TioYZZCrp9xVjl6kagKixg1Jvjl612m Jh4+4JhNMKeuBYqUynTlxmzc08luEH4n8mjLz+VCOMAxiKYng2YAB4szNs2lbwU5W11aapU6PWw w90JslMLmLhKrWKWIVUk7HIikNfogEBJf2kcPorojrZwPzlR/7kw7d1pMktfCxinDC9UKWuM3nZ +z3D7aSHKcx+zUsV4Mpn6aEX2jkIiJeJfxCTn8ew+95ZOQYrsgUQC3QfajVfZ0/BnTn7OAmBjNQ ohGBLJh4daL5e1lcrTf1WnQ1b6+5Wxiv5GHermdQRoeXjh2FFh0l19/nFZFn4PS4Bob7Bchjz9+ ++OZj2sF7iOmdjFl7R+IVOKFIML7psfNZ6NwLh0btsauJrn5+Jlx72RU8KLgXJMPqiPIBzB1BQy R4l/9FHqriHmXUy0x0e3nUoyv6tfGgxzdxN0UOU9whhrHNCnmjcc5KpgHHqSzvN/yX9XE+7+VkC /bMDyI= X-Google-Smtp-Source: AGHT+IEkle3Aedn8bDPfM1pY+gT60i1GGrDtnwwhXFWz4lhrlTzQc7O4Vpi2GjatmMtuPbCsPor0UQ== X-Received: by 2002:a05:6402:274c:b0:634:a992:a2 with SMTP id 4fb4d7f45d1cf-634a99206c6mr14634515a12.1.1759150598194; Mon, 29 Sep 2025 05:56:38 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:37 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 09/10] bitbake-setup: add support for specifying branches in repo checkouts Date: Mon, 29 Sep 2025 14:56:15 +0200 Message-Id: <20250929125616.1751116-9-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250929125616.1751116-1-alex.kanavin@gmail.com> References: <20250929125616.1751116-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18108 From: Alexander Kanavin Previously bitbake-setup was checking out 'detached commits' using fetcher's nobranch feature, as that is the only option when only a revision is in the config. Branches are optional, but beneficial, as - checkout directory will be on a branch, making it easier for users to understand where they are if they need to make changes (also bitbake will print branch information instead of saying 'HEAD:sha'). - supply chain security! Enforcing a branch means any specified revision has to be on it, and no one can sneak in (accidentally or deliberately!) some dangling commit, or something from their private branch in the same repo. Signed-off-by: Alexander Kanavin --- bin/bitbake-setup | 12 ++++++++++-- .../configurations/oe-nodistro.conf.json | 3 +++ .../configurations/poky-master.conf.json | 4 ++++ lib/bb/tests/setup.py | 11 ++++++----- 4 files changed, 23 insertions(+), 7 deletions(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index ce5cd5e65..8ceacada9 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -86,13 +86,17 @@ def checkout_layers(layers, layerdir, d): r_remote = r_data['git-remote'] rev = r_remote['rev'] + branch = r_remote.get('branch', None) remotes = r_remote['remotes'] for remote in remotes: type,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) print(" {}".format(r_name)) - fetcher = bb.fetch.Fetch(["{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir)], d) + if branch: + fetcher = bb.fetch.Fetch(["{};protocol={};rev={};branch={};destsuffix={}".format(fetchuri,type,rev,branch,repodir)], d) + else: + fetcher = bb.fetch.Fetch(["{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir)], d) do_fetch(fetcher, layerdir) if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')): @@ -444,12 +448,16 @@ def are_layers_changed(layers, layerdir, d): r_remote = r_data['git-remote'] rev = r_remote['rev'] + branch = r_remote.get('branch', None) remotes = r_remote['remotes'] for remote in remotes: type,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) - fetcher = bb.fetch.FetchData("{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir), d) + if branch: + fetcher = bb.fetch.FetchData("{};protocol={};rev={};branch={};destsuffix={}".format(fetchuri,type,rev,branch,repodir), d) + else: + fetcher = bb.fetch.FetchData("{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir), d) upstream_revision = fetcher.method.latest_revision(fetcher, d, 'default') rev_parse_result = bb.process.run('git -C {} rev-parse HEAD'.format(os.path.join(layerdir, repodir))) local_revision = rev_parse_result[0].strip() diff --git a/default-registry/configurations/oe-nodistro.conf.json b/default-registry/configurations/oe-nodistro.conf.json index f4a7492f3..7619738b1 100644 --- a/default-registry/configurations/oe-nodistro.conf.json +++ b/default-registry/configurations/oe-nodistro.conf.json @@ -8,6 +8,7 @@ "uri": "git://git.openembedded.org/bitbake;protocol=https" } }, + "branch": "master", "rev": "master" }, "path": "bitbake" @@ -19,6 +20,7 @@ "uri": "git://git.openembedded.org/openembedded-core;protocol=https" } }, + "branch": "master", "rev": "master" }, "path": "openembedded-core" @@ -30,6 +32,7 @@ "uri": "git://git.yoctoproject.org/yocto-docs;protocol=https" } }, + "branch": "master", "rev": "master" }, "path": "yocto-docs" diff --git a/default-registry/configurations/poky-master.conf.json b/default-registry/configurations/poky-master.conf.json index 227816983..3f1299179 100644 --- a/default-registry/configurations/poky-master.conf.json +++ b/default-registry/configurations/poky-master.conf.json @@ -8,6 +8,7 @@ "uri": "git://git.openembedded.org/bitbake;protocol=https" } }, + "branch": "master", "rev": "master" }, "path": "bitbake" @@ -19,6 +20,7 @@ "uri": "git://git.openembedded.org/openembedded-core;protocol=https" } }, + "branch": "master", "rev": "master" }, "path": "openembedded-core" @@ -30,6 +32,7 @@ "uri": "git://git.yoctoproject.org/meta-yocto;protocol=https" } }, + "branch": "master", "rev": "master" }, "path": "meta-yocto" @@ -41,6 +44,7 @@ "uri": "git://git.yoctoproject.org/yocto-docs;protocol=https" } }, + "branch": "master", "rev": "master" }, "path": "yocto-docs" diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index fb2c15f54..329a0c525 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -87,7 +87,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) bbsetup = os.path.abspath(os.path.dirname(__file__) + "/../../../bin/bitbake-setup") return bb.process.run("{} --global-settings {} {}".format(bbsetup, os.path.join(self.tempdir, 'global-config'), cmd)) - def add_json_config_to_registry(self, name, rev): + def add_json_config_to_registry(self, name, rev, branch): config = """ { "sources": { @@ -98,6 +98,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) "uri": "file://%s" } }, + "branch": "%s", "rev": "%s" }, "path": "test-repo" @@ -134,7 +135,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) }, "version": "1.0" } -""" % (self.testrepopath, rev) +""" % (self.testrepopath, branch, rev) os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True) with open(os.path.join(self.registrypath, name), 'w') as f: f.write(config) @@ -208,12 +209,12 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) self.assertNotIn("test-config-1", out[0]) self.assertNotIn("test-config-2", out[0]) - json_1 = self.add_json_config_to_registry('test-config-1.conf.json', 'master') + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', 'master', 'master') out = self.runbbsetup("list") self.assertIn("test-config-1", out[0]) self.assertNotIn("test-config-2", out[0]) - json_2 = self.add_json_config_to_registry('config-2/test-config-2.conf.json', 'master') + json_2 = self.add_json_config_to_registry('config-2/test-config-2.conf.json', 'master', 'master') out = self.runbbsetup("list --write-json={}".format(os.path.join(self.tempdir, "test-configs.json"))) self.assertIn("test-config-1", out[0]) self.assertIn("test-config-2", out[0]) @@ -270,7 +271,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) branch = "another-branch" self.git('checkout -b {}'.format(branch), cwd=self.testrepopath) self.add_file_to_testrepo('test-file', test_file_content) - json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch) + json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch, branch) for c in ('gadget','gizmo','gadget-notemplate','gizmo-notemplate'): buildpath = os.path.join(self.tempdir, 'bitbake-builds', 'test-config-1-{}'.format(c)) os.environ['BBPATH'] = os.path.join(buildpath, 'build') From patchwork Mon Sep 29 12:56:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39BF9CAC5BE for ; Mon, 29 Sep 2025 12:56:45 +0000 (UTC) Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) by mx.groups.io with SMTP id smtpd.web11.52702.1759150600613969950 for ; Mon, 29 Sep 2025 05:56:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=GCad9kQw; spf=pass (domain: gmail.com, ip: 209.85.208.44, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-634cdb5ed4bso4910945a12.2 for ; Mon, 29 Sep 2025 05:56:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759150599; x=1759755399; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B+TZO3Ots/Smbk4S3CECLp+WN8bxQf4cC8blJNg/c2A=; b=GCad9kQwywZNtlN+5zlKG8sahY3WqvilhekeQqmJvVpjeaNiUKmNW+AmESicfpc2+q BqkOKBed7hDsDBOsBjLPCGppOoG7dMdaggzylZMehrByGDgAzxmj/KzHHXZIbRv8ZKC6 B7rPbv3Ixd+fKS3NvCnbMw+3B3jjCFnPSJ0JXHWivQKp9uMuL/7M6xlfHl/iTFVyYp4I KtmIna7gR6pVnPdKsZ70y+ue9enTz0UixXxZuGTLxCLgf4DrpHfXc8P2oU6MLdexsMB2 MWHDkC6Ao4tq3usIOttPkuiwMpQdUBRQpeghQd8ecFn1kYDQnockiEBN+O8tV6MGTfV5 Inmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759150599; x=1759755399; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B+TZO3Ots/Smbk4S3CECLp+WN8bxQf4cC8blJNg/c2A=; b=m/okQMURra9S9l/c5iM0rY4gkPtnX7Tdm6NC+H7qNc8fHvLFHABXZ094V1MvrE6gGv s3E+0xPPWIhMC3un+K/gkR/htTspAtV9aY6T1xBIe2X0/FmULTYrtpiZjnY5XprAHxpb ym3jbRuedQQcWSgfMf/fFAUIUk9ftua3LjEnkob+lFojhWduvoV4dBqeGpgtV8lLl4H5 XewOuHKWitTPF609fRTGAi9q+pBU0oSxylB+9SNOMS87k2z3TK0jOHG56LaXt5BNiHMk PiKZYFC5p24sgmDFcctdAZ9C39ZFaeYC5TLuqxsagABKZdjaFkAz26PFhmllHlPAkyqi dPfA== X-Gm-Message-State: AOJu0Yxld1xBAfzD0T8Yk+jm0TJqK0MghG+lO0M9kIsu+zzpcUskg15E CeUaHiqQg3Vd5aDzggDpTWDfzauDA75pjmxN66HMq4wMNPalm49zpqVmOMuJUg== X-Gm-Gg: ASbGncsMoJbGh39MRpHNI+hGnGbHP/vW3+L/LG6yFyfr6uBrbplU7GfhEAoES8cya5n DYon5YUdms9Ou9qqVGL7osN5g6QSxZYNXbYls6fgK8/HrH6rmwGjisczz9a/K0T7cy2lAoCFtms iBJtI0CbFL70tWfdPxJGhWzVqQlu6zzcl5L3jaMYR51UD/MstymUJ/4fKddVtx/NsV54EqNGE7R 6F3af8Bk1tWRJczb8mV24c2ZxZhU1060D7QHQARJ/O4rhONXssH1fkDE5DTKqdhz9RmT3IDYMr2 vFzvdPnhlXJWDmmpGoqV61+VKsKbYouDJ0TfiMgHxJ0ZcZ8cfFDT5shOx0xMAQ1oy2IyQr+PT08 LAX7+TEzj1AzX1QGFz24psEQTmeVhKgt8cXS4OwQaB9FwQCTl/sVYt65eNYN9fnBIAuOj/Wbs6z /QOTcoRLvgUv3poBwQuyd5Ge3glG/bO/Rkg32QoYjkPcfPQ/C1jgHnXbouW0F4pDcot7Na X-Google-Smtp-Source: AGHT+IEDHmtnXQCFIQ1auUg0G/JN1CnSZoCWoWjsQgzcZqFPaRSsk05EJbhjoyETO3Vwio+ItQ++RA== X-Received: by 2002:aa7:d603:0:b0:632:67ba:f9af with SMTP id 4fb4d7f45d1cf-6349fa08f1amr11665626a12.12.1759150598881; Mon, 29 Sep 2025 05:56:38 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-634b46dca59sm6575915a12.8.2025.09.29.05.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Sep 2025 05:56:38 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 10/10] bitbake-setup: improve robustness of loading/writing settings Date: Mon, 29 Sep 2025 14:56:16 +0200 Message-Id: <20250929125616.1751116-10-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250929125616.1751116-1-alex.kanavin@gmail.com> References: <20250929125616.1751116-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 29 Sep 2025 12:56:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18109 From: Alexander Kanavin Particularly: - ensure global settings command line argument is always expanded to full path - ensure any errors that happen when loading settings are reported at that point, otherwise we get an empty dictionary and cryptic key errors later Signed-off-by: Alexander Kanavin --- bin/bitbake-setup | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index 8ceacada9..1fbb65494 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -635,11 +635,11 @@ def load_settings(top_dir, non_interactive): settings_path = default_settings_path(top_dir) settings = configparser.ConfigParser() print('Loading settings from\n {}\n'.format(settings_path)) - settings.read([settings_path]) + settings.read_file(open(settings_path)) return settings def global_settings_path(args): - return args.global_settings if args.global_settings else os.path.join(os.path.expanduser('~'), '.config', 'bitbake-setup', 'config') + return os.path.abspath(args.global_settings) if args.global_settings else os.path.join(os.path.expanduser('~'), '.config', 'bitbake-setup', 'config') def write_global_settings(settings_path, force_replace, non_interactive=True): if not os.path.exists(settings_path) or force_replace: @@ -673,7 +673,7 @@ def load_global_settings(settings_path, non_interactive): settings = configparser.ConfigParser() print('Loading global settings from\n {}\n'.format(settings_path)) - settings.read([settings_path]) + settings.read_file(open(settings_path)) return settings def change_settings(top_dir, new_settings):