| Message ID | 20250813151523.1855287-1-alex.kanavin@gmail.com |
|---|---|
| State | Accepted, archived |
| Commit | b96154aeb1fc89184ac245e0d68e6e726fe80c04 |
| Headers | show |
| Series | [1/3] bitbake-setup: add the initial implementation | expand |
On Wed, 2025-08-13 at 17:15 +0200, Alexander Kanavin via lists.openembedded.org wrote: > From: Alexander Kanavin <alex@linutronix.de> > > Preamble > ======== > > The latest iteration of this patchset is available at > https://github.com/kanavin/bitbake > I recommend taking the patches from there to ensure that > you are not trying out outdated code. > > For the rationale and design guidelines please see this message: > https://lists.openembedded.org/g/openembedded-architecture/message/1913 > > Left out for now but will be done later: > > - handling site-specific settings. There was a discussion about it: > > ===== > > Finally, a question, how do you recommend to handle my local SSTATE_DIR/DL_DIR? > > Setup with bbsetup, copy a site.conf with these config inside the build, then > > the proper build? ... or is there a better way? > > This is an unresolved question. I'm open to proposals. Or code. > > I guess it will be something like 'bitbake-setup change-setting > default site-conf /path/to/site.conf', and then bitbake-setup will add > that to every build that is initialized. Site.conf need not list site > settings directly, it too can add a fragment maintained in a layer. > That way CI settings can be specified and maintained in a layer, but > they don't need to be added to a json config, as that would impose > them on everyone using that config to reproduce builds locally. Thanks for continuing to work on this, I like it a lot and it feels a lot better than the previous version. From a "what it is doing" standpoint, I think we're getting there. Where it still needs work is in the "telling the user what it is doing" part. I say this as someone who knows bitbake/OE quite well and I still had to look at the files on disk to work out where things are. A big chunk of this is actually whitespace and formatting on the screen. Putting paths onto newlines with indentation will help readability, as will some blank newlines to separate out things. I think we should probably be "properly" integrating into the user's HOME, i.e. create ~/.config/bitbake/bitbake-setup.conf and ~/.cache/bitbake/downloads by default out the box too, we may as well make bitbake appear a "proper" citizen if used this way. Let me describe what I think it needs to "say" to a new user and see what you think. I suspect the interaction should look something like: ''' $ bitbake-setup init No previous bitbake setup configuration detected. Configure bitbake-setup in /home/XXXX/.config/bitbake y/n? Configured a common source downloads cache (DL_DIR) at XXXX. A common site.conf file to use in builds can be created at: /home/XXXX/.config/bitbake/site.conf <symlink this into a builds site.conf position?> No configuration registry specified, using the default of: XXXX Fetching configuration registry into: YYYY Configure a build at XXXX ? <let the user change/confirm the path> Available configurations: 0. yocto-master-nested-configs (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 (defined with nested configurations) 1. yocto-master-options (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined with options) Please select one of the above configurations by its number: 1 Selecting the only available bitbake configuration poky <Add blank newline> Available target machines: 0. machine/qemux86-64 1. machine/qemuarm64 2. machine/qemuriscv64 Please select one of the above options by its number: 0 <Add blank newline> Available distributions: 0. distro/poky 1. distro/poky-altcfg 2. distro/poky-tiny Please select one of the above options by its number: 0 <Add blank newline> Run 'bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64' to select this configuration non-interactively. <Add newline> Initializing a build in: /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64 <Add blank newline> Fetching into: /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers bitbake openembedded-core meta-yocto yocto-docs <Add blank newline> New build directory created: /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build <Add blank newline> Usage instructions and additional information: /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README <Add blank newline> To start using this build: source XXXX ''' What do you think? I'm throwing a few ideas around here... I'm throwing an idea of how to handle site.conf in there, which we could use to set DL_DIR and SSTATE_DIR defaults. I like the idea of some other bitbake-setup command to change those values. We could direct the user to run "bitbake-setup configure" before running init for the first time and move that setup piece there? Commandline overrides would make this part skipable if we have all the info needed? Cheers, Richard
I agree with Richard. This is really shaping up nicely. And I want to really thank you for all the effort and iterations to get to where we are today. I took the liberty of documenting how I currently use/see bitbake-setup: https://lists.yoctoproject.org/g/docs/message/7457 One of the things I came across is that build/conf/context-notes.txt is empty. This means when the user runs init-build-env they don't get the usual descriptive prompt. This can obviously be improved in another iteration. I resisted documenting both "yocto-master-nested-configs" and "yocto-master-options". I started with "yocto-master-nested-configs", which is "cleaner" or "simpler" to some extent and instead went with "yocto-master-options" which feels like it makes decisions the user expects (choosing a MACHINE and a DISTRO). I'm still on the fence about that. I've tried to resist adding new features and so on to allow you to do the initial development with your initial target audience. Maintainers are not the initial target audience, for instance. In my first attempt at documentation, I tried (like you intended) to address the "new to Yocto" audience and not experts. I hope that my initial attempt at documentation isn't too far off from being useful. Clearly another section explaining more about the way config-upstream.json works and other aspects such as bitbake-setup.conf is needed, but I wanted to address just the "how do I use this?" basics. My goal was to actually get folks to try it. Cheers, Tim On Wed, Aug 13, 2025 at 8:15 AM Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > From: Alexander Kanavin <alex@linutronix.de> > > Preamble > ======== > > The latest iteration of this patchset is available at > https://github.com/kanavin/bitbake > I recommend taking the patches from there to ensure that > you are not trying out outdated code. > > For the rationale and design guidelines please see this message: > https://lists.openembedded.org/g/openembedded-architecture/message/1913 > > Left out for now but will be done later: > > - handling site-specific settings. There was a discussion about it: > > ===== > > Finally, a question, how do you recommend to handle my local > SSTATE_DIR/DL_DIR? > > Setup with bbsetup, copy a site.conf with these config inside the build, > then > > the proper build? ... or is there a better way? > > This is an unresolved question. I'm open to proposals. Or code. > > I guess it will be something like 'bitbake-setup change-setting > default site-conf /path/to/site.conf', and then bitbake-setup will add > that to every build that is initialized. Site.conf need not list site > settings directly, it too can add a fragment maintained in a layer. > That way CI settings can be specified and maintained in a layer, but > they don't need to be added to a json config, as that would impose > them on everyone using that config to reproduce builds locally. > ===== > > - official configuration repository > > - documentation > > Amble *scratch* HOWTO > ===================== > > 1. If you don't know where to start, run 'bitbake-setup init'. > > Bitbake-setup will ask a few questions about available configuration > choices and set up a build: > > === > alex@alex-lx-laptop:~$ development/bitbake/bin/bitbake-setup init > Created a new settings file in > /home/alex/bitbake-builds/bitbake-setup.conf. > > Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf > > Fetching configuration registry git:// > github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main > into /home/alex/bitbake-builds/.bitbake-setup-cache/configurations > > Available configurations: > 0. yocto-master-nested-configs (future) Official Yocto configurations: > poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 (defined with nested > configurations) > 1. yocto-master-options (future) Official Yocto configurations: poky, > poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined with > options) > > Please select one of the above configurations by its number: > 1 > Selecting the only available bitbake configuration poky > Available target machines: > 0. machine/qemux86-64 > 1. machine/qemuarm64 > 2. machine/qemuriscv64 > Please select one of the above options by its number: > 0 > Available distributions: > 0. distro/poky > 1. distro/poky-altcfg > 2. distro/poky-tiny > Please select one of the above options by its number: > 0 > Run 'bitbake-setup yocto-master-options poky distro/poky > machine/qemux86-64' to select this configuration non-interactively. > Initializing a build in > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64 > Fetching layer/tool repository bitbake into > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/bitbake > Fetching layer/tool repository openembedded-core into > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/openembedded-core > Fetching layer/tool repository meta-yocto into > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/meta-yocto > Fetching layer/tool repository yocto-docs into > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/yocto-docs > ============================== > Setting up bitbake configuration in > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build > This bitbake configuration provides: Poky reference distro build > Usage instructions and additional information are in > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README > === > > Note: 'init' sub-command can also take a path or a URL with a > configuration file directly. > You can see how those files look like here: > https://github.com/kanavin/bitbake-setup-configurations > > 2. You can then source the bitbake environment and run bitbake to perform > builds as usual: > > alex@alex-lx-laptop:~$ . > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env > > Also, subsequent status/update commands will not require a separate > --build-dir argument telling > bitbake-setup where the build is. > > 3. To check if the build configuration needs to be updated, run: > === > alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup status > Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf. > > Fetching configuration registry git:// > github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main > into /home/alex/bitbake-builds/.bitbake-setup-cache/configurations > Configuration in /home/alex/bitbake-builds/poky-alex/ has not changed. > === > > If the configuration has changed, you will see the difference: > === > ... > Configuration in /home/alex/builds/poky-alex has changed: > --- /home/alex/builds/poky-alex/config/poky-alex.conf.json 2024-12-16 > 11:43:24.077446096 +0100 > +++ /home/alex/builds/poky-alex/config-tmp-asoubw5u/poky-alex.conf.json > 2024-12-16 11:47:43.237104405 +0100 > @@ -7,7 +7,7 @@ > "uri": "git://git.yoctoproject.org/poky-contrib" > } > }, > - "rev": "akanavin/sstate-for-all" > + "rev": "akanavin/bitbake-setup-testing" > }, > "path": "poky" > } > === > > If the configuration has not changed, but layer revisions referred to it > have (for example > if the configuration specifies a tip of a branch), you will see that too: > === > ... > Layer repository git://git.yoctoproject.org/poky-contrib checked out into > /home/alex/builds/poky-alex/layers/poky updated revision > akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to > aeb73e29379fe6007a8adc8d94c1ac18a93e68de > === > > 4. If the configuration has changed, you can bring it in sync with: > === > alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup update > Default parameter values are in /home/alex/.bitbake-setup/config - adjust > as needed. > > Fetching configuration repository git:// > github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main > into /home/alex/.bitbake-setup/configurations > Layer repository git://git.yoctoproject.org/poky-contrib checked out into > /home/alex/builds/poky-alex/layers/poky updated revision > akanavin/bitbake-setup-testing from > d174acad934f8ad1fe303abc5705733e15542859 to > a3d2ee10045f8c1151d680ad97994c5d6cf51ece > Fetching layer/tool repository poky into > /home/alex/builds/poky-alex/layers/poky > > Setting up bitbake configuration gadget in > /home/alex/bitbake-builds/poky-alex/build-gadget > Existing bitbake congfiguration directory renamed to > /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007 > The bitbake configuration has changed: > diff -uNr > /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf > /home/alex/builds/poky-alex/build-gadget/conf/local.conf > --- > /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf > 2024-12-16 11:47:51.865043102 +0100 > +++ /home/alex/builds/poky-alex/build-gadget/conf/local.conf 2024-12-16 > 11:50:07.811942847 +0100 > @@ -287,5 +287,3 @@ > # track the version of this file when it was generated. This can safely > be ignored if > # this doesn't mean anything to you. > CONF_VERSION = "2" > - > -TCLIBC = "musl" > > Bitbake configuration summary: This configuration is intended for building > gadget. > Usage instructions and additional information in > /home/alex/bitbake-builds/poky-alex/build-gadget/README > === > > Note that it will also rename/preserve the existing build/conf directory, > and print changes > in bitbake configuration (diff of content of build/conf/) if that has > changed. I can't > at the moment think of anything more clever that is also not much more > brittle or complex > to implement, but open to suggestions. > > Terminology > =========== > > - 'top directory' means the place under which bitbake-setup reads and > writes everything. bitbake-setup makes a promise to not touch anything > outside of > that, unless otherwise directed to by entries in settings (currently > there is one such setting for fetcher downloads for layers and config > registries). Top directory can be selected by an environment variable, a > command line option, > or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment > (e.g. we are in a bitbake environment), then the top directory is > deduced from that and doesn't need to be specified by hand. > > - 'settings' means bitbake-setup operational parameters that are > global to all builds under a top directory. E.g. the location of > configuration registry, or where the bitbake fetcher should place the > downloads (DL_DIR setting). Settings are stored in a .conf file in ini > format just under the top directory. > > - 'build' means a tree structure set up by 'bitbake-setup init', > consisting of, at least, a layers checkout, and a bitbake > build. It maps 1:1 to the json data it was constructed from, which is > called 'build configuration'. Build configurations are constructed from > generic configurations that may involve making one or more choices > about available options in them. Generic configurations are files, URLs > or are obtained from git repositories called 'config > registries', in which case they can be listed with 'bitbake-setup > list'. There can be multiple 'builds' under a top directory. Here are > two example generic configurations that showcase this: > > https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-options.conf.json > > https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-nested-configs.conf.json > > - 'bitbake-setup status' will tell if a build is in sync with > the generic configuration it was made from. 'bitbake-setup update' will > bring a build > in sync with a configuration if needed. > > - 'bitbake build' means a particular sub-tree inside a build that > bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR > by oe-init-build-env. conf/* in that tree is 'bitbake configuration'. > Bitbake configurations are constructed from templates and fragments, > with existing mechanisms provided by oe-core. The configuration file > format is specified such that other mechanisms to set up a > bitbake build can be added; there was a mention of ability to specify > local.conf content and a set of layers directly in a configuration. I > think that scales poorly compared to templates and fragments, but I > made sure alternative ways to configure a bitbake build are possible > to add in the future :) > > - 'source override' is a json file that can be used to modify revisions > and origins of layers that need to be checkout into a build (e.g. > when master branches need to be changed to master-next for purposes > of testing). Such a file is specified with a command-line option to 'init' > and an example can be seen here: > > https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-next.override.json > > This commit includes a fix by Ryan Eatmon <reatmon@ti.com>: > https://github.com/kanavin/bitbake/pull/1 > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > bin/bitbake-setup | 673 ++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 673 insertions(+) > create mode 100755 bin/bitbake-setup > > diff --git a/bin/bitbake-setup b/bin/bitbake-setup > new file mode 100755 > index 000000000..11569e67d > --- /dev/null > +++ b/bin/bitbake-setup > @@ -0,0 +1,673 @@ > +#!/usr/bin/env python3 > + > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > +import logging > +import os > +import sys > +import argparse > +import warnings > +import json > +import shutil > +import time > +import stat > +import tempfile > +import configparser > +import datetime > + > +default_registry = 'git:// > github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main > ' > + > +bindir = os.path.abspath(os.path.dirname(__file__)) > +sys.path[0:0] = [os.path.join(os.path.dirname(bindir), 'lib')] > + > +import bb.msg > +import bb.process > + > +logger = bb.msg.logger_create('bitbake-setup', sys.stdout) > + > +def cache_dir(top_dir): > + return os.path.join(top_dir, '.bitbake-setup-cache') > + > +def init_bb_cache(settings, args): > + dldir = settings["default"]["dl-dir"] > + bb_cachedir = os.path.join(cache_dir(args.top_dir), 'bitbake-cache') > + > + d = bb.data.init() > + d.setVar("DL_DIR", dldir) > + d.setVar("BB_CACHEDIR", bb_cachedir) > + d.setVar("__BBSRCREV_SEEN", "1") > + if args.no_network: > + d.setVar("BB_SRCREV_POLICY", "cache") > + bb.fetch.fetcher_init(d) > + return d > + > +def save_bb_cache(): > + bb.fetch2.fetcher_parse_save() > + bb.fetch2.fetcher_parse_done() > + > +def get_config_name(config): > + return os.path.basename(config).split('.')[0] > + > +def write_config(config, config_dir): > + with open(os.path.join(config_dir, "config-upstream.json"),'w') as s: > + json.dump(config, s, sort_keys=True, indent=4) > + > +def commit_config(config_dir): > + bb.process.run("git -C {} add .".format(config_dir)) > + bb.process.run("git -C {} commit -a -m 'Configuration at > {}'".format(config_dir, time.asctime())) > + > +def _write_layer_list(dest, repodirs): > + layers = [] > + for r in repodirs: > + for root, dirs, files in os.walk(os.path.join(dest,r)): > + if os.path.basename(root) == 'conf' and 'layer.conf' in files: > + layers.append(os.path.relpath(os.path.dirname(root), > dest)) > + layers_f = os.path.join(dest, ".oe-layers.json") > + with open(layers_f, 'w') as f: > + json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, > indent=4) > + > +def checkout_layers(layers, layerdir, d): > + repodirs = [] > + oesetupbuild = None > + for r_name in layers: > + r_data = layers[r_name] > + repodir = r_data["path"] > + repodirs.append(repodir) > + > + r_remote = r_data['git-remote'] > + rev = r_remote['rev'] > + remotes = r_remote['remotes'] > + > + for remote in remotes: > + type,host,path,user,pswd,params = > bb.fetch.decodeurl(remotes[remote]["uri"]) > + fetchuri = > bb.fetch.encodeurl(('git',host,path,user,pswd,params)) > + print("Fetching layer/tool repository {} into > {}".format(r_name, os.path.join(layerdir,repodir))) > + fetcher = > bb.fetch.Fetch(["{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir)], > d) > + do_fetch(fetcher, layerdir) > + > + if os.path.exists(os.path.join(layerdir, repodir, > 'scripts/oe-setup-build')): > + oesetupbuild = os.path.join(layerdir, repodir, > 'scripts/oe-setup-build') > + oeinitbuildenv = os.path.join(layerdir, repodir, > 'oe-init-build-env') > + > + _write_layer_list(layerdir, repodirs) > + > + if oesetupbuild: > + links = {'setup-build': oesetupbuild, 'oe-scripts': > os.path.dirname(oesetupbuild), 'init-build-env': oeinitbuildenv} > + for l,t in links.items(): > + symlink = os.path.join(layerdir, l) > + if os.path.lexists(symlink): > + os.remove(symlink) > + os.symlink(os.path.relpath(t,layerdir),symlink) > + > +def setup_bitbake_build(bitbake_config, layerdir, builddir): > + def _setup_build_conf(layers, build_conf_dir): > + os.makedirs(build_conf_dir) > + layers_s = "\n".join([" {} \\".format(os.path.join(layerdir,l)) > for l in layers]) > + bblayers_conf = """BBLAYERS ?= " \ > +{} > + " > +""".format(layers_s) > + with open(os.path.join(build_conf_dir, "bblayers.conf"), 'w') as > f: > + f.write(bblayers_conf) > + > + local_conf = """# > +# This file is intended for local configuration tweaks. > +# > +# If you would like to publish and share changes made to this file, > +# it is recommended to put them into a distro config, or to create > +# layer fragments from changes made here. > +# > +""" > + with open(os.path.join(build_conf_dir, "local.conf"), 'w') as f: > + f.write(local_conf) > + > + with open(os.path.join(build_conf_dir, "templateconf.cfg"), 'w') > as f: > + f.write("") > + > + with open(os.path.join(build_conf_dir, "conf-summary.txt"), 'w') > as f: > + f.write(bitbake_config["description"] + "\n") > + > + with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as > f: > + f.write("") > + > + def _make_init_build_env(builddir, initbuildenv): > + cmd = ". {} {}".format(initbuildenv, builddir) > + initbuild_in_builddir = os.path.join(builddir, 'init-build-env') > + with open(initbuild_in_builddir, 'w') as f: > + f.write(cmd) > + > + bitbake_builddir = os.path.join(builddir, "build") > + print("==============================") > + print("Setting up bitbake configuration in > {}".format(bitbake_builddir)) > + > + template = bitbake_config.get("oe-template") > + layers = bitbake_config.get("bb-layers") > + if not template and not layers: > + print("Bitbake configuration does not contain a reference to an > OpenEmbedded build template via 'oe-template' or a list of layers via > 'bb-layers'; please use oe-setup-build, oe-init-build-env or another > mechanism manually to complete the setup.") > + return > + oesetupbuild = os.path.join(layerdir, 'setup-build') > + if template and not os.path.exists(oesetupbuild): > + raise Exception("Cannot complete setting up a bitbake build > directory from OpenEmbedded template '{}' as oe-setup-build was not found > in any layers; please use oe-init-build-env manually.".format(template)) > + > + bitbake_confdir = os.path.join(bitbake_builddir, 'conf') > + backup_bitbake_confdir = bitbake_confdir + > "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) > + if os.path.exists(bitbake_confdir): > + os.rename(bitbake_confdir, backup_bitbake_confdir) > + > + if layers: > + _setup_build_conf(layers, bitbake_confdir) > + > + if template: > + bb.process.run("{} setup -c {} -b {} > --no-shell".format(oesetupbuild, template, bitbake_builddir)) > + else: > + initbuildenv = os.path.join(layerdir, 'init-build-env') > + if not os.path.exists(initbuildenv): > + print("Could not find oe-init-build-env in any of the layers; > please use another mechanism to initialize the bitbake environment") > + return > + _make_init_build_env(bitbake_builddir, > os.path.realpath(initbuildenv)) > + > + init_script = os.path.join(bitbake_builddir, "init-build-env") > + shell = os.path.basename(os.environ.get("SHELL","bash")) > + fragments = bitbake_config.get("oe-fragments", []) + > sorted(bitbake_config.get("oe-fragment-choices",{}).values()) > + if fragments: > + bb.process.run("{} -c '. {} && bitbake-config-build > enable-fragment {}'".format(shell, init_script, " ".join(fragments))) > + > + if os.path.exists(backup_bitbake_confdir): > + bitbake_config_diff = get_diff(backup_bitbake_confdir, > bitbake_confdir) > + if bitbake_config_diff: > + print("Existing bitbake configuration directory renamed to > {}".format(backup_bitbake_confdir)) > + print("The bitbake configuration has changed:") > + print(bitbake_config_diff) > + else: > + shutil.rmtree(backup_bitbake_confdir) > + > + print("This bitbake configuration provides: > {}".format(bitbake_config["description"])) > + > + readme = """{}\n\nAdditional information is in {} and {}\n > +Source the environment using '. {}' to run builds from the command line. > +The bitbake configuration files (local.conf, bblayers.conf and more) can > be found in {}/conf > +""".format( > + bitbake_config["description"], > + os.path.join(bitbake_builddir,'conf/conf-summary.txt'), > + os.path.join(bitbake_builddir,'conf/conf-notes.txt'), > + init_script, > + bitbake_builddir > + ) > + readme_file = os.path.join(bitbake_builddir, "README") > + with open(readme_file, 'w') as f: > + f.write(readme) > + print("Usage instructions and additional information are in > {}".format(readme_file)) > + > +def get_registry_config(registry_path, id): > + for root, dirs, files in os.walk(registry_path): > + for f in files: > + if f.endswith('.conf.json') and id == get_config_name(f): > + return os.path.join(root, f) > + raise Exception("Unable to find {} in available configurations; use > 'list' sub-command to see what is available".format(id)) > + > +def update_build(config, confdir, builddir, layerdir, d): > + layer_config = config["data"]["sources"] > + layer_overrides = config["source-overrides"]["sources"] > + for k,v in layer_overrides.items(): > + if k in layer_config: > + layer_config[k]["git-remote"] = v["git-remote"] > + checkout_layers(layer_config, layerdir, d) > + bitbake_config = config["bitbake-config"] > + setup_bitbake_build(bitbake_config, layerdir, builddir) > + > +def flatten_bitbake_configs(configs): > + def merge_configs(c1,c2): > + c_merged = {} > + for k,v in c2.items(): > + if k not in c1.keys(): > + c_merged[k] = v > + for k,v in c1.items(): > + if k not in c2.keys(): > + c_merged[k] = v > + else: > + c_merged[k] = c1[k] + c2[k] > + del c_merged['configurations'] > + return c_merged > + > + flattened_configs = [] > + for c in configs: > + if 'configurations' not in c: > + flattened_configs.append(c) > + else: > + for sub_c in flatten_bitbake_configs(c['configurations']): > + flattened_configs.append(merge_configs(c, sub_c)) > + return flattened_configs > + > +def choose_bitbake_config(configs, parameters, non_interactive): > + flattened_configs = flatten_bitbake_configs(configs) > + configs_dict = {i["name"]:i for i in flattened_configs} > + > + if parameters: > + config_id = parameters[0] > + if config_id not in configs_dict: > + raise Exception("Bitbake configuration {} not found; replace > with one of {}".format(config_id, configs_dict)) > + return configs_dict[config_id] > + > + enumerated_configs = list(enumerate(flattened_configs)) > + if len(enumerated_configs) == 1: > + only_config = flattened_configs[0] > + print("Selecting the only available bitbake configuration > {}".format(only_config["name"])) > + return only_config > + > + if non_interactive: > + raise Exception("Unable to choose from bitbake configurations in > non-interactive mode: {}".format(configs_dict)) > + > + print("Available bitbake configurations:") > + for n, config_data in enumerated_configs: > + print("{}. {}\t{}".format(n, config_data["name"], > config_data["description"])) > + print("\nPlease select one of the above bitbake configurations by its > number:") > + config_n = int(input()) > + return flattened_configs[config_n] > + > +def choose_config(configs, non_interactive): > + not_expired_configs = [k for k in configs.keys() if not > has_expired(configs[k].get("expires", None))] > + config_list = list(enumerate(not_expired_configs)) > + if len(config_list) == 1: > + only_config = config_list[0][1] > + print("Selecting the only available configuration > {}".format(only_config)) > + return only_config > + > + if non_interactive: > + raise Exception("Unable to choose from configurations in > non-interactive mode: {}".format(not_expired_configs)) > + > + print("\nAvailable configurations:") > + for n, config_name in config_list: > + config_data = configs[config_name] > + expiry_date = config_data.get("expires", None) > + config_desc = config_data["description"] > + if expiry_date: > + print("{}. {}\t{} (supported until {})".format(n, config_name, > config_desc, expiry_date)) > + else: > + print("{}. {}\t{}".format(n, config_name, config_desc)) > + print("\nPlease select one of the above configurations by its > number:") > + config_n = int(input()) > + return config_list[config_n][1] > + > +def choose_fragments(possibilities, parameters, non_interactive): > + choices = {} > + for k,v in possibilities.items(): > + choice = [o for o in v["options"] if o in parameters] > + if len(choice) > 1: > + raise Exception("Options specified on command line do not > allow a single selection from possibilities {}, please remove one or more > from {}".format(v["options"], parameters)) > + if len(choice) == 1: > + choices[k] = choice[0] > + continue > + > + if non_interactive: > + raise Exception("Unable to choose from options in > non-interactive mode: {}".format(v["options"])) > + > + print(v["description"]+ ":") > + options_enumerated = list(enumerate(v["options"])) > + for n,o in options_enumerated: > + print("{}. {}".format(n, o)) > + print("Please select one of the above options by its number:") > + option_n = int(input()) > + choices[k] = options_enumerated[option_n][1] > + return choices > + > +def obtain_config(settings, args, source_overrides, d): > + if args.config: > + config_id = args.config[0] > + config_parameters = args.config[1:] > + if os.path.exists(config_id): > + print("Reading configuration from local file > {}".format(config_id)) > + upstream_config = > {'type':'local','path':os.path.abspath(args.config),'name':get_config_name(config_id),'data':json.load(open(config_path))} > + elif config_id.startswith("http://") or > config_id.startswith("https://"): > + print("Reading configuration from network URI > {}".format(config_id)) > + import urllib.request > + with urllib.request.urlopen(config_id) as f: > + upstream_config = > {'type':'network','uri':config_id,'name':get_config_name(config_id),'data':json.load(f)} > + else: > + print("Looking up config {} in configuration > registry".format(config_id)) > + registry_path = > update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) > + registry_configs = list_registry(registry_path, > with_expired=True) > + if config_id not in registry_configs: > + raise Exception("Config {} not found in configuration > registry, re-run 'init' without parameters to choose from available > configurations.".format(config_id)) > + upstream_config = > {'type':'registry','registry':settings["default"]["registry"],'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} > + expiry_date = upstream_config['data'].get("expires", None) > + if has_expired(expiry_date): > + print("This configuration is no longer supported after {}. > Please consider changing to a supported configuration.".format(expiry_date)) > + else: > + registry_path = update_registry(settings["default"]["registry"], > cache_dir(args.top_dir), d) > + registry_configs = list_registry(registry_path, with_expired=True) > + config_id = choose_config(registry_configs, args.non_interactive) > + config_parameters = [] > + upstream_config = > {'type':'registry','registry':settings["default"]["registry"],'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} > + > + upstream_config['bitbake-config'] = > choose_bitbake_config(upstream_config['data']['bitbake-setup']['configurations'], > config_parameters, args.non_interactive) > + upstream_config['bitbake-config']['oe-fragment-choices'] = > choose_fragments(upstream_config['bitbake-config'].get('oe-fragments-one-of',{}), > config_parameters[1:], args.non_interactive) > + upstream_config['non-interactive-cmdline-options'] = " > ".join([upstream_config['name'], upstream_config['bitbake-config']['name']] > + sorted(upstream_config['bitbake-config']['oe-fragment-choices'].values())) > + upstream_config['source-overrides'] = source_overrides > + return upstream_config > + > +def init_config(settings, args, d): > + stdout = sys.stdout > + def handle_task_progress(event, d): > + rate = event.rate if event.rate else '' > + progress = event.progress if event.progress > 0 else 0 > + print("{}% {} ".format(progress, rate), > file=stdout, end='\r') > + > + source_overrides = json.load(open(args.source_overrides)) if > args.source_overrides else {'sources':{}} > + upstream_config = obtain_config(settings, args, source_overrides, d) > + print("Run 'bitbake-setup {}' to select this configuration > non-interactively.".format(upstream_config['non-interactive-cmdline-options'])) > + > + builddir = os.path.join(os.path.abspath(args.top_dir), > upstream_config['non-interactive-cmdline-options'].replace(" > ","-").replace("/","_")) > + if os.path.exists(builddir): > + print("Build already initialized in {}\nUse 'bitbake-setup > status' to check if it needs to be updated or 'bitbake-setup update' to > perform the update.".format(builddir)) > + return > + > + print("Initializing a build in {}".format(builddir)) > + > + os.makedirs(builddir) > + > + confdir = os.path.join(builddir, "config") > + layerdir = os.path.join(builddir, "layers") > + > + os.makedirs(confdir) > + os.makedirs(layerdir) > + > + bb.process.run("git -C {} init -b main".format(confdir)) > + bb.process.run("git -C {} commit --allow-empty -m 'Initial > commit'".format(confdir)) > + > + bb.event.register("bb.build.TaskProgress", handle_task_progress, > data=d) > + > + write_config(upstream_config, confdir) > + commit_config(confdir) > + update_build(upstream_config, confdir, builddir, layerdir, d) > + > + bb.event.remove("bb.build.TaskProgress", None) > + > +def get_diff(file1, file2): > + try: > + bb.process.run('diff -uNr {} {}'.format(file1, file2)) > + except bb.process.ExecutionError as e: > + if e.exitcode == 1: > + return e.stdout > + else: > + raise e > + return None > + > +def are_layers_changed(layers, layerdir, d): > + changed = False > + for r_name in layers: > + r_data = layers[r_name] > + repodir = r_data["path"] > + > + r_remote = r_data['git-remote'] > + rev = r_remote['rev'] > + remotes = r_remote['remotes'] > + > + for remote in remotes: > + type,host,path,user,pswd,params = > bb.fetch.decodeurl(remotes[remote]["uri"]) > + fetchuri = > bb.fetch.encodeurl(('git',host,path,user,pswd,params)) > + fetcher = > bb.fetch.FetchData("{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir), > d) > + upstream_revision = fetcher.method.latest_revision(fetcher, > d, 'default') > + rev_parse_result = bb.process.run('git -C {} rev-parse > HEAD'.format(os.path.join(layerdir, repodir))) > + local_revision = rev_parse_result[0].strip() > + if upstream_revision != local_revision: > + changed = True > + print('Layer repository {} checked out into {} updated > revision {} from {} to {}'.format(remotes[remote]["uri"], > os.path.join(layerdir, repodir), rev, local_revision, upstream_revision)) > + > + return changed > + > +def build_status(settings, args, d, update=False): > + builddir = args.build_dir > + > + confdir = os.path.join(builddir, "config") > + layerdir = os.path.join(builddir, "layers") > + > + current_upstream_config = json.load(open(os.path.join(confdir, > "config-upstream.json"))) > + > + args.config = > current_upstream_config['non-interactive-cmdline-options'].split() > + args.non_interactive = True > + source_overrides = current_upstream_config["source-overrides"] > + new_upstream_config = obtain_config(settings, args, source_overrides, > d) > + > + write_config(new_upstream_config, confdir) > + config_diff = bb.process.run('git -C {} diff'.format(confdir))[0] > + > + if config_diff: > + print('Configuration in {} has changed:\n{}'.format(builddir, > config_diff)) > + if update: > + commit_config(confdir) > + update_build(new_upstream_config, confdir, builddir, > layerdir, d) > + else: > + bb.process.run('git -C {} restore > config-upstream.json'.format(confdir)) > + return > + > + if are_layers_changed(current_upstream_config["data"]["sources"], > layerdir, d): > + if update: > + update_build(current_upstream_config, confdir, builddir, > layerdir, d) > + return > + > + print("Configuration in {} has not changed.".format(builddir)) > + > +def build_update(settings, args, d): > + build_status(settings, args, d, update=True) > + > +def do_fetch(fetcher, dir): > + # git fetcher simply dumps git output to stdout; in bitbake context > that is redirected to temp/log.do_fetch > + # and we need to set up smth similar here > + fetchlogdir = os.path.join(dir, 'logs') > + os.makedirs(fetchlogdir, exist_ok=True) > + fetchlog = os.path.join(fetchlogdir, > 'fetch_log.{}'.format(datetime.datetime.now().strftime("%Y%m%d%H%M%S"))) > + with open(fetchlog, 'a') as f: > + oldstdout = sys.stdout > + sys.stdout = f > + fetcher.download() > + fetcher.unpack(dir) > + sys.stdout = oldstdout > + > +def update_registry(registry, cachedir, d): > + registrydir = 'configurations' > + full_registrydir = os.path.join(cachedir, registrydir) > + print("Fetching configuration registry {} into {}".format(registry, > full_registrydir)) > + fetcher = bb.fetch.Fetch(["{};destsuffix={}".format(registry, > registrydir)], d) > + do_fetch(fetcher, cachedir) > + return full_registrydir > + > +def has_expired(expiry_date): > + if expiry_date: > + return datetime.datetime.now() > > datetime.datetime.fromisoformat(expiry_date) > + return False > + > +def list_registry(registry_path, with_expired): > + json_data = {} > + > + for root, dirs, files in os.walk(registry_path): > + for f in files: > + if f.endswith('.conf.json'): > + config_name = get_config_name(f) > + config_data = json.load(open(os.path.join(root, f))) > + config_desc = config_data["description"] > + expiry_date = config_data.get("expires", None) > + if expiry_date: > + if with_expired or not has_expired(expiry_date): > + json_data[config_name] = {"description": > config_desc, "expires": expiry_date} > + else: > + json_data[config_name] = {"description": config_desc} > + return json_data > + > +def list_configs(settings, args, d): > + registry_path = update_registry(settings["default"]["registry"], > cache_dir(args.top_dir), d) > + json_data = list_registry(registry_path, args.with_expired) > + print("\nAvailable configurations:") > + for config_name, config_data in json_data.items(): > + expiry_date = config_data.get("expires", None) > + config_desc = config_data["description"] > + if expiry_date: > + if args.with_expired or not has_expired(expiry_date): > + print("{}\t{} (supported until {})".format(config_name, > config_desc, expiry_date)) > + else: > + print("{}\t{}".format(config_name, config_desc)) > + print("\nRun 'init' with one of the above configuration identifiers > to set up a build.") > + > + if args.write_json: > + with open(args.write_json, 'w') as f: > + json.dump(json_data, f, sort_keys=True, indent=4) > + print("Available configurations written into > {}".format(args.write_json)) > + > +def default_settings_path(top_dir): > + return os.path.join(top_dir, 'bitbake-setup.conf') > + > +def write_settings(top_dir, force_replace): > + settings_path = default_settings_path(top_dir) > + if not os.path.exists(settings_path) or force_replace: > + if os.path.exists(settings_path): > + backup_conf = settings_path + > "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) > + os.rename(settings_path, backup_conf) > + print("Previous settings are in {}".format(backup_conf)) > + > + settings = configparser.ConfigParser() > + settings['default'] = { > + 'registry':default_registry, > + 'dl-dir':os.path.join(top_dir, > '.bitbake-setup-downloads'), > + } > + os.makedirs(os.path.dirname(settings_path), exist_ok=True) > + with open(settings_path, 'w') as settingsfile: > + settings.write(settingsfile) > + print('Created a new settings file in > {}.\n'.format(settings_path)) > + > +def load_settings(top_dir): > + # This creates a new settings file if it does not yet exist > + write_settings(top_dir, force_replace=False) > + > + settings_path = default_settings_path(top_dir) > + settings = configparser.ConfigParser() > + print('Loading settings from {}\n'.format(settings_path)) > + settings.read([settings_path]) > + return settings > + > +def change_settings(top_dir, new_settings): > + # This creates a new settings file if it does not yet exist > + write_settings(top_dir, force_replace=False) > + > + settings = load_settings(top_dir) > + for section, section_settings in new_settings.items(): > + for setting, value in section_settings.items(): > + settings[section][setting] = value > + print("Setting '{}' in section '{}' is changed to > '{}'".format(setting, section, value)) > + > + settings_path = default_settings_path(top_dir) > + with open(settings_path, 'w') as settingsfile: > + settings.write(settingsfile) > + print("New settings written to {}".format(settings_path)) > + return settings > + > +def get_build_dir_via_bbpath(): > + bbpath = os.environ.get('BBPATH') > + if bbpath: > + bitbake_dir = os.path.normpath(bbpath.split(':')[0]) > + if os.path.exists(os.path.join(bitbake_dir,'init-build-env')): > + build_dir = os.path.dirname(bitbake_dir) > + return build_dir > + return None > + > +def get_default_top_dir(): > + build_dir_via_bbpath = get_build_dir_via_bbpath() > + if build_dir_via_bbpath: > + top_dir = os.path.dirname(build_dir_via_bbpath) > + if os.path.exists(default_settings_path(top_dir)): > + return top_dir > + return os.environ.get('BITBAKE_SETUP_TOP_DIR') or > os.path.join(os.path.expanduser('~'), 'bitbake-builds') > + > +def main(): > + def add_top_dir_arg(parser): > + parser.add_argument('--top-dir', default=get_default_top_dir(), > help='Top level directory where builds are set up and downloaded > configurations and layers are cached for reproducibility and offline > builds, default is %(default)s, can be overriden via BITBAKE_SETUP_TOP_DIR > environment variable') > + > + def add_build_dir_arg(parser): > + build_dir = get_build_dir_via_bbpath() > + if build_dir: > + parser.add_argument('--build-dir', default=build_dir, > help="Path to the build, default is %(default)s via BBPATH") > + else: > + parser.add_argument('--build-dir', required=True, help="Path > to the build") > + > + parser = argparse.ArgumentParser( > + description="BitBake setup utility. Run with 'init' argument to > get started.", > + epilog="Use %(prog)s <subcommand> --help to get help on a > specific command" > + ) > + parser.add_argument('-d', '--debug', help='Enable debug output', > action='store_true') > + parser.add_argument('-q', '--quiet', help='Print only errors', > action='store_true') > + parser.add_argument('--color', choices=['auto', 'always', 'never'], > default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', > metavar='COLOR') > + parser.add_argument('--no-network', action='store_true', help='Do not > check whether configuration repositories and layer repositories have been > updated; use only the local cache.') > + > + subparsers = parser.add_subparsers() > + > + parser_list = subparsers.add_parser('list', help='List available > configurations') > + add_top_dir_arg(parser_list) > + parser_list.add_argument('--with-expired', action='store_true', > help='List also configurations that are no longer supported due to reaching > their end-of-life dates.') > + parser_list.add_argument('--write-json', action='store', help='Write > available configurations into a json file so they can be programmatically > processed.') > + parser_list.set_defaults(func=list_configs) > + > + parser_init = subparsers.add_parser('init', help='Select a > configuration and initialize a build from it') > + add_top_dir_arg(parser_init) > + parser_init.add_argument('config', nargs='*', help="path/URL/id to a > configuration file (use 'list' command to get available ids), followed by > configuration options. Bitbake-setup will ask to choose from available > choices if command line doesn't completely specify them.") > + parser_init.add_argument('--non-interactive', action='store_true', > help='Do not ask to interactively choose from available options; if > bitbake-setup cannot make a decision it will stop with a failure.') > + parser_init.add_argument('--source-overrides', action='store', > help='Override sources information (repositories/revisions) with values > from a local json file.') > + parser_init.set_defaults(func=init_config) > + > + parser_status = subparsers.add_parser('status', help='Check if the > build needs to be synchronized with configuration') > + add_build_dir_arg(parser_status) > + parser_status.set_defaults(func=build_status) > + > + parser_update = subparsers.add_parser('update', help='Update a build > to be in sync with configuration') > + add_build_dir_arg(parser_update) > + parser_update.set_defaults(func=build_update) > + > + parser_reset_settings = subparsers.add_parser('reset-settings', > help='Write a settings file with default values into the top level > directory (contains the location of build configuration registry, downloads > directory and other global settings)') > + add_top_dir_arg(parser_reset_settings) > + parser_reset_settings.set_defaults(func=write_settings) > + > + parser_change_setting = subparsers.add_parser('change-setting', > help='Change a setting in the settings file') > + add_top_dir_arg(parser_change_setting) > + parser_change_setting.add_argument('section', help="Section in a > settings file, typically 'default'") > + parser_change_setting.add_argument('key', help="Name of the setting") > + parser_change_setting.add_argument('value', help="Value of the > setting") > + parser_change_setting.set_defaults(func=change_settings) > + > + args = parser.parse_args() > + > + logging.basicConfig(stream=sys.stdout) > + if args.debug: > + logger.setLevel(logging.DEBUG) > + elif args.quiet: > + logger.setLevel(logging.ERROR) > + > + # Need to re-run logger_create with color argument > + # (will be the same logger since it has the same name) > + bb.msg.logger_create('bitbake-setup', output=sys.stdout, > + color=args.color, > + level=logger.getEffectiveLevel()) > + > + > + if hasattr(args, 'build_dir'): > + if not os.path.exists(os.path.join(args.build_dir,'build', > 'init-build-env')): > + print("Not a valid build directory: build/init-build-env does > not exist in {}".format(args.build_dir)) > + return > + # commands without --top-dir argument (status, update) still need > to know where > + # the top dir is, but it should be auto-deduced as parent of > args.build_dir > + if not hasattr(args, 'top_dir'): > + args.top_dir = > os.path.dirname(os.path.normpath(args.build_dir)) > + > + if 'func' in args: > + if args.func == write_settings: > + write_settings(args.top_dir, force_replace=True) > + elif args.func == change_settings: > + change_settings(args.top_dir, > {args.section:{args.key:args.value}}) > + else: > + settings = load_settings(args.top_dir) > + d = init_bb_cache(settings, args) > + args.func(settings, args, d) > + save_bb_cache() > + else: > + from argparse import Namespace > + parser.print_help() > + > +main() > -- > 2.39.5 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#17872): > https://lists.openembedded.org/g/bitbake-devel/message/17872 > Mute This Topic: https://lists.openembedded.org/mt/114685174/924729 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ > ticotimo@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Tue, 19 Aug 2025 at 16:22, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > Thanks for continuing to work on this, I like it a lot and it feels a > lot better than the previous version. From a "what it is doing" > standpoint, I think we're getting there. Where it still needs work is > in the "telling the user what it is doing" part. I say this as someone > who knows bitbake/OE quite well and I still had to look at the files on > disk to work out where things are. > > A big chunk of this is actually whitespace and formatting on the > screen. Putting paths onto newlines with indentation will help > readability, as will some blank newlines to separate out things. Thank. I didn't polish the UI as I wanted to get the 'new functionality' out as soon as possible. I also already want to tweak a few things in the output. By the way here are the slides I'm going to show in less than a week in OSS Amsterdam: https://osseu2025.sched.com/event/25VtR There's a nice hand-drawn chart of how bitbake-setup puts all the pieces together in creating a build/conf and layers checkout for bitbake to be able to run. If you can take a moment and take a look, I'd appreciate. > I think we should probably be "properly" integrating into the user's > HOME, i.e. create ~/.config/bitbake/bitbake-setup.conf and > ~/.cache/bitbake/downloads by default out the box too, we may as well > make bitbake appear a "proper" citizen if used this way. I'm not sure about using ~/.config and ~/.cache. I introduced the concept of 'top directory' (where everything else is) on purpose, such that there can be multiple top directories, each with its own bitbake-setup settings file and site.conf. If you look at the settings file: [default] registry = git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main dl-dir = /home/alex/bitbake-builds/.bitbake-setup-downloads there can be conceivable use cases where users would want to have multiple such files, each with its own registry location and dl-dir (and site.conf items in a separate file). E.g one for builds internal to their organization and another for upstream yocto work. > Let me describe what I think it needs to "say" to a new user and see > what you think. I suspect the interaction should look something like: > > ''' > $ bitbake-setup init > > No previous bitbake setup configuration detected. Configure bitbake-setup in /home/XXXX/.config/bitbake y/n? > > Configured a common source downloads cache (DL_DIR) at XXXX. > > A common site.conf file to use in builds can be created at: > /home/XXXX/.config/bitbake/site.conf > <symlink this into a builds site.conf position?> > > No configuration registry specified, using the default of: > XXXX > > Fetching configuration registry into: > YYYY > > Configure a build at XXXX ? <let the user change/confirm the path> The above configuration sequence is fine, except I'm reluctant to put the settings and site.conf into ~/.config as explained above. Maybe we could use that as 'default', but also be able to have a custom location? > Available configurations: > 0. yocto-master-nested-configs (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 (defined with nested configurations) > 1. yocto-master-options (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined with options) > > Please select one of the above configurations by its number: > 1 > Selecting the only available bitbake configuration poky > <Add blank newline> > Available target machines: > 0. machine/qemux86-64 > 1. machine/qemuarm64 > 2. machine/qemuriscv64 > Please select one of the above options by its number: > 0 > <Add blank newline> > Available distributions: > 0. distro/poky > 1. distro/poky-altcfg > 2. distro/poky-tiny > Please select one of the above options by its number: > 0 > <Add blank newline> > Run 'bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64' to select this configuration non-interactively. > <Add newline> > Initializing a build in: > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64 > <Add blank newline> > Fetching into: /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers > bitbake > openembedded-core > meta-yocto > yocto-docs > <Add blank newline> > New build directory created: > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build > <Add blank newline> > Usage instructions and additional information: > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README > <Add blank newline> > To start using this build: > source XXXX > ''' > > What do you think? I'm throwing a few ideas around here... These tweaks are fine, I'll do them. > I'm throwing an idea of how to handle site.conf in there, which we > could use to set DL_DIR and SSTATE_DIR defaults. I like the idea of > some other bitbake-setup command to change those values. We could > direct the user to run "bitbake-setup configure" before running init > for the first time and move that setup piece there? Commandline > overrides would make this part skipable if we have all the info needed? Yes, this could be a separate command, but init could run that sequence first as well when needed. site.conf handling I agree with. The only sticking point is ability to have multiple bitbake-setup settings and site.confs which I'd like to preserve Alex
On Wed, 2025-08-20 at 11:29 +0200, Alexander Kanavin wrote: > Thank. I didn't polish the UI as I wanted to get the 'new > functionality' out as soon as possible. I also already want to tweak a > few things in the output. > > By the way here are the slides I'm going to show in less than a week > in OSS Amsterdam: > https://osseu2025.sched.com/event/25VtR > > There's a nice hand-drawn chart of how bitbake-setup puts all the > pieces together in creating a build/conf and layers checkout for > bitbake to be able to run. If you can take a moment and take a look, > I'd appreciate. I will take a look, I can't right now but remind me if I don't remember. > > I think we should probably be "properly" integrating into the user's > > HOME, i.e. create ~/.config/bitbake/bitbake-setup.conf and > > ~/.cache/bitbake/downloads by default out the box too, we may as well > > make bitbake appear a "proper" citizen if used this way. > > I'm not sure about using ~/.config and ~/.cache. > > I introduced the concept of 'top directory' (where everything else is) > on purpose, such that there can be multiple top directories, each with > its own bitbake-setup settings file and site.conf. If you look at the > settings file: > > [default] > registry = git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main > dl-dir = /home/alex/bitbake-builds/.bitbake-setup-downloads > > there can be conceivable use cases where users would want to have > multiple such files, each with its own registry location and dl-dir > (and site.conf items in a separate file). E.g one for builds internal > to their organization and another for upstream yocto work. I agree we should allow this to be configured, I'm just thinking the .config/.cache directories might be the right place to put the defaults. This does move bitbake (or bitbake-setup really) to become more of a distro/system tool. I'm still not 100% sure if we should do that but using the standard locations by default would fit with that. > > Let me describe what I think it needs to "say" to a new user and see > > what you think. I suspect the interaction should look something like: > > > > ''' > > $ bitbake-setup init > > > > No previous bitbake setup configuration detected. Configure bitbake-setup in /home/XXXX/.config/bitbake y/n? > > > > Configured a common source downloads cache (DL_DIR) at XXXX. > > > > A common site.conf file to use in builds can be created at: > > /home/XXXX/.config/bitbake/site.conf > > <symlink this into a builds site.conf position?> > > > > No configuration registry specified, using the default of: > > XXXX > > > > Fetching configuration registry into: > > YYYY > > > > Configure a build at XXXX ? <let the user change/confirm the path> > > The above configuration sequence is fine, except I'm reluctant to put > the settings and site.conf into ~/.config as explained above. Maybe we > could use that as 'default', but also be able to have a custom > location? I do think making it configurable is fine. I was kind of assuming that would be the case. > > Available configurations: > > 0. yocto-master-nested-configs (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 (defined with nested configurations) > > 1. yocto-master-options (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined with options) > > > > Please select one of the above configurations by its number: > > 1 > > Selecting the only available bitbake configuration poky > > <Add blank newline> > > Available target machines: > > 0. machine/qemux86-64 > > 1. machine/qemuarm64 > > 2. machine/qemuriscv64 > > Please select one of the above options by its number: > > 0 > > <Add blank newline> > > Available distributions: > > 0. distro/poky > > 1. distro/poky-altcfg > > 2. distro/poky-tiny > > Please select one of the above options by its number: > > 0 > > <Add blank newline> > > Run 'bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64' to select this configuration non-interactively. > > <Add newline> > > Initializing a build in: > > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64 > > <Add blank newline> > > Fetching into: /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers > > bitbake > > openembedded-core > > meta-yocto > > yocto-docs > > <Add blank newline> > > New build directory created: > > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build > > <Add blank newline> > > Usage instructions and additional information: > > /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README > > <Add blank newline> > > To start using this build: > > source XXXX > > ''' > > > > What do you think? I'm throwing a few ideas around here... > > These tweaks are fine, I'll do them. > > > I'm throwing an idea of how to handle site.conf in there, which we > > could use to set DL_DIR and SSTATE_DIR defaults. I like the idea of > > some other bitbake-setup command to change those values. We could > > direct the user to run "bitbake-setup configure" before running init > > for the first time and move that setup piece there? Commandline > > overrides would make this part skipable if we have all the info needed? > > Yes, this could be a separate command, but init could run that > sequence first as well when needed. site.conf handling I agree with. > > The only sticking point is ability to have multiple bitbake-setup > settings and site.confs which I'd like to preserve See above, I'm fine with making it configurable. I'm curious if others have opinions. To be clear, the patch set is making huge progress and I think we're getting there. I'm really happy with the direction. As ever, the detail is now what we need to get right but it is very much going where I think we need it to go/be. Thanks! Cheers, Richard
On Wed, 20 Aug 2025, Richard Purdie via lists.openembedded.org wrote: > On Wed, 2025-08-20 at 11:29 +0200, Alexander Kanavin wrote: > > Thank. I didn't polish the UI as I wanted to get the 'new > > functionality' out as soon as possible. I also already want to tweak a > > few things in the output. > > > > By the way here are the slides I'm going to show in less than a week > > in OSS Amsterdam: > > https://osseu2025.sched.com/event/25VtR > > > > There's a nice hand-drawn chart of how bitbake-setup puts all the > > pieces together in creating a build/conf and layers checkout for > > bitbake to be able to run. If you can take a moment and take a look, > > I'd appreciate. > > I will take a look, I can't right now but remind me if I don't > remember. > > > > I think we should probably be "properly" integrating into the user's > > > HOME, i.e. create ~/.config/bitbake/bitbake-setup.conf and > > > ~/.cache/bitbake/downloads by default out the box too, we may as well > > > make bitbake appear a "proper" citizen if used this way. > > > > I'm not sure about using ~/.config and ~/.cache. > > > > I introduced the concept of 'top directory' (where everything else is) > > on purpose, such that there can be multiple top directories, each with > > its own bitbake-setup settings file and site.conf. If you look at the > > settings file: > > > > [default] > > registry = git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main > > dl-dir = /home/alex/bitbake-builds/.bitbake-setup-downloads > > > > there can be conceivable use cases where users would want to have > > multiple such files, each with its own registry location and dl-dir > > (and site.conf items in a separate file). E.g one for builds internal > > to their organization and another for upstream yocto work. > > I agree we should allow this to be configured, I'm just thinking the > .config/.cache directories might be the right place to put the > defaults. This does move bitbake (or bitbake-setup really) to become > more of a distro/system tool. I'm still not 100% sure if we should do > that but using the standard locations by default would fit with that. I can see where you're coming from, as this shadows how git config works, but the top directory behavior is something I've been on the fence about with bitbake-setup, and this seems like it'd maybe make that worse. With things as they are now, I fear I'll be having to document passing the --top-dir option in any bitbake-setup usage to avoid contamination of configurations between projects, and avoid users asking why they're having stuff show up in $HOME. I'd honestly prefer if it only operated in the current working directory by default like whisk and kas. Scott
On Wed, 2025-08-20 at 12:42 -0400, Scott Murray wrote: > On Wed, 20 Aug 2025, Richard Purdie via lists.openembedded.org wrote: > > On Wed, 2025-08-20 at 11:29 +0200, Alexander Kanavin wrote: > > > > I agree we should allow this to be configured, I'm just thinking the > > .config/.cache directories might be the right place to put the > > defaults. This does move bitbake (or bitbake-setup really) to become > > more of a distro/system tool. I'm still not 100% sure if we should do > > that but using the standard locations by default would fit with that. > > I can see where you're coming from, as this shadows how git config works, > but the top directory behavior is something I've been on the fence about > with bitbake-setup, and this seems like it'd maybe make that worse. With > things as they are now, I fear I'll be having to document passing the > --top-dir option in any bitbake-setup usage to avoid contamination of > configurations between projects, and avoid users asking why they're having > stuff show up in $HOME. I'd honestly prefer if it only operated in the > current working directory by default like whisk and kas. The only things I'd expect to come from this would be downloads and sstate, both of which should be shareable. Obviously users can do other things but I'm cautiously optimistic that if we properly document that, it could work quite well... There are probably ways we could sanity test site.conf specifically too. Cheers, Richard
On Wed, 20 Aug 2025 at 18:42, Scott Murray <scott.murray@konsulko.com> wrote: > I can see where you're coming from, as this shadows how git config works, > but the top directory behavior is something I've been on the fence about > with bitbake-setup, and this seems like it'd maybe make that worse. With > things as they are now, I fear I'll be having to document passing the > --top-dir option in any bitbake-setup usage to avoid contamination of > configurations between projects, and avoid users asking why they're having > stuff show up in $HOME. I'd honestly prefer if it only operated in the > current working directory by default like whisk and kas. You only need --top-dir if you're not in a bitbake environment yet. If you are, bitbake-setup will deduce top-dir from BBPATH. Alex
On Wed, 20 Aug 2025, Alexander Kanavin wrote: > On Wed, 20 Aug 2025 at 18:42, Scott Murray <scott.murray@konsulko.com> wrote: > > > I can see where you're coming from, as this shadows how git config works, > > but the top directory behavior is something I've been on the fence about > > with bitbake-setup, and this seems like it'd maybe make that worse. With > > things as they are now, I fear I'll be having to document passing the > > --top-dir option in any bitbake-setup usage to avoid contamination of > > configurations between projects, and avoid users asking why they're having > > stuff show up in $HOME. I'd honestly prefer if it only operated in the > > current working directory by default like whisk and kas. > > You only need --top-dir if you're not in a bitbake environment yet. If > you are, bitbake-setup will deduce top-dir from BBPATH. Sure, but my point is that's what I'll be doing every time I set up any project (and maybe reconfiguring it?), so effectively for me the default behavior is not useful. I doubt I am alone in doing all my development work on a different disk than where /home is on my dev and build machines. Scott
On Wed, 20 Aug 2025 at 22:15, Scott Murray <scott.murray@konsulko.com> wrote: > Sure, but my point is that's what I'll be doing every time I set up > any project (and maybe reconfiguring it?), so effectively for me the > default behavior is not useful. I doubt I am alone in doing all my > development work on a different disk than where /home is on my dev > and build machines. Then there's the option of setting BITBAKE_SETUP_TOP_DIR in environment. Alex
On Wed, 20 Aug 2025, Alexander Kanavin wrote: > On Wed, 20 Aug 2025 at 22:15, Scott Murray <scott.murray@konsulko.com> wrote: > > Sure, but my point is that's what I'll be doing every time I set up > > any project (and maybe reconfiguring it?), so effectively for me the > > default behavior is not useful. I doubt I am alone in doing all my > > development work on a different disk than where /home is on my dev > > and build machines. > > Then there's the option of setting BITBAKE_SETUP_TOP_DIR in environment. I'll not bother pressing this further since you seem set on the current design, but my original concern that I pointed out was that I'll always having to configure that, and likely document doing the need to do so for users of any project I may want to use bitbake-setup for. The default behaviors of other tools in this space do not require that. Scott
On Wed, 2025-08-20 at 16:49 -0400, Scott Murray wrote: > On Wed, 20 Aug 2025, Alexander Kanavin wrote: > > > On Wed, 20 Aug 2025 at 22:15, Scott Murray <scott.murray@konsulko.com> wrote: > > > Sure, but my point is that's what I'll be doing every time I set up > > > any project (and maybe reconfiguring it?), so effectively for me the > > > default behavior is not useful. I doubt I am alone in doing all my > > > development work on a different disk than where /home is on my dev > > > and build machines. > > > > Then there's the option of setting BITBAKE_SETUP_TOP_DIR in environment. > > I'll not bother pressing this further since you seem set on the current > design, but my original concern that I pointed out was that I'll always > having to configure that, and likely document doing the need to do so for > users of any project I may want to use bitbake-setup for. The default > behaviors of other tools in this space do not require that. You'd always have to configure it because you want no external interference in builds even if that was limited to sstate/sstate mirrors/dl_dir? Or you don't believe dl_dir/sstate can be shared between builds/projects? I do understand your point but I also suspect there is some level of common user setting we should start trying to take account of. If that means adding some sanity check mechanism to only allow certain variables from site.conf, that is something we can look at. Cheers, Richard
On Wed, 20 Aug 2025, Richard Purdie wrote: > On Wed, 2025-08-20 at 16:49 -0400, Scott Murray wrote: > > On Wed, 20 Aug 2025, Alexander Kanavin wrote: > > > > > On Wed, 20 Aug 2025 at 22:15, Scott Murray <scott.murray@konsulko.com> wrote: > > > > Sure, but my point is that's what I'll be doing every time I set up > > > > any project (and maybe reconfiguring it?), so effectively for me the > > > > default behavior is not useful. I doubt I am alone in doing all my > > > > development work on a different disk than where /home is on my dev > > > > and build machines. > > > > > > Then there's the option of setting BITBAKE_SETUP_TOP_DIR in environment. > > > > I'll not bother pressing this further since you seem set on the current > > design, but my original concern that I pointed out was that I'll always > > having to configure that, and likely document doing the need to do so for > > users of any project I may want to use bitbake-setup for. The default > > behaviors of other tools in this space do not require that. > > You'd always have to configure it because you want no external > interference in builds even if that was limited to sstate/sstate > mirrors/dl_dir? Or you don't believe dl_dir/sstate can be shared > between builds/projects? I will always have to configure it because I don't want those in some default directory in $HOME, and there's going to be times I don't want them shared at all. With respect to sharing, IMO it depends on the project. I tend to keep downloads and sstate split by branch, as master branch has so much churn that it complicates pruning (which is to some degree an unsolved problem that everyone has their own ad hoc approach for). I also tend to not share sstate across significantly different customer projects that use less common layers, again because it makes pruning easier. > I do understand your point but I also suspect there is some level of > common user setting we should start trying to take account of. If that > means adding some sanity check mechanism to only allow certain > variables from site.conf, that is something we can look at. I'm not sure I follow your train of thought here? If you mean not allowing overriding DL_DIR or SSTATE_DIR from site.conf, I'd be strongly against that idea. Scott
On Wed, 2025-08-20 at 19:58 -0400, Scott Murray wrote: > On Wed, 20 Aug 2025, Richard Purdie wrote: > > > On Wed, 2025-08-20 at 16:49 -0400, Scott Murray wrote: > > > On Wed, 20 Aug 2025, Alexander Kanavin wrote: > > > > > > > On Wed, 20 Aug 2025 at 22:15, Scott Murray <scott.murray@konsulko.com> wrote: > > > > > Sure, but my point is that's what I'll be doing every time I set up > > > > > any project (and maybe reconfiguring it?), so effectively for me the > > > > > default behavior is not useful. I doubt I am alone in doing all my > > > > > development work on a different disk than where /home is on my dev > > > > > and build machines. > > > > > > > > Then there's the option of setting BITBAKE_SETUP_TOP_DIR in environment. > > > > > > I'll not bother pressing this further since you seem set on the current > > > design, but my original concern that I pointed out was that I'll always > > > having to configure that, and likely document doing the need to do so for > > > users of any project I may want to use bitbake-setup for. The default > > > behaviors of other tools in this space do not require that. > > > > You'd always have to configure it because you want no external > > interference in builds even if that was limited to sstate/sstate > > mirrors/dl_dir? Or you don't believe dl_dir/sstate can be shared > > between builds/projects? > > I will always have to configure it because I don't want those in some > default directory in $HOME, and there's going to be times I don't want > them shared at all. Keep in mind there are two levels to the config. One level is the basic site.conf, the second is the use of defaults in .cache. Personally, I think we should put DL_DIR/SSTATE_DIR where the user asks, defaulting to $HOME/.cache. Like you, I have dedicated specific spaces for these but that is fine and I can just set those in my site.conf and be happy, I won't need to specify it on the command every time. > With respect to sharing, IMO it depends on the > project. I tend to keep downloads and sstate split by branch, as > master branch has so much churn that it complicates pruning (which is to > some degree an unsolved problem that everyone has their own ad hoc > approach for). I also tend to not share sstate across significantly > different customer projects that use less common layers, again because > it makes pruning easier. state pruning is a valid concern and I do tend to do that myself with older releases but we should really work to improve the cleanup tools too. I'm not sure that issue should change the base design. > > I do understand your point but I also suspect there is some level of > > common user setting we should start trying to take account of. If that > > means adding some sanity check mechanism to only allow certain > > variables from site.conf, that is something we can look at. > > I'm not sure I follow your train of thought here? If you mean not > allowing overriding DL_DIR or SSTATE_DIR from site.conf, I'd be strongly > against that idea. I'm meaning only allowing setting certain variables from site.conf such as DL_DIR and SSTATE_DIR. Cheers, Richard
--- /home/alex/builds/poky-alex/config/poky-alex.conf.json 2024-12-16 11:43:24.077446096 +0100 +++ /home/alex/builds/poky-alex/config-tmp-asoubw5u/poky-alex.conf.json 2024-12-16 11:47:43.237104405 +0100 @@ -7,7 +7,7 @@ "uri": "git://git.yoctoproject.org/poky-contrib" } }, - "rev": "akanavin/sstate-for-all" + "rev": "akanavin/bitbake-setup-testing" }, "path": "poky" } === If the configuration has not changed, but layer revisions referred to it have (for example if the configuration specifies a tip of a branch), you will see that too: === ... Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to aeb73e29379fe6007a8adc8d94c1ac18a93e68de === 4. If the configuration has changed, you can bring it in sync with: === alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup update Default parameter values are in /home/alex/.bitbake-setup/config - adjust as needed. Fetching configuration repository git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/.bitbake-setup/configurations Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/bitbake-setup-testing from d174acad934f8ad1fe303abc5705733e15542859 to a3d2ee10045f8c1151d680ad97994c5d6cf51ece Fetching layer/tool repository poky into /home/alex/builds/poky-alex/layers/poky Setting up bitbake configuration gadget in /home/alex/bitbake-builds/poky-alex/build-gadget Existing bitbake congfiguration directory renamed to /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007 The bitbake configuration has changed: diff -uNr /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf /home/alex/builds/poky-alex/build-gadget/conf/local.conf --- /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf 2024-12-16 11:47:51.865043102 +0100 +++ /home/alex/builds/poky-alex/build-gadget/conf/local.conf 2024-12-16 11:50:07.811942847 +0100 @@ -287,5 +287,3 @@ # track the version of this file when it was generated. This can safely be ignored if # this doesn't mean anything to you. CONF_VERSION = "2" - -TCLIBC = "musl" Bitbake configuration summary: This configuration is intended for building gadget. Usage instructions and additional information in /home/alex/bitbake-builds/poky-alex/build-gadget/README === Note that it will also rename/preserve the existing build/conf directory, and print changes in bitbake configuration (diff of content of build/conf/) if that has changed. I can't at the moment think of anything more clever that is also not much more brittle or complex to implement, but open to suggestions. Terminology =========== - 'top directory' means the place under which bitbake-setup reads and writes everything. bitbake-setup makes a promise to not touch anything outside of that, unless otherwise directed to by entries in settings (currently there is one such setting for fetcher downloads for layers and config registries). Top directory can be selected by an environment variable, a command line option, or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment (e.g. we are in a bitbake environment), then the top directory is deduced from that and doesn't need to be specified by hand. - 'settings' means bitbake-setup operational parameters that are global to all builds under a top directory. E.g. the location of configuration registry, or where the bitbake fetcher should place the downloads (DL_DIR setting). Settings are stored in a .conf file in ini format just under the top directory. - 'build' means a tree structure set up by 'bitbake-setup init', consisting of, at least, a layers checkout, and a bitbake build. It maps 1:1 to the json data it was constructed from, which is called 'build configuration'. Build configurations are constructed from generic configurations that may involve making one or more choices about available options in them. Generic configurations are files, URLs or are obtained from git repositories called 'config registries', in which case they can be listed with 'bitbake-setup list'. There can be multiple 'builds' under a top directory. Here are two example generic configurations that showcase this: https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-options.conf.json https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-nested-configs.conf.json - 'bitbake-setup status' will tell if a build is in sync with the generic configuration it was made from. 'bitbake-setup update' will bring a build in sync with a configuration if needed. - 'bitbake build' means a particular sub-tree inside a build that bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR by oe-init-build-env. conf/* in that tree is 'bitbake configuration'. Bitbake configurations are constructed from templates and fragments, with existing mechanisms provided by oe-core. The configuration file format is specified such that other mechanisms to set up a bitbake build can be added; there was a mention of ability to specify local.conf content and a set of layers directly in a configuration. I think that scales poorly compared to templates and fragments, but I made sure alternative ways to configure a bitbake build are possible to add in the future :) - 'source override' is a json file that can be used to modify revisions and origins of layers that need to be checkout into a build (e.g. when master branches need to be changed to master-next for purposes of testing). Such a file is specified with a command-line option to 'init' and an example can be seen here: https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-next.override.json This commit includes a fix by Ryan Eatmon <reatmon@ti.com>: https://github.com/kanavin/bitbake/pull/1 Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- bin/bitbake-setup | 673 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 673 insertions(+) create mode 100755 bin/bitbake-setup diff --git a/bin/bitbake-setup b/bin/bitbake-setup new file mode 100755 index 000000000..11569e67d --- /dev/null +++ b/bin/bitbake-setup @@ -0,0 +1,673 @@ +#!/usr/bin/env python3 + +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import sys +import argparse +import warnings +import json +import shutil +import time +import stat +import tempfile +import configparser +import datetime + +default_registry = 'git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main' + +bindir = os.path.abspath(os.path.dirname(__file__)) +sys.path[0:0] = [os.path.join(os.path.dirname(bindir), 'lib')] + +import bb.msg +import bb.process + +logger = bb.msg.logger_create('bitbake-setup', sys.stdout) + +def cache_dir(top_dir): + return os.path.join(top_dir, '.bitbake-setup-cache') + +def init_bb_cache(settings, args): + dldir = settings["default"]["dl-dir"] + bb_cachedir = os.path.join(cache_dir(args.top_dir), 'bitbake-cache') + + d = bb.data.init() + d.setVar("DL_DIR", dldir) + d.setVar("BB_CACHEDIR", bb_cachedir) + d.setVar("__BBSRCREV_SEEN", "1") + if args.no_network: + d.setVar("BB_SRCREV_POLICY", "cache") + bb.fetch.fetcher_init(d) + return d + +def save_bb_cache(): + bb.fetch2.fetcher_parse_save() + bb.fetch2.fetcher_parse_done() + +def get_config_name(config): + return os.path.basename(config).split('.')[0] + +def write_config(config, config_dir): + with open(os.path.join(config_dir, "config-upstream.json"),'w') as s: + json.dump(config, s, sort_keys=True, indent=4) + +def commit_config(config_dir): + bb.process.run("git -C {} add .".format(config_dir)) + bb.process.run("git -C {} commit -a -m 'Configuration at {}'".format(config_dir, time.asctime())) + +def _write_layer_list(dest, repodirs): + layers = [] + for r in repodirs: + for root, dirs, files in os.walk(os.path.join(dest,r)): + if os.path.basename(root) == 'conf' and 'layer.conf' in files: + layers.append(os.path.relpath(os.path.dirname(root), dest)) + layers_f = os.path.join(dest, ".oe-layers.json") + with open(layers_f, 'w') as f: + json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4) + +def checkout_layers(layers, layerdir, d): + repodirs = [] + oesetupbuild = None + for r_name in layers: + r_data = layers[r_name] + repodir = r_data["path"] + repodirs.append(repodir) + + r_remote = r_data['git-remote'] + rev = r_remote['rev'] + remotes = r_remote['remotes'] + + for remote in remotes: + type,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + print("Fetching layer/tool repository {} into {}".format(r_name, os.path.join(layerdir,repodir))) + fetcher = bb.fetch.Fetch(["{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir)], d) + do_fetch(fetcher, layerdir) + + if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')): + oesetupbuild = os.path.join(layerdir, repodir, 'scripts/oe-setup-build') + oeinitbuildenv = os.path.join(layerdir, repodir, 'oe-init-build-env') + + _write_layer_list(layerdir, repodirs) + + if oesetupbuild: + links = {'setup-build': oesetupbuild, 'oe-scripts': os.path.dirname(oesetupbuild), 'init-build-env': oeinitbuildenv} + for l,t in links.items(): + symlink = os.path.join(layerdir, l) + if os.path.lexists(symlink): + os.remove(symlink) + os.symlink(os.path.relpath(t,layerdir),symlink) + +def setup_bitbake_build(bitbake_config, layerdir, builddir): + def _setup_build_conf(layers, build_conf_dir): + os.makedirs(build_conf_dir) + layers_s = "\n".join([" {} \\".format(os.path.join(layerdir,l)) for l in layers]) + bblayers_conf = """BBLAYERS ?= " \ +{} + " +""".format(layers_s) + with open(os.path.join(build_conf_dir, "bblayers.conf"), 'w') as f: + f.write(bblayers_conf) + + local_conf = """# +# This file is intended for local configuration tweaks. +# +# If you would like to publish and share changes made to this file, +# it is recommended to put them into a distro config, or to create +# layer fragments from changes made here. +# +""" + with open(os.path.join(build_conf_dir, "local.conf"), 'w') as f: + f.write(local_conf) + + with open(os.path.join(build_conf_dir, "templateconf.cfg"), 'w') as f: + f.write("") + + with open(os.path.join(build_conf_dir, "conf-summary.txt"), 'w') as f: + f.write(bitbake_config["description"] + "\n") + + with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as f: + f.write("") + + def _make_init_build_env(builddir, initbuildenv): + cmd = ". {} {}".format(initbuildenv, builddir) + initbuild_in_builddir = os.path.join(builddir, 'init-build-env') + with open(initbuild_in_builddir, 'w') as f: + f.write(cmd) + + bitbake_builddir = os.path.join(builddir, "build") + print("==============================") + print("Setting up bitbake configuration in {}".format(bitbake_builddir)) + + template = bitbake_config.get("oe-template") + layers = bitbake_config.get("bb-layers") + if not template and not layers: + print("Bitbake configuration does not contain a reference to an OpenEmbedded build template via 'oe-template' or a list of layers via 'bb-layers'; please use oe-setup-build, oe-init-build-env or another mechanism manually to complete the setup.") + return + oesetupbuild = os.path.join(layerdir, 'setup-build') + if template and not os.path.exists(oesetupbuild): + raise Exception("Cannot complete setting up a bitbake build directory from OpenEmbedded template '{}' as oe-setup-build was not found in any layers; please use oe-init-build-env manually.".format(template)) + + bitbake_confdir = os.path.join(bitbake_builddir, 'conf') + backup_bitbake_confdir = bitbake_confdir + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + if os.path.exists(bitbake_confdir): + os.rename(bitbake_confdir, backup_bitbake_confdir) + + if layers: + _setup_build_conf(layers, bitbake_confdir) + + if template: + bb.process.run("{} setup -c {} -b {} --no-shell".format(oesetupbuild, template, bitbake_builddir)) + else: + initbuildenv = os.path.join(layerdir, 'init-build-env') + if not os.path.exists(initbuildenv): + print("Could not find oe-init-build-env in any of the layers; please use another mechanism to initialize the bitbake environment") + return + _make_init_build_env(bitbake_builddir, os.path.realpath(initbuildenv)) + + init_script = os.path.join(bitbake_builddir, "init-build-env") + shell = os.path.basename(os.environ.get("SHELL","bash")) + fragments = bitbake_config.get("oe-fragments", []) + sorted(bitbake_config.get("oe-fragment-choices",{}).values()) + if fragments: + bb.process.run("{} -c '. {} && bitbake-config-build enable-fragment {}'".format(shell, init_script, " ".join(fragments))) + + if os.path.exists(backup_bitbake_confdir): + bitbake_config_diff = get_diff(backup_bitbake_confdir, bitbake_confdir) + if bitbake_config_diff: + print("Existing bitbake configuration directory renamed to {}".format(backup_bitbake_confdir)) + print("The bitbake configuration has changed:") + print(bitbake_config_diff) + else: + shutil.rmtree(backup_bitbake_confdir) + + print("This bitbake configuration provides: {}".format(bitbake_config["description"])) + + readme = """{}\n\nAdditional information is in {} and {}\n +Source the environment using '. {}' to run builds from the command line. +The bitbake configuration files (local.conf, bblayers.conf and more) can be found in {}/conf +""".format( + bitbake_config["description"], + os.path.join(bitbake_builddir,'conf/conf-summary.txt'), + os.path.join(bitbake_builddir,'conf/conf-notes.txt'), + init_script, + bitbake_builddir + ) + readme_file = os.path.join(bitbake_builddir, "README") + with open(readme_file, 'w') as f: + f.write(readme) + print("Usage instructions and additional information are in {}".format(readme_file)) + +def get_registry_config(registry_path, id): + for root, dirs, files in os.walk(registry_path): + for f in files: + if f.endswith('.conf.json') and id == get_config_name(f): + return os.path.join(root, f) + raise Exception("Unable to find {} in available configurations; use 'list' sub-command to see what is available".format(id)) + +def update_build(config, confdir, builddir, layerdir, d): + layer_config = config["data"]["sources"] + layer_overrides = config["source-overrides"]["sources"] + for k,v in layer_overrides.items(): + if k in layer_config: + layer_config[k]["git-remote"] = v["git-remote"] + checkout_layers(layer_config, layerdir, d) + bitbake_config = config["bitbake-config"] + setup_bitbake_build(bitbake_config, layerdir, builddir) + +def flatten_bitbake_configs(configs): + def merge_configs(c1,c2): + c_merged = {} + for k,v in c2.items(): + if k not in c1.keys(): + c_merged[k] = v + for k,v in c1.items(): + if k not in c2.keys(): + c_merged[k] = v + else: + c_merged[k] = c1[k] + c2[k] + del c_merged['configurations'] + return c_merged + + flattened_configs = [] + for c in configs: + if 'configurations' not in c: + flattened_configs.append(c) + else: + for sub_c in flatten_bitbake_configs(c['configurations']): + flattened_configs.append(merge_configs(c, sub_c)) + return flattened_configs + +def choose_bitbake_config(configs, parameters, non_interactive): + flattened_configs = flatten_bitbake_configs(configs) + configs_dict = {i["name"]:i for i in flattened_configs} + + if parameters: + config_id = parameters[0] + if config_id not in configs_dict: + raise Exception("Bitbake configuration {} not found; replace with one of {}".format(config_id, configs_dict)) + return configs_dict[config_id] + + enumerated_configs = list(enumerate(flattened_configs)) + if len(enumerated_configs) == 1: + only_config = flattened_configs[0] + print("Selecting the only available bitbake configuration {}".format(only_config["name"])) + return only_config + + if non_interactive: + raise Exception("Unable to choose from bitbake configurations in non-interactive mode: {}".format(configs_dict)) + + print("Available bitbake configurations:") + for n, config_data in enumerated_configs: + print("{}. {}\t{}".format(n, config_data["name"], config_data["description"])) + print("\nPlease select one of the above bitbake configurations by its number:") + config_n = int(input()) + return flattened_configs[config_n] + +def choose_config(configs, non_interactive): + not_expired_configs = [k for k in configs.keys() if not has_expired(configs[k].get("expires", None))] + config_list = list(enumerate(not_expired_configs)) + if len(config_list) == 1: + only_config = config_list[0][1] + print("Selecting the only available configuration {}".format(only_config)) + return only_config + + if non_interactive: + raise Exception("Unable to choose from configurations in non-interactive mode: {}".format(not_expired_configs)) + + print("\nAvailable configurations:") + for n, config_name in config_list: + config_data = configs[config_name] + expiry_date = config_data.get("expires", None) + config_desc = config_data["description"] + if expiry_date: + print("{}. {}\t{} (supported until {})".format(n, config_name, config_desc, expiry_date)) + else: + print("{}. {}\t{}".format(n, config_name, config_desc)) + print("\nPlease select one of the above configurations by its number:") + config_n = int(input()) + return config_list[config_n][1] + +def choose_fragments(possibilities, parameters, non_interactive): + choices = {} + for k,v in possibilities.items(): + choice = [o for o in v["options"] if o in parameters] + if len(choice) > 1: + raise Exception("Options specified on command line do not allow a single selection from possibilities {}, please remove one or more from {}".format(v["options"], parameters)) + if len(choice) == 1: + choices[k] = choice[0] + continue + + if non_interactive: + raise Exception("Unable to choose from options in non-interactive mode: {}".format(v["options"])) + + print(v["description"]+ ":") + options_enumerated = list(enumerate(v["options"])) + for n,o in options_enumerated: + print("{}. {}".format(n, o)) + print("Please select one of the above options by its number:") + option_n = int(input()) + choices[k] = options_enumerated[option_n][1] + return choices + +def obtain_config(settings, args, source_overrides, d): + if args.config: + config_id = args.config[0] + config_parameters = args.config[1:] + if os.path.exists(config_id): + print("Reading configuration from local file {}".format(config_id)) + upstream_config = {'type':'local','path':os.path.abspath(args.config),'name':get_config_name(config_id),'data':json.load(open(config_path))} + elif config_id.startswith("http://") or config_id.startswith("https://"): + print("Reading configuration from network URI {}".format(config_id)) + import urllib.request + with urllib.request.urlopen(config_id) as f: + upstream_config = {'type':'network','uri':config_id,'name':get_config_name(config_id),'data':json.load(f)} + else: + print("Looking up config {} in configuration registry".format(config_id)) + registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + if config_id not in registry_configs: + raise Exception("Config {} not found in configuration registry, re-run 'init' without parameters to choose from available configurations.".format(config_id)) + upstream_config = {'type':'registry','registry':settings["default"]["registry"],'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + expiry_date = upstream_config['data'].get("expires", None) + if has_expired(expiry_date): + print("This configuration is no longer supported after {}. Please consider changing to a supported configuration.".format(expiry_date)) + else: + registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + registry_configs = list_registry(registry_path, with_expired=True) + config_id = choose_config(registry_configs, args.non_interactive) + config_parameters = [] + upstream_config = {'type':'registry','registry':settings["default"]["registry"],'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))} + + upstream_config['bitbake-config'] = choose_bitbake_config(upstream_config['data']['bitbake-setup']['configurations'], config_parameters, args.non_interactive) + upstream_config['bitbake-config']['oe-fragment-choices'] = choose_fragments(upstream_config['bitbake-config'].get('oe-fragments-one-of',{}), config_parameters[1:], args.non_interactive) + upstream_config['non-interactive-cmdline-options'] = " ".join([upstream_config['name'], upstream_config['bitbake-config']['name']] + sorted(upstream_config['bitbake-config']['oe-fragment-choices'].values())) + upstream_config['source-overrides'] = source_overrides + return upstream_config + +def init_config(settings, args, d): + stdout = sys.stdout + def handle_task_progress(event, d): + rate = event.rate if event.rate else '' + progress = event.progress if event.progress > 0 else 0 + print("{}% {} ".format(progress, rate), file=stdout, end='\r') + + source_overrides = json.load(open(args.source_overrides)) if args.source_overrides else {'sources':{}} + upstream_config = obtain_config(settings, args, source_overrides, d) + print("Run 'bitbake-setup {}' to select this configuration non-interactively.".format(upstream_config['non-interactive-cmdline-options'])) + + builddir = os.path.join(os.path.abspath(args.top_dir), upstream_config['non-interactive-cmdline-options'].replace(" ","-").replace("/","_")) + if os.path.exists(builddir): + print("Build already initialized in {}\nUse 'bitbake-setup status' to check if it needs to be updated or 'bitbake-setup update' to perform the update.".format(builddir)) + return + + print("Initializing a build in {}".format(builddir)) + + os.makedirs(builddir) + + confdir = os.path.join(builddir, "config") + layerdir = os.path.join(builddir, "layers") + + os.makedirs(confdir) + os.makedirs(layerdir) + + bb.process.run("git -C {} init -b main".format(confdir)) + bb.process.run("git -C {} commit --allow-empty -m 'Initial commit'".format(confdir)) + + bb.event.register("bb.build.TaskProgress", handle_task_progress, data=d) + + write_config(upstream_config, confdir) + commit_config(confdir) + update_build(upstream_config, confdir, builddir, layerdir, d) + + bb.event.remove("bb.build.TaskProgress", None) + +def get_diff(file1, file2): + try: + bb.process.run('diff -uNr {} {}'.format(file1, file2)) + except bb.process.ExecutionError as e: + if e.exitcode == 1: + return e.stdout + else: + raise e + return None + +def are_layers_changed(layers, layerdir, d): + changed = False + for r_name in layers: + r_data = layers[r_name] + repodir = r_data["path"] + + r_remote = r_data['git-remote'] + rev = r_remote['rev'] + remotes = r_remote['remotes'] + + for remote in remotes: + type,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) + fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params)) + fetcher = bb.fetch.FetchData("{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir), d) + upstream_revision = fetcher.method.latest_revision(fetcher, d, 'default') + rev_parse_result = bb.process.run('git -C {} rev-parse HEAD'.format(os.path.join(layerdir, repodir))) + local_revision = rev_parse_result[0].strip() + if upstream_revision != local_revision: + changed = True + print('Layer repository {} checked out into {} updated revision {} from {} to {}'.format(remotes[remote]["uri"], os.path.join(layerdir, repodir), rev, local_revision, upstream_revision)) + + return changed + +def build_status(settings, args, d, update=False): + builddir = args.build_dir + + confdir = os.path.join(builddir, "config") + layerdir = os.path.join(builddir, "layers") + + current_upstream_config = json.load(open(os.path.join(confdir, "config-upstream.json"))) + + args.config = current_upstream_config['non-interactive-cmdline-options'].split() + args.non_interactive = True + source_overrides = current_upstream_config["source-overrides"] + new_upstream_config = obtain_config(settings, args, source_overrides, d) + + write_config(new_upstream_config, confdir) + config_diff = bb.process.run('git -C {} diff'.format(confdir))[0] + + if config_diff: + print('Configuration in {} has changed:\n{}'.format(builddir, config_diff)) + if update: + commit_config(confdir) + update_build(new_upstream_config, confdir, builddir, layerdir, d) + else: + bb.process.run('git -C {} restore config-upstream.json'.format(confdir)) + return + + if are_layers_changed(current_upstream_config["data"]["sources"], layerdir, d): + if update: + update_build(current_upstream_config, confdir, builddir, layerdir, d) + return + + print("Configuration in {} has not changed.".format(builddir)) + +def build_update(settings, args, d): + build_status(settings, args, d, update=True) + +def do_fetch(fetcher, dir): + # git fetcher simply dumps git output to stdout; in bitbake context that is redirected to temp/log.do_fetch + # and we need to set up smth similar here + fetchlogdir = os.path.join(dir, 'logs') + os.makedirs(fetchlogdir, exist_ok=True) + fetchlog = os.path.join(fetchlogdir, 'fetch_log.{}'.format(datetime.datetime.now().strftime("%Y%m%d%H%M%S"))) + with open(fetchlog, 'a') as f: + oldstdout = sys.stdout + sys.stdout = f + fetcher.download() + fetcher.unpack(dir) + sys.stdout = oldstdout + +def update_registry(registry, cachedir, d): + registrydir = 'configurations' + full_registrydir = os.path.join(cachedir, registrydir) + print("Fetching configuration registry {} into {}".format(registry, full_registrydir)) + fetcher = bb.fetch.Fetch(["{};destsuffix={}".format(registry, registrydir)], d) + do_fetch(fetcher, cachedir) + return full_registrydir + +def has_expired(expiry_date): + if expiry_date: + return datetime.datetime.now() > datetime.datetime.fromisoformat(expiry_date) + return False + +def list_registry(registry_path, with_expired): + json_data = {} + + for root, dirs, files in os.walk(registry_path): + for f in files: + if f.endswith('.conf.json'): + config_name = get_config_name(f) + config_data = json.load(open(os.path.join(root, f))) + config_desc = config_data["description"] + expiry_date = config_data.get("expires", None) + if expiry_date: + if with_expired or not has_expired(expiry_date): + json_data[config_name] = {"description": config_desc, "expires": expiry_date} + else: + json_data[config_name] = {"description": config_desc} + return json_data + +def list_configs(settings, args, d): + registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + json_data = list_registry(registry_path, args.with_expired) + print("\nAvailable configurations:") + for config_name, config_data in json_data.items(): + expiry_date = config_data.get("expires", None) + config_desc = config_data["description"] + if expiry_date: + if args.with_expired or not has_expired(expiry_date): + print("{}\t{} (supported until {})".format(config_name, config_desc, expiry_date)) + else: + print("{}\t{}".format(config_name, config_desc)) + print("\nRun 'init' with one of the above configuration identifiers to set up a build.") + + if args.write_json: + with open(args.write_json, 'w') as f: + json.dump(json_data, f, sort_keys=True, indent=4) + print("Available configurations written into {}".format(args.write_json)) + +def default_settings_path(top_dir): + return os.path.join(top_dir, 'bitbake-setup.conf') + +def write_settings(top_dir, force_replace): + settings_path = default_settings_path(top_dir) + if not os.path.exists(settings_path) or force_replace: + if os.path.exists(settings_path): + backup_conf = settings_path + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + os.rename(settings_path, backup_conf) + print("Previous settings are in {}".format(backup_conf)) + + settings = configparser.ConfigParser() + settings['default'] = { + 'registry':default_registry, + 'dl-dir':os.path.join(top_dir, '.bitbake-setup-downloads'), + } + os.makedirs(os.path.dirname(settings_path), exist_ok=True) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + print('Created a new settings file in {}.\n'.format(settings_path)) + +def load_settings(top_dir): + # This creates a new settings file if it does not yet exist + write_settings(top_dir, force_replace=False) + + settings_path = default_settings_path(top_dir) + settings = configparser.ConfigParser() + print('Loading settings from {}\n'.format(settings_path)) + settings.read([settings_path]) + return settings + +def change_settings(top_dir, new_settings): + # This creates a new settings file if it does not yet exist + write_settings(top_dir, force_replace=False) + + settings = load_settings(top_dir) + for section, section_settings in new_settings.items(): + for setting, value in section_settings.items(): + settings[section][setting] = value + print("Setting '{}' in section '{}' is changed to '{}'".format(setting, section, value)) + + settings_path = default_settings_path(top_dir) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + print("New settings written to {}".format(settings_path)) + return settings + +def get_build_dir_via_bbpath(): + bbpath = os.environ.get('BBPATH') + if bbpath: + bitbake_dir = os.path.normpath(bbpath.split(':')[0]) + if os.path.exists(os.path.join(bitbake_dir,'init-build-env')): + build_dir = os.path.dirname(bitbake_dir) + return build_dir + return None + +def get_default_top_dir(): + build_dir_via_bbpath = get_build_dir_via_bbpath() + if build_dir_via_bbpath: + top_dir = os.path.dirname(build_dir_via_bbpath) + if os.path.exists(default_settings_path(top_dir)): + return top_dir + return os.environ.get('BITBAKE_SETUP_TOP_DIR') or os.path.join(os.path.expanduser('~'), 'bitbake-builds') + +def main(): + def add_top_dir_arg(parser): + parser.add_argument('--top-dir', default=get_default_top_dir(), help='Top level directory where builds are set up and downloaded configurations and layers are cached for reproducibility and offline builds, default is %(default)s, can be overriden via BITBAKE_SETUP_TOP_DIR environment variable') + + def add_build_dir_arg(parser): + build_dir = get_build_dir_via_bbpath() + if build_dir: + parser.add_argument('--build-dir', default=build_dir, help="Path to the build, default is %(default)s via BBPATH") + else: + parser.add_argument('--build-dir', required=True, help="Path to the build") + + parser = argparse.ArgumentParser( + description="BitBake setup utility. Run with 'init' argument to get started.", + epilog="Use %(prog)s <subcommand> --help to get help on a specific command" + ) + parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true') + parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true') + parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR') + parser.add_argument('--no-network', action='store_true', help='Do not check whether configuration repositories and layer repositories have been updated; use only the local cache.') + + subparsers = parser.add_subparsers() + + parser_list = subparsers.add_parser('list', help='List available configurations') + add_top_dir_arg(parser_list) + parser_list.add_argument('--with-expired', action='store_true', help='List also configurations that are no longer supported due to reaching their end-of-life dates.') + parser_list.add_argument('--write-json', action='store', help='Write available configurations into a json file so they can be programmatically processed.') + parser_list.set_defaults(func=list_configs) + + parser_init = subparsers.add_parser('init', help='Select a configuration and initialize a build from it') + add_top_dir_arg(parser_init) + parser_init.add_argument('config', nargs='*', help="path/URL/id to a configuration file (use 'list' command to get available ids), followed by configuration options. Bitbake-setup will ask to choose from available choices if command line doesn't completely specify them.") + parser_init.add_argument('--non-interactive', action='store_true', help='Do not ask to interactively choose from available options; if bitbake-setup cannot make a decision it will stop with a failure.') + parser_init.add_argument('--source-overrides', action='store', help='Override sources information (repositories/revisions) with values from a local json file.') + parser_init.set_defaults(func=init_config) + + parser_status = subparsers.add_parser('status', help='Check if the build needs to be synchronized with configuration') + add_build_dir_arg(parser_status) + parser_status.set_defaults(func=build_status) + + parser_update = subparsers.add_parser('update', help='Update a build to be in sync with configuration') + add_build_dir_arg(parser_update) + parser_update.set_defaults(func=build_update) + + parser_reset_settings = subparsers.add_parser('reset-settings', help='Write a settings file with default values into the top level directory (contains the location of build configuration registry, downloads directory and other global settings)') + add_top_dir_arg(parser_reset_settings) + parser_reset_settings.set_defaults(func=write_settings) + + parser_change_setting = subparsers.add_parser('change-setting', help='Change a setting in the settings file') + add_top_dir_arg(parser_change_setting) + parser_change_setting.add_argument('section', help="Section in a settings file, typically 'default'") + parser_change_setting.add_argument('key', help="Name of the setting") + parser_change_setting.add_argument('value', help="Value of the setting") + parser_change_setting.set_defaults(func=change_settings) + + args = parser.parse_args() + + logging.basicConfig(stream=sys.stdout) + if args.debug: + logger.setLevel(logging.DEBUG) + elif args.quiet: + logger.setLevel(logging.ERROR) + + # Need to re-run logger_create with color argument + # (will be the same logger since it has the same name) + bb.msg.logger_create('bitbake-setup', output=sys.stdout, + color=args.color, + level=logger.getEffectiveLevel()) + + + if hasattr(args, 'build_dir'): + if not os.path.exists(os.path.join(args.build_dir,'build', 'init-build-env')): + print("Not a valid build directory: build/init-build-env does not exist in {}".format(args.build_dir)) + return + # commands without --top-dir argument (status, update) still need to know where + # the top dir is, but it should be auto-deduced as parent of args.build_dir + if not hasattr(args, 'top_dir'): + args.top_dir = os.path.dirname(os.path.normpath(args.build_dir)) + + if 'func' in args: + if args.func == write_settings: + write_settings(args.top_dir, force_replace=True) + elif args.func == change_settings: + change_settings(args.top_dir, {args.section:{args.key:args.value}}) + else: + settings = load_settings(args.top_dir) + d = init_bb_cache(settings, args) + args.func(settings, args, d) + save_bb_cache() + else: + from argparse import Namespace + parser.print_help() + +main()