| Message ID | 20221110092433.2874130-1-alex@linutronix.de |
|---|---|
| State | New |
| Headers | show |
| Series | oe-setup-build: add a tool for discovering config templates and setting up builds | expand |
On Thu, 2022-11-10 at 10:24 +0100, Alexander Kanavin wrote: > This is the last (I believe) piece of the puzzle in setting up builds from nothing > without having to write custom scripts or use external tools. > > After layers have been fetched and placed into their respective locations, one would > surely want to proceed to the actual build, and here's how: > > 1. Without arguments or with 'list-config-templates', the tool simply walks the ../.. > of its location (which is the parent dir of poky/oe-core), and prints what templates it > has found, as seen below. If the following is not enough information, > adding '-v' will also print conf-notes.txt for each of the templates: > > $ oe-setup-build > Available build configuration templates (re-run with -v to see their descriptions): > > /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/poky/meta-poky/conf/templates/default > will create a build configuration in /srv/work/alex/build-meta-poky-default > > /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gizmo > will create a build configuration in /srv/work/alex/build-meta-alex-configuration-gizmo > > /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gadget > will create a build configuration in /srv/work/alex/build-meta-alex-configuration-gadget > > 2. Then the user picks one command of the above and runs it. This will land them in a shell ready to run bitbake: > > $ oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gizmo > Running: TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/configuration-gizmo . /srv/work/alex/poky/oe-init-build-env /srv/work/alex/build-meta-alex-configuration-gizmo && /bin/bash > You had no conf/local.conf file. This configuration file has therefore been > created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gizmo/local.conf.sample > You may wish to edit it to, for example, select a different MACHINE (target > hardware). See conf/local.conf for more information as common configuration > options are commented. > > You had no conf/bblayers.conf file. This configuration file has therefore been > created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gizmo/bblayers.conf.sample > To add additional metadata layers into your configuration please add entries > to conf/bblayers.conf. > > The Yocto Project has extensive documentation about OE including a reference > manual which can be found at: > https://docs.yoctoproject.org > > For more information about OpenEmbedded see the website: > https://www.openembedded.org/ > > This configuration template will set up a build for the purposes of supporting gizmo. > Please refer to meta-alex/README for additional details and available bitbake targets. > > 3. The full set of command line options is: > > $ oe-setup-build -h > usage: oe-setup-build [-h] {list-config-templates,setup-build-env} ... > > A script that discovers available build configuration templates and sets up a build environment based on one of them > > positional arguments: > {list-config-templates,setup-build-env} > list-config-templates > List available configuration templates > setup-build-env Set up a build environment and open a shell session with it, ready to run builds. > > optional arguments: > -h, --help show this help message and exit > > $ oe-setup-build list-config-templates -h > usage: oe-setup-build list-config-templates [-h] [--topdir TOPDIR] [-v] > > optional arguments: > -h, --help show this help message and exit > --topdir TOPDIR Where to look for available build configuration templates (default is /srv/work/alex). > -v Print a description for each available build configuration template. > > $ oe-setup-build setup-build-env -h > usage: oe-setup-build setup-build-env [-h] -c template_path [-b build_path] [--no-shell] > > optional arguments: > -h, --help show this help message and exit > -c template_path Use a build configuration template in template_path to set up a build environment (run this script with 'list-config-templates' to see what is available) > -b build_path Set up a build directory in build_path (run this script with 'list-config-templates' to see where it would be by default) > --no-shell Create a build directory but do not start a shell session with the build environment from it. > > 4. There's also a selftest that runs a basic check for template discovery and build setup, > and an added hint in oe-setup-layers about how to proceed (as it is really not user-friendly > to fetch the layer repos successfully and then exit without a word). > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> I appreciate I've taken a long time to respond to this, sorry about that. I'd hoped others might give feedback but that hasn't happened and something about the patch doesn't "feel" right to me. The trouble is that as soon as I say that, you'd understandably want to know what it is and I'm having trouble quantifying it. Firstly, let me explain why I'm holding this change to a particularly high standard. The issue the namespaces, this isn't some random layer, it is in the core and it is adding commands to our core tools. As such they need to be right. They need to ensure the project scales in the right ways and ensure we have the ability to add functionality where needed in the future. Sometimes there is a case for adding something and then incrementally improving but in this specific case that is harder. I think part of my concern is that the tool isn't working with the layers as everything else does. This results in a bit of an ugly commandline interface and poor capabilities of the tool. By that I mean that ideally we shouldn't be walking TOPDIR but we should be be iterating over the layers looking for config templates in each one. A config template in one layer could override another using the standard approach layers have to override each other. The challenge here is that bblayers.conf is coming from the templates, so we can't parse anything until we at least have bblayers.conf to say which layers to parse. This leads to a few different questions: a) Should bblayers.conf be autogenerated? b) Should oe-setup-layers write bblayers.conf? c) Should oe-setup-layers write the json somewhere so that the list of layers can be known? d) Should oe-setup-build run in two stages, establishing bblayers.conf, then once present, list the configs? e) Should bblayers.conf be moved somewhere other than the build directory? I appreciate that the tool works with what we have today, however I think what we have today is part of the problem we need to fix. If we add this patch, we're then adding more pieces we're "stuck" to supporting. There is also an implied challenge here of the different stages of "checkout the repos", "document those checkouts in bblayers.conf" and "here is a separate build directory". The layer config file is where information from the two are mixed but I suspect there is something different we could do. For example, if the config fragment says "I need layers X, Y" but the layer setup json might define Z as well. We'd need a markup for "layers X, Y, Z are available" from oe-setup-layers. oe-setup-build could then iterate the layers, show the configs, then generate a setup for configX which only needed layers Y and Z configured. Part of the issue here is also the handoff between initial setup and continued development. If we can solve that issue neatly it would be a huge win for the project and it is exactly the kind of thing we'd want to have a tool in core doing. Going back to this patch, I think the paths needed are a sign that we need more data being stored for the tools to act upon to allow for more intelligent configuration of the builds. That would then naturally lead into what I've talked about elsewhere, with the idea of "pre-canned" config/setup and ultimately that would allow us to drop things like the combo-layer poky. Hopefully that gives some idea of why I'm a bit reluctant to add this patch in this particular namespace. The oe-setup-layers piece is doing something which is unlikely to fundamentally change (git clone/checkouts), but this piece is where we have things we do need to improve. Cheers, Richard
On Mon, 16 Jan 2023 at 14:30, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > There is also an implied challenge here of the different stages of > "checkout the repos", "document those checkouts in bblayers.conf" and > "here is a separate build directory". The layer config file is where > information from the two are mixed but I suspect there is something > different we could do. > > For example, if the config fragment says "I need layers X, Y" but the > layer setup json might define Z as well. We'd need a markup for "layers > X, Y, Z are available" from oe-setup-layers. oe-setup-build could then > iterate the layers, show the configs, then generate a setup for configX > which only needed layers Y and Z configured. What I would like to try is have oe-setup-layers first clone/checkout the layer repositories (according to what is in the supplied json), then walk the resulting directories looking for conf/layer.conf, e.g. determine what actual layers are in the checkout and where they are. Then this list would be written out to a file in the top level directory containing the layer repos for further usage by other tools. E.g.: /path/to/setup-layers --destdir path/to/checkout/ would write a path/to/checkout/oe-layers.txt (final filename TBD) containing layer paths relative to the file itself, e.g. (final format also TBD) meta-intel meta-arm meta-openembedded/meta-oe meta-openembedded/meta-networking and the location of oe-setup-build into a runnable shell script path/to/checkout/setup-build.sh: #!/bin/sh poky/scripts/oe-setup-build --layers-list ./oe-layers.txt $* This leaves bblayers.conf(.sample) unchanged and exactly as it was - a template/build item consumed by bitbake only and used to list layers active in a specific build. Alex
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index b048948386..328af91532 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -124,6 +124,18 @@ class BitbakeLayers(OESelftestTestCase): fullpath = os.path.join(layerpath, "conf", "templates", "buildconf-1", f) self.assertTrue(os.path.exists(fullpath), "Template configuration file {} not found".format(fullpath)) + cmd = 'oe-setup-build list-config-templates --topdir {}'.format(layerpath) + result = runCmd(cmd) + cond = "scripts/oe-setup-build setup-build-env -c " in result.output and "test-bitbakelayer-layercreate/conf/templates/buildconf-1" in result.output + self.assertTrue(cond, "Incorrect output from {}: {}".format(cmd, result.output)) + + # rather than hardcode the build setup cmdline here, let's actually run what the tool suggests to the user + cmd = None + for l in result.output.splitlines(): + if "scripts/oe-setup-build setup-build-env -c " in l: + cmd = l + " --no-shell" + result = runCmd(cmd) + def get_recipe_basename(self, recipe): recipe_file = "" result = runCmd("bitbake-layers show-recipes -f %s" % recipe) diff --git a/scripts/oe-setup-build b/scripts/oe-setup-build new file mode 100755 index 0000000000..7c125425ec --- /dev/null +++ b/scripts/oe-setup-build @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import argparse +import json +import os +import subprocess + +def defaulttop(): + return os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')) + +def makebuildpath(topdir, templatelocation, template): + return os.path.join(topdir, "build-{}-{}".format(os.path.basename(os.path.abspath(os.path.join(templatelocation,'..','..'))), template)) + +def print_templates(args): + topdir = args.topdir + verbose = args.v + print("Available build configuration templates (re-run with -v to see their descriptions):\n") + for (dir, dirs, files) in os.walk(topdir): + if dir.endswith('conf/templates'): + for d in dirs: + print("{} setup-build-env -c {}\nwill create a build configuration in {}\n".format(__file__, os.path.join(dir, d), makebuildpath(args.topdir, dir, d))) + if verbose: + if os.path.join(dir, d).endswith('meta-poky/conf/templates/default'): + print("Description: this is the reference configuration of the poky reference distribution (choose this if you are uncertain).") + elif os.path.join(dir, d).endswith('meta/conf/templates/default'): + print("Description: this is the reference configuration of the openembedded-core layer (choose this if you are uncertain).") + else: + print("Description:", open(os.path.join(dir, d, 'conf-notes.txt')).read()) + print("---") + + # Do not recurse into build directories; they can be enormous + if 'conf' in dirs and 'bblayers.conf' in os.listdir(os.path.join(dir, 'conf')): + dirs.clear() + # Do not recurse into sstate-cache or downloads similarly + if 'universal' in dirs and '00' in dirs: + dirs.clear() + if 'uninative' in dirs and 'git2' in dirs: + dirs.clear() + +def setup_build_env(args): + template = args.c + builddir = args.b + no_shell = args.no_shell + if not builddir: + builddir = makebuildpath(defaulttop(), os.path.dirname(template), os.path.basename(template)) + coredir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) + cmd = "TEMPLATECONF={} . {} {}".format(template, os.path.join(coredir, 'oe-init-build-env'), builddir) + if not no_shell: + cmd = cmd + " && {}".format(os.environ['SHELL']) + print("Running:", cmd) + subprocess.run(cmd, shell=True, executable=os.environ['SHELL']) + +parser = argparse.ArgumentParser(description="A script that discovers available build configuration templates and sets up a build environment based on one of them") +subparsers = parser.add_subparsers() +parser_list_templates = subparsers.add_parser('list-config-templates', help='List available configuration templates') + +parser_list_templates.add_argument("--topdir", default=defaulttop(), help='Where to look for available build configuration templates (default is {}).'.format(defaulttop())) +parser_list_templates.add_argument('-v', action='store_true', + help='Print a description for each available build configuration template.') +parser_list_templates.set_defaults(func=print_templates) + +parser_setup_env = subparsers.add_parser('setup-build-env', help='Set up a build environment and open a shell session with it, ready to run builds.') +parser_setup_env.add_argument('-c', metavar='template_path', required=True, help="Use a build configuration template in template_path to set up a build environment (run this script with 'list-config-templates' to see what is available)") +parser_setup_env.add_argument('-b', metavar='build_path', help="Set up a build directory in build_path (run this script with 'list-config-templates' to see where it would be by default)") +parser_setup_env.add_argument('--no-shell', action='store_true', + help='Create a build directory but do not start a shell session with the build environment from it.') +parser_setup_env.set_defaults(func=setup_build_env) + +args = parser.parse_args() +if 'func' in args: + args.func(args) +else: + from argparse import Namespace + print_templates(Namespace(topdir=defaulttop(), v=False)) diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers index 6ecaffed75..7fe250c557 100755 --- a/scripts/oe-setup-layers +++ b/scripts/oe-setup-layers @@ -17,6 +17,7 @@ import os import subprocess def _do_checkout(args, json): + oesetupbuild = None layers = json['sources'] for l_name in layers: l_data = layers[l_name] @@ -53,6 +54,12 @@ def _do_checkout(args, json): print("Running '{}' in {}".format(cmd, layerdir)) subprocess.check_output(cmd, shell=True, cwd=layerdir) + if os.path.exists(os.path.join(layerdir, 'scripts/oe-setup-build')): + oesetupbuild = os.path.join(layerdir, 'scripts/oe-setup-build') + + if oesetupbuild: + print("\nRun '{} list-config-templates' to list available build configuration templates and set up a build from one of them.".format(oesetupbuild)) + parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/") parser.add_argument('--force-bootstraplayer-checkout', action='store_true',
This is the last (I believe) piece of the puzzle in setting up builds from nothing without having to write custom scripts or use external tools. After layers have been fetched and placed into their respective locations, one would surely want to proceed to the actual build, and here's how: 1. Without arguments or with 'list-config-templates', the tool simply walks the ../.. of its location (which is the parent dir of poky/oe-core), and prints what templates it has found, as seen below. If the following is not enough information, adding '-v' will also print conf-notes.txt for each of the templates: $ oe-setup-build Available build configuration templates (re-run with -v to see their descriptions): /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/poky/meta-poky/conf/templates/default will create a build configuration in /srv/work/alex/build-meta-poky-default /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gizmo will create a build configuration in /srv/work/alex/build-meta-alex-configuration-gizmo /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gadget will create a build configuration in /srv/work/alex/build-meta-alex-configuration-gadget 2. Then the user picks one command of the above and runs it. This will land them in a shell ready to run bitbake: $ oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gizmo Running: TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/configuration-gizmo . /srv/work/alex/poky/oe-init-build-env /srv/work/alex/build-meta-alex-configuration-gizmo && /bin/bash You had no conf/local.conf file. This configuration file has therefore been created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gizmo/local.conf.sample You may wish to edit it to, for example, select a different MACHINE (target hardware). See conf/local.conf for more information as common configuration options are commented. You had no conf/bblayers.conf file. This configuration file has therefore been created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gizmo/bblayers.conf.sample To add additional metadata layers into your configuration please add entries to conf/bblayers.conf. The Yocto Project has extensive documentation about OE including a reference manual which can be found at: https://docs.yoctoproject.org For more information about OpenEmbedded see the website: https://www.openembedded.org/ This configuration template will set up a build for the purposes of supporting gizmo. Please refer to meta-alex/README for additional details and available bitbake targets. 3. The full set of command line options is: $ oe-setup-build -h usage: oe-setup-build [-h] {list-config-templates,setup-build-env} ... A script that discovers available build configuration templates and sets up a build environment based on one of them positional arguments: {list-config-templates,setup-build-env} list-config-templates List available configuration templates setup-build-env Set up a build environment and open a shell session with it, ready to run builds. optional arguments: -h, --help show this help message and exit $ oe-setup-build list-config-templates -h usage: oe-setup-build list-config-templates [-h] [--topdir TOPDIR] [-v] optional arguments: -h, --help show this help message and exit --topdir TOPDIR Where to look for available build configuration templates (default is /srv/work/alex). -v Print a description for each available build configuration template. $ oe-setup-build setup-build-env -h usage: oe-setup-build setup-build-env [-h] -c template_path [-b build_path] [--no-shell] optional arguments: -h, --help show this help message and exit -c template_path Use a build configuration template in template_path to set up a build environment (run this script with 'list-config-templates' to see what is available) -b build_path Set up a build directory in build_path (run this script with 'list-config-templates' to see where it would be by default) --no-shell Create a build directory but do not start a shell session with the build environment from it. 4. There's also a selftest that runs a basic check for template discovery and build setup, and an added hint in oe-setup-layers about how to proceed (as it is really not user-friendly to fetch the layer repos successfully and then exit without a word). Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- meta/lib/oeqa/selftest/cases/bblayers.py | 12 ++++ scripts/oe-setup-build | 79 ++++++++++++++++++++++++ scripts/oe-setup-layers | 7 +++ 3 files changed, 98 insertions(+) create mode 100755 scripts/oe-setup-build