| Message ID | 20220905182141.1091053-1-alex@linutronix.de |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] oe-setup-build: add a tool for discovering config templates and setting up builds | expand |
Hi Alex, Thanks for your work on this! I have a few comments about it. On 2022-09-05, Alexander Kanavin <alex.kanavin@gmail.com> wrote: > This is the last (I believe) piece of the puzzle in setting up builds from nothing > without having to write custom scripts or use external tools. People can already setup the build-env without writing custom scripts or using external tools. AFAICT, instead of users doing: $ TEMPLATECONF=/some/template/dir . /path/to/oe-init-build-env /my/build/dir you want them to do: $ /path/to/oe-setup-build -c /some/template/dir -b /my/build/dir As a yocto user, I am fine with how things have been. But I understand that for new users, settings env-vars and sourcing files is unusual (using the sdk relies on nearly the same procedure). So if your motivation is to change from setting TEMPLATECONF and sourcing oe-init-build-env to using an intuitive tool, then that should be clearly stated in the commit message. It is not correct to say that it cannot be setup without this tool because we've been doing it for many years now. :-) > After layers have been fetched and placed into their respective locations, one would > surely want to proceed to the actual build, and here's how: > > 1. Without arguments, the tool simply walks the ../.. of its location (which is the parent dir > of poky/oe-core), and prints what templates has found. If the following is not enough information, > adding '-v' will also print conf-notes.txt for each of the templates: For me this is not an intuitive interface. Below is a list of changes that I would suggest: 1. The tool should be called "oe-setup-build-env". This matches the source file "oe-init-build-env" and makes it clear that only a build environment is setup and nothing will be "built". 2. The "-c" argument is required (also in your code), so not specifying it should be an error. Upon printing the error message, the tool could provide the help. Something like: $ oe-setup-build-env error: no template path specified with `-c' usage: oe-setup-build-env -c template_path [-h] [--topdir TOPDIR] [-v] [-b build_path] [--no-shell] A script that discovers available build configuration templates and sets up a build environment based on one of them optional arguments: -h, --help show this help message and exit --topdir TOPDIR Where to look for available build configuration templates (default is ../..). -v Print a description for each available build configuration template. -c template_path Use build configuration template in template_path to set up a build (run this script without arguments or with -v to see what is available) -b build_path Set up a build in build_path (run this script without arguments or with -v to see where it would be by default) --no-shell Create a build but do not start a shell session with it. (Or maybe template_path could be a positional argument instead of specified with -c? Not sure.) 3. In the help output I think it is important to always talk about build environments instead of builds. For example: Use build configuration template in template_path to set up a build environment Set up a build environment in build_path Create a build environment but do not start a shell session within it 4. Listing available templates is a nice feature, but this alternate mode of operation should have its own argument. Something like -l List available build configuration templates. 5. Your comments hint that this tool might be a second step for users of the new "oe-setup-layers" tool. That tool uses a configuration file that has an explicit list of layers that are used. But this tool recursively searches for templates in _any_ layer under topdir. If this tool is indeed meant as a second step to "oe-setup-layers" then perhaps it should also support using the configuration file so that it knows which layers it is allowed to list (or maybe even use). --jsondata File containing the layer data in json format With the json data available, the tool could also report if the layers are dirty or are using a different commit, before setting up the build environment. John Ogness
Thanks for the review, comments below. On Tue, 6 Sept 2022 at 12:20, John Ogness <john.ogness@linutronix.de> wrote: > So if your motivation is to change from setting TEMPLATECONF and > sourcing oe-init-build-env to using an intuitive tool, then that should > be clearly stated in the commit message. It is not correct to say that > it cannot be setup without this tool because we've been doing it for > many years now. :-) The motivation for the tool is primarily in adding a way to discover and learn about available configuration templates which until now has been completely absent. In fact I'd say that is the primary purpose of the tool. Setting up builds is secondary, and if someone wants to continue using '. oe-init-build-env', I am totally fine with that - the tool merely provides a convenient shortcut for that with a separate shell session, and reasonable defaults for choosing where the build dir would be, unless specified explicitly via -b. > > After layers have been fetched and placed into their respective locations, one would > > surely want to proceed to the actual build, and here's how: > > > > 1. Without arguments, the tool simply walks the ../.. of its location (which is the parent dir > > of poky/oe-core), and prints what templates has found. If the following is not enough information, > > adding '-v' will also print conf-notes.txt for each of the templates: > > For me this is not an intuitive interface. Below is a list of changes > that I would suggest: > > 1. The tool should be called "oe-setup-build-env". This matches the > source file "oe-init-build-env" and makes it clear that only a build > environment is setup and nothing will be "built". Sure, I will change that. > 2. The "-c" argument is required (also in your code), so not specifying > it should be an error. Upon printing the error message, the tool could > provide the help. Something like: > > $ oe-setup-build-env > error: no template path specified with `-c' > > usage: oe-setup-build-env -c template_path [-h] [--topdir TOPDIR] [-v] [-b build_path] [--no-shell] > > A script that discovers available build configuration templates and sets up a build environment based on one of them > > optional arguments: > -h, --help show this help message and exit > --topdir TOPDIR Where to look for available build configuration templates (default is ../..). > -v Print a description for each available build configuration template. > -c template_path Use build configuration template in template_path to set up a build (run this script without arguments or with -v to see what is available) > -b build_path Set up a build in build_path (run this script without arguments or with -v to see where it would be by default) > --no-shell Create a build but do not start a shell session with it. > > (Or maybe template_path could be a positional argument instead of > specified with -c? Not sure.) This is where I have to say no. As stated, the primary purpose is to find out what is available, and the tool should do something useful in that aspect when run without any arguments. Which is what it does - '-c' is not actually required :) > 3. In the help output I think it is important to always talk about > build environments instead of builds. For example: > > Use build configuration template in template_path to set up a build environment > > Set up a build environment in build_path > > Create a build environment but do not start a shell session within it Yes, I will correct the terminology. > 4. Listing available templates is a nice feature, but this alternate > mode of operation should have its own argument. Something like > > -l List available build configuration templates. See above please. > 5. Your comments hint that this tool might be a second step for users of > the new "oe-setup-layers" tool. That tool uses a configuration file that > has an explicit list of layers that are used. But this tool recursively > searches for templates in _any_ layer under topdir. If this tool is > indeed meant as a second step to "oe-setup-layers" then perhaps it > should also support using the configuration file so that it knows which > layers it is allowed to list (or maybe even use). > > --jsondata File containing the layer data in json format > > With the json data available, the tool could also report if the layers > are dirty or are using a different commit, before setting up the build > environment. To get the layer descriptions in json accepted and merged, I had to drop all configuration-related data from it, and keep it strictly about repos checkout and placement into correct paths. Originally, it had a list of machines, distros and config templates, per layer, and the list of layers; now all of it is gone, and only the repos remain (and repo != layer, as repos can and do contain multiple layers). There was a lot of resistance against mixing the layer setup with build setup into the same json and same tools. We can probably come up with a more tightly controlled template discovery and buildenv setup if proven necessary, but this one just works when all checked out layers are in a single top folder just above poky/oe-core, so why not keep things simple, and do away with config files altogether? Please keep in mind the target audience: it is not you or me. This is intended, first, and foremost, for first-timers, and the expectation is that distro maintainers set things up for them (layer config, and build templates), so that nobody has to 'learn yocto' or read documentation (that includes READMEs in product/company layers too), or poke around the confusing layer tree to get from a newly created empty 'my-yocto-stuff' directory to having flashable target artifacts. Alex
On Tue, 6 Sept 2022 at 13:24, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > 4. Listing available templates is a nice feature, but this alternate > > mode of operation should have its own argument. Something like > > > > -l List available build configuration templates. > > See above please. On second thought... the two modes of operation should be more clearly separated, perhaps with git-style subcommands. As long as the default without arguments remains discovery :) Alex
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded- > core@lists.openembedded.org> On Behalf Of Alexander Kanavin > Sent: den 6 september 2022 13:32 > To: Alexander Kanavin <alex.kanavin@gmail.com> > Cc: John Ogness <john.ogness@linutronix.de>; OE-core <openembedded- > core@lists.openembedded.org>; Alexander Kanavin <alex@linutronix.de> > Subject: Re: [OE-core] [PATCH 1/2] oe-setup-build: add a tool for > discovering config templates and setting up builds > > On Tue, 6 Sept 2022 at 13:24, Alexander Kanavin via > lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> > wrote: > > > 4. Listing available templates is a nice feature, but this alternate > > > mode of operation should have its own argument. Something like > > > > > > -l List available build configuration templates. > > > > See above please. > > On second thought... the two modes of operation should be more clearly > separated, perhaps with git-style subcommands. As long as the default > without arguments remains discovery :) > > Alex Given that the tool is called oe-setup-build and not oe-discover-build, making the discovery mode the default is actually a bit weird. Subcommands sounds fine to me though. Maybe you should call it oe-build-env instead, since it has multiple modes? //Peter
On Tue, 6 Sept 2022 at 14:25, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote: > Given that the tool is called oe-setup-build and not oe-discover-build, > making the discovery mode the default is actually a bit weird. > Subcommands sounds fine to me though. Maybe you should call it > oe-build-env instead, since it has multiple modes? I'd like to keep it within 'oe-setup-..' family though: oe-setup-layers, oe-setup-build, and then later oe-setup as a general integrated one-stop thing that ties it all together: oe-setup kirkstone oe-setup master +docker +systemd oe-setup yoe riscv64 oe-setup webos oe-setup some-custom.json Like RP suggested. I think having discovery functionality in something called 'setup' is actually fine, as the setup actually happens from the available choices, and one would want to know about them first. I'll certainly switch to subcommands; this will allow adding config fragment management as well. Alex
Alright, v2 is out. Changes are: switch to subcommands, and use 'build environment' or 'build directory' in place of ambigous 'build'. Alex On Tue, 6 Sept 2022 at 14:35, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > On Tue, 6 Sept 2022 at 14:25, Peter Kjellerstedt > <peter.kjellerstedt@axis.com> wrote: > > > Given that the tool is called oe-setup-build and not oe-discover-build, > > making the discovery mode the default is actually a bit weird. > > Subcommands sounds fine to me though. Maybe you should call it > > oe-build-env instead, since it has multiple modes? > > I'd like to keep it within 'oe-setup-..' family though: > oe-setup-layers, oe-setup-build, and then later oe-setup as a general > integrated one-stop thing that ties it all together: > > oe-setup kirkstone > oe-setup master +docker +systemd > oe-setup yoe riscv64 > oe-setup webos > oe-setup some-custom.json > > Like RP suggested. > > I think having discovery functionality in something called 'setup' is > actually fine, as the setup actually happens from the available > choices, and one would want to know about them first. > > I'll certainly switch to subcommands; this will allow adding config > fragment management as well. > > Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#170351): https://lists.openembedded.org/g/openembedded-core/message/170351 > Mute This Topic: https://lists.openembedded.org/mt/93485066/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Mon, 2022-09-05 at 20:21 +0200, Alexander Kanavin wrote: > This is the last (I believe) piece of the puzzle in setting up builds > from nothing without having to write custom scripts or use external > tools. > > After layers have been fetched and placed into their respective > locations, one would surely want to proceed to the actual build, and > here's how: To set expectations, I'm probably not going to merge this for 4.1. I think we've taken some really good steps in the direction we need so far but we don't want to rush this. I haven't had the time needed to look at these patches properly due to everything else that was breaking. That doesn't stop review ad refining the patches ready for 4.2 opening. I'm open to representations if everyone thinks this is ready but I get the feeling many are distracted atm. Cheers, Richard
On Wed, 7 Sept 2022 at 11:15, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > To set expectations, I'm probably not going to merge this for 4.1. I > think we've taken some really good steps in the direction we need so > far but we don't want to rush this. I haven't had the time needed to > look at these patches properly due to everything else that was > breaking. > > That doesn't stop review ad refining the patches ready for 4.2 opening. > I'm open to representations if everyone thinks this is ready but I get > the feeling many are distracted atm. I'm fine with that; I'd like to add config fragments to it as well, to make the case for this tool stronger. The plan is to split up local.conf.sample.extended into those fragments as a showcase/testbed for the feature. The other two patches should be trivial to merge. Alex
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index c6bd5a1f6a..1dd3046aa6 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -122,6 +122,18 @@ class BitbakeLayers(OESelftestTestCase): fullpath = os.path.join(layerpath, "conf", "templates", "buildconf-1", f) self.assertTrue(os.path.exists(fullpath), "Template configuration file {} not found".format(fullpath)) + cmd = 'oe-setup-build --topdir {}'.format(layerpath) + result = runCmd(cmd) + cond = "scripts/oe-setup-build -c " in result.output and "test-bitbakelayer-layercreate/conf/templates/buildconf-1" in result.output + self.assertTrue(cond, "Incorrect output from {}: {}".format(cmd, result.output)) + + # rather than hardcode the build setup cmdline here, let's actually run what the tool suggests to the user + cmd = None + for l in result.output.splitlines(): + if "scripts/oe-setup-build -c " in l: + cmd = l + " --no-shell" + result = runCmd(cmd) + def get_recipe_basename(self, recipe): recipe_file = "" result = runCmd("bitbake-layers show-recipes -f %s" % recipe) diff --git a/scripts/oe-setup-build b/scripts/oe-setup-build new file mode 100755 index 0000000000..56bbd24b26 --- /dev/null +++ b/scripts/oe-setup-build @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import argparse +import json +import os +import subprocess + +def makebuildpath(topdir, templatelocation, template): + return os.path.join(topdir, "build-{}-{}".format(os.path.basename(os.path.abspath(os.path.join(templatelocation,'..','..'))), template)) + +def print_templates(topdir, verbose): + print("Available build configuration templates (re-run with -v to see their descriptions):\n") + for (dir, dirs, files) in os.walk(topdir): + if dir.endswith('conf/templates'): + for d in dirs: + print("{} -c {}\nwill create a build in {}\n".format(__file__, os.path.join(dir, d), makebuildpath(args.topdir, dir, d))) + if verbose: + if os.path.join(dir, d).endswith('meta-poky/conf/templates/default'): + print("Description: this is the reference configuration of the poky reference distribution (choose this if you are uncertain).") + elif os.path.join(dir, d).endswith('meta/conf/templates/default'): + print("Description: this is the reference configuration of the openembedded-core layer (choose this if you are uncertain).") + else: + print("Description:", open(os.path.join(dir, d, 'conf-notes.txt')).read()) + print("---") + + # Do not recurse into build directories; they can be enormous + if 'conf' in dirs and 'bblayers.conf' in os.listdir(os.path.join(dir, 'conf')): + dirs.clear() + # Do not recurse into sstate-cache or downloads similarly + if 'universal' in dirs and '00' in dirs: + dirs.clear() + if 'uninative' in dirs and 'git2' in dirs: + dirs.clear() + +def setup_build(defaulttop, template, builddir, no_shell): + if not builddir: + builddir = makebuildpath(defaulttop, os.path.dirname(template), os.path.basename(template)) + coredir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) + cmd = "TEMPLATECONF={} . {} {}".format(template, os.path.join(coredir, 'oe-init-build-env'), builddir) + if not no_shell: + cmd = cmd + " && {}".format(os.environ['SHELL']) + print("Running:", cmd) + subprocess.run(cmd, shell=True, executable=os.environ['SHELL']) + +parser = argparse.ArgumentParser(description="A script that discovers available build configuration templates and sets up a build environment based on one of them") + +defaulttop = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')) +parser.add_argument("--topdir", default=defaulttop, help='Where to look for available build configuration templates (default is {}).'.format(defaulttop)) + +parser.add_argument('-v', action='store_true', + help='Print a description for each available build configuration template.') + +parser.add_argument('-c', metavar='template_path', help="Use build configuration template in template_path to set up a build (run this script without arguments or with -v to see what is available)") +parser.add_argument('-b', metavar='build_path', help="Set up a build in build_path (run this script without arguments or with -v to see where it would be by default)") +parser.add_argument('--no-shell', action='store_true', + help='Create a build but do not start a shell session with it.') + +args = parser.parse_args() + +if not args.c: + print_templates(args.topdir, args.v) +else: + setup_build(defaulttop, args.c, args.b, args.no_shell) diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers index 6ecaffed75..8336037630 100755 --- a/scripts/oe-setup-layers +++ b/scripts/oe-setup-layers @@ -17,6 +17,7 @@ import os import subprocess def _do_checkout(args, json): + oesetupbuild = None layers = json['sources'] for l_name in layers: l_data = layers[l_name] @@ -53,6 +54,12 @@ def _do_checkout(args, json): print("Running '{}' in {}".format(cmd, layerdir)) subprocess.check_output(cmd, shell=True, cwd=layerdir) + if os.path.exists(os.path.join(layerdir, 'scripts/oe-setup-build')): + oesetupbuild = os.path.join(layerdir, 'scripts/oe-setup-build') + + if oesetupbuild: + print("\nRun {} to list available build configuration templates and set up a build from one of them".format(oesetupbuild)) + parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/") parser.add_argument('--force-bootstraplayer-checkout', action='store_true',
This is the last (I believe) piece of the puzzle in setting up builds from nothing without having to write custom scripts or use external tools. After layers have been fetched and placed into their respective locations, one would surely want to proceed to the actual build, and here's how: 1. Without arguments, the tool simply walks the ../.. of its location (which is the parent dir of poky/oe-core), and prints what templates has found. If the following is not enough information, adding '-v' will also print conf-notes.txt for each of the templates: $ oe-setup-build Available build configuration templates (re-run with -v to see their descriptions): /srv/work/alex/poky/scripts/oe-setup-build -c /srv/work/alex/poky/meta-poky/conf/templates/default will create a build in /srv/work/alex/build-meta-poky-default /srv/work/alex/poky/scripts/oe-setup-build -c /srv/work/alex/meta-alex/conf/templates/configuration-gizmo will create a build in /srv/work/alex/build-meta-alex-configuration-gizmo /srv/work/alex/poky/scripts/oe-setup-build -c /srv/work/alex/meta-alex/conf/templates/configuration-gadget will create a build in /srv/work/alex/build-meta-alex-configuration-gadget 2. Then the users picks one of the above and runs it. This will land them in a shell ready to run bitbake: $ oe-setup-build -c /srv/work/alex/meta-alex/conf/templates/configuration-gizmo Running: TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/configuration-gizmo . /srv/work/alex/poky/oe-init-build-env /srv/work/alex/build-meta-alex-configuration-gizmo && /bin/bash You had no conf/local.conf file. This configuration file has therefore been created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gizmo/local.conf.sample You may wish to edit it to, for example, select a different MACHINE (target hardware). See conf/local.conf for more information as common configuration options are commented. You had no conf/bblayers.conf file. This configuration file has therefore been created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gizmo/bblayers.conf.sample To add additional metadata layers into your configuration please add entries to conf/bblayers.conf. The Yocto Project has extensive documentation about OE including a reference manual which can be found at: https://docs.yoctoproject.org For more information about OpenEmbedded see the website: https://www.openembedded.org/ This configuration template will set up a build for the purposes of supporting gizmo. Please refer to meta-alex/README for additional details and available bitbake targets. 3. The full set of command line options is: $ oe-setup-build -h usage: oe-setup-build [-h] [--topdir TOPDIR] [-v] [-c template_path] [-b build_path] [--no-shell] A script that discovers available build configuration templates and sets up a build environment based on one of them optional arguments: -h, --help show this help message and exit --topdir TOPDIR Where to look for available build configuration templates (default is /srv/work/alex). -v Print a description for each available build configuration template. -c template_path Use build configuration template in template_path to set up a build (run this script without arguments or with -v to see what is available) -b build_path Set up a build in build_path (run this script without arguments or with -v to see where it would be by default) --no-shell Create a build but do not start a shell session with it. 4. There's also a selftest that runs a basic check for template discovery and build setup, and an added hint in oe-setup-layers about how to proceed (as it is really not user-friendly to fetch the layer repos successfully and then exit without a word). Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- meta/lib/oeqa/selftest/cases/bblayers.py | 12 +++++ scripts/oe-setup-build | 68 ++++++++++++++++++++++++ scripts/oe-setup-layers | 7 +++ 3 files changed, 87 insertions(+) create mode 100755 scripts/oe-setup-build