| Message ID | 20230510095726.4090311-2-alex@linutronix.de |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] scripts/oe-setup-layers: write a list of layer paths into the checkout's top dir | expand |
On Wed, 2023-05-10 at 11:57 +0200, Alexander Kanavin wrote: > This is the last (I believe) piece of the puzzle in setting up builds from nothing > without having to write custom scripts or use external tools. > > After layers have been fetched and placed into their respective locations by > oe-setup-layers, one would surely want to proceed to the actual build, and here's how: > > 1. Without arguments or with 'list-config-templates', the tool reads available layers > from .oe-layers.json file (written out by oe-setup-layers) and prints what > templates it has found, as seen below. If the following is not enough information, > adding '-v' will also print conf-notes.txt for each of the templates: > > $ oe-setup-build > Available build configuration templates (re-run with -v to see their descriptions): > > /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/poky/meta-poky/conf/templates/default > will create a build configuration in /srv/work/alex/build-meta-poky-default > > /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gizmo > will create a build configuration in /srv/work/alex/build-meta-alex-configuration-gizmo > > /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gadget > will create a build configuration in /srv/work/alex/build-meta-alex-configuration-gadget I've just been looking at this again and I'm still not convinced this is right. In particular, the above output worries me a lot, partly as I barely understand it and I suspect if I struggle, I won't be the only one. With this kind of core tool, the "interface" to the user needs to be really clear and the problem here is that it isn't. Being more specific, from the above it is listing some commands I could run, but the user is being left to work out what the difference is for themselves. If I print a list like: a) poky default b) gizmo config c) gadget config then ask the user to select one, it is clear as the user is drawn to the key information. In the above text output, you have to know where to look in the commands to guess the pieces. The "/srv/work/alex" string is confusing the output and in many ways even filtering that out would help reduce the "superfluous" text but the issue does run deeper than that. As another example, perhaps I shouldn't need the -v to get at least some summary of what each one does? Cheers, Richard
On Mon, 31 Jul 2023 at 12:56, Richard Purdie <rpurdie@rpsys.net> wrote: > I've just been looking at this again and I'm still not convinced this > is right. In particular, the above output worries me a lot, partly as I > barely understand it and I suspect if I struggle, I won't be the only > one. > > With this kind of core tool, the "interface" to the user needs to be > really clear and the problem here is that it isn't. > > Being more specific, from the above it is listing some commands I could > run, but the user is being left to work out what the difference is for > themselves. If I print a list like: > > a) poky default > b) gizmo config > c) gadget config > > then ask the user to select one, it is clear as the user is drawn to > the key information. In the above text output, you have to know where > to look in the commands to guess the pieces. > > The "/srv/work/alex" string is confusing the output and in many ways > even filtering that out would help reduce the "superfluous" text but > the issue does run deeper than that. > > As another example, perhaps I shouldn't need the -v to get at least > some summary of what each one does? Thanks, I'm glad this wasn't forgotten (or quietly shelved :). I kinda announced in my EOSS talk that there should be a nicer way for newcomers to set up builds than oe-init-build-env and that we're working on it. For no reason at all I was trying to avoid adding an interactive mode :), but prompting users to make a selection will make things a lot smoother. And each configuration should have a stable and sweet shortcut name (at least, within a specific layer checkout), so that awkward full paths are completely avoided. Maybe {template-directory-name}-{layer-name}. I.e. 'default-meta-poky' or 'gizmo-meta-alex'. Also, content of conf-notes.txt in oe-core/poky should be moved to the first-time banner in scripts/oe-setup-builddir, and the files should only point out that "This template sets up a default build configuration used by Poky reference distribution" or similar. Otherwise the tool will print what these files have now, which is not helpful. Should we be fine with just conf-notes.txt containing the template description as a freeform plaintext, or is there a use case for more structured template metadata? Alex
On Mon, 2023-07-31 at 13:31 +0200, Alexander Kanavin wrote: > On Mon, 31 Jul 2023 at 12:56, Richard Purdie <rpurdie@rpsys.net> wrote: > > > I've just been looking at this again and I'm still not convinced this > > is right. In particular, the above output worries me a lot, partly as I > > barely understand it and I suspect if I struggle, I won't be the only > > one. > > > > With this kind of core tool, the "interface" to the user needs to be > > really clear and the problem here is that it isn't. > > > > Being more specific, from the above it is listing some commands I could > > run, but the user is being left to work out what the difference is for > > themselves. If I print a list like: > > > > a) poky default > > b) gizmo config > > c) gadget config > > > > then ask the user to select one, it is clear as the user is drawn to > > the key information. In the above text output, you have to know where > > to look in the commands to guess the pieces. > > > > The "/srv/work/alex" string is confusing the output and in many ways > > even filtering that out would help reduce the "superfluous" text but > > the issue does run deeper than that. > > > > As another example, perhaps I shouldn't need the -v to get at least > > some summary of what each one does? > > Thanks, I'm glad this wasn't forgotten (or quietly shelved :). I kinda > announced in my EOSS talk that there should be a nicer way for > newcomers to set up builds than oe-init-build-env and that we're > working on it. > > For no reason at all I was trying to avoid adding an interactive mode > :), but prompting users to make a selection will make things a lot > smoother. And each configuration should have a stable and sweet > shortcut name (at least, within a specific layer checkout), so that > awkward full paths are completely avoided. Maybe > {template-directory-name}-{layer-name}. I.e. 'default-meta-poky' or > 'gizmo-meta-alex'. Just to be clear I'm not saying it needs to be interactive but that we need to think carefully about how the information is presented so that it is easy to understand. You could think about using the layer name in these contexts as layers should be defining them at this point? Yes, getting the layer name means parsing but we can likely handle that at this point as we're in python. > Also, content of conf-notes.txt in oe-core/poky should be moved to the > first-time banner in scripts/oe-setup-builddir, and the files should > only point out that "This template sets up a default build > configuration used by Poky reference distribution" or similar. > Otherwise the tool will print what these files have now, which is not > helpful. > > Should we be fine with just conf-notes.txt containing the template > description as a freeform plaintext, or is there a use case for more > structured template metadata? Could/should it be a variable in layer.conf? Cheers, Richard
On Mon, 31 Jul 2023 at 14:04, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > You could think about using the layer name in these contexts as layers > should be defining them at this point? > > Yes, getting the layer name means parsing but we can likely handle that > at this point as we're in python. You mean getting BBFILE_COLLECTIONS value out of conf/layer.conf? That should be fairly easy with standard python, yes. > > Should we be fine with just conf-notes.txt containing the template > > description as a freeform plaintext, or is there a use case for more > > structured template metadata? > > Could/should it be a variable in layer.conf? I'm not quite following - what kind of variable would that be, and with what kind of value? Each template needs its own set of metadata (or freeform description in conf-notes.txt as a minimum), so layer.conf is probably not the place? Alex
On Mon, 2023-07-31 at 14:15 +0200, Alexander Kanavin wrote: > On Mon, 31 Jul 2023 at 14:04, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > You could think about using the layer name in these contexts as layers > > should be defining them at this point? > > > > Yes, getting the layer name means parsing but we can likely handle that > > at this point as we're in python. > > You mean getting BBFILE_COLLECTIONS value out of conf/layer.conf? That > should be fairly easy with standard python, yes. I think we should be able to use bits of bitbake to help too? > > > Should we be fine with just conf-notes.txt containing the template > > > description as a freeform plaintext, or is there a use case for more > > > structured template metadata? > > > > Could/should it be a variable in layer.conf? > > I'm not quite following - what kind of variable would that be, and > with what kind of value? Each template needs its own set of metadata > (or freeform description in conf-notes.txt as a minimum), so > layer.conf is probably not the place? That indeed doesn't make sense and I'm not thinking clearly. I feel pressured to get you some reply, long overdue but obviously I didn't spend enough time working through the details, sorry :(. The conf-notes.txt has a specific purpose and I don't like the idea of trying to make it to two things badly. We therefore need to put this information into which is likely a new location, probably a new conf file within each template directory? Ultimately perhaps we put conf-notes into the new config file too as a separate variable? That text file was good as a quick hack a decade ago but we probably need to rethink how some of this works now. That can come in due course though. Establishing a location for new information about each template is probably the right first step. Cheers, Richard
On Mon, 31 Jul 2023 at 14:42, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > > Yes, getting the layer name means parsing but we can likely handle that > > > at this point as we're in python. > > > > You mean getting BBFILE_COLLECTIONS value out of conf/layer.conf? That > > should be fairly easy with standard python, yes. > > I think we should be able to use bits of bitbake to help too? But how? To use bitbake a build must be set up first. The script sets up the build. When the script executes, bitbake won't yet run. > The conf-notes.txt has a specific purpose and I don't like the idea of > trying to make it to two things badly. We therefore need to put this > information into which is likely a new location, probably a new conf > file within each template directory? So perhaps: conf-description.txt - a short (one line ideally) description of what the configuration is for, shown by oe-setup-build to help the users choose one configuration from several. conf-notes.txt - a kind of freeform 'readme' for the configuration explaining how to use it, and which are the key bitbake targets etc, shown after the selection is made, when a build is set up from a selected template. Alex
On Mon, 2023-07-31 at 16:14 +0200, Alexander Kanavin wrote: > On Mon, 31 Jul 2023 at 14:42, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > Yes, getting the layer name means parsing but we can likely handle that > > > > at this point as we're in python. > > > > > > You mean getting BBFILE_COLLECTIONS value out of conf/layer.conf? That > > > should be fairly easy with standard python, yes. > > > > I think we should be able to use bits of bitbake to help too? > > But how? To use bitbake a build must be set up first. The script sets > up the build. When the script executes, bitbake won't yet run. To use bitbake itself, yes. To use tinfoil, or library code from lib/bb, it doesn't necessarily need that. You don't want bitbake itself anyway. > > The conf-notes.txt has a specific purpose and I don't like the idea of > > trying to make it to two things badly. We therefore need to put this > > information into which is likely a new location, probably a new conf > > file within each template directory? > > So perhaps: > conf-description.txt - a short (one line ideally) description of what > the configuration is for, shown by oe-setup-build to help the users > choose one configuration from several. > conf-notes.txt - a kind of freeform 'readme' for the configuration > explaining how to use it, and which are the key bitbake targets etc, > shown after the selection is made, when a build is set up from a > selected template. I'm still thinking we should try and get away from these text files... Cheers, Richard
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 65426a67829..73fefb5d195 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -151,12 +151,12 @@ class BitbakeLayers(OESelftestTestCase): self.validate_layersjson(jsonfile) # The revision-under-test may not necessarily be available on the remote server, - # so replace it with a revision that has a yocto-4.0 tag. + # so replace it with a revision that has a yocto-4.1 tag. import json with open(jsonfile) as f: data = json.load(f) for s in data['sources']: - data['sources'][s]['git-remote']['rev'] = '00cfdde791a0176c134f31e5a09eff725e75b905' + data['sources'][s]['git-remote']['rev'] = '5200799866b92259e855051112520006e1aaaac0' with open(jsonfile, 'w') as f: json.dump(data, f) @@ -164,3 +164,22 @@ class BitbakeLayers(OESelftestTestCase): result = runCmd('{}/setup-layers --destdir {}'.format(self.testlayer_path, testcheckoutdir)) layers_json = os.path.join(testcheckoutdir, ".oe-layers.json") self.assertTrue(os.path.exists(layers_json), "File {} not found in test layer checkout".format(layers_json)) + + # As setup-layers checkout out an old revision of poky, there is no setup-build symlink, + # and we need to run oe-setup-build directly from the current poky tree under test + oe_setup_build = os.path.join(get_bb_var('COREBASE'), 'scripts/oe-setup-build') + oe_setup_build_l = os.path.join(testcheckoutdir, 'setup-build') + os.symlink(oe_setup_build,oe_setup_build_l) + + cmd = '{} list-config-templates --layerlist {}'.format(oe_setup_build_l, layers_json) + result = runCmd(cmd) + cond = "setup-build setup-build-env -c " in result.output and "conf/templates/default" in result.output + self.assertTrue(cond, "Incorrect output from {}: {}".format(cmd, result.output)) + + # rather than hardcode the build setup cmdline here, let's actually run what the tool suggests to the user + cmd = None + for l in result.output.splitlines(): + if "setup-build setup-build-env -c " in l: + cmd = l + " --no-shell" + self.assertIsNotNone(cmd, "Could not find the command to set up a build in the output: {}".format(result.output)) + result = runCmd(cmd) diff --git a/scripts/oe-setup-build b/scripts/oe-setup-build new file mode 100755 index 00000000000..4cfc002de3f --- /dev/null +++ b/scripts/oe-setup-build @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import argparse +import json +import os +import subprocess + +def defaultlayers(): + return os.path.abspath(os.path.join(os.path.dirname(__file__), '.oe-layers.json')) + +def makebuildpath(topdir, templatelocation, template): + return os.path.join(topdir, "build-{}-{}".format(os.path.basename(os.path.abspath(os.path.join(templatelocation,'..','..'))), template)) + +def print_templates(args): + layers_file = args.layerlist + verbose = args.v + if not os.path.exists(layers_file): + print("List of layers {} does not exist; were they set up using the setup-layers script?".format(layers_file)) + return + layers_list = json.load(open(layers_file))["layers"] + print("Available build configuration templates (re-run with 'list-config-templates -v' to see their descriptions):\n") + for layer in layers_list: + template_dir = os.path.join(os.path.dirname(layers_file), layer, 'conf','templates') + if os.path.exists(template_dir): + for d in os.listdir(template_dir): + if not os.path.isdir(os.path.join(template_dir,d)): + continue + print("{} setup-build-env -c {}\nwill create a build configuration in {}\n".format(__file__, os.path.join(template_dir, d), makebuildpath(os.path.dirname(defaultlayers()), template_dir, d))) + if verbose: + if os.path.join(template_dir, d).endswith('meta-poky/conf/templates/default'): + print("Description: this is the reference configuration of the poky reference distribution (choose this if you are uncertain).") + elif os.path.join(template_dir, d).endswith('meta/conf/templates/default'): + print("Description: this is the reference configuration of the openembedded-core layer (choose this if you are uncertain).") + else: + print("Description:", open(os.path.join(template_dir, d, 'conf-notes.txt')).read()) + print("---") + +def setup_build_env(args): + template = args.c + builddir = args.b + no_shell = args.no_shell + if not builddir: + builddir = makebuildpath(os.path.dirname(defaultlayers()), os.path.dirname(template), os.path.basename(template)) + coredir = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), '..')) + cmd = "TEMPLATECONF={} . {} {}".format(template, os.path.join(coredir, 'oe-init-build-env'), builddir) + if not no_shell: + cmd = cmd + " && {}".format(os.environ['SHELL']) + print("Running:", cmd) + subprocess.run(cmd, shell=True, executable=os.environ['SHELL']) + +parser = argparse.ArgumentParser(description="A script that discovers available build configuration templates and sets up a build environment based on one of them") +subparsers = parser.add_subparsers() +parser_list_templates = subparsers.add_parser('list-config-templates', help='List available configuration templates') + +parser_list_templates.add_argument("--layerlist", default=defaultlayers(), help='Where to look for available layers (as written out by setup-layers script) (default is {}).'.format(defaultlayers())) +parser_list_templates.add_argument('-v', action='store_true', + help='Print a description for each available build configuration template.') +parser_list_templates.set_defaults(func=print_templates) + +parser_setup_env = subparsers.add_parser('setup-build-env', help='Set up a build environment and open a shell session with it, ready to run builds.') +parser_setup_env.add_argument('-c', metavar='template_path', required=True, help="Use a build configuration template in template_path to set up a build environment (run this script with 'list-config-templates' to see what is available)") +parser_setup_env.add_argument('-b', metavar='build_path', help="Set up a build directory in build_path (run this script with 'list-config-templates' to see where it would be by default)") +parser_setup_env.add_argument('--no-shell', action='store_true', + help='Create a build directory but do not start a shell session with the build environment from it.') +parser_setup_env.set_defaults(func=setup_build_env) + +args = parser.parse_args() +if 'func' in args: + args.func(args) +else: + from argparse import Namespace + print_templates(Namespace(layerlist=defaultlayers(), v=False)) diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers index cbf94ddb08b..97e13fc03be 100755 --- a/scripts/oe-setup-layers +++ b/scripts/oe-setup-layers @@ -63,6 +63,7 @@ def _write_layer_list(dest, repodirs): def _do_checkout(args, json): repos = json['sources'] repodirs = [] + oesetupbuild = None for r_name in repos: r_data = repos[r_name] repodir = os.path.abspath(os.path.join(args['destdir'], r_data['path'])) @@ -108,9 +109,18 @@ def _do_checkout(args, json): if _contains_submodules(repodir): print("Repo {} contains submodules, use 'git submodule update' to ensure they are up to date".format(repodir)) + if os.path.exists(os.path.join(repodir, 'scripts/oe-setup-build')): + oesetupbuild = os.path.join(repodir, 'scripts/oe-setup-build') _write_layer_list(args['destdir'], repodirs) + if oesetupbuild: + oesetupbuild_symlink = os.path.join(args['destdir'], 'setup-build') + if os.path.exists(oesetupbuild_symlink): + os.remove(oesetupbuild_symlink) + os.symlink(os.path.relpath(oesetupbuild,args['destdir']),oesetupbuild_symlink) + print("\nRun '{}' to list available build configuration templates and set up a build from one of them.".format(oesetupbuild_symlink)) + parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/") parser.add_argument('--force-bootstraplayer-checkout', action='store_true',
This is the last (I believe) piece of the puzzle in setting up builds from nothing without having to write custom scripts or use external tools. After layers have been fetched and placed into their respective locations by oe-setup-layers, one would surely want to proceed to the actual build, and here's how: 1. Without arguments or with 'list-config-templates', the tool reads available layers from .oe-layers.json file (written out by oe-setup-layers) and prints what templates it has found, as seen below. If the following is not enough information, adding '-v' will also print conf-notes.txt for each of the templates: $ oe-setup-build Available build configuration templates (re-run with -v to see their descriptions): /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/poky/meta-poky/conf/templates/default will create a build configuration in /srv/work/alex/build-meta-poky-default /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gizmo will create a build configuration in /srv/work/alex/build-meta-alex-configuration-gizmo /srv/work/alex/poky/scripts/oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gadget will create a build configuration in /srv/work/alex/build-meta-alex-configuration-gadget 2. Then the user picks one command of the above and runs it. This will land them in a shell ready to run bitbake: $ oe-setup-build setup-build-env -c /srv/work/alex/meta-alex/conf/templates/configuration-gizmo Running: TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/configuration-gizmo . /srv/work/alex/poky/oe-init-build-env /srv/work/alex/build-meta-alex-configuration-gizmo && /bin/bash (...standard Yocto banner omitted...) This configuration template will set up a build for the purposes of supporting gizmo. Please refer to meta-alex/README for additional details and available bitbake targets. 3. The full set of command line options is: $ oe-setup-build -h usage: oe-setup-build [-h] {list-config-templates,setup-build-env} ... A script that discovers available build configuration templates and sets up a build environment based on one of them positional arguments: {list-config-templates,setup-build-env} list-config-templates List available configuration templates setup-build-env Set up a build environment and open a shell session with it, ready to run builds. optional arguments: -h, --help show this help message and exit $ oe-setup-build list-config-templates -h usage: oe-setup-build list-config-templates [-h] [--layerlist LAYERLIST] [-v] optional arguments: -h, --help show this help message and exit --layerlist LAYERLIST Where to look for available layers (as written out by setup-layers script) (default is /srv/storage/alex/yocto/build-64-alt/.oe-layers.json). -v Print a description for each available build configuration template. $ oe-setup-build setup-build-env -h usage: oe-setup-build setup-build-env [-h] -c template_path [-b build_path] [--no-shell] optional arguments: -h, --help show this help message and exit -c template_path Use a build configuration template in template_path to set up a build environment (run this script with 'list-config-templates' to see what is available) -b build_path Set up a build directory in build_path (run this script with 'list-config-templates' to see where it would be by default) --no-shell Create a build directory but do not start a shell session with the build environment from it. 4. There's an an added hint in oe-setup-layers about how to proceed (as it is really not user-friendly to fetch the layer repos successfully and then exit without a word), and a symlink to the script from the top level layer checkout directory. 5. The selftest to check layer setup has been adjusted to run a basic check for template discovery and build setup. The revision of poky to be cloned has been bumped to 4.1, as that's the first version with a default template in a standard location. Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- meta/lib/oeqa/selftest/cases/bblayers.py | 23 ++++++- scripts/oe-setup-build | 77 ++++++++++++++++++++++++ scripts/oe-setup-layers | 10 +++ 3 files changed, 108 insertions(+), 2 deletions(-) create mode 100755 scripts/oe-setup-build