From patchwork Fri Jul 8 19:17:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 10027 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06981C43334 for ; Fri, 8 Jul 2022 19:17:51 +0000 (UTC) Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) by mx.groups.io with SMTP id smtpd.web09.3183.1657307862534450326 for ; Fri, 08 Jul 2022 12:17:42 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=gFgD1EX5; spf=pass (domain: gmail.com, ip: 209.85.218.42, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f42.google.com with SMTP id d2so39281122ejy.1 for ; Fri, 08 Jul 2022 12:17:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=tAHvvi0tcp2CwRb8DdPGpiIqK/FDZrp20nu1EOvi9Lc=; b=gFgD1EX5Im25CS4kp2ZrgjAfI6mmjKqsSxjCdkqVEhEYhuxdkf5D081IoWhBGvTvaX df3MFR6V+68FXRQnaqKrdN9+aPOOHb7zk2vg3/o4bIiDNr2gRvHrqxZZ16IpG+JmoamZ K5VQraKLDGWxeTv3cd7ADSokr5RcR+7UxZBMQxEyJLUZeLdDbzXRpw3Rty6vZL6bOZ2C +frGr7ScmYgqXdve2PhY33GR9xr2keEdS3UEJXorRXQRzOJyJ3SzZNzjam7CDsgNqPRJ ZkiKpjjOEgPmHoqr5l5v+CYwj5EJrFQbDT7RT8VmTULSZRTt3zwqAxpL/33d62H5ElP9 3VSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=tAHvvi0tcp2CwRb8DdPGpiIqK/FDZrp20nu1EOvi9Lc=; b=7NXxwm2SXuzDNjQRYqwnOdGzcArd9xxQmWojhh1DeaMr85WlN7jBdM9jtci6avZIaq WlPurs6lKdPW/ZSHpji/vk8hTsNGgRXB8Yp7lySwzN1XBfpNk+NA0pfpCiXtoxBvU2w+ k0NcAsrHsbP/jo61CmMzJqC+gX1+oU+O/L6p0tc+HeHiBVqgksaO6gglMecKHYJFjzHp RzEd+0TdcuB02BWLDm+PSMIta8qosxoN81iMfL9qPLsrI5uFVBLAn1zRg8PHU7zXpvde z1g6SaL2TNqwrHBHl2T3TedKae2qKT/DixW5DbjYxQgury8++kgfpEdAIl49Lu5yNarA ryyA== X-Gm-Message-State: AJIora8E7EsNZkh4u4fjwoyAQCWD/M1ssxjJ/jEtk6p1HVXWnFFCk+OO UZbHhUo1LRAJ/J7WcjOT5L4EKPYIxbCAbA== X-Google-Smtp-Source: AGRyM1srRMVkkDpUAmIkkcgmRSB7bseSERvA6sgER7LxoZD3pNTW++m4GXKHYHwTLY3pVsubQC7MqA== X-Received: by 2002:a17:906:31d9:b0:726:a68b:b666 with SMTP id f25-20020a17090631d900b00726a68bb666mr5026502ejf.159.1657307860872; Fri, 08 Jul 2022 12:17:40 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bl24-20020a170906c25800b00704757b1debsm20465705ejb.9.2022.07.08.12.17.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Jul 2022 12:17:39 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Joshua Watt , Alexander Kanavin Subject: [RFC PATCH 1/3] meta/files: add layer setup JSON schema and example Date: Fri, 8 Jul 2022 21:17:28 +0200 Message-Id: <20220708191730.3413011-1-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Jul 2022 19:17:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/167828 From: Joshua Watt Defines a common schema for layer setup that can be consumed by tools to know how to fetch and assemble layers for end users. Also includes an example of the layer setup that constructs poky/meta-intel/imaginary product layer for reference. The schema can be used to validate a layer setup file with the commands: $ python3 -m pip install jsonschema $ jsonschema -i meta/files/layers.example.json meta/files/layers.schema.json Signed-off-by: Joshua Watt Alex: I made the following modifications to Joshua's original commit: - moved the files from meta/lib to meta/files - the example json showcases a multi-repo, multi-layer setup with additional configurations and machines, instead of just poky - closer to a typical product - the schema is modified so that: -- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items. This allows using them as dicts inside Python, and makes the json more compact and readable. -- added 'contains_this_file' property to source object -- added 'buildconfigs', 'machines' and 'distros' properties to layer objects. -- replaced 'remote' property with a 'oneOf' definition for git with a specific 'git-remote' property. 'oneOf' is problematic when schema validation fails: the diagnostic is only that none of oneOf variants matched, which is too non-specific. -- added 'describe' property to 'git-remote' object. FIXME: - oe-selftest that validates the example json against the schema using python3-jsonschema-native Signed-off-by: Alexander Kanavin --- meta/files/layers.example.json | 115 ++++++++++++++++++++++++++++++ meta/files/layers.schema.json | 125 +++++++++++++++++++++++++++++++++ 2 files changed, 240 insertions(+) create mode 100644 meta/files/layers.example.json create mode 100644 meta/files/layers.schema.json diff --git a/meta/files/layers.example.json b/meta/files/layers.example.json new file mode 100644 index 0000000000..f606567b9d --- /dev/null +++ b/meta/files/layers.example.json @@ -0,0 +1,115 @@ +{ + "sources": { + "meta-alex": { + "contains_this_file": true, + "git-remote": { + "branch": "master", + "describe": "", + "remotes": { + "remote-alex": { + "uri": "https://github.com/kanavin/meta-alex" + } + }, + "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5" + }, + "layers": { + "meta-alex": { + "buildconfigs": { + "conf/templates/configuration-gadget": {}, + "conf/templates/configuration-gizmo": {} + }, + "subpath": "" + } + }, + "path": "meta-alex" + }, + "meta-intel": { + "git-remote": { + "branch": "master", + "describe": "15.0-hardknott-3.3-310-g0a96edae", + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/meta-intel" + } + }, + "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a" + }, + "layers": { + "meta-intel": { + "machines": { + "intel-core2-32": {}, + "intel-corei7-64": {}, + "intel-skylake-64": {} + }, + "subpath": "" + } + }, + "path": "meta-intel" + }, + "poky": { + "git-remote": { + "branch": "akanavin/setup-layers", + "describe": "4.1_M1-374-g9dda719b2a", + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/poky" + }, + "poky-contrib": { + "uri": "ssh://git@push.yoctoproject.org/poky-contrib" + } + }, + "rev": "9dda719b2a4727a4d43a6ab8d9e23f8ca68790ec" + }, + "layers": { + "meta": { + "distros": { + "defaultsetup": {} + }, + "machines": { + "qemuarm": {}, + "qemuarm64": {}, + "qemuarmv5": {}, + "qemumips": {}, + "qemumips64": {}, + "qemuppc": {}, + "qemuppc64": {}, + "qemuriscv32": {}, + "qemuriscv64": {}, + "qemux86": {}, + "qemux86-64": {} + }, + "subpath": "meta" + }, + "meta-poky": { + "buildconfigs": { + "conf": {} + }, + "distros": { + "poky": {}, + "poky-altcfg": {}, + "poky-bleeding": {}, + "poky-tiny": {} + }, + "subpath": "meta-poky" + }, + "meta-selftest": { + "machines": { + "qemux86copy": {} + }, + "subpath": "meta-selftest" + }, + "meta-yocto-bsp": { + "machines": { + "beaglebone-yocto": {}, + "edgerouter": {}, + "genericx86": {}, + "genericx86-64": {} + }, + "subpath": "meta-yocto-bsp" + } + }, + "path": "poky" + } + }, + "version": "1.0" +} \ No newline at end of file diff --git a/meta/files/layers.schema.json b/meta/files/layers.schema.json new file mode 100644 index 0000000000..ba46e00b52 --- /dev/null +++ b/meta/files/layers.schema.json @@ -0,0 +1,125 @@ +{ + "description": "OpenEmbedder Layer Setup Manifest", + "type": "object", + "additionalProperties": false, + "required": [ + "version" + ], + "properties": { + "version": { + "description": "The version of this document; currently '1.0'", + "enum": ["1.0"] + }, + "sources": { + "description": "The dict of layer sources", + "type": "object", + "patternProperties": { ".*" : { + "type": "object", + "description": "The upstream source from which a set of layers may be fetched", + "additionalProperties": false, + "required": [ + "path" + ], + "properties": { + "path": { + "description": "The path where this layer source will be placed when fetching", + "type": "string" + }, + "description": { + "description": "A description of this layer source", + "type": "string" + }, + "contains_this_file": { + "description": "Whether the directory with the layer source also contains this json description. Tools may want to skip the checkout of the source then.", + "type": "boolean" + }, + "layers": { + "description": "The dict of layers to be used from this upstream source", + "type": "object", + "patternProperties": { ".*" : { + "description": "A layer from the upstream source", + "type": "object", + "additionalProperties": false, + "properties": { + "subpath": { + "description": "The subpath (relative to the source root) for this layer. Omit if the source root is the layer path", + "type": "string" + }, + "buildconfigs": { + "description": "A dict of template build configurations in the layer, keyed by relative path to the layer root", + "type": "object", + "patternProperties": { ".*" : { + "description": "Properties of a build confuguration template", + "type": "object", + "additionalProperties": false, + "properties":{} + }} + }, + "machines": { + "description": "A dict of machine definitions in the layer", + "type": "object", + "patternProperties": { ".*" : { + "description": "Properties of a machine definition", + "type": "object", + "additionalProperties": false, + "properties":{} + }} + }, + "distros": { + "description": "A dict of distro definitions in the layer", + "type": "object", + "patternProperties": { ".*" : { + "description": "Properties of a distro definition", + "type": "object", + "additionalProperties": false, + "properties":{} + }} + } + } + }} + }, + "git-remote": { + "description": "A remote git source from which to fetch", + "type": "object", + "additionalProperties": false, + "required": [ + "rev" + ], + "properties": { + "branch": { + "description": "The git branch to fetch (optional)", + "type": "string" + }, + "rev": { + "description": "The git revision to checkout", + "type": "string" + }, + "describe": { + "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).", + "type": "string" + }, + "remotes": { + "description": "The dict of git remotes to add to this repository", + "type": "object", + "patternProperties": { ".*" : { + "description": "A git remote", + "type": "object", + "addtionalProperties": false, + "required": [ + "uri" + ], + "properties": { + "uri": { + "description": "The URI for the remote", + "type": "string" + } + } + }} + } + } + } + } + } + }} + } +} From patchwork Fri Jul 8 19:17:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 10025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 075B2CCA47F for ; Fri, 8 Jul 2022 19:17:51 +0000 (UTC) Received: from mail-ed1-f47.google.com (mail-ed1-f47.google.com [209.85.208.47]) by mx.groups.io with SMTP id smtpd.web10.3126.1657307863336035779 for ; Fri, 08 Jul 2022 12:17:43 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=b36OHO7C; spf=pass (domain: gmail.com, ip: 209.85.208.47, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f47.google.com with SMTP id y8so22212099eda.3 for ; Fri, 08 Jul 2022 12:17:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=X4+rf0hzLkg2pFXUUDn+nKCK8OEOMFIx18xUNHOTuio=; b=b36OHO7CXUCy/0+I5rDiP/1c/NdL3MkErYv5u3+W77R94nFT3EzMbsEd/9PmXsRra0 0ML0JyQbZgcTXtrIspXxXVhShoqch0s+XhSq8KHvy1SPbJjJPYKKUYXP0LMxF/a6Io9w +KGrEF0+c5UX1c93dJZCIrMC+urKSL6whTQwNUpNmK0cd4x79D3No8z63IWaCfG2AuE9 6NF48+ll0H5mfwe5+ZFUNDrwIOBUMZng5A7GBz8dyUJxBq3PlePbXojOZLC+wimO/wsj ozuk/WgJS1kBbEDvN3d2RCArzR1TzVUG/a2VjTDao2MAlknO27iu2hrrJv8psEXAXHGF 94Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X4+rf0hzLkg2pFXUUDn+nKCK8OEOMFIx18xUNHOTuio=; b=Ea1SClUIIWAyQhNXorJ6b7DGBpi4w2pRGPRUsVnrMHrjQkgqTIA81JoyektDF+Mbtv p5uBnPhGzNr/eijnXXSSH/sDyYGcEZSdiUY1qDNjdazcjbedT5ajwDWJAXNptHbO6JVd uAs+avOwNDtHR1tsAAj8yQFxFxgdkKHLXd+jGCeONjdjCs0tAd44IeHoXHTipBB6WUv9 Q1DOTJ262unWu33mhPYXv5CAut/f06AZaH0ErQhK6UiLqU+goAKB7KxCOROQlIj30lBr XvlLhEzcEBNWWA8mpcwvJeV5JZnmHrF6OD802QZNCVZiHzoO0FY5oKIBGJ0y9ruOi4Uo DbDw== X-Gm-Message-State: AJIora+HtYLs4qWiVD81C/zy9rqIhvk0PbWLAQQpwz7Z/Kh5Jd5ubUKW 9nJ4qOaWtyGLzHI8PzV8BhiO++G7qJKYsg== X-Google-Smtp-Source: AGRyM1vnDdFHd6+4O6NLF5hCL4AZ+cbt1E9arX5SK+d6CcNjKTmIylOao3yNeiB6RnNJ1ya59m4G4g== X-Received: by 2002:a05:6402:3708:b0:433:2d3b:ed5 with SMTP id ek8-20020a056402370800b004332d3b0ed5mr6799865edb.246.1657307861769; Fri, 08 Jul 2022 12:17:41 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bl24-20020a170906c25800b00704757b1debsm20465705ejb.9.2022.07.08.12.17.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Jul 2022 12:17:41 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 2/3] bitbake-layers: add ability to save current layer repository configuration into a json file Date: Fri, 8 Jul 2022 21:17:29 +0200 Message-Id: <20220708191730.3413011-2-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220708191730.3413011-1-alex@linutronix.de> References: <20220708191730.3413011-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Jul 2022 19:17:51 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/167829 This addresses a long standing gap in the core offering: there is no tooling to capture the currently configured layers with their revisions, or restore the layers from a configuration file (without using external tools, some of which aren't particularly suitable for the task). This plugin addresses the 'capture' part. How to save a layer configuration: a) Running with default choices: NOTE: Starting bitbake server... NOTE: Created /srv/work/alex/meta-alex/setup-layers.json NOTE: Created /srv/work/alex/meta-alex/setup-layers b) Command line options: alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h NOTE: Starting bitbake server... usage: bitbake-layers create-layers-setup [-h] [--output-prefix OUTPUT_PREFIX] [--json-only] destdir Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. positional arguments: destdir Directory where to write the output (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). optional arguments: -h, --help show this help message and exit --output-prefix OUTPUT_PREFIX, -o OUTPUT_PREFIX File name prefix for the output files, if the default (setup-layers) is undesirable. --json-only Write only the layer configuruation in json format. Otherwise, also a copy of poky/scripts/oe-setup-layers is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json. FIXME: - oe-selftest that validates the generated json against the schema - handling hierarchical layer repository trees Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/makesetup.py | 135 +++++++++++++++++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 meta/lib/bblayers/makesetup.py diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py new file mode 100644 index 0000000000..36de32b6f9 --- /dev/null +++ b/meta/lib/bblayers/makesetup.py @@ -0,0 +1,135 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import stat +import sys +import shutil +import json + +import bb.utils +import bb.process + +from bblayers.common import LayerPlugin + +logger = logging.getLogger('bitbake-layers') + +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) + +import oe.buildcfg + +def plugin_init(plugins): + return MakeSetupPlugin() + +class MakeSetupPlugin(LayerPlugin): + + def _write_python(self, input, output): + with open(input) as f: + script = f.read() + with open(output, 'w') as f: + f.write(script) + st = os.stat(output) + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + + def _write_json(self, repos, output): + with open(output, 'w') as f: + json.dump(repos, f, sort_keys=True, indent=4) + + def _get_repo_path(self, layer_path): + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) + return repo_path.strip() + + def _get_remotes(self, repo_path): + remotes = {} + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) + for r in remotes_list.split(): + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) + remotes[r] = {'uri':uri.strip()} + return remotes + + def _get_describe(self, repo_path): + try: + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) + except bb.process.ExecutionError: + return "" + return describe.strip() + + def _get_confs(self, conf_path): + try: + files = os.listdir(conf_path) + except: + return [] + return {f.replace(".conf",""):{} for f in files if f.endswith(".conf")} + + def _get_distros(self, layer_path): + return self._get_confs(os.path.join(layer_path, "conf/distro")) + + def _get_machines(self, layer_path): + return self._get_confs(os.path.join(layer_path, "conf/machine")) + + def _get_buildconfigs(self, layerpath): + return {os.path.relpath(dir, start=layerpath):{} for (dir, subdirs, files) in os.walk(layerpath) if 'local.conf.sample' in files and 'bblayers.conf.sample' in files} + + def _make_repo_config(self, destdir): + repos = {} + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + if l_name == 'workspace': + continue + if l_ismodified: + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l_name,path=l_path)) + return + repo_path = self._get_repo_path(l_path) + if repo_path not in repos.keys(): + repos[repo_path] = {'path':os.path.basename(repo_path),'layers':{},'git-remote':{'rev':l_rev, 'branch':l_branch, 'remotes':self._get_remotes(repo_path), 'describe':self._get_describe(repo_path)}} + if not repos[repo_path]['git-remote']['remotes']: + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) + return + if repo_path in os.path.abspath(destdir): + repos[repo_path]['contains_this_file'] = True + repos[repo_path]['layers'][l_name] = {'subpath':l_path.replace(repo_path,'')[1:]} + distros = self._get_distros(l_path) + machines = self._get_machines(l_path) + buildconfigs = self._get_buildconfigs(l_path) + if distros: + repos[repo_path]['layers'][l_name]['distros'] = distros + if machines: + repos[repo_path]['layers'][l_name]['machines'] = machines + if buildconfigs: + repos[repo_path]['layers'][l_name]['buildconfigs'] = buildconfigs + + repo_dirs = set([os.path.dirname(p) for p in repos.keys()]) + if len(repo_dirs) > 1: + logger.error("Layer repositories are not all in the same parent directory: {repo_dirs}. They need to be relocated into the same directory.".format(repo_dirs=repo_dirs)) + return + + repos_nopaths = {} + for r in repos.keys(): + r_nopath = os.path.basename(r) + repos_nopaths[r_nopath] = repos[r] + return repos_nopaths + + def do_make_setup(self, args): + """ Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. """ + repos = self._make_repo_config(args.destdir) + json = {"version":"1.0","sources":repos} + if not repos: + return + output = args.output_prefix or "setup-layers" + output = os.path.join(os.path.abspath(args.destdir),output) + self._write_json(json, output + ".json") + logger.info('Created {}.json'.format(output)) + if not args.json_only: + self._write_python(os.path.join(os.path.dirname(__file__),'../../../scripts/oe-setup-layers'), output) + logger.info('Created {}'.format(output)) + + def register_commands(self, sp): + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) + parser_setup_layers.add_argument('destdir', + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') + parser_setup_layers.add_argument('--output-prefix', '-o', + help='File name prefix for the output files, if the default (setup-layers) is undesirable.') + parser_setup_layers.add_argument('--json-only', action='store_true', + help='Write only the layer configuruation in json format. Otherwise, also a copy of poky/scripts/oe-setup-layers is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json.') From patchwork Fri Jul 8 19:17:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 10026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0D88C433EF for ; Fri, 8 Jul 2022 19:17:50 +0000 (UTC) Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by mx.groups.io with SMTP id smtpd.web12.3240.1657307864415063056 for ; Fri, 08 Jul 2022 12:17:44 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=RQfu7tFz; spf=pass (domain: gmail.com, ip: 209.85.218.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f51.google.com with SMTP id u12so39186757eja.8 for ; Fri, 08 Jul 2022 12:17:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CbsJaZVUhPVgC2f93ekdnunuF6zDeajZZDkrpuwxtKk=; b=RQfu7tFzDpDBsqkLQ2f36b0SDF3ZPj6EE+g72yQVipB8/OsD7jm5QO8RLh99P+7NTI 8dVe3q69RVC0WpWItz2GsejKw2+40xv2n2rtCx9QElLRP90qodb/hK77W4hGWOBUea+e HJPDqkQnDowAc7LeleTSj0vivQNap4R4cfp0tlYGBxG3WxWkJiaynCpa+h9IqfUwiJvX rGs2SZSWILMJjqOALHsTUfjwOQrDa2fgLeyJPUyeL9mlTH7BEmUB0G+XN5RrXEqeDuwW CjAPSuUeFGU/q5B9HQg5PgVv78rFfw/aHLTV34YRPJSG0tq+Y0O4SrPx3Ox+NJ455Kjw X1tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CbsJaZVUhPVgC2f93ekdnunuF6zDeajZZDkrpuwxtKk=; b=OJiuBmwVQTR9egdCDGjc8DnisPrSGE7BAExFqC9yoBgcx6Pv/WNYlh6IwJUD16hw1E sfihIKIqUA6rxSA+3jfqk/geafdFHEbk/d3oaWlINHbBOv7ghX12RYkyrot/PjMvdNxr g0yejNERqNzLFDXxGODddue9+6JeDWS4XctNxd82XtUEqflxmGoJ4y5jCxDF0EWG67aU 1ElVNdUkjHHMMsnjpcjFziVRdCa1UUSBZ8LPm4KHiS/60+pmNNT/3qqV9dkYrVoraCz1 HpjlAC6pqV39m5pFYCStEEjBYZsXyQLSkZs8Xj5QIwDWQ4R9opcGzOvETgKuoMEBUtlu MliA== X-Gm-Message-State: AJIora/ZoHxgjm/df5MG8R7j5foENYHVPIHn1hGQI00jGW5MJrEPeAC6 6PPMLolVPCSTEclZBT5cj/pDXl+bM8NEyg== X-Google-Smtp-Source: AGRyM1ucBLEkEqtzQmnbFh3H1WqPrXPKnb5BAKBVY0LOpSIyQID/FQ7fjmqE0uCBOehpgrJ3hhoC4Q== X-Received: by 2002:a17:906:8a41:b0:72b:20fe:80ea with SMTP id gx1-20020a1709068a4100b0072b20fe80eamr5067436ejc.316.1657307862599; Fri, 08 Jul 2022 12:17:42 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id bl24-20020a170906c25800b00704757b1debsm20465705ejb.9.2022.07.08.12.17.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Jul 2022 12:17:42 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [RFC PATCH 3/3] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file Date: Fri, 8 Jul 2022 21:17:30 +0200 Message-Id: <20220708191730.3413011-3-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220708191730.3413011-1-alex@linutronix.de> References: <20220708191730.3413011-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Jul 2022 19:17:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/167830 This script can be used directly from poky, or can be copied directly into a layer or any other repository - it is self-suffucient and requires only python3 and git on the host where it will run. It is also copied by the bitbake-layers layers-setup plugin together with the json, unless requested otherwise. 1. How to restore the layers from the saved configuration: a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. (use 'bitbake-layers create-layer-setup' from the previous commit to create them) b) Running with default options: (note: this will work to update an existing checkout as well) alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky Available build configurations: /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gadget /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gizmo /srv/work/alex/my-build/poky/meta-poky/conf You can set up a build with: TEMPLATECONF=one/of/the/above . /srv/work/alex/my-build/poky/oe-init-build-env 2. Command line options: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --help show this help message and exit --force-bootstraplayer-checkout Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). FIXME: - oe-selftest that replicates steps in 1a and checks that the outcome is usable. Signed-off-by: Alexander Kanavin --- scripts/oe-setup-layers | 85 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 scripts/oe-setup-layers diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers new file mode 100755 index 0000000000..c94829214a --- /dev/null +++ b/scripts/oe-setup-layers @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +# +# This file was copied from poky/scripts/oe-setup-layers by running +# +# bitbake-layers create-layers-setup destdir +# +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. +# + +import argparse +import json +import os +import subprocess + +def _do_checkout(args, json): + layers = json['sources'] + buildconfs = [] + oecorepath = "" + for l_name in layers: + l_data = layers[l_name] + layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) + + for ll_name in l_data['layers']: + if ll_name == 'meta': + oecorepath = layerdir + ll_data = l_data['layers'][ll_name] + if 'buildconfigs' in ll_data: + for c in ll_data['buildconfigs']: + buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) + + if 'contains_this_file' in l_data.keys(): + force_arg = 'force_bootstraplayer_checkout' + if not args[force_arg]: + print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) + continue + l_remote = l_data['git-remote'] + rev = l_remote['rev'] + desc = l_remote['describe'] + if not desc: + desc = rev[:10] + branch = l_remote['branch'] + remotes = l_remote['remotes'] + + print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch)) + cmd = 'git init -q {}'.format(layerdir) + print("Running '{}'".format(cmd)) + subprocess.check_output(cmd, text=True, shell=True) + + for remote in remotes: + cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri']) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, text=True, shell=True, cwd=layerdir) + + cmd = "git fetch -q {} || true".format(remote) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, text=True, shell=True, cwd=layerdir) + + cmd = 'git checkout -q {}'.format(rev) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, text=True, shell=True, cwd=layerdir) + + + print("\nAvailable build configurations:\n{}\n".format("\n".join(buildconfs))) + print("You can set up a build with:\nTEMPLATECONF=one/of/the/above . {}/oe-init-build-env".format(oecorepath)) + + +parser = argparse.ArgumentParser(description='A self contained python script that fetches all the needed layers and sets them to correct revisions') + +parser.add_argument('--force-bootstraplayer-checkout', action='store_true', + help='Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).') + +try: + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) +except subprocess.CalledProcessError as e: + defaultdest = os.path.abspath(".") + +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) +parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) + +args = parser.parse_args() + +with open(args.jsondata) as f: + json = json.load(f) + +_do_checkout(vars(args), json)