From patchwork Fri Jul 29 14:10:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 10739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BF50C19F29 for ; Fri, 29 Jul 2022 14:11:04 +0000 (UTC) Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) by mx.groups.io with SMTP id smtpd.web09.8432.1659103858739268061 for ; Fri, 29 Jul 2022 07:10:59 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=YgWTqgLv; spf=pass (domain: gmail.com, ip: 209.85.218.42, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f42.google.com with SMTP id l23so8741772ejr.5 for ; Fri, 29 Jul 2022 07:10:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=1iRBnJxSRy8QPB2LwHsDZ1tJ2uAWrdOHgRRj+diNQI8=; b=YgWTqgLv+C9RCUjuRxZl9iK6oszQmWG1VzAmGlyK2guiXArxB9hkMMvryUZDpgH+hA XG72oIcweB5BDtRF+2XgFwyc9SiSeCcmnL5nu89Qg8HaBhfC2bIm/FYr37RZb5m6CXpd 1HMoThjEWYWyzdDpojWHnPlzv5YQmSjD8/LcBb9r/gAomYTt94x06SgzeVEwHaRzaomw ryhLOZbeO1zrFyW4dDUO7aZC/g51YQPh6vdJNh0KfraEq6dAJZNr+7dHn0g13yulro0w LmhHnW83c8gaEVwpLpJqtkr9X+8DUljKr+Zi8Fl3op2XsMnXnNp8X7/1QUIFuW9l6LDS +/NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=1iRBnJxSRy8QPB2LwHsDZ1tJ2uAWrdOHgRRj+diNQI8=; b=vkqiPUU286Ca8uwLjy6Otc0aJl8BE0HNWKm07Cj6VpzVENhYNdPOvMpwcVvxBVOwAS M8eVYmcneSQJdX2cbxBuVc3m72M3ERjqRr0C+olliA4yx7NpiAUfMW+Yhiv+IRi25edZ aqs3Iym49Gg/Nzzthr6Gq17+oNdAtwS36vNBgy7bsR4trsJfm159V6wJUEmTmYUx2AT4 1eWEmNa2Rbr7/Q+/cybIR4K0YUb0NbxAoV4L/0LR8mEervhaI4hNoQXdCD36VXnEiSgk vI7Nz+qhEO44e8dQPmuMmXJoYBjod5JG4OhIS0eld2NqHhpmX44tjjvcPprbYhITEfQ1 nV2A== X-Gm-Message-State: AJIora+MPn2Yqa/g57rnt7o/PWQ70cUWEmGD9IzwDXZIAd/zZOMTKpoB K3PA2BsKI0nXTKNXw5nVbXgRwIlAsg0= X-Google-Smtp-Source: AGRyM1uR1hGZbCd1kDcIeMPRt0Rkx2NUcGTF93tUPXem9obJx2WI63BdpeBvZ+ia/4yOXHTQO3XlAA== X-Received: by 2002:a17:907:2c5b:b0:72b:54bd:40f8 with SMTP id hf27-20020a1709072c5b00b0072b54bd40f8mr2942707ejc.137.1659103856901; Fri, 29 Jul 2022 07:10:56 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id fm15-20020a1709072acf00b0072b14836087sm1744227ejc.103.2022.07.29.07.10.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Jul 2022 07:10:55 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 1/6] scripts/oe-setup-builddir: make it known where configurations come from Date: Fri, 29 Jul 2022 16:10:44 +0200 Message-Id: <20220729141049.835733-1-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 29 Jul 2022 14:11:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/168628 "with some default values" isn't clear; if the user wants to change or inspect the default values and their history, we should help them find where they are. This becomes especially important when using template configs other than poky's. Signed-off-by: Alexander Kanavin --- scripts/oe-setup-builddir | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir index 54048e62ec..b4fec4d120 100755 --- a/scripts/oe-setup-builddir +++ b/scripts/oe-setup-builddir @@ -74,9 +74,9 @@ fi if [ ! -r "$BUILDDIR/conf/local.conf" ]; then cat < X-Patchwork-Id: 10737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BBDAC19F2C for ; Fri, 29 Jul 2022 14:11:04 +0000 (UTC) Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) by mx.groups.io with SMTP id smtpd.web10.8439.1659103859431809447 for ; Fri, 29 Jul 2022 07:10:59 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=d7A5+7x2; spf=pass (domain: gmail.com, ip: 209.85.218.42, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f42.google.com with SMTP id z23so8741157eju.8 for ; Fri, 29 Jul 2022 07:10:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PedJ4+0l7BYB8YQgOqr6wuv3BK6lXGxzxnG0296TTu8=; b=d7A5+7x2eiw1ethaffVkrry0759ZlncpM7UrNwnrQzmkcpgYUWBLkeVj6EX6nE5Dsx DXp/j6gjsC0SU/H7ONy9p24h3gl5D9Ds+lGdjG6OdYLMY27K6wNfhYzcNKubQecTqPJa 1C9A9WQR8Q+9LFmTEc4GKqRlphmcQvIC7spWVPnCUWe4sqUFBQQ6s4kkIrH0rg2aeGbF emnVaCu6Ot0gJlth3w5Gks4X9CulK4sPA8rM54ixwLl4rSVR2B0zSxV3LbKXO5/NCn0J n0RhPffkaBq7YT45mO1d7hpmV+SGvL833oXvnV55aw4nMePC3yvNul61xjayaCayA65A w3Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PedJ4+0l7BYB8YQgOqr6wuv3BK6lXGxzxnG0296TTu8=; b=hfIlKQ0J1qNfiplq7ffi/7AT8d4gQHoQQQGymO7l+dtjzI78TZa1X0F4IK+EvBeGbX 6Rv5PYMcf8pTcphIKwhQFCTEdy0+tlcKu8Bx5my91PnAkMuVz1SDXAXPo/oGGtDCjUMS 37tds/sw0xtdXkT4O2OmHUrQRGy29REhSmivxENt3cG0ghTAIoOaHrFyl0P81GKjEiEq TWP1OzFO+SZBjG/II4rmUEWe3QgStrfNBaoOngP00o0MmCnfw+Jbzxl9jXkSxQQalWci ZUKZu0L/6VTGFMUK0NyIUNvBPetFe7wpCTxK8TgTjv1HwTY0tq7GDGxE1FIxSMiSlXOM WHjA== X-Gm-Message-State: AJIora8BeAdDjErTUP6wWc/3ix6ECdMuuZZtgCs1jOVsr+Pm1Ozxce/j N2Vs8uEBdgVnlKaCMEok+i6KzVTFx34= X-Google-Smtp-Source: AGRyM1voDApITz1K+guLhADlD5MWL8B5lMs3p4m/w7giHcCvT8HMoSiORD4l1PZt52VCuSCTQoiYRQ== X-Received: by 2002:a17:907:7fa2:b0:72b:60c4:a04e with SMTP id qk34-20020a1709077fa200b0072b60c4a04emr3055599ejc.290.1659103857866; Fri, 29 Jul 2022 07:10:57 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id fm15-20020a1709072acf00b0072b14836087sm1744227ejc.103.2022.07.29.07.10.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Jul 2022 07:10:57 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 2/6] bitbake-layers: add a command to save the active build configuration as a template into a layer Date: Fri, 29 Jul 2022 16:10:45 +0200 Message-Id: <20220729141049.835733-2-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220729141049.835733-1-alex@linutronix.de> References: <20220729141049.835733-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 29 Jul 2022 14:11:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/168629 This is the reverse of setting up a build by pointing TEMPLATECONF to a directory with a template and running '. oe-init-build-env': this takes the config files from build/conf, replaces site-specific paths in bblayers.conf with ##OECORE##-relative paths, and copies the config files into a specified layer under a specified template name. In many or perhaps most cases such static prefabricated configurations (that require no further editing) are just enough, and I believe they should be offered by the official configuration management. On the other hand, generating build configurations with a sufficiently versatile tool is a far more complex problem, and one we should try to tackle once we see where and how static configs fall short. Tooling to discover and select these templates when setting up a build will be provided later on. How to use: alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers save-build-conf ../../meta-alex/ test-1 NOTE: Starting bitbake server... NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1 Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt You can try out the configuration with TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1 alex@Zen2:/srv/work/alex/poky/build-layersetup$ Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/buildconf.py | 83 ++++++++++++++++++++++++ meta/lib/oeqa/selftest/cases/bblayers.py | 5 ++ 2 files changed, 88 insertions(+) create mode 100644 meta/lib/bblayers/buildconf.py diff --git a/meta/lib/bblayers/buildconf.py b/meta/lib/bblayers/buildconf.py new file mode 100644 index 0000000000..ece8836d5a --- /dev/null +++ b/meta/lib/bblayers/buildconf.py @@ -0,0 +1,83 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import stat +import sys +import shutil +import json + +import bb.utils +import bb.process + +from bblayers.common import LayerPlugin + +logger = logging.getLogger('bitbake-layers') + +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) + +import oe.buildcfg + +def plugin_init(plugins): + return BuildConfPlugin() + +class BuildConfPlugin(LayerPlugin): + notes_fixme = """FIXME: Please place here the description of this build configuration. +It will be shown to the users when they set up their builds via TEMPLATECONF. +""" + + def _save_conf(self, templatename, templatepath, oecorepath, relpaths_to_oecore): + confdir = os.path.join(os.environ["BBPATH"], "conf") + destdir = os.path.join(templatepath, "conf", "templates", templatename) + os.makedirs(destdir, exist_ok=True) + + with open(os.path.join(confdir, "local.conf")) as src: + with open(os.path.join(destdir, "local.conf.sample"), 'w') as dest: + dest.write(src.read()) + + with open(os.path.join(confdir, "bblayers.conf")) as src: + with open(os.path.join(destdir, "bblayers.conf.sample"), 'w') as dest: + bblayers_data = src.read() + + for (abspath, relpath) in relpaths_to_oecore: + bblayers_data = bblayers_data.replace(abspath, "##OEROOT##/" + relpath) + dest.write(bblayers_data) + + with open(os.path.join(destdir, "conf-notes.txt"), 'w') as dest: + dest.write(self.notes_fixme) + + logger.info("""Configuration template placed into {} +Please review the files in there, and particularly provide a configuration description in {} +You can try out the configuration with +TEMPLATECONF={} . {}/oe-init-build-env build-try-{}""" +.format(destdir, os.path.join(destdir, "conf-notes.txt"), destdir, oecorepath, templatename)) + + def do_save_build_conf(self, args): + """ Save the currently active build configuration (conf/local.conf, conf/bblayers.conf) as a template into a layer.\n This template can later be used for setting up builds via TEMPLATECONF. """ + repos = {} + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) + targetlayer = None + oecore = None + + for l in layers: + if l[0] == os.path.abspath(args.layerpath): + targetlayer = l[0] + if l[1] == 'meta': + oecore = os.path.dirname(l[0]) + + if not targetlayer: + logger.error("Layer {} not in one of the currently enabled layers:\n{}".format(args.layerpath, "\n".join([l[0] for l in layers]))) + elif not oecore: + logger.error("Openembedded-core not in one of the currently enabled layers:\n{}".format("\n".join([l[0] for l in layers]))) + else: + relpaths_to_oecore = [(l[0], os.path.relpath(l[0], start=oecore)) for l in layers] + self._save_conf(args.templatename, targetlayer, oecore, relpaths_to_oecore) + + def register_commands(self, sp): + parser_build_conf = self.add_command(sp, 'save-build-conf', self.do_save_build_conf, parserecipes=False) + parser_build_conf.add_argument('layerpath', + help='The path to the layer where the configuration template should be saved.') + parser_build_conf.add_argument('templatename', + help='The name of the configuration template.') diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 7d74833f61..8aa45e432e 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -111,6 +111,11 @@ class BitbakeLayers(OESelftestTestCase): self.assertEqual(bb_vars['BBFILE_PRIORITY_%s' % layername], str(priority), 'BBFILE_PRIORITY_%s != %d' % (layername, priority)) + result = runCmd('bitbake-layers save-build-conf {} {}'.format(layerpath, "buildconf-1")) + for f in ('local.conf.sample', 'bblayers.conf.sample', 'conf-notes.txt'): + fullpath = os.path.join(layerpath, "conf", "templates", "buildconf-1", f) + self.assertTrue(os.path.exists(fullpath), "Template configuration file {} not found".format(fullpath)) + def get_recipe_basename(self, recipe): recipe_file = "" result = runCmd("bitbake-layers show-recipes -f %s" % recipe) From patchwork Fri Jul 29 14:10:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 10740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E351C19F2B for ; Fri, 29 Jul 2022 14:11:04 +0000 (UTC) Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) by mx.groups.io with SMTP id smtpd.web12.8423.1659103860890781851 for ; Fri, 29 Jul 2022 07:11:01 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=TeOYbdxE; spf=pass (domain: gmail.com, ip: 209.85.218.48, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f48.google.com with SMTP id fy29so8689943ejc.12 for ; Fri, 29 Jul 2022 07:11:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CsxuF0CddlrXuIzGokcZRzcfRJB0l07nqE/LEGiWM0I=; b=TeOYbdxEqekVZ3QK//XPSSkHIFv9XH8ssycRdlTYi9i59v0nhi/d9k8MIZDst8zqxf jODlLQMhat9q8InTs1PXBrOSKSe/AbRphfp6e9qm8x7cENfIwdp5TFbJ6+RGbebnWsJs 1qmCevHbZhhSSho8veBWgkYS/rBhI1uJi5jUAjx6CmEiG4uYzCrSY0GAD38vRB0a+LQp LplCuLkIFNEGxWDoXcBzzuUkz74rLe3vyqc2H9WWPvC6YpatZ4BWBO4d1VVD0ZxfTU6o sPYbaHt2IsqFxMI2YrMaR9CDnAPjGqysHeFxZISpu5KcF8BXsmel47EUjdknnd3xj7Jl R6BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CsxuF0CddlrXuIzGokcZRzcfRJB0l07nqE/LEGiWM0I=; b=VMluf7eGIjc3CnMubbCNnytHKAk9/v0EA2MBCT1Hy52zLW/9dX5hyJ0RKtCE77zTrV vcJ82FsyksYLc6hkvl5J6mmnJKtxTx/FVGDVbHGsZAs8U/9souA0wpqTS3CmVQd8QAtC guBDL7XnCn5krJ1HDIwa1B+n9E1izM3CpwG5vlT873BLBxBsRd2kJ+pbvSebFNom+ogH j4943pRLtX8CwwHxxAyS71L1zRvbRhVIeTL9sjUsBa1GaZdWTEtp3mfcwzzlEOg0T8jK guAyKJ5ydCxvp6tsQPKYe/U/3yXwud9obIfdWMDsJwgRiw0Lp4CmGU8A4trSWe1BH9GP U8HQ== X-Gm-Message-State: AJIora/Nq5E8lO1k4FKarksT5xma+qBEgw8oIUBUFHW0MTKM9yw1Gsce zgih6/L/ATOw1LnrD0JC9IAbrByCIT8= X-Google-Smtp-Source: AGRyM1sjOfbzHLTGv3Ebj/mt2+mtU24yuawyZF907UKDI2bTsh39K6b1cfWNYWg4CZxpnQ8rhcnh+Q== X-Received: by 2002:a17:907:67b0:b0:72b:972c:53df with SMTP id qu48-20020a17090767b000b0072b972c53dfmr3148414ejc.37.1659103859177; Fri, 29 Jul 2022 07:10:59 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id fm15-20020a1709072acf00b0072b14836087sm1744227ejc.103.2022.07.29.07.10.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Jul 2022 07:10:58 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Joshua Watt , Alexander Kanavin Subject: [PATCH 3/6] meta/files: add layer setup JSON schema and example Date: Fri, 29 Jul 2022 16:10:46 +0200 Message-Id: <20220729141049.835733-3-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220729141049.835733-1-alex@linutronix.de> References: <20220729141049.835733-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 29 Jul 2022 14:11:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/168630 From: Joshua Watt Defines a common schema for layer setup that can be consumed by tools to know how to fetch and assemble layers for end users. Also includes an example of the layer setup that constructs poky/meta-intel/imaginary product layer for reference. The schema can be used to validate a layer setup file with the commands: $ python3 -m pip install jsonschema $ jsonschema -i meta/files/layers.example.json meta/files/layers.schema.json Signed-off-by: Joshua Watt Alex: I made the following modifications to Joshua's original commit: - moved the files from meta/lib to meta/files - the example json showcases a multi-repo, multi-layer setup with additional configurations and machines, instead of just poky - closer to a typical product - added oe-selftest that validates the example json against the schema using python3-jsonschema-native - the schema is modified so that: -- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items. This allows using them as dicts inside Python, and makes the json more compact and readable. -- added 'contains_this_file' property to source object -- added 'buildconfigs', 'machines' and 'distros' properties to layer objects. -- replaced 'remote' property with a 'oneOf' definition for git with a specific 'git-remote' property. 'oneOf' is problematic when schema validation fails: the diagnostic is only that none of oneOf variants matched, which is too non-specific. -- added 'describe' property to 'git-remote' object. -- removed description property for a layer source: it is not clear how to add that when auto-generating the json Signed-off-by: Alexander Kanavin --- meta/files/layers.example.json | 115 +++++++++++++++++++++ meta/files/layers.schema.json | 121 +++++++++++++++++++++++ meta/lib/oeqa/selftest/cases/bblayers.py | 16 ++- 3 files changed, 251 insertions(+), 1 deletion(-) create mode 100644 meta/files/layers.example.json create mode 100644 meta/files/layers.schema.json diff --git a/meta/files/layers.example.json b/meta/files/layers.example.json new file mode 100644 index 0000000000..b6e35c615d --- /dev/null +++ b/meta/files/layers.example.json @@ -0,0 +1,115 @@ +{ + "sources": { + "meta-alex": { + "contains_this_file": true, + "git-remote": { + "branch": "master", + "describe": "", + "remotes": { + "remote-alex": { + "uri": "https://github.com/kanavin/meta-alex" + } + }, + "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5" + }, + "layers": { + "meta-alex": { + "buildconfigs": { + "conf/templates/configuration-gadget": {}, + "conf/templates/configuration-gizmo": {} + }, + "subpath": "" + } + }, + "path": "meta-alex" + }, + "meta-intel": { + "git-remote": { + "branch": "master", + "describe": "15.0-hardknott-3.3-310-g0a96edae", + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/meta-intel" + } + }, + "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a" + }, + "layers": { + "meta-intel": { + "machines": { + "intel-core2-32": {}, + "intel-corei7-64": {}, + "intel-skylake-64": {} + }, + "subpath": "" + } + }, + "path": "meta-intel" + }, + "poky": { + "git-remote": { + "branch": "akanavin/setup-layers", + "describe": "4.1_M1-374-g9dda719b2a", + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/poky" + }, + "poky-contrib": { + "uri": "ssh://git@push.yoctoproject.org/poky-contrib" + } + }, + "rev": "9dda719b2a4727a4d43a6ab8d9e23f8ca68790ec" + }, + "layers": { + "meta": { + "distros": { + "defaultsetup": {} + }, + "machines": { + "qemuarm": {}, + "qemuarm64": {}, + "qemuarmv5": {}, + "qemumips": {}, + "qemumips64": {}, + "qemuppc": {}, + "qemuppc64": {}, + "qemuriscv32": {}, + "qemuriscv64": {}, + "qemux86": {}, + "qemux86-64": {} + }, + "subpath": "meta" + }, + "meta-poky": { + "buildconfigs": { + "conf": {} + }, + "distros": { + "poky": {}, + "poky-altcfg": {}, + "poky-bleeding": {}, + "poky-tiny": {} + }, + "subpath": "meta-poky" + }, + "meta-selftest": { + "machines": { + "qemux86copy": {} + }, + "subpath": "meta-selftest" + }, + "meta-yocto-bsp": { + "machines": { + "beaglebone-yocto": {}, + "edgerouter": {}, + "genericx86": {}, + "genericx86-64": {} + }, + "subpath": "meta-yocto-bsp" + } + }, + "path": "poky" + } + }, + "version": "1.0" +} diff --git a/meta/files/layers.schema.json b/meta/files/layers.schema.json new file mode 100644 index 0000000000..2603fae8f7 --- /dev/null +++ b/meta/files/layers.schema.json @@ -0,0 +1,121 @@ +{ + "description": "OpenEmbedder Layer Setup Manifest", + "type": "object", + "additionalProperties": false, + "required": [ + "version" + ], + "properties": { + "version": { + "description": "The version of this document; currently '1.0'", + "enum": ["1.0"] + }, + "sources": { + "description": "The dict of layer sources", + "type": "object", + "patternProperties": { ".*" : { + "type": "object", + "description": "The upstream source from which a set of layers may be fetched", + "additionalProperties": false, + "required": [ + "path" + ], + "properties": { + "path": { + "description": "The path where this layer source will be placed when fetching", + "type": "string" + }, + "contains_this_file": { + "description": "Whether the directory with the layer source also contains this json description. Tools may want to skip the checkout of the source then.", + "type": "boolean" + }, + "layers": { + "description": "The dict of layers to be used from this upstream source", + "type": "object", + "patternProperties": { ".*" : { + "description": "A layer from the upstream source", + "type": "object", + "additionalProperties": false, + "properties": { + "subpath": { + "description": "The subpath (relative to the source root) for this layer. Omit if the source root is the layer path", + "type": "string" + }, + "buildconfigs": { + "description": "A dict of template build configurations in the layer, keyed by relative path to the layer root", + "type": "object", + "patternProperties": { ".*" : { + "description": "Properties of a build confuguration template", + "type": "object", + "additionalProperties": false, + "properties":{} + }} + }, + "machines": { + "description": "A dict of machine definitions in the layer", + "type": "object", + "patternProperties": { ".*" : { + "description": "Properties of a machine definition", + "type": "object", + "additionalProperties": false, + "properties":{} + }} + }, + "distros": { + "description": "A dict of distro definitions in the layer", + "type": "object", + "patternProperties": { ".*" : { + "description": "Properties of a distro definition", + "type": "object", + "additionalProperties": false, + "properties":{} + }} + } + } + }} + }, + "git-remote": { + "description": "A remote git source from which to fetch", + "type": "object", + "additionalProperties": false, + "required": [ + "rev" + ], + "properties": { + "branch": { + "description": "The git branch to fetch (optional)", + "type": "string" + }, + "rev": { + "description": "The git revision to checkout", + "type": "string" + }, + "describe": { + "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).", + "type": "string" + }, + "remotes": { + "description": "The dict of git remotes to add to this repository", + "type": "object", + "patternProperties": { ".*" : { + "description": "A git remote", + "type": "object", + "addtionalProperties": false, + "required": [ + "uri" + ], + "properties": { + "uri": { + "description": "The URI for the remote", + "type": "string" + } + } + }} + } + } + } + } + } + }} + } +} diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 8aa45e432e..4f4952f3eb 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -6,12 +6,16 @@ import os import re import oeqa.utils.ftools as ftools -from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars +from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake from oeqa.selftest.case import OESelftestTestCase class BitbakeLayers(OESelftestTestCase): + def setUpLocal(self): + bitbake("python3-jsonschema-native") + bitbake("-c addto_recipe_sysroot python3-jsonschema-native") + def test_bitbakelayers_layerindexshowdepends(self): result = runCmd('bitbake-layers layerindex-show-depends meta-poky') find_in_contents = re.search("openembedded-core", result.output) @@ -126,3 +130,13 @@ class BitbakeLayers(OESelftestTestCase): self.assertTrue(os.path.isfile(recipe_file), msg = "Can't find recipe file for %s" % recipe) return os.path.basename(recipe_file) + + def validate_layersjson(self, json): + python = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-jsonschema-native'), 'nativepython3') + jsonvalidator = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-jsonschema-native'), 'jsonschema') + jsonschema = os.path.join(get_bb_var('COREBASE'), 'meta/files/layers.schema.json') + result = runCmd("{} {} -i {} {}".format(python, jsonvalidator, json, jsonschema)) + + def test_validate_examplelayersjson(self): + json = os.path.join(get_bb_var('COREBASE'), "meta/files/layers.example.json") + self.validate_layersjson(json) From patchwork Fri Jul 29 14:10:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 10738 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B10EC00144 for ; Fri, 29 Jul 2022 14:11:04 +0000 (UTC) Received: from mail-ej1-f47.google.com (mail-ej1-f47.google.com [209.85.218.47]) by mx.groups.io with SMTP id smtpd.web12.8425.1659103861993777675 for ; Fri, 29 Jul 2022 07:11:02 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=kmQ/h3hr; spf=pass (domain: gmail.com, ip: 209.85.218.47, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f47.google.com with SMTP id sz17so8733094ejc.9 for ; Fri, 29 Jul 2022 07:11:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0dDqFtaP0FFfSqzIoXwx2hoyxMlP101mCMxyJc6rpBY=; b=kmQ/h3hrrirAFjgBunTw1HUaLulf04SUtqZ14guloEtBi2NsxuR3FwTkz8OYwpKxZ9 XLF7TFzladciWmwIVz3UA9f8lqLTWzhLZ0O6hYxPkYA1/akucJ0b9fDdYuKyXOaJFXmX wopLCM4PsQXEUoHmplyXKfYBiOY1NHepcWtm16Fi3fsj4yIDlgCBibSd4LRDrLlO4Y0z w00KjAnSluh7xeKqvhHMVpfYVlxoMFEcCkvO8oTNBt/NfPAHXQ4Svf3s1lo1WASFfxl6 Kg0jvxe1liBX6C/Sx9ZYXf04/hO5E4p3W0Ja8ZY9U4dQXS4I0DQ63Z6cLb2LP0h598Mf B0Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0dDqFtaP0FFfSqzIoXwx2hoyxMlP101mCMxyJc6rpBY=; b=T6oaonwcVpZ7SvvzXXgHF16vDDu7x0zBiITRnX3a/QRxlELXwlpy6JyRPXLHDT69c0 A25KfwfNetGK6lPOKiEzdSwGBRvMFTevOUrj3AQKEAESKfYXiA3rq/F8QVv01DJ9BE+O NAsN1NbgHrwaJL1CHPI8sl1MLupaRlO4efxccUebmX6OayaIiMnRxTxbCk3ac66TYEtb lEvAfp3LswoGDbCyjeXQp3cRaGAhgJuz9mCTJGSBKUuKtGMlgh5waZqpE7qaNdGNpcFu uekJd7GvZaDUkv3WklxvhjDWyn/qEzlvV+Mh7AYJEtA4Nie4OcNmqCoPoIW/gyxAx4LC jOvw== X-Gm-Message-State: AJIora9us7o6ao6qftEqASDINQUXLaJ/jdiq6vNTlwl3hlWZO0yW24tq NVTe2upbUVuW16+gaL+cUSQEITF0yv4= X-Google-Smtp-Source: AGRyM1un+LOJk1rOtfhYBnuXpyci15SD+0aZoV27werBCXLCabzsmAoyNq5SePtZ/UzPoAsJUVhqHA== X-Received: by 2002:a17:906:4598:b0:72f:643c:db45 with SMTP id qs24-20020a170906459800b0072f643cdb45mr3058735ejc.194.1659103860221; Fri, 29 Jul 2022 07:11:00 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id fm15-20020a1709072acf00b0072b14836087sm1744227ejc.103.2022.07.29.07.10.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Jul 2022 07:10:59 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 4/6] bitbake-layers: add ability to save current layer repository configuration into a json file Date: Fri, 29 Jul 2022 16:10:47 +0200 Message-Id: <20220729141049.835733-4-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220729141049.835733-1-alex@linutronix.de> References: <20220729141049.835733-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 29 Jul 2022 14:11:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/168631 This addresses a long standing gap in the core offering: there is no tooling to capture the currently configured layers with their revisions, or restore the layers from a configuration file (without using external tools, some of which aren't particularly suitable for the task). This plugin addresses the 'capture' part. How to save a layer configuration: a) Running with default choices: NOTE: Starting bitbake server... NOTE: Created /srv/work/alex/meta-alex/setup-layers.json NOTE: Created /srv/work/alex/meta-alex/setup-layers b) Command line options: alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h NOTE: Starting bitbake server... usage: bitbake-layers create-layers-setup [-h] [--output-prefix OUTPUT_PREFIX] [--json-only] destdir Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. positional arguments: destdir Directory where to write the output (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). optional arguments: -h, --help show this help message and exit --output-prefix OUTPUT_PREFIX, -o OUTPUT_PREFIX File name prefix for the output files, if the default (setup-layers) is undesirable. --json-only Write only the layer configuruation in json format. Otherwise, also a copy of poky/scripts/oe-setup-layers is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json. Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/makesetup.py | 139 +++++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 meta/lib/bblayers/makesetup.py diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py new file mode 100644 index 0000000000..b7f62900f4 --- /dev/null +++ b/meta/lib/bblayers/makesetup.py @@ -0,0 +1,139 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import stat +import sys +import shutil +import json + +import bb.utils +import bb.process + +from bblayers.common import LayerPlugin + +logger = logging.getLogger('bitbake-layers') + +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) + +import oe.buildcfg + +def plugin_init(plugins): + return MakeSetupPlugin() + +class MakeSetupPlugin(LayerPlugin): + + def _write_python(self, input, output): + with open(input) as f: + script = f.read() + with open(output, 'w') as f: + f.write(script) + st = os.stat(output) + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + + def _write_json(self, repos, output): + with open(output, 'w') as f: + json.dump(repos, f, sort_keys=True, indent=4) + + def _get_repo_path(self, layer_path): + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) + return repo_path.strip() + + def _get_remotes(self, repo_path): + remotes = {} + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) + for r in remotes_list.split(): + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) + remotes[r] = {'uri':uri.strip()} + return remotes + + def _get_describe(self, repo_path): + try: + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) + except bb.process.ExecutionError: + return "" + return describe.strip() + + def _get_confs(self, conf_path): + try: + files = os.listdir(conf_path) + except: + return [] + return {f.replace(".conf",""):{} for f in files if f.endswith(".conf")} + + def _get_distros(self, layer_path): + return self._get_confs(os.path.join(layer_path, "conf/distro")) + + def _get_machines(self, layer_path): + return self._get_confs(os.path.join(layer_path, "conf/machine")) + + def _get_buildconfigs(self, layerpath): + return {os.path.relpath(dir, start=layerpath):{} for (dir, subdirs, files) in os.walk(layerpath) if 'local.conf.sample' in files and 'bblayers.conf.sample' in files} + + def _make_repo_config(self, destdir): + repos = {} + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) + try: + destdir_repo = self._get_repo_path(destdir) + except bb.process.ExecutionError: + destdir_repo = None + + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + if l_name == 'workspace': + continue + if l_ismodified: + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l_name,path=l_path)) + return + repo_path = self._get_repo_path(l_path) + if repo_path not in repos.keys(): + repos[repo_path] = {'path':os.path.basename(repo_path),'layers':{},'git-remote':{'rev':l_rev, 'branch':l_branch, 'remotes':self._get_remotes(repo_path), 'describe':self._get_describe(repo_path)}} + if repo_path == destdir_repo: + repos[repo_path]['contains_this_file'] = True + if not repos[repo_path]['git-remote']['remotes'] and not repos[repo_path]['contains_this_file']: + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) + return + repos[repo_path]['layers'][l_name] = {'subpath':l_path.replace(repo_path,'')[1:]} + distros = self._get_distros(l_path) + machines = self._get_machines(l_path) + buildconfigs = self._get_buildconfigs(l_path) + if distros: + repos[repo_path]['layers'][l_name]['distros'] = distros + if machines: + repos[repo_path]['layers'][l_name]['machines'] = machines + if buildconfigs: + repos[repo_path]['layers'][l_name]['buildconfigs'] = buildconfigs + + top_path = os.path.commonpath([os.path.dirname(r) for r in repos.keys()]) + + repos_nopaths = {} + for r in repos.keys(): + r_nopath = os.path.basename(r) + repos_nopaths[r_nopath] = repos[r] + r_relpath = os.path.relpath(r, top_path) + repos_nopaths[r_nopath]['path'] = r_relpath + return repos_nopaths + + def do_make_setup(self, args): + """ Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. """ + repos = self._make_repo_config(args.destdir) + json = {"version":"1.0","sources":repos} + if not repos: + raise Exception("Could not determine layer sources") + output = args.output_prefix or "setup-layers" + output = os.path.join(os.path.abspath(args.destdir),output) + self._write_json(json, output + ".json") + logger.info('Created {}.json'.format(output)) + if not args.json_only: + self._write_python(os.path.join(os.path.dirname(__file__),'../../../scripts/oe-setup-layers'), output) + logger.info('Created {}'.format(output)) + + def register_commands(self, sp): + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) + parser_setup_layers.add_argument('destdir', + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') + parser_setup_layers.add_argument('--output-prefix', '-o', + help='File name prefix for the output files, if the default (setup-layers) is undesirable.') + parser_setup_layers.add_argument('--json-only', action='store_true', + help='Write only the layer configuruation in json format. Otherwise, also a copy of poky/scripts/oe-setup-layers is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json.') From patchwork Fri Jul 29 14:10:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 10741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BF9EC3F6B0 for ; Fri, 29 Jul 2022 14:11:04 +0000 (UTC) Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by mx.groups.io with SMTP id smtpd.web10.8441.1659103863093712892 for ; Fri, 29 Jul 2022 07:11:03 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=BUEOW2co; spf=pass (domain: gmail.com, ip: 209.85.218.54, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f54.google.com with SMTP id rq15so2746455ejc.10 for ; Fri, 29 Jul 2022 07:11:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Je+jwr6vCvljNDFaZne0LW229Np6Sy8Lj/nZt9Dq0I4=; b=BUEOW2coXK3UIoqlEbBFNeRYkVr2lowNaPhSFBOzn6Her0JhQ2AWE7wek/NVGftcmg GMsiZ/Au2RL4K8V8HWa/VfJo1UgjZMv0ta9WoGJgjNj3gfwA/nmTHy6LJ1P35hlLR7wv Ygs+vsi+/+pRvuuU/2h8sEewzX5eRzBmNFTnkvQPDbdOH/qpBFnylST6hHSdNhphVNLu OEMs4/zufptYBe9Dh5DX7Yl/rKAFZJoos/bN49Nwpo2NlG/vzJxqKAEhD2ZLZ7ObU+zM HzTBY2iCLvqd3lHtoR4tkKkheWACHcuwX04hYIaRV3bPuMFun2+qkg6oGszC+Hsv0Tga ySHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Je+jwr6vCvljNDFaZne0LW229Np6Sy8Lj/nZt9Dq0I4=; b=whxSLyhzHcDat9XRKhXAdlvoxY2aAJujG5YxL8LXVo4LdegkNIZqSIb9zokJs50dHV bxp9Scr2TrcRhw55XVr4j3f36NHxaeO+cuv+ZMq5P8zR9049OgUMjcLfjOGxYL5y/nTe HMTKP5OsvaWTQvX2l1IMVSGuEGJU55bTSwv1rxK0sTV0f+uKLAx+8nWlqicZQRE7JsqT kDEjJBL7GsvP0MSqIKEzermyifZ95eo5PI9PeiEBq6Zk2zKquNXWcepAEZK57KRcS1bO 0/ixXgDUQ3OQtmmxNLLVWtGnRD8xmxrpDdK/cSifcQwAsxBz19Qw7dbuD7l5C4iuvpZP gdrA== X-Gm-Message-State: AJIora+HLurYWbiSMDXz0PXyJK/eSiacJ71vSplor9/BBtdq84meZMWJ iv1zknTF9DIJDinizl/Tv9WGD5CQL6I= X-Google-Smtp-Source: AGRyM1v4puYUYNiPB13nOnng9ba9EeZouZzbFRgyOZmWz2qHl4aCEm1JxH/M4sE1N+e5AA81+mNhhA== X-Received: by 2002:a17:906:730c:b0:72b:4f83:51c1 with SMTP id di12-20020a170906730c00b0072b4f8351c1mr3044945ejc.515.1659103861276; Fri, 29 Jul 2022 07:11:01 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id fm15-20020a1709072acf00b0072b14836087sm1744227ejc.103.2022.07.29.07.11.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Jul 2022 07:11:00 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 5/6] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file Date: Fri, 29 Jul 2022 16:10:48 +0200 Message-Id: <20220729141049.835733-5-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220729141049.835733-1-alex@linutronix.de> References: <20220729141049.835733-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 29 Jul 2022 14:11:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/168632 This script can be used directly from poky, or can be copied directly into a layer or any other repository - it is self-suffucient and requires only python3 and git on the host where it will run. It is also copied by the bitbake-layers layers-setup plugin together with the json, unless requested otherwise. 1. How to restore the layers from the saved configuration: a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. (use 'bitbake-layers create-layer-setup' from the previous commit to create them) b) Running with default options: (note: this will work to update an existing checkout as well) alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky Available build configurations: /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gadget /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gizmo /srv/work/alex/my-build/poky/meta-poky/conf You can set up a build with: TEMPLATECONF=one/of/the/above . /srv/work/alex/my-build/poky/oe-init-build-env 2. Command line options: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --help show this help message and exit --force-bootstraplayer-checkout Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). Signed-off-by: Alexander Kanavin --- scripts/oe-setup-layers | 85 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 scripts/oe-setup-layers diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers new file mode 100755 index 0000000000..017c400ad6 --- /dev/null +++ b/scripts/oe-setup-layers @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +# +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running +# +# bitbake-layers create-layers-setup destdir +# +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. +# + +import argparse +import json +import os +import subprocess + +def _do_checkout(args, json): + layers = json['sources'] + buildconfs = [] + oecorepath = "" + for l_name in layers: + l_data = layers[l_name] + layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) + + for ll_name in l_data['layers']: + if ll_name == 'meta': + oecorepath = layerdir + ll_data = l_data['layers'][ll_name] + if 'buildconfigs' in ll_data: + for c in ll_data['buildconfigs']: + buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) + + if 'contains_this_file' in l_data.keys(): + force_arg = 'force_bootstraplayer_checkout' + if not args[force_arg]: + print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) + continue + l_remote = l_data['git-remote'] + rev = l_remote['rev'] + desc = l_remote['describe'] + if not desc: + desc = rev[:10] + branch = l_remote['branch'] + remotes = l_remote['remotes'] + + print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch)) + cmd = 'git init -q {}'.format(layerdir) + print("Running '{}'".format(cmd)) + subprocess.check_output(cmd, text=True, shell=True) + + for remote in remotes: + cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri']) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, text=True, shell=True, cwd=layerdir) + + cmd = "git fetch -q {} || true".format(remote) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, text=True, shell=True, cwd=layerdir) + + cmd = 'git checkout -q {}'.format(rev) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, text=True, shell=True, cwd=layerdir) + + + print("\nAvailable build configurations:\n{}\n".format("\n".join(buildconfs))) + print("You can set up a build with:\nTEMPLATECONF=one/of/the/above . {}/oe-init-build-env".format(oecorepath)) + + +parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/") + +parser.add_argument('--force-bootstraplayer-checkout', action='store_true', + help='Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).') + +try: + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', text=True, shell=True, cwd=os.path.dirname(__file__))) +except subprocess.CalledProcessError as e: + defaultdest = os.path.abspath(".") + +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) +parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) + +args = parser.parse_args() + +with open(args.jsondata) as f: + json = json.load(f) + +_do_checkout(vars(args), json) From patchwork Fri Jul 29 14:10:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 10742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37060C00144 for ; Fri, 29 Jul 2022 14:11:14 +0000 (UTC) Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by mx.groups.io with SMTP id smtpd.web09.8434.1659103863962244586 for ; Fri, 29 Jul 2022 07:11:04 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=QbMqoKTV; spf=pass (domain: gmail.com, ip: 209.85.208.41, mailfrom: alex.kanavin@gmail.com) Received: by mail-ed1-f41.google.com with SMTP id m8so5959300edd.9 for ; Fri, 29 Jul 2022 07:11:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Fv5EfvI5qxIjQnh7ixNMs3R4YsMVashWDi4a7VrUflg=; b=QbMqoKTVWcsTgoTOy8Pv9WJqQrhqlduWwhRdYzU7GMizLO1jlub3n9BCrNT9UIXt+H roc5Fi++4cpfnVMdg+SQUUsFYtCq6i18MlvGj4I75ufvJFTrbXBgSIEE94rvvMSgBf92 wcRumlZU4XcOPDM+zTuOivNlV5zITgdTeu1FaCX29N2cZHHGNC/wMHWrT+G1M5ADDRp1 nmCLPHIwqdMlsppU1t92kH/WAKx3HvixZQXtPy5T6r+NjCx6Jseu8eanSZ+wsFCJqB7W 3v7kYgs2OFOzSZk+FQw+Oq+4gpa2LV2ggInLjqtIze9RbAy9OPTTA93y3aMWZqQ8qZvb Gmeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Fv5EfvI5qxIjQnh7ixNMs3R4YsMVashWDi4a7VrUflg=; b=dxzbeAzWxb7ucI630rdi0zwXtQeWttxO8sxdFcz8jsM83iFvsWhcpD08B8l8/idkGH K9mF8MBGquNefduoC88ellK/ePHxC7t9t2Cidiuz3MwtZvOGz/n6Uh0MRLCoVAmX4K7q i/UYJeo97cNw6+SX0QFGpRtXajRjomCTFPKom3t6QleRqa2bWccS022jXBIyb7wMQtu4 5SiA3IkHleO3O8oRSlMhLQZdfon/0a/tlvmzNGiODNQ9Odnx3RwoCYh7VBGMoCau34fU iu5rwFZXFVe/mqpDkz3KRNJEGzCGI5bztg0dxVaJHENaNRJitBySDe57VFAkQBJF89xF gHAw== X-Gm-Message-State: AJIora8Tk9gFRyvspAgfF2wPdeF56Jt6iKfpUTwmQstkTYToF8jxMi92 kBohMyjiW18rpr59pQMXFxAvjfCDnSY= X-Google-Smtp-Source: AGRyM1tuCgu3c1Ekk5n1j9u7qHEOwAmVbwKkb0C6C9RhXSdyHIg+s/W+ePgQWm9St3arS/6zW115fg== X-Received: by 2002:a50:ec0c:0:b0:43c:fd26:404d with SMTP id g12-20020a50ec0c000000b0043cfd26404dmr3787769edr.359.1659103862535; Fri, 29 Jul 2022 07:11:02 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id fm15-20020a1709072acf00b0072b14836087sm1744227ejc.103.2022.07.29.07.11.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Jul 2022 07:11:01 -0700 (PDT) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 6/6] selftest/bblayers: add a test for creating a layer setup and using it to restore the layers Date: Fri, 29 Jul 2022 16:10:49 +0200 Message-Id: <20220729141049.835733-6-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220729141049.835733-1-alex@linutronix.de> References: <20220729141049.835733-1-alex@linutronix.de> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 29 Jul 2022 14:11:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/168633 This does a basic run-through of the bitbake-layers plugin, and the resulting json layer config and the layer setup script that uses it. Only poky is actually fetched by the script. Signed-off-by: Alexander Kanavin --- meta/lib/oeqa/selftest/cases/bblayers.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 4f4952f3eb..1f5c576356 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -140,3 +140,15 @@ class BitbakeLayers(OESelftestTestCase): def test_validate_examplelayersjson(self): json = os.path.join(get_bb_var('COREBASE'), "meta/files/layers.example.json") self.validate_layersjson(json) + + def test_bitbakelayers_setup(self): + result = runCmd('bitbake-layers create-layers-setup {}'.format(self.testlayer_path)) + jsonfile = os.path.join(self.testlayer_path, "setup-layers.json") + self.validate_layersjson(jsonfile) + + testcheckoutdir = os.path.join(self.builddir, 'test-layer-checkout') + result = runCmd('{}/setup-layers --destdir {}'.format(self.testlayer_path, testcheckoutdir)) + # May not necessarily be named 'poky' + pokydir = os.listdir(testcheckoutdir)[0] + testcheckoutfile = os.path.join(testcheckoutdir, pokydir, "oe-init-build-env") + self.assertTrue(os.path.exists(testcheckoutfile), "File {} not found in test layer checkout".format(testcheckoutfile))