From patchwork Wed Oct 8 16:56:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1AB4CCA470 for ; Wed, 8 Oct 2025 16:57:19 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web11.741.1759942633206843410 for ; Wed, 08 Oct 2025 09:57:13 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=fhIXnzhW; spf=pass (domain: gmail.com, ip: 209.85.221.48, mailfrom: alex.kanavin@gmail.com) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-3ecdf2b1751so111093f8f.0 for ; Wed, 08 Oct 2025 09:57:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759942631; x=1760547431; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=qZDg81WwS8Xsw5QplX9hErCH6HsyfyBJ+UWOH/LzIPg=; b=fhIXnzhWngRPsMAB4Os2CJ4q2SzfHWc6RhzGYM2D+wxH+eAmnADh1QboRV7KJgBQs0 yTPM8jE4DcisSgGq4/YOCgx3aRvTTOMKo3JHYKRf2Mlhz8IosQDsIWv4kcbftYEbBrFd jPsw0NifCXqvITk1bo1HR/S7AmVpU2dshqGWFhQRQ+zG9SFZmgbl4JKP06yzm/xJgswX BPpX/hkk2CBChx+Mn7RaHUQuqbL93zXUxhl3VePKqWyqyxGIzXrVR6ppsyQ7jcfXdqVU Qvu2Yj6MHQmjeWRdOYcbn+Bl8/37cO1EHNNi508NjJsVu4Cgc3cQavbzeKCoJOtNt6NS 2DQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759942631; x=1760547431; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=qZDg81WwS8Xsw5QplX9hErCH6HsyfyBJ+UWOH/LzIPg=; b=dJIjpfL63URcFbVcNUETeDAz9+rD6OoLdGYG9nU1IQ0oxQ9Gs3zAcQT6fy+dl/sb/P UGr+sqLHYaaqpd+XvA0RFBl5nGPrPkqP1mYwWjb/nEROd8Y4l5bhLKBxla77F/zdqk6t TsI6VgXZiQNRX4Zo3dZ4OMGvSknfFSKhxMvL6UesJu/9/BKh1S0TvAdTOyceO9nzFCZV +448H1VoZa43CuHbSstkNvRY54jruCJJFbGNxadjhYlJLtWXTb3+v3FUt4RgNB9Ye3fQ 1Ap86CRxl44WwNbIRlz+rJN4CXt35TuE4GnAgUrLBRaUc2zPOQEhkUm5f5yjn9rkMjlZ vwaQ== X-Gm-Message-State: AOJu0YxpJD8qjHzOey0jSABomwvUvOxdkLlVmGwrlarLnum2Cwp8Uqw8 SSfdb+/pKGOCLqNl0wPo3WYd1pWlExnkWDzKiZPsS3sb1UQTQrK7KflV9ckUrQ== X-Gm-Gg: ASbGncvKTJU+ubYAMJgdudCItzT9x0asbbgh46W/+GEX22yrXqrGq0qWRHDlA3tpAT2 xteidaLF+h6mTe6UvVUg8yRhz0l1QcRx3NwEM/jQQIABDEUXUH80gZaaJ20jjlN2n8eSrwmV9HE Ozp3ea2ptsJuJZ0i+4LuHPTslEhwj/Z5i+BXz9PKORCWmM6Lo3SBUnToO9TooK81wzCTlczpqsC 9Tdfg2s0SRyQboT3GUwQQBdxb4QSe0QSJcg9ZfpCaBc6cPZ9crXRjf2CHD4hpAFQZ0KQobJHHhy FtToO6wH9EhlNyXu/9fjO2yexcLGv5zgJxbPPAG6Lpz0SPwaOtOEVYHyJJDpez3qHK8HlAocrFd iVS0ErUFtchMjW6/wyIAatNKcMWBcIyK6pjF3+putEb3WaUpVVz+72IpP/cleY+P6Sb3xOzVoZ/ LFTsmHc6GclOzpFBb0o3+woXwmzYiXkAWpNP9SxeoFoMPWerlXILPueH8DnGdAFt5gjAepV5lRe nwoZ9cL+Q== X-Google-Smtp-Source: AGHT+IF3+HT7QudTIjAoGkGXBsuXi1X6R/ADjQ5cQ4PGrp0GztelUmnHXH+hq2pCiv/LgeYs+BJ6aQ== X-Received: by 2002:a5d:5d85:0:b0:425:7e45:a4df with SMTP id ffacd0b85a97d-42666ac60c3mr2694653f8f.11.1759942631313; Wed, 08 Oct 2025 09:57:11 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4255d8a6b77sm30740453f8f.6.2025.10.08.09.57.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Oct 2025 09:57:10 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Johannes Schneider Subject: [PATCH 1/4] bitbake-setup: support adding environment-passthroughs to the init-build-env Date: Wed, 8 Oct 2025 18:56:56 +0200 Message-Id: <20251008165659.1884881-1-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 08 Oct 2025 16:57:19 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18145 From: Johannes Schneider This patch adds support for extending the BB_ENV_PASSTHROUGH_ADDITIONS environment variable from within the `init-build-env` wrapper script - generated by either oe-core's oe-setup-build, or by `bitbake-setup` - based on per-configuration JSON settings. This enables CI workflows to inject environment-specific data - such as build number, host, build type, or credentials required to fetch from certain SRC_URIs - which cannot be captured via configuration fragments alone. These variables are now handled early in the setup process and exported directly into the build environment. Example: "bb-env-passthrough-additions": [ "ACME_DIR", "ARTIFACTORY_TOKEN", "ARTIFACTORY_USERNAME", "GITHUB_TOKEN", "GITHUB_PROTOCOL", "KEY" ] the resulting 'init-build-env' would then be: # environment passthrough added by bitbake-setup export BB_ENV_PASSTHROUGH_ADDITIONS=" \ $BB_ENV_PASSTHROUGH_ADDITIONS \ ACME_DIR \ ARTIFACTORY_TOKEN \ ARTIFACTORY_USERNAME \ GITHUB_TOKEN \ GITHUB_PROTOCOL \ KEY" # init-build-env wrapper created by bitbake-setup . /tmp/acme_master-acme-distro_acme-machine_bang/layers/openembedded-core/oe-init-build-env /tmp/bitbake-setup/gs/acme_master-acme-distro_acme-machine_bang/build Signed-off-by: Johannes Schneider --- bin/bitbake-setup | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index e7b955213..7878cd939 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -162,8 +162,31 @@ def setup_bitbake_build(bitbake_config, layerdir, builddir, thisdir): builddir = os.path.realpath(builddir) cmd = "cd {}\nset {}\n. ./oe-init-build-env\n".format(oeinitbuildenvdir, builddir) initbuild_in_builddir = os.path.join(builddir, 'init-build-env') + with open(initbuild_in_builddir, 'w') as f: - f.write(cmd) + f.write("# init-build-env wrapper created by bitbake-setup\n") + f.write(cmd + '\n') + + def _prepend_passthrough_to_init_build_env(builddir): + env = bitbake_config.get("bb-env-passthrough-additions") + if not env: + return + + initbuild_in_builddir = os.path.join(builddir, 'init-build-env') + with open(initbuild_in_builddir) as f: + content = f.read() + + joined = " \\\n".join(env) + env = "export BB_ENV_PASSTHROUGH_ADDITIONS=\" \\\n" + env += "${BB_ENV_PASSTHROUGH_ADDITIONS} \\\n" + env += joined + env += '"' + + with open(initbuild_in_builddir, 'w') as f: + f.write("# environment passthrough added by bitbake-setup\n") + f.write(env + '\n') + f.write('\n') + f.write(content) bitbake_builddir = os.path.join(builddir, "build") print("Setting up bitbake configuration in\n {}\n".format(bitbake_builddir)) @@ -194,6 +217,8 @@ def setup_bitbake_build(bitbake_config, layerdir, builddir, thisdir): return _make_init_build_env(bitbake_builddir, os.path.realpath(oeinitbuildenvdir)) + _prepend_passthrough_to_init_build_env(bitbake_builddir) + siteconf_symlink = os.path.join(bitbake_confdir, "site.conf") siteconf = os.path.normpath(os.path.join(builddir, '..', "site.conf")) if os.path.lexists(siteconf_symlink): From patchwork Wed Oct 8 16:56:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D379FCCD187 for ; Wed, 8 Oct 2025 16:57:19 +0000 (UTC) Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by mx.groups.io with SMTP id smtpd.web10.791.1759942633914194187 for ; Wed, 08 Oct 2025 09:57:14 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=YVpleJ91; spf=pass (domain: gmail.com, ip: 209.85.221.53, mailfrom: alex.kanavin@gmail.com) Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-3f99ac9acc4so120989f8f.3 for ; Wed, 08 Oct 2025 09:57:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759942632; x=1760547432; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QmJW1kqlnN1S2CnsBugrVRuV2iWPSXdpG228dMj/vCs=; b=YVpleJ91VzZX11sckKPbdoFUadGiGEih1b1bMW2bXz4fqxWkG8xpYruNryrnKrTsKi C71G9+C8wWM2+voGZ2HcTOrcmlmSVL/1/eQV54n6S/6xc702Ys1r+aZpHeRj+PFED6/n uLo5LN+tSszcFeCWZSZhthejteN71xDxX5CgzbJhaTEsxgFK0x5Xptqj90d15Lg/ouS4 VzW+vtBXFRspAnEy0psNx+NQkSU9uySZkFLPN/1N3vA6JrLqUsBMP2xPmhgxOl7qZ22u +rA1B2byno3sKeFslvOa2iDvVuGMX9UX2qXORlsbfKg3eHHL4Hwn/1tqd5bMqvyZ5C1u z5Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759942632; x=1760547432; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QmJW1kqlnN1S2CnsBugrVRuV2iWPSXdpG228dMj/vCs=; b=XSaq0Ar/vYcNU5ymUdsV0wedu8CLljHc26nmrEakE4m8eaPs1qRQJ6iAZcjofgZYA9 KcLTQ/94jlHE1ErQI7xVZRnbDtJc1pZE7ok0Cl/faVxp3EW/vs1Hz4hhoYOXwbKOnWwp WT2mH6wqTscn+pogHIYyoWQWXPSrLqvo8+v7QTL5POB5gt8g/j1geUOv7c+1/Xkrp41v 4UqtaSRi7Ab1tOCI24p1N/5YhxXgdKJHyX7swvRKA1+Pd7kKIITTG2WmYfelRWSblqx4 +/onaWMdOZfXHbdYsQkZUEJIBjSbHWXXkbh4XbNk6mBkg8//8UxL11h4ssL5+hMIQN7N 740g== X-Gm-Message-State: AOJu0YwFs48xusOR3m1fs1I4XNw1MjIMEbQzwQXf2GQASJKiQO+xmc66 PTyZ20Rl7hHCre1r5kgO8SuwQe+3uN5uzaLOTAL4tRzF1D+d1FF47A8XbuOeoQ== X-Gm-Gg: ASbGnct4ouCkOVb/WLU67ZVGJdK6Pe7SkLXRBn3oam1z6Ni+F9bSizZHXkUnYbciuKi 83krUQZvApPo6MIpEIIN1FWaXgdmp8WB2dlMvokg9De/JB13NHopDPBfN5I81W3veFCc09DpKEE 4OrZOkXlTYl6ynzGvuL0YhZIequa7LPk0YsRCFI8hX/podjG4rc0mzktcZ5E0myYE0CadMu2EH1 dC8wKbH+VivhpFsC3GpfTVRfPlNff9W+3yeDnPc9ef1p2xcqylImspAdrIeLv9KEbQbzjt/unav V3P1BCvzIsp7i7SUTBimJ1ayWpwIOjMFaxlpQSYhk4F4/qRYFFv++LpnSzKh0SfB3nOkxh3x3By lYqRMxyuVhuic+X6HtH1w/X88+VUR1gHhcMk7a9mN4osMAoPDzjpfDqWaWyVRTegBb6CoTwhIsL VRXgY9HmfvHvsEQWDfRCtoinfJi9KiaC2iB/nSNUaQb+zfG+Xwwh8scFjMtqQLp0VsfHtwW5zG2 LLDMO/1Sg== X-Google-Smtp-Source: AGHT+IEGjHtkuXGQcnihcopD2WQpT8y29mZ3zo5GphgzOVw6q80P+ckc4Ski0BWcUki8Wsp/m1Kd/w== X-Received: by 2002:a05:6000:2501:b0:40d:86c9:5c9e with SMTP id ffacd0b85a97d-4266e7dfddfmr2685216f8f.40.1759942632149; Wed, 08 Oct 2025 09:57:12 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4255d8a6b77sm30740453f8f.6.2025.10.08.09.57.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Oct 2025 09:57:11 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Johannes Schneider Subject: [PATCH 2/4] bitbake-setup: tests: add a template-only test configuration Date: Wed, 8 Oct 2025 18:56:57 +0200 Message-Id: <20251008165659.1884881-2-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20251008165659.1884881-1-alex.kanavin@gmail.com> References: <20251008165659.1884881-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 08 Oct 2025 16:57:19 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18146 From: Johannes Schneider bitbake-setup conf.json can be purely template driven, to setup older yocto LTS (e.g. scarthgap) based layer-collections - where fragment support was not yet present in oe-core+bitbake. Add a configuration for that scenario and add it to the test routine. Signed-off-by: Johannes Schneider --- lib/bb/tests/setup.py | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index 747a9b733..c1e753b14 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -119,6 +119,11 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) "oe-template": "test-configuration-gizmo", "oe-fragments": ["test-fragment-2"] }, + { + "name": "gizmo-no-fragment", + "description": "Gizmo no-fragment template-only build configuration", + "oe-template": "test-configuration-gizmo" + }, { "name": "gadget-notemplate", "description": "Gadget notemplate build configuration", @@ -189,8 +194,9 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) else: self.assertIn(os.path.join(buildpath, "layers", l), bblayers) - for f in bitbake_config["oe-fragments"]: - self.assertTrue(os.path.exists(os.path.join(bb_conf_path, f))) + if 'oe-fragment' in bitbake_config.keys(): + for f in bitbake_config["oe-fragments"]: + self.assertTrue(os.path.exists(os.path.join(bb_conf_path, f))) def test_setup(self): @@ -247,8 +253,16 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) self.add_file_to_testrepo('test-file', test_file_content) # test-config-1 is tested as a registry config, test-config-2 as a local file - test_configurations = {'test-config-1': {'cmdline': 'test-config-1', 'buildconfigs':('gadget','gizmo','gadget-notemplate','gizmo-notemplate')}, - 'test-config-2': {'cmdline': os.path.join(self.registrypath,'config-2/test-config-2.conf.json'), 'buildconfigs': ('gadget','gizmo','gadget-notemplate','gizmo-notemplate', 'gizmo-notemplate-with-thisdir') } } + test_configurations = {'test-config-1': {'cmdline': 'test-config-1', + 'buildconfigs':('gadget','gizmo', + 'gizmo-no-fragment', + 'gadget-notemplate','gizmo-notemplate')}, + 'test-config-2': {'cmdline': os.path.join(self.registrypath,'config-2/test-config-2.conf.json'), + 'buildconfigs': ('gadget','gizmo', + 'gizmo-no-fragment', + 'gadget-notemplate','gizmo-notemplate', + 'gizmo-notemplate-with-thisdir')} + } for cf, v in test_configurations.items(): for c in v['buildconfigs']: out = self.runbbsetup("init --non-interactive {} {}".format(v['cmdline'], c)) @@ -272,13 +286,16 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) prev_test_file_content = test_file_content test_file_content = 'modified\n' self.add_file_to_testrepo('test-file', test_file_content) - for c in ('gadget','gizmo','gadget-notemplate','gizmo-notemplate'): + for c in ('gadget', 'gizmo', + 'gizmo-env-passthrough', + 'gizmo-no-fragment', + 'gadget-notemplate', 'gizmo-notemplate'): buildpath = os.path.join(self.tempdir, 'bitbake-builds', 'test-config-1-{}'.format(c)) os.environ['BBPATH'] = os.path.join(buildpath, 'build') out = self.runbbsetup("status") self.assertIn("Layer repository file://{} checked out into {}/layers/test-repo updated revision master from".format(self.testrepopath, buildpath), out[0]) out = self.runbbsetup("update") - if c in ('gadget','gizmo'): + if c in ('gadget', 'gizmo'): self.assertIn("Existing bitbake configuration directory renamed to {}/build/conf-backup.".format(buildpath), out[0]) self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0]) with open(os.path.join(buildpath, 'config', "config-upstream.json")) as f: @@ -294,14 +311,16 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) self.git('checkout -b {}'.format(branch), cwd=self.testrepopath) self.add_file_to_testrepo('test-file', test_file_content) json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch, branch) - for c in ('gadget','gizmo','gadget-notemplate','gizmo-notemplate'): + for c in ('gadget', 'gizmo', + 'gizmo-no-fragment', + 'gadget-notemplate', 'gizmo-notemplate'): buildpath = os.path.join(self.tempdir, 'bitbake-builds', 'test-config-1-{}'.format(c)) os.environ['BBPATH'] = os.path.join(buildpath, 'build') out = self.runbbsetup("status") self.assertIn("Configuration in {} has changed:".format(buildpath), out[0]) self.assertIn('- "rev": "master"\n+ "rev": "another-branch"', out[0]) out = self.runbbsetup("update") - if c in ('gadget','gizmo'): + if c in ('gadget', 'gizmo'): self.assertIn("Existing bitbake configuration directory renamed to {}/build/conf-backup.".format(buildpath), out[0]) self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0]) with open(os.path.join(buildpath, 'config', "config-upstream.json")) as f: From patchwork Wed Oct 8 16:56:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71852 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D085DCAC5BB for ; Wed, 8 Oct 2025 16:57:19 +0000 (UTC) Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by mx.groups.io with SMTP id smtpd.web11.744.1759942634632407948 for ; Wed, 08 Oct 2025 09:57:14 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=dl6U4e0q; spf=pass (domain: gmail.com, ip: 209.85.221.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-3ee64bc6b85so151656f8f.3 for ; Wed, 08 Oct 2025 09:57:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759942633; x=1760547433; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m/Vhtg76ygXqDf/DoCmJBgG2CcU3cr/K7NUEm77PXA8=; b=dl6U4e0qaMUTldl++8JtT9y5tcFbypyCYW4iHck5kv6Adr5LvglFfj94o9FrnIJ2V3 zzS+irvnmFCTWwqhq6vWybVebKplmyxpZGUHfvMxfpBRLliP83su4bU3wATEZijeXDwI av6Nu5QD2F+swVreV7lYGCge8nbQhOaHHSn0lzt9My5tyOXKTCdIEReq9L+KlfGCeYL4 dZapod8NVEV5eQgP6xUvT4Z4m933+iuJt4CXa7hzEwn8Qn7ycOCCdYqtG56fi1tZNNgD JEW5GUF+sx/wUPVG6IOW83GR/v7Vgc4Et8BPU5y3UzaNeKCWmNUMi9/0nDKWQiuXYCds mNIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759942633; x=1760547433; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m/Vhtg76ygXqDf/DoCmJBgG2CcU3cr/K7NUEm77PXA8=; b=J8NLi40Nx+rhKGP1npfIwUONzMZ9v404W1yizSamlk/uX3Wk0iljtSb4ctse1jURBZ OFgSdKIBCLnfWR6StZFwCOANWsi7Ddt/uj7yhhH61+R//7cKEZ02KttTpDnoeWnEO2nc kUY7KRZhWV0QBM+GpcWG6ip8PoL5iU3Ga2Wx1I2sfsAWMorEk90i7GD2iOJ9MbgGzVj/ vJJLTr11HyFxILGKlN5tx02evVDpNqgFVFxik58/aoVABD24sjefPIYiGB4db84CyLuE qpMU+os4mj+OCTkekbKI0jLZS7WDTq4IrYmQaDhDJeUv5B9b+79uE3lmHVDHbVP3DM/i gYpQ== X-Gm-Message-State: AOJu0Yx2ZiW2hSgZrqhH7K0gFSTWxmqU1ENqWSyqTnL4sEjrUUPZ3aNy pRukhIKPXtTaAb1e5XqjVC0Xa9OaJoyrjAFwwUIAuab4qbVgvKR/SFxeJzSbUQ== X-Gm-Gg: ASbGnctRIwVkHCoK6lSRoi0C1bAVMamejrlvQjEmCk6OC6IKkYxdt4fCSv/Rb7+pvMh Tvryh66LUQTzbCS/0gF33GQ4oCAjCoMEmLzL3EGuPB0wlWLGF6oD3BmiN/KP+MbOnNvza6p10N/ 7Ko8kfHgvT/6agovAai4LXIfwSPvdla5NmFEsOUSAKayPnsyUQk2X8l1lxtF6CunMvmtlPtXT/j pQ8Xg9adh/NX3mDxMPWqN3TEojCtpyKUe9cPAduwKneUeUpbInUkzTd6NUwjGYkX9E6qz8elU6w q/sk5YtQAJvZ7vXCrcYN2wPjGr8zLcs277Oe7ZXF0t3N21I1YmrY+qICXlKPRPa7IB9r4DaULjk Z475rx3YtHh4mot0XGyGTf42VwaMftALGcyMQO2dgbZ2cRbG6iIG5bYvbDK+8OimzaGhYV3na7B DCQmnn3ppebBIi7nAQJKImqPWMh87XJLMnksrnFiSPMtWseZQNbHSAYa2bv2YJk+MDEi1mAsCSA lTnRe7ntg== X-Google-Smtp-Source: AGHT+IELP1j2OjlgF6fqHaU2IGfAorwC97Sax5VOV3IZiC4XXPcBQLBdCapvteDaW0zNUKl88l/UVQ== X-Received: by 2002:a05:6000:2dc9:b0:425:8c06:b78a with SMTP id ffacd0b85a97d-4266e8e6820mr2556654f8f.59.1759942632932; Wed, 08 Oct 2025 09:57:12 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4255d8a6b77sm30740453f8f.6.2025.10.08.09.57.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Oct 2025 09:57:12 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Johannes Schneider Subject: [PATCH 3/4] bitbake-setup: tests: add environment-passthrough Date: Wed, 8 Oct 2025 18:56:58 +0200 Message-Id: <20251008165659.1884881-3-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20251008165659.1884881-1-alex.kanavin@gmail.com> References: <20251008165659.1884881-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 08 Oct 2025 16:57:19 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18147 From: Johannes Schneider Add a test configuration to cover the 'bb-env-passthrough-additions' conf.json key, and add it to the test routine. Signed-off-by: Johannes Schneider --- lib/bb/tests/setup.py | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index c1e753b14..495d1da20 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -119,6 +119,17 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) "oe-template": "test-configuration-gizmo", "oe-fragments": ["test-fragment-2"] }, + { + "name": "gizmo-env-passthrough", + "description": "Gizmo build configuration with environment-passthrough", + "bb-layers": ["layerC","layerD/meta-layer"], + "oe-fragments": ["test-fragment-1"], + "bb-env-passthrough-additions": [ + "BUILD_ID", + "BUILD_DATE", + "BUILD_SERVER" + ] + }, { "name": "gizmo-no-fragment", "description": "Gizmo no-fragment template-only build configuration", @@ -198,6 +209,15 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) for f in bitbake_config["oe-fragments"]: self.assertTrue(os.path.exists(os.path.join(bb_conf_path, f))) + if 'bb-environment-passthrough' in bitbake_config.keys(): + with open(os.path.join(bb_build_path, 'init-build-env'), 'r') as f: + init_build_env = f.read() + self.assertTrue('BB_ENV_PASSTHROUGH_ADDITIONS' in init_build_env) + self.assertTrue('BUILD_ID' in init_build_env) + self.assertTrue('BUILD_DATE' in init_build_env) + self.assertTrue('BUILD_SERVER' in init_build_env) + # a more throrough test could be to initialize a bitbake build-env, export FOO to the shell environment, set the env-passthrough on it and finally check against 'bitbake-getvar FOO' + def test_setup(self): # unset BBPATH to ensure tests run in isolation from the existing bitbake environment @@ -255,10 +275,12 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) # test-config-1 is tested as a registry config, test-config-2 as a local file test_configurations = {'test-config-1': {'cmdline': 'test-config-1', 'buildconfigs':('gadget','gizmo', + 'gizmo-env-passthrough', 'gizmo-no-fragment', 'gadget-notemplate','gizmo-notemplate')}, 'test-config-2': {'cmdline': os.path.join(self.registrypath,'config-2/test-config-2.conf.json'), 'buildconfigs': ('gadget','gizmo', + 'gizmo-env-passthrough', 'gizmo-no-fragment', 'gadget-notemplate','gizmo-notemplate', 'gizmo-notemplate-with-thisdir')} @@ -312,6 +334,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) self.add_file_to_testrepo('test-file', test_file_content) json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch, branch) for c in ('gadget', 'gizmo', + 'gizmo-env-passthrough', 'gizmo-no-fragment', 'gadget-notemplate', 'gizmo-notemplate'): buildpath = os.path.join(self.tempdir, 'bitbake-builds', 'test-config-1-{}'.format(c)) From patchwork Wed Oct 8 16:56:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 71853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFF46CCD183 for ; Wed, 8 Oct 2025 16:57:19 +0000 (UTC) Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by mx.groups.io with SMTP id smtpd.web11.747.1759942635954495336 for ; Wed, 08 Oct 2025 09:57:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Y1H4cNsN; spf=pass (domain: gmail.com, ip: 209.85.221.54, mailfrom: alex.kanavin@gmail.com) Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-3ee13baf2e1so121089f8f.3 for ; Wed, 08 Oct 2025 09:57:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759942634; x=1760547434; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Tn1SxDNwrChFeNJbHBsIYc4kQcbOEY+C/agGZjl02FM=; b=Y1H4cNsNURCaiUcPddd0mWgISVZXYNZjyT+SeN6Pm48KmQ1ssARXVo25GOBF8M8oMK rQ8nq4binA6Js5GWWyof97JOwv9dtCAILHDJcE19voQGjXOBGPoidnudXyVwpZd765mF Rkm+ZCch45oAg5sgEA4R0PdPohZ4oip+UG2nK0LasL6W7qZ9ORVvvGIRALMHKt7Q78PZ ZogHDUK1qceXm3vfkPch/nEQsnG9bRILFo4eCHWsBHEZ6DE66MssjKAqNGnCv31pp3pF AGbuW6m99kQY+ifvKF8E1shyu6Sbl5kpxuym5H46zRpLLlMGgsiFig/Y0UWal6wkDlzj DqgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759942634; x=1760547434; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Tn1SxDNwrChFeNJbHBsIYc4kQcbOEY+C/agGZjl02FM=; b=nK/c8q01hVhesFsHJYqjvP8QI76muBW+EKD4KdrWR8XG4KFEGhpUf3yFm/3HumHLEV hV7ADEEUMfhzs5XugT+3Cdv1k+/4SohAwmWdbhZ5N9W8BHMK7OkR5QINh6BFbrEN55Fi CahL+hXVe8UwTrGS3gchqSgmmwU4Uy9leaW7RV+t0uIw9wsu7zRZhaT3R6MhRvHiToFW 9qEE4X3exT7oiMkcwtss7h+hiq8hLa6pXSKhg9CJuYJYWd8bPw3kbm/uhfg+eutIQ2Nv MlI1tCoGWk4xJ5QL4Ob59LscTQWgwoW0rnW1qjySdQy/dPKx3wgjU9XbCmh2dZJHZH4D OANw== X-Gm-Message-State: AOJu0YyNa6ivT2cJxOF2L+izKXQE9KrkXwiK+fcfex+Hf1iCQnTYGZmT sOSWQj5GCKQlNeqGyCsqPArLXMIkiYNcliuzLNZs6kqTTPZv4rqNwHqVUFreGg== X-Gm-Gg: ASbGnctuOWX32PuZ2ynL6ipQgzSCdT+RHmUbsp1mDpDbS0L4NUVHY+DExemjU8/6NKa JWKO+Lbo8XEYfrL7a8+t/bJMrkcqW7jhm5dXtuEUiNPr2wTVyA7SbWL6wHYfo5BeNmLVThYok7d jSNkAY+TKg/+4sJJq1FyCrh2FqNn13GgZrCBLpzsGrX8OKuPD6SPkXIHnohblBRTI7vghuEIOD7 rraVjrJ0QbR8q47wBkWVvslEjXTph8xrwqls6vSMGIJvK7kkNudTv5U82zRU3/DIZRMUW9bjvwb d/YmrH1LJPDPYirfmFY7BJr7BqM58V498RialojuEyH03nDAdW1ttXZPe7arnEmA1B00EU1XYDq a0fYG9NkoMQbTuKGPOcquO/0adzzyJuV+u05eQRvWQIJd2FUGJ+1gPRyxFJFVH58gOAo1njHk6s N+/S2x70/Dii+CWYHasEHOAzK9uKQiESAdN4qI3oxOLsdPIoygyGKdigZ/iF60GVE7NxY558qs/ Lb0evNODw== X-Google-Smtp-Source: AGHT+IGEXbRmdyezD1N0DtVWH8H5J6D8IDMwwxSVzj11ijLzK4ds4cZyRhWZMn/1wbP5lmMarTA3+w== X-Received: by 2002:a5d:5887:0:b0:425:7591:7072 with SMTP id ffacd0b85a97d-4265ef60236mr2359873f8f.0.1759942633794; Wed, 08 Oct 2025 09:57:13 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4255d8a6b77sm30740453f8f.6.2025.10.08.09.57.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Oct 2025 09:57:13 -0700 (PDT) From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 4/4] bitbake-setup: rework the settings handling Date: Wed, 8 Oct 2025 18:56:59 +0200 Message-Id: <20251008165659.1884881-4-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20251008165659.1884881-1-alex.kanavin@gmail.com> References: <20251008165659.1884881-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 08 Oct 2025 16:57:19 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/18148 From: Alexander Kanavin This is the outcome of various discussions, suggestions and pull requests on github. What has specifically changed? 1. The sources for the settings are no longer separated, but are stacked and given priorities, from highest to lowest: a. '--setting section key value' on the command line b. a settings file in the top directory c. a global settings file in ~/.config/bitbake-setup/ (or in a file pointed to by --global-settings) Any setting can be in any of these three locations (other than top dir name and prefix which do not make sense in the settings file in the top directory). 2. A global settings file must contain all of the needed settings, while a settings file in the top directory can be empty (and this is how they are written out if they do not exist). Specifically, both dl-dir and registry settings have been relocated to the global file, and dl-dir defaults to ~/.cache/bitbake-setup/downloads, rather than somewhere in top dir. 3. The file name for both global and top dir settings is now 'settings.conf'. 4. --top-dir-prefix and --top-dir-name options have been removed and superseded by a generic, universal --setting option. 5. 'install-settings' command has been removed, as it is no longer does anything useful, and is superseded by the 'setting' command (see below). 'install-global-settings' has been retained, to be able to have a set of global defaults that can be changed without initializing a build. 6. 'change-setting', 'change-global-setting' and 'install-settings' have all been replaced by a single 'setting' command that mimics 'git config' in its parameters: a. Changing a setting: bitbake-setup setting [--global] default dl-dir /path/to/downloads b. Removing a setting: bitbake-setup setting [--global] --unset default dl-dir Signed-off-by: Alexander Kanavin --- bin/bitbake-setup | 287 +++++++++++++++++++++--------------------- lib/bb/tests/setup.py | 12 +- 2 files changed, 147 insertions(+), 152 deletions(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index 7878cd939..e9e73a927 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -32,9 +32,9 @@ logger = bb.msg.logger_create('bitbake-setup', sys.stdout) def cache_dir(top_dir): return os.path.join(top_dir, '.bitbake-setup-cache') -def init_bb_cache(settings, args): +def init_bb_cache(top_dir, settings, args): dldir = settings["default"]["dl-dir"] - bb_cachedir = os.path.join(cache_dir(args.top_dir), 'bitbake-cache') + bb_cachedir = os.path.join(cache_dir(top_dir), 'bitbake-cache') d = bb.data.init() d.setVar("DL_DIR", dldir) @@ -389,7 +389,7 @@ def choose_fragments(possibilities, parameters, non_interactive, skip_selection) choices[k] = options_enumerated[option_n][1] return choices -def obtain_config(settings, args, source_overrides, d): +def obtain_config(top_dir, settings, args, source_overrides, d): if args.config: config_id = args.config[0] config_parameters = args.config[1:] @@ -407,7 +407,7 @@ def obtain_config(settings, args, source_overrides, d): upstream_config = {'type':'network','uri':config_id,'name':get_config_name(config_id),'data':json.load(f)} else: print("Looking up config {} in configuration registry".format(config_id)) - registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + registry_path = update_registry(settings["default"]["registry"], cache_dir(top_dir), d) registry_configs = list_registry(registry_path, with_expired=True) if config_id not in registry_configs: raise Exception("Config {} not found in configuration registry, re-run 'init' without parameters to choose from available configurations.".format(config_id)) @@ -416,7 +416,7 @@ def obtain_config(settings, args, source_overrides, d): if has_expired(expiry_date): print("This configuration is no longer supported after {}. Please consider changing to a supported configuration.".format(expiry_date)) else: - registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) + registry_path = update_registry(settings["default"]["registry"], cache_dir(top_dir), d) registry_configs = list_registry(registry_path, with_expired=True) config_id = choose_config(registry_configs, args.non_interactive) config_parameters = [] @@ -429,7 +429,7 @@ def obtain_config(settings, args, source_overrides, d): upstream_config['skip-selection'] = args.skip_selection return upstream_config -def init_config(settings, args, d): +def init_config(top_dir, settings, args, d): stdout = sys.stdout def handle_task_progress(event, d): rate = event.rate if event.rate else '' @@ -437,10 +437,10 @@ def init_config(settings, args, d): print("{}% {} ".format(progress, rate), file=stdout, end='\r') source_overrides = json.load(open(args.source_overrides)) if args.source_overrides else {'sources':{}} - upstream_config = obtain_config(settings, args, source_overrides, d) + upstream_config = obtain_config(top_dir, settings, args, source_overrides, d) print("\nRun 'bitbake-setup init --non-interactive {}' to select this configuration non-interactively.\n".format(" ".join(upstream_config['non-interactive-cmdline-options']))) - builddir = os.path.join(os.path.abspath(args.top_dir), args.build_dir_name or "{}-{}".format(upstream_config['name']," ".join(upstream_config['non-interactive-cmdline-options'][1:]).replace(" ","-").replace("/","_"))) + builddir = os.path.join(os.path.abspath(top_dir), args.build_dir_name or "{}-{}".format(upstream_config['name']," ".join(upstream_config['non-interactive-cmdline-options'][1:]).replace(" ","-").replace("/","_"))) if os.path.exists(os.path.join(builddir, "layers")): print("Build already initialized in {}\nUse 'bitbake-setup status' to check if it needs to be updated or 'bitbake-setup update' to perform the update.".format(builddir)) return @@ -511,7 +511,7 @@ def are_layers_changed(layers, layerdir, d): return changed -def build_status(settings, args, d, update=False): +def build_status(top_dir, settings, args, d, update=False): builddir = args.build_dir confdir = os.path.join(builddir, "config") @@ -523,7 +523,7 @@ def build_status(settings, args, d, update=False): args.non_interactive = True args.skip_selection = current_upstream_config['skip-selection'] source_overrides = current_upstream_config["source-overrides"] - new_upstream_config = obtain_config(settings, args, source_overrides, d) + new_upstream_config = obtain_config(top_dir, settings, args, source_overrides, d) write_config(new_upstream_config, confdir) config_diff = bb.process.run('git -C {} diff'.format(confdir))[0] @@ -544,8 +544,8 @@ def build_status(settings, args, d, update=False): print("\nConfiguration in {} has not changed.".format(builddir)) -def build_update(settings, args, d): - build_status(settings, args, d, update=True) +def build_update(top_dir, settings, args, d): + build_status(top_dir, settings, args, d, update=True) def do_fetch(fetcher, dir): # git fetcher simply dumps git output to stdout; in bitbake context that is redirected to temp/log.do_fetch @@ -595,8 +595,8 @@ def list_registry(registry_path, with_expired): json_data[config_name] = {"description": config_desc} return json_data -def list_configs(settings, args, d): - registry_path = update_registry(settings["default"]["registry"], cache_dir(args.top_dir), d) +def list_configs(top_dir, settings, args, d): + registry_path = update_registry(settings["default"]["registry"], cache_dir(top_dir), d) json_data = list_registry(registry_path, args.with_expired) print("\nAvailable configurations:") for config_name, config_data in json_data.items(): @@ -614,7 +614,7 @@ def list_configs(settings, args, d): json.dump(json_data, f, sort_keys=True, indent=4) print("Available configurations written into {}".format(args.write_json)) -def install_buildtools(settings, args, d): +def install_buildtools(top_dir, settings, args, d): buildtools_install_dir = os.path.join(args.build_dir, 'buildtools') if os.path.exists(buildtools_install_dir): if not args.force: @@ -634,117 +634,113 @@ def install_buildtools(settings, args, d): subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True) def default_settings_path(top_dir): - return os.path.join(top_dir, 'bitbake-setup.conf') + return os.path.join(top_dir, 'settings.conf') -def write_settings(top_dir, force_replace, non_interactive=True): +def create_settings(top_dir, non_interactive=True): settings_path = default_settings_path(top_dir) - if not os.path.exists(settings_path) or force_replace: - - settings = configparser.ConfigParser() - settings['default'] = { - 'registry':default_registry, - 'dl-dir':os.path.join(top_dir, '.bitbake-setup-downloads'), - } - os.makedirs(os.path.dirname(settings_path), exist_ok=True) - - siteconfpath = os.path.join(top_dir, 'site.conf') - print('Configuration registry set to\n {}\n'.format(settings['default']['registry'])) - print('Bitbake-setup download cache (DL_DIR) set to\n {}\n'.format(settings['default']['dl-dir'])) - print('A new settings file will be created in\n {}\n'.format(settings_path)) - print('A common site.conf file will be created, please edit or replace before running builds\n {}\n'.format(siteconfpath)) - if not non_interactive: - y_or_n = input('Bitbake-setup will be configured with the above settings in {}, (y/N): '.format(top_dir)) - if y_or_n != 'y': - print("\nYou can run 'bitbake-setup install-settings' to edit them before setting up builds") - exit() - print() - - if os.path.exists(settings_path): - backup_conf = settings_path + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) - os.rename(settings_path, backup_conf) - print("Previous settings are in {}".format(backup_conf)) - with open(settings_path, 'w') as settingsfile: - settings.write(settingsfile) - - if os.path.exists(siteconfpath): - backup_siteconf = siteconfpath + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) - os.rename(siteconfpath, backup_siteconf) - print("Previous settings are in {}".format(backup_siteconf)) - with open(siteconfpath, 'w') as siteconffile: - siteconffile.write('# This file is intended for build host-specific bitbake settings\n') + settings = configparser.ConfigParser() + settings['default'] = { + } + os.makedirs(os.path.dirname(settings_path), exist_ok=True) + + siteconfpath = os.path.join(top_dir, 'site.conf') + print('A new empty settings file will be created in (you can add settings to it to override defaults from the global settings file)\n {}\n'.format(settings_path)) + print('A common site.conf file will be created, please edit or replace before running builds\n {}\n'.format(siteconfpath)) + if not non_interactive: + y_or_n = input('Bitbake-setup will be configured with the above settings in {}, (y/N): '.format(top_dir)) + if y_or_n != 'y': + print("\nYou can run 'bitbake-setup install-settings' to edit them before setting up builds") + exit() + print() -def load_settings(top_dir, non_interactive): - # This creates a new settings file if it does not yet exist - write_settings(top_dir, force_replace=False, non_interactive=non_interactive) + if os.path.exists(settings_path): + backup_conf = settings_path + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + os.rename(settings_path, backup_conf) + print("Previous settings are in {}".format(backup_conf)) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + + if os.path.exists(siteconfpath): + backup_siteconf = siteconfpath + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + os.rename(siteconfpath, backup_siteconf) + print("Previous settings are in {}".format(backup_siteconf)) + with open(siteconfpath, 'w') as siteconffile: + siteconffile.write('# This file is intended for build host-specific bitbake settings\n') +def load_settings(top_dir, non_interactive): settings_path = default_settings_path(top_dir) + if not os.path.exists(settings_path): + create_settings(top_dir, non_interactive=non_interactive) + settings = configparser.ConfigParser() print('Loading settings from\n {}\n'.format(settings_path)) settings.read_file(open(settings_path)) return settings def global_settings_path(args): - return os.path.abspath(args.global_settings) if args.global_settings else os.path.join(os.path.expanduser('~'), '.config', 'bitbake-setup', 'config') - -def write_global_settings(settings_path, force_replace, non_interactive=True): - if not os.path.exists(settings_path) or force_replace: - - settings = configparser.ConfigParser() - settings['default'] = { - 'top-dir-prefix':os.path.expanduser('~'), - 'top-dir-name':'bitbake-builds' - } - os.makedirs(os.path.dirname(settings_path), exist_ok=True) - print('Configuring global settings in\n {}\n'.format(settings_path)) - print('Top directory prefix (where all top level directories are created) set to\n {}\n'.format(settings['default']['top-dir-prefix'])) - print('Top directory name (this is added to the top directory prefix to form a top directory where builds are set up) set to\n {}\n'.format(settings['default']['top-dir-name'])) - if not non_interactive: - y_or_n = input('Write out the global settings as specified above (y/N)? ') - if y_or_n != 'y': - print("\nYou can run 'bitbake-setup install-global-settings' to edit them before setting up builds") - exit() - print() - - if os.path.exists(settings_path): - backup_conf = settings_path + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) - os.rename(settings_path, backup_conf) - print("Previous global settings are in {}".format(backup_conf)) - with open(settings_path, 'w') as settingsfile: - settings.write(settingsfile) + return os.path.abspath(args.global_settings) if args.global_settings else os.path.join(os.path.expanduser('~'), '.config', 'bitbake-setup', 'settings.conf') + +def create_global_settings(settings_path, non_interactive=True): + settings = configparser.ConfigParser() + settings['default'] = { + 'top-dir-prefix':os.path.expanduser('~'), + 'top-dir-name':'bitbake-builds', + 'registry':default_registry, + 'dl-dir':os.path.join(os.path.expanduser('~'), '.cache', 'bitbake-setup', 'downloads'), + } + os.makedirs(os.path.dirname(settings_path), exist_ok=True) + print('Configuring global settings in\n {}\n'.format(settings_path)) + print('Top directory prefix (where all top level directories are created) set to\n {}\n'.format(settings['default']['top-dir-prefix'])) + print('Top directory name (this is added to the top directory prefix to form a top directory where builds are set up) set to\n {}\n'.format(settings['default']['top-dir-name'])) + print('Configuration registry set to\n {}\n'.format(settings['default']['registry'])) + print('Bitbake-setup download cache (DL_DIR) set to\n {}\n'.format(settings['default']['dl-dir'])) + if not non_interactive: + y_or_n = input('Write out the global settings as specified above (y/N)? ') + if y_or_n != 'y': + print("\nYou can run 'bitbake-setup install-global-settings' to edit them before setting up builds") + exit() + print() + + if os.path.exists(settings_path): + backup_conf = settings_path + "-backup.{}".format(time.strftime("%Y%m%d%H%M%S")) + os.rename(settings_path, backup_conf) + print("Previous global settings are in {}".format(backup_conf)) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) def load_global_settings(settings_path, non_interactive): - # This creates a new settings file if it does not yet exist - write_global_settings(settings_path, force_replace=False, non_interactive=non_interactive) + if not os.path.exists(settings_path): + create_global_settings(settings_path, non_interactive=non_interactive) settings = configparser.ConfigParser() print('Loading global settings from\n {}\n'.format(settings_path)) settings.read_file(open(settings_path)) return settings -def change_settings(top_dir, new_settings): - settings = load_settings(top_dir, non_interactive=True) - for section, section_settings in new_settings.items(): - for setting, value in section_settings.items(): - settings[section][setting] = value - print("Setting '{}' in section '{}' is changed to '{}'".format(setting, section, value)) +def change_setting(settings_path, settings, args): + if args.section and args.key and args.value: + settings[args.section][args.key] = args.value + print("Setting '{}' in section '{}' is changed to '{}'".format(args.key, args.section, args.value)) + if args.unset: + section = args.unset[0] + setting = args.unset[1] + if section in settings.keys() and setting in settings[section].keys(): + del settings[section][setting] + print("Setting '{} in section '{}' is removed".format(setting, section)) - settings_path = default_settings_path(top_dir) with open(settings_path, 'w') as settingsfile: settings.write(settingsfile) print("New settings written to {}".format(settings_path)) - return settings -def change_global_settings(settings_path, new_settings): - settings = load_global_settings(settings_path, non_interactive=True) - for section, section_settings in new_settings.items(): - for setting, value in section_settings.items(): - settings[section][setting] = value - print("Setting '{}' in section '{}' is changed to '{}'".format(setting, section, value)) +def setting_global(args): + settings = load_global_settings(global_settings_path(args), args.non_interactive) + settings_path = global_settings_path(args) + change_setting(settings_path, settings, args) - with open(settings_path, 'w') as settingsfile: - settings.write(settingsfile) - print("New global settings written to {}".format(settings_path)) - return settings +def setting(top_dir, args): + settings = load_settings(top_dir, args.non_interactive) + settings_path = default_settings_path(top_dir) + change_setting(settings_path, settings, args) def get_build_dir_via_bbpath(): bbpath = os.environ.get('BBPATH') @@ -755,7 +751,7 @@ def get_build_dir_via_bbpath(): return build_dir return None -def get_top_dir(args, global_settings): +def get_top_dir(args, settings): build_dir_via_bbpath = get_build_dir_via_bbpath() if build_dir_via_bbpath: top_dir = os.path.dirname(build_dir_via_bbpath) @@ -763,20 +759,25 @@ def get_top_dir(args, global_settings): return top_dir if hasattr(args, 'build_dir'): - # commands without --top-dir-prefix/name arguments (status, update) still need to know where - # the top dir is, but it should be auto-deduced as parent of args.build_dir top_dir = os.path.dirname(os.path.normpath(args.build_dir)) return top_dir - top_dir_prefix = args.top_dir_prefix if args.top_dir_prefix else global_settings['default']['top-dir-prefix'] - top_dir_name = args.top_dir_name if args.top_dir_name else global_settings['default']['top-dir-name'] + top_dir_prefix = settings['default']['top-dir-prefix'] + top_dir_name = settings['default']['top-dir-name'] return os.path.join(top_dir_prefix, top_dir_name) -def main(): - def add_top_dir_arg(parser): - parser.add_argument('--top-dir-prefix', help='Top level directory prefix. This is where all top level directories are created.') - parser.add_argument('--top-dir-name', help='Top level directory name. Together with the top directory prefix this forms a top directory where builds are set up and downloaded configurations and layers are cached for reproducibility and offline builds.') +def merge_settings(global_settings, local_settings, cmdline_settings): + all_settings = global_settings + for section, section_settings in local_settings.items(): + for setting, value in section_settings.items(): + all_settings[section][setting] = value + + for (section, setting, value) in cmdline_settings: + all_settings[section][setting] = value + + return all_settings +def main(): def add_build_dir_arg(parser): build_dir = get_build_dir_via_bbpath() if build_dir: @@ -792,18 +793,17 @@ def main(): parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true') parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR') parser.add_argument('--no-network', action='store_true', help='Do not check whether configuration repositories and layer repositories have been updated; use only the local cache.') - parser.add_argument('--global-settings', action='store', help='Path to the global settings file where defaults for top directory prefix and name can be specified') + parser.add_argument('--global-settings', action='store', help='Path to the global settings file.') + parser.add_argument('--setting', default=[], action='append', nargs=3, help='Modify a setting (for this bitbake-setup invocation only), for example "--setting default top-dir-prefix /path/to/top/dir".') subparsers = parser.add_subparsers() parser_list = subparsers.add_parser('list', help='List available configurations') - add_top_dir_arg(parser_list) parser_list.add_argument('--with-expired', action='store_true', help='List also configurations that are no longer supported due to reaching their end-of-life dates.') parser_list.add_argument('--write-json', action='store', help='Write available configurations into a json file so they can be programmatically processed.') parser_list.set_defaults(func=list_configs) parser_init = subparsers.add_parser('init', help='Select a configuration and initialize a build from it') - add_top_dir_arg(parser_init) parser_init.add_argument('config', nargs='*', help="path/URL/id to a configuration file (use 'list' command to get available ids), followed by configuration options. Bitbake-setup will ask to choose from available choices if command line doesn't completely specify them.") parser_init.add_argument('--non-interactive', action='store_true', help='Do not ask to interactively choose from available options; if bitbake-setup cannot make a decision it will stop with a failure.') parser_init.add_argument('--source-overrides', action='store', help='Override sources information (repositories/revisions) with values from a local json file.') @@ -824,25 +824,16 @@ def main(): parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.') parser_install_buildtools.set_defaults(func=install_buildtools) - parser_install_settings = subparsers.add_parser('install-settings', help='Write a settings file with default values into the top level directory (contains the location of build configuration registry, downloads directory and other settings specific to a top directory)') - add_top_dir_arg(parser_install_settings) - parser_install_settings.set_defaults(func=write_settings) - - parser_install_global_settings = subparsers.add_parser('install-global-settings', help='Write a global settings file with default values (contains the default prefix and name of the top directory)') - parser_install_global_settings.set_defaults(func=write_global_settings) - - parser_change_setting = subparsers.add_parser('change-setting', help='Change a setting in the settings file') - add_top_dir_arg(parser_change_setting) - parser_change_setting.add_argument('section', help="Section in a settings file, typically 'default'") - parser_change_setting.add_argument('key', help="Name of the setting") - parser_change_setting.add_argument('value', help="Value of the setting") - parser_change_setting.set_defaults(func=change_settings) + parser_install_global_settings = subparsers.add_parser('install-global-settings', help='Write a global settings file with default values') + parser_install_global_settings.set_defaults(func=create_global_settings) - parser_change_global_setting = subparsers.add_parser('change-global-setting', help='Change a setting in the global settings file') - parser_change_global_setting.add_argument('section', help="Section in a global settings file, typically 'default'") - parser_change_global_setting.add_argument('key', help="Name of the setting") - parser_change_global_setting.add_argument('value', help="Value of the setting") - parser_change_global_setting.set_defaults(func=change_global_settings) + parser_setting = subparsers.add_parser('setting', help='Set or unset a setting in a setting file (e.g. the default prefix and name of the top directory, the location of build configuration registry, downloads directory and other settings specific to a top directory)') + parser_setting.add_argument('section', nargs='?', help="Section in a settings file, typically 'default'") + parser_setting.add_argument('key', nargs='?', help="Name of the setting") + parser_setting.add_argument('value', nargs='?', help="Value of the setting") + parser_setting.add_argument('--global', action='store_true', help="Modify the setting in a global settings file, rather than one specific to a top directory") + parser_setting.add_argument('--unset', nargs=2, help="Unset a setting, e.g. 'bitbake-setup setting --unset default registry' would revert to the registry setting in a global settings file") + parser_setting.set_defaults(func=setting) args = parser.parse_args() @@ -859,11 +850,8 @@ def main(): level=logger.getEffectiveLevel()) if 'func' in args: - if args.func == write_global_settings: - write_global_settings(global_settings_path(args), force_replace=True) - return - elif args.func == change_global_settings: - change_global_settings(global_settings_path(args), {args.section:{args.key:args.value}}) + if args.func == create_global_settings: + create_global_settings(global_settings_path(args)) return if hasattr(args, 'build_dir'): @@ -874,19 +862,24 @@ def main(): if not hasattr(args, 'non_interactive'): args.non_interactive = True + if args.func == setting and vars(args)['global']: + setting_global(args) + return + global_settings = load_global_settings(global_settings_path(args), args.non_interactive) - args.top_dir = get_top_dir(args, global_settings) + top_dir = get_top_dir(args, merge_settings(global_settings, {}, args.setting)) - print('Bitbake-setup is using {} as top directory (can be changed with --top-dir-prefix/name arguments or by setting them in {}).\n'.format(args.top_dir, global_settings_path(args))) - if args.func == write_settings: - write_settings(args.top_dir, force_replace=True) - elif args.func == change_settings: - change_settings(args.top_dir, {args.section:{args.key:args.value}}) - else: - settings = load_settings(args.top_dir, args.non_interactive) - d = init_bb_cache(settings, args) - args.func(settings, args, d) - save_bb_cache() + if args.func == setting: + setting(top_dir, args) + return + + print('Bitbake-setup is using {} as top directory (can be changed by setting top dir prefix and name in {}).\n'.format(top_dir, global_settings_path(args))) + + settings = load_settings(top_dir, args.non_interactive) + settings = merge_settings(global_settings, settings, args.setting) + d = init_bb_cache(top_dir, settings, args) + args.func(top_dir, settings, args, d) + save_bb_cache() else: from argparse import Namespace parser.print_help() diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index 495d1da20..22edda40e 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -235,16 +235,18 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) out = self.runbbsetup("install-global-settings") settings_path = "{}/global-config".format(self.tempdir) self.assertIn(settings_path, out[0]) - out = self.runbbsetup("change-global-setting default top-dir-prefix {}".format(self.tempdir)) + out = self.runbbsetup("setting --global default top-dir-prefix {}".format(self.tempdir)) self.assertIn("Setting 'top-dir-prefix' in section 'default' is changed to", out[0]) - self.assertIn("New global settings written to".format(settings_path), out[0]) + self.assertIn("New settings written to".format(settings_path), out[0]) + out = self.runbbsetup("setting --global default dl-dir {}".format(os.path.join(self.tempdir, 'downloads'))) + self.assertIn("Setting 'dl-dir' in section 'default' is changed to", out[0]) + self.assertIn("New settings written to".format(settings_path), out[0]) # check that writing settings works and then adjust them to point to # test registry repo - out = self.runbbsetup("install-settings") - settings_path = "{}/bitbake-builds/bitbake-setup.conf".format(self.tempdir) + out = self.runbbsetup("setting default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)) + settings_path = "{}/bitbake-builds/settings.conf".format(self.tempdir) self.assertIn(settings_path, out[0]) - out = self.runbbsetup("change-setting default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)) self.assertIn("Setting 'registry' in section 'default' is changed to", out[0]) self.assertIn("New settings written to".format(settings_path), out[0])