From patchwork Mon Jul 6 22:05:33 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91874 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B73B0C43458 for ; Mon, 6 Jul 2026 22:06:44 +0000 (UTC) Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.161473.1783375597337593838 for ; Mon, 06 Jul 2026 15:06:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=TrtGqzM4; spf=pass (domain: gmail.com, ip: 209.85.210.170, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-845ea8924a2so3469661b3a.1 for ; Mon, 06 Jul 2026 15:06:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375597; x=1783980397; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=RHc2v7P6S3eXsGYj8ncD03YtPXXdbXRMlxd+lLp0sgM=; b=TrtGqzM4aQvaH/9aThDZpF1BdT18KKcUO95Erx4e5SLmIlIEfuliKYjRxhR0SurhQS 3pMGmRpmrX6HW44Ei0Izu8gPwGZzbCFaOZNbwxcMuaoCmbK8sBZKX/Y0N6VReb/d2sRk XWU5cKHdB7sO/f+mL1CElh/fF8kX0RSAVoHqQCmYOJKuSoozl5qegOZhiZ4Ha4QS4cN9 FD8ZDLGbozGYLgZLRSKeSBTAwS4t9EW7B7kTH6ateozZE+A4vVS54g9XfiomxKQDuGyR G1n/+ByNmVbnZ1nQtKfbvtrf+rFgw+2VQ47bP3wyfIPMFQ8fqdG3lKUX2S4od3W9p7L+ V6lQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375597; x=1783980397; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=RHc2v7P6S3eXsGYj8ncD03YtPXXdbXRMlxd+lLp0sgM=; b=XIb64EIbTFq4+LRX2x2LMLfGqNf5GSqGaeh1g0VXSjbhzfZQ06RQuSlmmiNkJDFXH/ 4k23qh613yc97s0zuBaiw65ga8aSnCrBRpMssN/R2rnkDyDJt5tPkDFrCv+DEZTlPy2f eojNC52q5CyMokN4HP6gMaa5pYOrzsp19O8wbW+PsAfdHQcIoXlNEmgaIG459E/gWuEI pONbpSqUJfQiLxZowwtFv+FY1kTqc5iHqn1gnI0nOx+euseyrC/n9dpQxI+pE2o+V0qw 9EgVqGMWvQvjT1wTKsZRHF4NtVOB9ms5MEmn1Cw1USkEVC/boy7pGHXL0qAS8OGvxNPg boEQ== X-Gm-Message-State: AOJu0Yx9wm34smOe+8kl9mQuXI39V1P1/RWv1/yvM41gkGTsNNM0bZzf ByyLB3vRybuxatj12eN7XSA9ImHAn8bk8OF1te073hDy0YrSi3/8AXhsZbpJjA== X-Gm-Gg: AfdE7cnM0rhtBCXeHuXN0TkhDOFsx3vQR84Y7IuCm3dVoQTHYNmdWBr0hMUmx4H6rn8 PBkBVCk+ipN6rL2GJJcSezfsNQUp3ocCGiZmZH567TiVrsK25tvjD3ULmcZjRx/dU+my3OIIOt8 Pt22f+XLO5olInd8UGqnKAufkBX4kC0RfamI7/ZtAzU2+SOOeRQy5enWnHDiPqLbGR8TOlAE4H6 tRHSLwumGICSbOEaL5D/cwtQ27W/1zRaG5KswbNZmvV5q+3Bnmrkfsgn8/z8gZ+iU4HtP1HcaBZ FpE+UoswZtEcH0NbQRvbloRFXiypoU6vzfSQhGPSjf7NtEKcpFqA3m5j9cP0Xb5Iz+bexIrcunf YvHk7UZzVkZjOVpjwMXQwIUuNULE3WFW5LMYgQ/Uy30ooDxhkkHcUhaS0LyYndJFuwM/vzQsI7j 77fCItpsg7CiTyyx2/ClCbxY2BxifNwZMzJX5yL43pIJXUUKuN8CYWi7MZmOCLiA/HfuRPv86Nd FOMG9rYBA+BBWjzzffk1pE= X-Received: by 2002:a05:6a00:2e24:b0:847:8625:f7bb with SMTP id d2e1a72fcca58-84826e2393cmr2304731b3a.60.1783375596587; Mon, 06 Jul 2026 15:06:36 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.06.32 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:06:32 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 01/11] scripts: add vcontainer-tarball setup, integration, and publishing Date: Mon, 6 Jul 2026 15:05:33 -0700 Message-ID: <496be2c920bf0bffb4c0881f0818ff858175ed2d.1783371914.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:06:44 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4431 Introduce the vcontainer-tarball SDK plumbing used by container build jobs. The vcontainer-tarball is a meta-virtualization-derived SDK (modelled after buildtools-tarball) that ships the container build toolchain so worker jobs do not need to rebuild it for every step. * scripts/utils.py: add setup_vcontainer_tarball(), and add an env_glob keyword argument to setup_tools_tarball() and enable_tools_tarball() so the vcontainer-tarball can source its specific environment-setup-ci file rather than the universal glob. * scripts/run-config: source the vcontainer-tarball environment for build-targets when "vcontainer" parameter is passed in, similar to "extratools" behavior. * scripts/publish-artefacts: publish the vcontainer-tarball artefact so downstream test jobs can fetch a stable SDK. AI-Generated: Claude Cowork Opus 4.8 Signed-off-by: Tim Orling --- scripts/publish-artefacts | 5 +++++ scripts/run-config | 11 +++++++++++ scripts/utils.py | 20 ++++++++++++++++---- 3 files changed, 32 insertions(+), 4 deletions(-) diff --git a/scripts/publish-artefacts b/scripts/publish-artefacts index e56e131..0e820e9 100755 --- a/scripts/publish-artefacts +++ b/scripts/publish-artefacts @@ -146,5 +146,10 @@ case "$target" in sha256sums $TMPDIR/deploy/images/qemux86-64 cp -R --no-dereference --preserve=links $TMPDIR/deploy/images/qemux86-64/*qemux86* $DEST/patchtest ;; + "vcontainer-tarball") + mkdir -p $DEST/vcontainer-tarball + sha256sums $TMPDIR/deploy/sdk + cp -R --no-dereference --preserve=links $TMPDIR/deploy/sdk/*vcontainer* $DEST/vcontainer-tarball + ;; esac diff --git a/scripts/run-config b/scripts/run-config index e896234..90a5996 100755 --- a/scripts/run-config +++ b/scripts/run-config @@ -153,6 +153,17 @@ else: if args.phase == "init" and args.stepname == "buildtools": sys.exit(0) +vcontainer = utils.getconfigvar("vcontainer", ourconfig, args.target) +if jcfg: + if vcontainer: + addentry("vcontainer", "Setup vcontainer tarball", "init") +elif vcontainer: + # vcontainer is opt-in per target via the "vcontainer" config variable, + # so this is a no-op for targets which don't set it + utils.setup_vcontainer_tarball(ourconfig, args.target, args.builddir + "/../vcontainer-tarball") + if args.phase == "init" and args.stepname == "vcontainer": + sys.exit(0) + extratools = utils.getconfigvar("extratools", ourconfig, args.target) if jcfg: if extratools: diff --git a/scripts/utils.py b/scripts/utils.py index a4dd12e..bddc715 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -456,8 +456,8 @@ def sha256_file(filename): pass return method.hexdigest() -def enable_tools_tarball(btdir, name): - btenv = glob.glob(btdir + "/environment-setup*") +def enable_tools_tarball(btdir, name, env_glob="/environment-setup*"): + btenv = glob.glob(btdir + env_glob) print("Using %s %s" % (name, btenv)) # We either parse or wrap all our execution calls, rock and a hard place :( with open(btenv[0], "r") as f: @@ -474,6 +474,18 @@ def enable_tools_tarball(btdir, name): if line in os.environ: del os.environ[line] +# Unlike buildtools (a host/worker property, keyed by worker name globs), +# the vcontainer-tarball is only needed by specific jobs (e.g. +# containers-library), so it is keyed off the target/builder via a +# per-target "vcontainer" config variable, following the extratools pattern. +def setup_vcontainer_tarball(ourconfig, target, vcdir, checkonly=False): + vctarball = getconfigvar("vcontainer", ourconfig, target) or None + + if checkonly: + return vctarball + + setup_tools_tarball(ourconfig, vcdir, vctarball, name="vcontainer-tarball", env_glob="/environment-setup-ci") + def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): bttarball = None if "buildtools" in ourconfig and workername: @@ -488,7 +500,7 @@ def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): setup_tools_tarball(ourconfig, btdir, bttarball) -def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): +def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools", env_glob="/environment-setup*"): btenv = None if bttarball: @@ -557,7 +569,7 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): if not os.path.exists(btdir): print("Extracting %s %s" % (name, bttarball)) subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) - enable_tools_tarball(btdir, name) + enable_tools_tarball(btdir, name, env_glob) def get_string_from_version(version, milestone=None, rc=None): """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists, From patchwork Mon Jul 6 22:05:34 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4F7FC44500 for ; Mon, 6 Jul 2026 22:06:44 +0000 (UTC) Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.162430.1783375600958320813 for ; Mon, 06 Jul 2026 15:06:41 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=ERB5kb0c; spf=pass (domain: gmail.com, ip: 209.85.210.178, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-8478fe07f65so3584019b3a.0 for ; Mon, 06 Jul 2026 15:06:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375600; x=1783980400; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=9+gFaOja+ZxdtARCjaXQkdq5mL9J6nnmick6B/WZ7hE=; b=ERB5kb0c99a8A3iwPE5gEop1gMzRiCHA0wP7+qN648OG66SD07cIJ45G7Tg/G4KeBi DGTR/V2s4XBhVKGPsLTZuJayzwP1Z+ibAspA4XY8DbSzWlpN1w2fr4A5jO7p1AHQfnzs a9PgMJweHFU3Mf9DWPoDwHB09Y+aOjBx4+90cvdJORNhU4vgskhZqkqwa5Fe/P+Idzwc bzbE85ys7gfMZ9xKlw7/v97RhLZIBIHrfWjZMcI8NdP09MSXIKel3Mc6ucaQHtAyWrip CRpb8sMRBkS0dnPE43FwW5cSvXgNIUB9s5T7gG5u0K2fEMdN0OqLFLWAaOK1oe0VkxO1 UGLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375600; x=1783980400; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=9+gFaOja+ZxdtARCjaXQkdq5mL9J6nnmick6B/WZ7hE=; b=Zmnx75YU91ZFOj+xOd1LNa7ZpHmIvlp8gfXHNAPYDTdUj1/4NapU3foymxzeGoxgQN jt9TAr8oDvC5kpPw8xcjep5o2amOlQVSI/gMqXBD8an0x+XxqFz6cxBsz3p6XvLgV+v1 6HAvCdECncK9eW2kUX5RCIyYZRtOq8AFCk2lOuqO5YuT+BZsOo00gYnpSSsL6aV0YqTW dyN4RAwX2G4xvI2p2ePMbQgNFmZUosdayMFkeahKFhz4ur7f/UILIQrwd16vw6aBSTbY zeLLhJWTCbVNsPFC4mCld/HgtliHeQ1YuU6/vD+rRUKUmHXZoJ+/wpj/jNIFZR7/2Nqh FM4Q== X-Gm-Message-State: AOJu0YxehNvWkWPb6YGDFYaR9HsFzIqK7sQR4KdtQr2326lbGCk++HVr 6T6tUXoUrWdfOXxRkHCD9ZMV6MhUnp6HHklx82E3DvZkdfpR107HhM0P6vtVDQ== X-Gm-Gg: AfdE7cm6HEIItDkCAvkfWp70yA1LAv+a3sgyAUZC+ivg30Th5hIvB9/1V7EXiGV+slk IsU/YDDIRCjp5qOT6kwMhmUkCxzcFyOhW7GiWZDxh0B2eErh0EifOHJx2+RLa0nzmiSbAP03f34 VmytpL0QjARKhTRty83BYMnnJM0d+TG1xY5B/quebvbos8xhqkqG+6RmicL8U67hs8nn+UR+pr/ sCkDjcexL/mADilIZYKOk6t6jlt6NmajZGgBIAKrstiQXI2hy793RNFlo4WAmJCQf+ubnUoP+hD h4kCldd363HGIBJesp7Rrig0OqVn8qKp+FvEk8BiJaEpdVDdlRMHgilgvNdPblHkd1t9ob2uamt vACI4bdTpoyMfHwjEl/37fg27LIshnO0QQT1Rl96ijVPCO7jcwhey1gvdlAxJ9F80kO7ZS64W48 QGS+0LD0EbxquvQAheoSkn7xv80xjp1wMJkQXtLyAMQlUrAakSdtZDfOu/+NbZKUgQocxdcxP7T BCCHdSNbOt3 X-Received: by 2002:a05:6a00:995:b0:847:949e:ea6f with SMTP id d2e1a72fcca58-84826bfc073mr2420336b3a.4.1783375600212; Mon, 06 Jul 2026 15:06:40 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.06.37 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:06:38 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 02/11] config.json: add vcontainer-tarball build target Date: Mon, 6 Jul 2026 15:05:34 -0700 Message-ID: <21daf90c9799e7049ebb34f24925e26d311852af.1783371914.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:06:44 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4432 Add the vcontainer-tarball build target which produces the meta-virtualization SDK used by downstream container build/test jobs. Modelled after the buildtools-tarball target. Following meta-virtualization/docs/build-profiles.md add "require conf/distro/include/meta-virt-host.conf" to extravars, which appends required DISTRO_FEATURES and sets appropriate BBMULTICONFIG. The target places the extravars in the per-step config, and includes a publish-artefacts step so the resulting SDK tarball is staged for reuse by container-tests and other consumers. Since meta-virtualization is the first layer removed by 'remove-layers', our BBMULTICONFIGs become invalid immediately after layer removal and throw an error. In scripts/run-config, EXTRACMDS runs after BBTARGETS, but before 'remove-layers'. Use 'sed' to remove the "require ...meta-virt-host.conf" line. The error in the 'remove-layers' step can be summarized as: bb.parse.ParseError: ParseError at /home/pokybuild/yocto-worker/vcontainer-tarball/build/layers/openembedded-core/meta/conf/bitbake.conf:824: Could not include required file conf/multiconfig/vruntime-aarch64.conf Signed-off-by: Tim Orling --- config.json | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/config.json b/config.json index f4ead18..972682d 100644 --- a/config.json +++ b/config.json @@ -1830,6 +1830,31 @@ }, "toaster" : { "EXTRACMDS" : ["${SCRIPTSDIR}/run-toaster-tests ${HELPERBUILDDIR} ${HELPERBUILDDIR}/../layers/bitbake"] + }, + "vcontainer-tarball": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Build vcontainer-tarballs", + "BBTARGETS" : "vcontainer-tarball", + "extravars" : [ + "require conf/distro/include/meta-virt-host.conf", + "INIT_MANAGER = 'systemd'" + ], + "EXTRACMDS" : ["sed -i '/meta-virt-host.conf/d' ${HELPERBUILDDIR}/conf/auto.conf"] + }, + "step2" : { + "shortname" : "Publish vcontainer SDK for test reuse", + "EXTRACMDS" : [ + "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" + ] + } } }, "repo-defaults" : { From patchwork Mon Jul 6 22:05:35 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA960C43458 for ; Mon, 6 Jul 2026 22:06:54 +0000 (UTC) Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.161477.1783375605134296675 for ; Mon, 06 Jul 2026 15:06:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=gWaogGpx; spf=pass (domain: gmail.com, ip: 209.85.215.171, mailfrom: ticotimo@gmail.com) Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-c9eefcf9175so3081756a12.3 for ; Mon, 06 Jul 2026 15:06:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375604; x=1783980404; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=h1RUgvlqNXA8m8aSMS6aYHUavNnebZzesXCZwYhtOSE=; b=gWaogGpxCtGcDhoArvfjDAbHRCXf5m/ZV66afH6V8+3rW3zRZqqaTcNJ3zHUrEIMxL tCKumH5AkA2DnddD5QYv6h2rqKt7yausNo60DoW8ENmFFjQ5VRADDJm7LCDyvu+YoqJ+ aQC4mUnYR8QMOjPGNpFoAD8QQYIFXe8RTbUTel6vRx/WTWEFdh7vuaegCRZV5qTQDxg1 6AK2kVFrPEiNY3a94cYgLfd8LVQzwQL4Ekk7TQkpQBfc2qGaWkmeXk6KdZpJiRqn4tkE mcDUXbpHIhfy+uCQ/ceJ0nRBhy+hqGevIMgaCyafRw/5RZAvPffAHYH5praifw/yGksK M/Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375604; x=1783980404; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=h1RUgvlqNXA8m8aSMS6aYHUavNnebZzesXCZwYhtOSE=; b=RN4bSHvIqgswHGyaSldo4sidKvPTOU2HIwiBHhCmoNn3yxQabBXvAchwZO4UTa8jnt JEIjQBJmbAwBasTB5jfDC+c8UmQKR4WG2EmwyS/NMlN4rAfXrrmTKNcDrZ4/smRwapio U7bvlM3OSVNImW+CGxG8lDxrmDIEsS0KngerZVhBMp76w+cKkmCEyZYObwyVMGghtb4L nghu/UO+aQaQc9agZcExfpHC70vd8Uu32PFflItwC1DPtBZEAkmFkIt8C4JMx8gpZXEQ LcA5plQk1SpkIWsgGpL13BzG/w0HI7FOfXDkYxHKoBaLiUosp0O7+WVYw62VZy/u7ANo YWHQ== X-Gm-Message-State: AOJu0YxgWj9k9BW3Uek9rcZFVzKKdhnVj5gQ1Jqse7HGxq7bKjfEv0J9 15MkcUR0be7+GjL+bwV5i2xwfRAgb5DDiBLQHNIyyiqMqor/N+iQywHC4Lzbag== X-Gm-Gg: AfdE7clFtkFmXyA6LfqwAaAKuS8pJ0cAj0Ixp2oJPy61ktZAWwn7EbnZEhb7UXpQi5E mXEsDCxnWt7wg/kPntj1LFhTFFbEcVIucMckl0lXQh1DvCKz95E+qucwqk5pMYJZ9FIfKwjeEBM NUPjeLNsB/U1GqYaE0coMXlpsUD4gAaZPm/ZRWDefqfP4oQOiaR0PIiAWeP0HojI2lWC3P8Rqdu 3S1vjcUP+cbv7yL1qogeSczfV/iSf1C3zwYEmxJ1VFKaBcwfkoBrzkWElEA+R2o4dpsDmt0Z/R7 I/b54+up/VHtZgoaDeEkpFnroJcVFGXE7wOEepEPL1AoXdsIjS8vcjrLzSHWOFiJzATPDyJa9HY q5anTt3bdvASvtQ4HgjeXkG6SA4GcK869ozFl52Kb0TQRCIuBlrXJ70cAx5aR8w6c45wQZ9Reay 46qZYICWsJ5xjualowd5apDCuwlXV5Hogzqc2KLR4DXcAkApqHq6+SbPtVneFp31jivOpO3JZ3q /ulbP6VVsuto/2xjGukj1I= X-Received: by 2002:a05:6a20:914d:b0:3bf:6c08:284d with SMTP id adf61e73a8af0-3c08ef83fabmr3053107637.60.1783375604248; Mon, 06 Jul 2026 15:06:44 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.06.41 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:06:42 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 03/11] scripts: add run-vcontainer-tests for meta-virtualization Date: Mon, 6 Jul 2026 15:05:35 -0700 Message-ID: <33fd6cf552070e50949e104771ed72746ff5921b.1783371914.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:06:54 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4433 Add scripts/run-vcontainer-tests, the test runner used by the vcontainers-test job. It sources the vcontainer-tarball SDK, discovers the meta-virtualization pytest suite, and runs a configurable set of suites (vdkr, vpdmn, memres) against the checked-out layers. Suites can be selected per-step so the top-level 'vcontainer-tests' job runs the container engine agnostic tests: - tests/test_container_cross_install.py - tests/test_container_registry_script.py - tests/test_vcontainer_auth_config.py - tests/test_multiarch_oci.py - tests/test_multilayer_oci.py The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective suites (including memres for each container engine): - tests/test_vdkr.py - tests/test_vdkr_registry.py and - tests/test_vpdmn.py The purpose of this script is to test the just built vcontainer-tarball, so that it can be considered "known-good" and be published for use by container building jobs that need the vcontainer-tarball installed. AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling --- config.json | 31 +++++ scripts/run-vcontainer-tests | 212 +++++++++++++++++++++++++++++++++++ 2 files changed, 243 insertions(+) create mode 100755 scripts/run-vcontainer-tests diff --git a/config.json b/config.json index 972682d..ae13c5b 100644 --- a/config.json +++ b/config.json @@ -1855,6 +1855,37 @@ "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" ] } + }, + "vcontainer-tests": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Run vcontainer pytest suite", + "NOBUILDTOOLS" : 1, + "EXTRACMDS" : [ + "${SCRIPTSDIR}/run-vcontainer-tests -s vcontainer -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}" + ] + }, + "step2" : { + "shortname" : "Run vdkr pytest suite", + "NOBUILDTOOLS" : 1, + "EXTRACMDS" : [ + "${SCRIPTSDIR}/run-vcontainer-tests -s vdkr -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}" + ] + }, + "step3" : { + "shortname" : "Run vpdmn pytest suite", + "NOBUILDTOOLS" : 1, + "EXTRACMDS" : [ + "${SCRIPTSDIR}/run-vcontainer-tests -s vpdmn -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}" + ] + } } }, "repo-defaults" : { diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests new file mode 100755 index 0000000..a7dbab3 --- /dev/null +++ b/scripts/run-vcontainer-tests @@ -0,0 +1,212 @@ +#!/bin/bash +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Run meta-virtualization pytest test suites against the vcontainer +# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake +# step. +# +# Usage: +# run-vcontainer-tests -s -b -m [options] +# +# Required: +# -s suite name: one of "vcontainer", "vdkr", "vpdmn" +# -b bitbake build directory (${BUILDDIR}) +# -m path to the meta-virtualization layer +# +# Optional: +# -S path to the vcontainer standalone SDK installer. The +# autobuilder -tests jobs share the SDK produced by the +# separate vcontainer-tarball builder. When unset, falls +# back to looking in /tmp/deploy/sdk. +# -e where to extract the standalone SDK tarball +# (default: /vcontainer-test-extracted) +# -i path to an OCI image directory (enables vdkr/vpdmn +# import tests) +# -a target architecture for vdkr/vpdmn tests (default: x86_64) +# -r directory to copy pytest artefacts (junit xml / log) to +# -k pytest marker filter (default excludes long running / +# infrastructure dependent tests) +# -h show this help and exit +# +# The script is intentionally conservative: any pytest tests that cannot run +# in the CI environment (those marked "slow", "network", "boot") are skipped +# so that the autobuilder step completes without needing network access. Those +# can be re-enabled via -k before invocation. +# +# It is assumed that /dev/kvm is writable by the CI user running the tests, +# since the performance is significantly faster with 'memres'. +# + +set -e +set -u +set -o pipefail +set -x + +usage() { + cat >&2 <<'EOF' +Usage: run-vcontainer-tests -s -b -m [options] + +Required: + -s vcontainer | vdkr | vpdmn + -b bitbake build directory + -m path to the meta-virtualization layer + +Optional: + -S path to the vcontainer standalone SDK installer + -e where to extract the standalone SDK tarball + -i path to an OCI image directory + -a target architecture (default: x86_64) + -r directory to copy pytest artefacts to + -k pytest marker filter + -h show this help and exit +EOF +} + +suite="" +builddir="" +metavirtdir="" +sdk_tarball="" +extract_dir="" +oci_image="" +arch="" +results_dir="" +marker_filter="not slow and not network and not boot and not incus and not k3s" + +while getopts ":s:b:m:S:e:i:a:r:k:h" opt; do + case "$opt" in + s) suite="$OPTARG" ;; + b) builddir="$OPTARG" ;; + m) metavirtdir="$OPTARG" ;; + S) sdk_tarball="$OPTARG" ;; + e) extract_dir="$OPTARG" ;; + i) oci_image="$OPTARG" ;; + a) arch="$OPTARG" ;; + r) results_dir="$OPTARG" ;; + k) marker_filter="$OPTARG" ;; + h) usage; exit 0 ;; + :) echo "ERROR: option -$OPTARG requires an argument" >&2; usage; exit 2 ;; + \?) echo "ERROR: unknown option -$OPTARG" >&2; usage; exit 2 ;; + esac +done + +if [ -z "$suite" ] || [ -z "$builddir" ] || [ -z "$metavirtdir" ]; then + echo "ERROR: -s, -b and -m are required" >&2 + usage + exit 2 +fi + +builddir=$(realpath "$builddir") +metavirtdir=$(realpath "$metavirtdir") + +if [ ! -d "$metavirtdir/tests" ]; then + echo "ERROR: meta-virtualization tests directory not found at $metavirtdir/tests" >&2 + exit 1 +fi + +# Locate the vcontainer standalone SDK tarball. Prefer an explicitly-provided +# SDK (-S), and fall back to looking in the local build's +# deploy/sdk directory when running stand-alone. +if [ -n "$sdk_tarball" ]; then + if [ ! -f "$sdk_tarball" ]; then + echo "ERROR: SDK installer '$sdk_tarball' is set but not a file" >&2 + exit 1 + fi +else + sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh" + if [ ! -f "$sdk_tarball" ]; then + # Try to find any matching tarball in case naming changed (e.g. versioned) + alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh 2>/dev/null | head -n1 || true) + if [ -n "$alt" ]; then + sdk_tarball="$alt" + else + echo "ERROR: vcontainer standalone SDK not found." >&2 + echo " Pass -S with an existing SDK installer, or" >&2 + echo " build vcontainer-tarball so $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2 + exit 1 + fi + fi +fi + +extract_dir="${extract_dir:-$builddir/vcontainer-test-extracted}" +rm -rf "$extract_dir" +mkdir -p "$(dirname "$extract_dir")" + +# Self-extracting installer (silent, -y agrees to license, -d picks dir) +"$sdk_tarball" -d "$extract_dir" -y + +# Prepare a Python venv so we don't pollute the worker's system packages. +python3 -m venv "$builddir/meta-virt-test-venv" +# shellcheck disable=SC1091 +source "$builddir/meta-virt-test-venv/bin/activate" +# Avoid warnings by upgrading pip; install pytest/pexpect into the venv via pip. +export PIP_DISABLE_PIP_VERSION_CHECK=1 +python3 -m pip install --quiet -r "$metavirtdir/tests/requirements.txt" + +# Per-suite test file selection. Uses -k/-m for fine-grained filtering and +# keeps the CLI small for logging clarity. +case "$suite" in + vdkr) + test_files=( + "tests/test_vdkr.py" + "tests/test_vdkr_registry.py" + ) + ;; + vpdmn) + test_files=( + "tests/test_vpdmn.py" + ) + ;; + vcontainer) + # Broad vcontainer/bbclass/tooling coverage that doesn't require the + # vdkr/vpdmn CLI harness to be running. + test_files=( + "tests/test_container_cross_install.py" + "tests/test_container_registry_script.py" + "tests/test_vcontainer_auth_config.py" + "tests/test_multiarch_oci.py" + "tests/test_multilayer_oci.py" + ) + ;; + *) + echo "ERROR: unknown suite '$suite' (expected vcontainer|vdkr|vpdmn)" >&2 + exit 2 + ;; +esac + +pytest_args=( + -v + --tb=short + -m "$marker_filter" + --vdkr-dir "$extract_dir" + --junitxml="$builddir/pytest-$suite-results.xml" +) + +# Allow tests that consume an OCI image (import/save/load) to find one. +if [ -n "$oci_image" ] && [ -d "$oci_image" ]; then + pytest_args+=(--oci-image "$oci_image") +fi + +# Pass architecture through when set (default is x86_64). +if [ -n "$arch" ]; then + pytest_args+=(--arch "$arch") +fi + +cd "$metavirtdir" +# Don't let a single failing test kill the whole step - collect the junit +# report, then surface the exit code via the junit file + exit status. +set +e +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}" +rc=$? +set -e + +# Copy artefacts to the results dir if one was provided. +if [ -n "$results_dir" ]; then + mkdir -p "$results_dir" + cp -f "$builddir/pytest-$suite-results.xml" "$results_dir/" 2>/dev/null || true + if [ -f /tmp/pytest-vcontainer.log ]; then + cp -f /tmp/pytest-vcontainer.log "$results_dir/pytest-$suite.log" || true + fi +fi + +exit $rc From patchwork Mon Jul 6 22:05:36 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD46C44500 for ; Mon, 6 Jul 2026 22:06:54 +0000 (UTC) Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.162433.1783375610468321323 for ; Mon, 06 Jul 2026 15:06:50 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=bIom9D1T; spf=pass (domain: gmail.com, ip: 209.85.214.182, mailfrom: ticotimo@gmail.com) Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2cacd69a9c0so29844925ad.1 for ; Mon, 06 Jul 2026 15:06:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375610; x=1783980410; darn=lists.yoctoproject.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+zE59Ut3mVzR786qcfGxStb+/atlqzzjvoDh2+mMjQI=; b=bIom9D1TE9HaTmzTcnqAZ0M/bSIFQZf+b1jpvVgByH/g9LrrSGv8UKiKr1Ui60ct8h JddmusivuRW37WF+K3jGOBZfQo10zxKr6nnst9EHIrbqJNJ1hsd90J57u/5D/5eHLgA4 Jo+YOHBv8FCYNmlYvGiS4opBXVJtEveTO6JTlTpLnCdfV3UJNzHhVI4GekfXzEBoe7VB auLXWiaq6Ra9d1+Issx/1uvY3QCWkfGwVhHFX1VpccaeXUPsm9+6OstF+kltkRpD7qoQ CJUkqz+y/2S3wflb6+jcXA6QlhvL96qTPyZ9wKHHig8OgJ+v9Lrx69xO7zLb6DtW0YYc H2YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375610; x=1783980410; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+zE59Ut3mVzR786qcfGxStb+/atlqzzjvoDh2+mMjQI=; b=YIEWuJXsYmtwGQ2kZ9nZ2OUVTBuh5pPZMf6/15A1kuxRD80csmEO1bRnYDGt3hdliQ jc9vIURWyYRvjIBfa0GwBu3UJjQR4s9Nh0lKNRoW1UJBjuki8nVnQ7EElHcne/TgroFX 4ntciRHEEkR7z/mCas0LVogftslRFvZj7jU38k5cUH/UlkyTqfOdvwq8hAn4B6ln256Y SSK98sanzsPsZ1HvA1nkqO0jytapDk1WpvTXmOR00HCRdHqDUQgTD0o0ds9d7/r4Es5O cooctgN1OlBdnoAdvNkVS7dAkONjiIDoMO/gHx4Z1MT6tgraWhobh6XYDvJAyer5EK1B 57WQ== X-Gm-Message-State: AOJu0YwcXDOqZa5rFnRmjACpIyFBmvI1aA/npKSHyDsRC4YncirBNcdG EJXGip/SqWrrQudCehM+9FXLwD9ESVDdYKZ0XIeTL0N4atTM6KqFmTWgpBopCA== X-Gm-Gg: AfdE7ck8uBNzm0BV7FGB4DIqy8emHDFfyb02ffkPdroSvrCay0QHX9KaeVwj3rumpVJ +BIk2AXBSiAIPh7vodOyMcnkC5XtqdPYx+dL4Urp8ZWvCwjANn9SdOXDD0lfdOS6UEuaDIRxKJs K2QkCrfiSqHQd5bL9dPxbhlADw8f4ZnmCODhvSclcvW/F1SBQU/Rtz1U5BzNRnYU6Z4TH+t8Zqv RFsqWHA9YTa96ISfTXfKox3meQipP76UIFM5XcgsgmW+rOUBLlvJxbznYb5wtp7lPyQMtdYXhlS GSTPBLF/y2gTv0R7BqhrTCDUbCAsGpTJ0/XQtdwJ95mJVOHKcEuBzPZ/v3w6RwyUe97s3H0GAWN uZwPgNTliQIlREhxQBBMiFsRCCWHsKZU17eochARx0n0gBKktOFB7RKiTHE4vSfnp097zSRO1Je 6OkhuaBmjuxkzLlD30uOpjDeaJ9JIotEG+EwEnfd44JrxOaWHN1tP/CrZ92bw23spc3SLJ6E0U7 pr5SFT8JmxVqV6oPhVYFtg= X-Received: by 2002:a05:6a21:6e83:b0:3bf:c2ce:972b with SMTP id adf61e73a8af0-3c08ef2d3dcmr2777592637.50.1783375608498; Mon, 06 Jul 2026 15:06:48 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.06.45 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:06:45 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 04/11] scripts: add container registry push, auth, tagging, runtime selection Date: Mon, 6 Jul 2026 15:05:36 -0700 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:06:54 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4434 Add the push-containers infrastructure that drives the post-build steps for the 'containers-library' job. After each build step the runtime container store is harvested and pushed to one or more registries with derived per-step tags. * config.json: add CONTAINER_REGISTRIES, CONTAINER_AUTH_CONFIG, CONTAINER_RUNTIME, CONTAINER_TAG_CMDS, CONTAINER_VERSION_RECIPE and CONTAINER_IMAGE_MAP configuration knobs. * scripts/run-config: drive push-containers as a post-step action (by running scripts/run-push-containers). * scripts/run-push-containers: Tags are generated from recipe and distro metadata (yocto- tag uses major.minor on snapshots and full PV on releases) with CONTAINER_VERSION_RECIPE allowing a step to source PV from a different recipe than the image itself. * Skip pushing when building the container image failed * Registry auth is staged via .../config.json or podman .../auth.json using CONTAINER_AUTH_CONFIG, replacing an interactive login that could hang. CONTAINER_RUNTIME picks between vdkr (Docker-compatible) and vpdmn (Podman) runtimes. * Robustness: skip gracefully when no registries are configured, fix the OCI directory path, handle memres already running, and avoid hanging when memres has not yet come up. * The vcontainer-tarball bundles its own python3/site-packages; sourcing its environment-setup-ci puts them on PATH and shadows the buildbot-venv (e.g. its websockets<10 breaks bitbake's hashserv). It must never touch the build steps' PATH, so there is no job-wide "vcontainer" init step: the tarball is set up (downloaded/extracted on demand, cached) and its env sourced only in the synthesized steps that actually use its runtime wrappers -- push-containers (vpdmn/vdkr 'vimport' for single-arch; multiarch uses skopeo-native from the build). AI-Generated: Claude Cowork Opus 4.8 Signed-off-by: Tim Orling --- config.json | 5 + scripts/run-config | 24 +++-- scripts/run-push-containers | 180 ++++++++++++++++++++++++++++++++++++ 3 files changed, 201 insertions(+), 8 deletions(-) create mode 100755 scripts/run-push-containers diff --git a/config.json b/config.json index ae13c5b..c6ee372 100644 --- a/config.json +++ b/config.json @@ -43,6 +43,11 @@ "BUILDINFOVARS" : ["INHERIT += 'image-buildinfo'", "IMAGE_BUILDINFO_VARS:append = ' IMAGE_BASENAME IMAGE_NAME'"], "WRITECONFIG" : true, "SENDERRORS" : true, + "CONTAINER_RUNTIME" : "vpdmn", + "CONTAINER_REGISTRIES" : [], + "CONTAINER_TAGS" : ["latest"], + "CONTAINER_TAG_CMDS" : [], + "CONTAINER_IMAGE_MAP" : {}, "extravars" : [ "SANITY_TESTED_DISTROS = ''", "BB_HASHSERVE = '${AUTOBUILDER_HASHSERV}'", diff --git a/scripts/run-config b/scripts/run-config index 90a5996..391db3e 100755 --- a/scripts/run-config +++ b/scripts/run-config @@ -154,15 +154,11 @@ else: sys.exit(0) vcontainer = utils.getconfigvar("vcontainer", ourconfig, args.target) -if jcfg: - if vcontainer: - addentry("vcontainer", "Setup vcontainer tarball", "init") -elif vcontainer: - # vcontainer is opt-in per target via the "vcontainer" config variable, - # so this is a no-op for targets which don't set it +# The vcontainer-tarball bundles its own python3/site-packages; avoid +# shadowing the buildbot-venv, except in steps that require 'vpdmn' or +# 'vkdr'. This avoids build failures on 'buildtools' workers. +if not jcfg and vcontainer and args.stepname in ("push-containers"): utils.setup_vcontainer_tarball(ourconfig, args.target, args.builddir + "/../vcontainer-tarball") - if args.phase == "init" and args.stepname == "vcontainer": - sys.exit(0) extratools = utils.getconfigvar("extratools", ourconfig, args.target) if jcfg: @@ -195,6 +191,8 @@ utils.mkdir(errordir) errorlogs = set() +push_containers = properties.get("push_containers", False) + def log_file_contents(filename, builddir, stepnum, stepname): logfile = logname(builddir, stepnum, stepname) with open(logfile, "a") as outf, open(filename, "r") as f: @@ -313,6 +311,16 @@ def handle_stepnum(stepnum): hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets)) bitbakecmd(args.builddir, "bitbake %s -k" % (sanitytargets), report, stepnum, args.stepname) + # Push container images to registries when push_containers is enabled. + # The push logic itself lives in scripts/run-push-containers. + container_images = utils.getconfigdict("CONTAINER_IMAGE_MAP", ourconfig, args.target, stepnum) + if container_images and push_containers: + if jcfg: + addstepentry("push-containers", "Push containers", shortdesc, desc, str(container_images), str(stepnum)) + elif args.stepname == "push-containers": + hp.printheader("Step %s/%s: Pushing container images %s" % (stepnum, maxsteps, list(container_images.keys()))) + bitbakecmd(args.builddir, "%s/run-push-containers %s %s" % (scriptsdir, args.target, stepnum), report, stepnum, args.stepname) + # Run any extra commands specified cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum) if jcfg: diff --git a/scripts/run-push-containers b/scripts/run-push-containers new file mode 100755 index 0000000..00d87ed --- /dev/null +++ b/scripts/run-push-containers @@ -0,0 +1,180 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Push container images built by a target step to the configured registries. +# +# Invoked by run-config for 'push-containers' steps with the OE build +# environment already sourced (bitbake and the vcontainer runtime wrappers +# must be on PATH). All configuration comes from the target/step config: +# +# CONTAINER_IMAGE_MAP - dict of image recipe -> registry image name +# CONTAINER_RUNTIME - vpdmn (default) or vdkr +# CONTAINER_REGISTRIES - list of registries to push to +# CONTAINER_TAGS - list of static tags (e.g. latest) +# CONTAINER_TAG_CMDS - extra shell to populate _EXTRA_TAGS +# CONTAINER_VERSION_RECIPE - recipe whose PV provides the version tag +# CONTAINER_AUTH_CONFIG - registry auth file staged into the guest +# + +import subprocess +import sys + +import utils + +parser = utils.ArgParser(description='Pushes container images for a target step to the configured registries.') + +parser.add_argument('target', + help="The target build name") +parser.add_argument('stepnum', + type=int, + help="The step number within the target") + +args = parser.parse_args() + +ourconfig = utils.loadconfig() + +container_images = utils.getconfigdict("CONTAINER_IMAGE_MAP", ourconfig, args.target, args.stepnum) +if not container_images: + print("No CONTAINER_IMAGE_MAP for %s step %s, nothing to push" % (args.target, args.stepnum)) + sys.exit(0) + +registries = utils.getconfiglist("CONTAINER_REGISTRIES", ourconfig, args.target, args.stepnum) +if not registries: + utils.printheader("push-containers skipped — CONTAINER_REGISTRIES is empty, no containers pushed") + sys.exit(0) + +runtime = utils.getconfigvar("CONTAINER_RUNTIME", ourconfig, args.target, args.stepnum) or "vpdmn" +static_tags = utils.getconfiglist("CONTAINER_TAGS", ourconfig, args.target, args.stepnum) +auth_config = utils.getconfigvar("CONTAINER_AUTH_CONFIG", ourconfig, args.target, args.stepnum) +if not auth_config: + if runtime == "vpdmn": + auth_config = "${HOME}/.config/containers/auth.json" + else: + auth_config = "${HOME}/.docker/config.json" + +utils.printheader("Pushing container images %s" % list(container_images.keys())) + +script = [ + "set -e", + "test -w /dev/kvm || { echo 'ERROR: /dev/kvm is not writable, cannot push containers'; exit 1; }", + # Always bring up a fresh memres VM in the foreground. + # + # 'memres status' only checks that the QEMU PID in daemon.pid + # is alive (see daemon_is_running()/daemon_status() in + # meta-virtualization's vrunner.sh); it returns 0 as soon as + # QEMU forks, so a hung/partially-booted VM from a previous + # run — or a VM in mid-boot — is reported as healthy. The + # subsequent 'login'/'vimport'/'push' commands then hang on + # the unresponsive daemon socket. + # + # 'memres restart' is synchronous: it does stop+start and + # runs a PING/PONG readiness probe against the daemon socket + # (120s timeout), exiting non-zero if the VM never answers. + # Running it in the foreground gives us a trustworthy ready + # signal via its exit code, so we can drop the status-poll + # loop entirely. + # + # Install an EXIT trap first so we always tear the daemon + # down, even if bitbake -e / vimport / push fails mid-step + # under 'set -e'. The trap is armed before the restart so + # a restart failure also triggers cleanup. + # + # Registry auth is staged into the guest at VM boot via + # the global '--config' flag — vrunner.sh's setup_auth_share() + # copies $AUTH_CONFIG onto a read-only 9p share, and + # vdkr-init.sh / vpdmn-init.sh's install_auth_config() + # installs it at /root/.docker/config.json (vdkr) or + # /run/containers/0/auth.json (vpdmn) inside the guest. + # Subsequent 'push' calls use those creds directly, so no + # explicit 'login' step is needed. Calling 'login' would + # actually hang under the autobuilder (no PTY): when the + # memres daemon is running, vcontainer-common.sh dispatches + # login via '--daemon-interactive' and blocks reading the + # password from stdin (see login case in vcontainer-common.sh). + "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime, + "%s-$(arch) --config %s memres restart ' suffix on AUTOREV/dev recipes — Docker + # reference format does not allow '+' in tags, and the + # base PV is what consumers expect. + # + # DISTRO_VERSION needs context-sensitive handling. Poky's + # DISTRO_VERSION resolves to '${PV}+snapshot-${METADATA_REVISION}' + # off a tag and just '${PV}' on a release tag. The '+' in + # the snapshot form is illegal in a Docker tag, but more + # importantly the patch level on a snapshot build (e.g. + # '6.0.99' between 6.0 and 6.1) is a moving target that + # doesn't correspond to any real release — only the + # major.minor line is meaningful. So: + # - snapshot build (DISTRO_VERSION contains '+') → tag + # with major.minor only, e.g. 'yocto-6.0'. + # - release-tag build (no '+') → tag with the full + # version, e.g. 'yocto-5.0.5' from the yocto-5.0.5 tag. + script += [ + "_BBENV=$(bitbake -e %s 2>/dev/null) || true" % recipe, + "_PV=$(echo \"$_BBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')", + "_DISTRO_CODENAME=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_CODENAME=/{ print $2; exit }')", + "_DISTRO_VERSION_RAW=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_VERSION=/{ print $2; exit }')", + "case \"$_DISTRO_VERSION_RAW\" in", + " *+*) _DISTRO_VERSION=$(echo \"${_DISTRO_VERSION_RAW%%+*}\" | cut -d. -f1,2) ;;", + " *) _DISTRO_VERSION=\"$_DISTRO_VERSION_RAW\" ;;", + "esac", + "_DEPLOY_DIR_IMAGE=$(echo \"$_BBENV\" | awk -F'\"' '/^DEPLOY_DIR_IMAGE=/{ print $2; exit }')", + "_EXTRA_TAGS=\"\"", + ] + if version_recipe: + # When the image recipe's PV is a wrapper-style + # placeholder (e.g. app-container-python_1.0.0.bb, + # whose 1.0.0 is meaningless to a downstream user), + # CONTAINER_VERSION_RECIPE points at the recipe whose + # PV is actually meaningful for the resulting tag — + # typically the language runtime or app being packaged + # (e.g. python3 -> 3.14.x). Override _PV from that + # recipe; image-recipe state still drives + # DEPLOY_DIR_IMAGE and DISTRO_* since those are + # environment-wide. + script += [ + "_VBBENV=$(bitbake -e %s 2>/dev/null) || true" % version_recipe, + "_PV=$(echo \"$_VBBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')", + ] + script += tag_cmds + script.append( + "_TAGS=\"%s $_PV $_DISTRO_CODENAME yocto-$_DISTRO_VERSION $_EXTRA_TAGS\"" % " ".join(static_tags) + ) + # Only push an image whose build actually produced an OCI artefact. + # build-targets runs 'bitbake ... -k', so a failed image build does + # not abort the build step or the other images, and the failure is + # already reported there. If we let the push proceed, 'vimport' of + # the missing ${recipe}-latest-oci would fail under 'set -e' and + # abort the whole push step — taking down the images that *did* + # build with it. So skip a missing image here (warn, don't fail) + # and let the successfully-built ones publish. + script += [ + "_OCI_IMAGE=${_DEPLOY_DIR_IMAGE}/%s-latest-oci" % recipe, + "if [ ! -e \"$_OCI_IMAGE\" ]; then", + " echo \"WARNING: %s did not build (no OCI image at $_OCI_IMAGE), skipping push\"" % recipe, + "else", + ] + for registry in registries: + # No per-registry 'login': credentials were staged into + # the guest by '--config' on 'memres restart' above. + script += [ + " for _tag in $_TAGS; do", + " %s-$(arch) vimport ${_DEPLOY_DIR_IMAGE}/%s-latest-oci %s/%s:${_tag}" % (runtime, recipe, registry, image), + " %s-$(arch) push %s/%s:${_tag}" % (runtime, registry, image), + " done", + ] + script.append("fi") +# Tear-down is handled by the EXIT trap installed above. +utils.flush() +sys.exit(subprocess.call(["/bin/bash", "-c", "\n".join(script)])) From patchwork Mon Jul 6 22:05:37 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9705C43458 for ; Mon, 6 Jul 2026 22:07:04 +0000 (UTC) Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.162435.1783375615856913724 for ; Mon, 06 Jul 2026 15:06:55 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=TbYEFmuw; spf=pass (domain: gmail.com, ip: 209.85.210.172, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-847e6f03df8so4311650b3a.0 for ; Mon, 06 Jul 2026 15:06:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375615; x=1783980415; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=apFaqJcux5yubEOSjwEytbZacgvToIJohWLkaO52AJ4=; b=TbYEFmuwt/goTjn2L94BWcOTWWwGjoJc8z0NPZDgTnHJimfFfYPTZQD7dWEJptTb2l dYjMtQrFxW2ZNzRHtxh34WD0FOBxO+0u1ZzkyPnjg1LVLiqBNQ9fwxFkV+OYRiqN029f TqzTR1Ig4WE/N/owZFo2MdCKf0VE4eV7AFA/GxvQBwCehDgnS0lB1wPt2N0EI7NOXkUo yBS+CSMWh8/lSIT+l/YlJBmVPMKbLHVaRUgtzLiWxCq8I5/M4saK806Qe0IXSfGfUjIU 40qjm2iSNj26LdhtP2wu8PZ4NdWISwfg6NSK/GjAvB5T3DaJmkhSKp9yOUgRvex5syo5 hBLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375615; x=1783980415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=apFaqJcux5yubEOSjwEytbZacgvToIJohWLkaO52AJ4=; b=iprQquVxw8DbkqGy/SyKIVa+23BZQVlTyX1k3GaCysnErxU62NZ7Lpwm5yKQj9y9TW cdzBi+HCPhHBEToIJuwp+1YdCukcVhxNetBf4W3GZerYEcqAx9wsFP3THn90AVxkjWt2 T2iSFANgKgiekVJEDugoppKYuhH5Loj+zRv9I13P9GooUkKqfXyT7UxOZ1gq4Dpozsha vClQ0SOmeIXI1T4WrOmVKKcS9XK+JXfPdJ0FadsjGdJfewRjH6Y5BSK1xaR9Co7cWx5j sKmKdTMZgkkW7SMxWsMNCcZYxQcWr9HH+T1yP2HsJ4cRtf8zpBoSk+32gI2YNXQBt7xf Nu/Q== X-Gm-Message-State: AOJu0YzekDKKskLr9tfBDawjrT07AT3Yp5aSUGnGsXeTH0URSeCA8zcM O7evVOaFr4dvK9HvXMgyG2MsmQPYZG0ubClRe7uB0ghFW8ML+kFR/7GXI8LyBA== X-Gm-Gg: AfdE7clDPXmMIGKJrfNhSbj5k0GAH08Co6+9HLwqolVuCPgXLGUR0K05qn5tl4TVaLo robpQRc/nevOPNd1oxKabD2VXeCYozQPunvxrSk+EecyerHOO+9IIVZZ8tGuiZM4HYJDPxD5hLj LNcZwmSDZrlI3ZKIoJMkLrQErZTNGKl5HAMRLy+NFA6gAUHwq9/MO96iPAjv5/nI9ORqDAL/iSR 6ImvCX3kFTAf9GQCCjKHUg+cWSVHfRHLWT8QV701J06pXl1XSwZ13zknCrOr94p7uSp03JCZ5Hi HTXV33HcO63dZPC0Xnv7XGTHPb8X5sOrFJqB0R75mD9J51uH45rxsQFkbu9GB1dJ7W0tcZFfcZz kFIqbIeXbjrT0Mx66j7gR3hRC2L+tJFzbGz6yMVOkGYJJZscFfx22H7QW07nZ06x3I3UmuJYKe3 RFOIMfoEgtVX55ErWKo179g+B9ETEi0OToTSH6p/+6Fqt194z0ephc6xxWGdKJlqNbjfGL/tmce J6A2D96x7Gb X-Received: by 2002:a05:6a00:2e29:b0:847:9200:21f8 with SMTP id d2e1a72fcca58-84826a83449mr2593962b3a.0.1783375615129; Mon, 06 Jul 2026 15:06:55 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.06.49 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:06:49 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 05/11] config.json: add 'containers-library' build job Date: Mon, 6 Jul 2026 15:05:37 -0700 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:07:04 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4435 Add 'containers-library' build job that build container images on top of the vcontainer-tarball SDK: * original container images from the intial "container-cross-install" branch of meta-virtualization - container-base - app-container-curl * additional images modelled after docker.io/library/* - app-container-python - app-container-mosquitto - app-container-valkey - app-container-nginx * Tag containers with versions based on the recipe to which they are mapped, e.g. python:3, python:3.14, python:3.14.5. Also tag with DISTRO_CODENAME and DISTRO_VERSION, e.g. wrynose and yocto-6.0. * Similar to vcontainer-tarball job, follow the guidance in meta-virtualization/docs/build-profiles.md and add "require conf/distro/include/meta-virt-host.conf" to extravars. This adds the required DISTRO_FEATURES and BBMULTICONFIG (vruntime-*, container-*). It also fixes warnings due to BBMASKing, by setting the appropriate BBFILE_PATTERN_IGNORE_EMPTY on layers which are intentionally masked. Use 'sed' in EXTRACMDS to remove the added 'require' line from conf/auto.conf or else remove-layers step throws a file not found error. * Append 'usrmerge' to DISTRO_FEATURES and set INIT_MANAGER to systemd Signed-off-by: Tim Orling --- config.json | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/config.json b/config.json index c6ee372..adabd57 100644 --- a/config.json +++ b/config.json @@ -1891,6 +1891,65 @@ "${SCRIPTSDIR}/run-vcontainer-tests -s vpdmn -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}" ] } + }, + "containers-library": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-openembedded/meta-webserver", + "${BUILDDIR}/../meta-virtualization" + ], + "extravars" : [ + "require conf/distro/include/meta-virt-host.conf", + "DISTRO_FEATURES:append = ' usrmerge'", + "INIT_MANAGER = 'systemd'" + ], + "EXTRACMDS" : ["sed -i '/meta-virt-host.conf/d' ${HELPERBUILDDIR}/conf/auto.conf"], + "vcontainer" : "${VCONTAINER_TARBALL_URL}", + "CONTAINER_TAG_CMDS" : [ + "_PV_MAJOR=$(echo $_PV | cut -d. -f1)", + "_PV_MAJOR_MINOR=$(echo $_PV | cut -d. -f1,2)", + "_EXTRA_TAGS=\"$_PV_MAJOR $_PV_MAJOR_MINOR\"" + ], + "step1" : { + "shortname" : "Build 'base' container", + "BBTARGETS" : "container-base", + "CONTAINER_IMAGE_MAP" : {"container-base": "base"}, + "CONTAINER_VERSION_RECIPE" : "base-files" + }, + "step2" : { + "shortname" : "Build 'curl' container", + "BBTARGETS" : "app-container-curl", + "CONTAINER_IMAGE_MAP" : {"app-container-curl": "curl"}, + "CONTAINER_VERSION_RECIPE" : "curl" + }, + "step3" : { + "shortname" : "Build 'python' container", + "BBTARGETS" : "app-container-python", + "CONTAINER_IMAGE_MAP" : {"app-container-python": "python"}, + "CONTAINER_VERSION_RECIPE" : "python3" + }, + "step4" : { + "shortname" : "Build 'mosquitto' container", + "BBTARGETS" : "app-container-mosquitto", + "CONTAINER_IMAGE_MAP" : {"app-container-mosquitto": "mosquitto"}, + "CONTAINER_VERSION_RECIPE" : "mosquitto" + }, + "step5" : { + "shortname" : "Build 'valkey' container", + "BBTARGETS" : "app-container-valkey", + "CONTAINER_IMAGE_MAP" : {"app-container-valkey": "valkey"}, + "CONTAINER_VERSION_RECIPE" : "valkey" + }, + "step6" : { + "shortname" : "Build 'nginx' container", + "BBTARGETS" : "app-container-nginx", + "CONTAINER_IMAGE_MAP" : {"app-container-nginx": "nginx"}, + "CONTAINER_VERSION_RECIPE" : "nginx" + } } }, "repo-defaults" : { From patchwork Mon Jul 6 22:05:38 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6AAAC44500 for ; Mon, 6 Jul 2026 22:07:04 +0000 (UTC) Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.161485.1783375620176697237 for ; Mon, 06 Jul 2026 15:07:00 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=ON7rNd9u; spf=pass (domain: gmail.com, ip: 209.85.210.179, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-847d1e9db22so3866327b3a.2 for ; Mon, 06 Jul 2026 15:07:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375619; x=1783980419; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=g5UZJPQX+RKDfv7IWxMkfC1RxiCXyXKO+7Td8ClJeRs=; b=ON7rNd9uRovXf9UkxT+QgULxKdT+0nkdlc2XrO68JtnLMyPlHi6jZtZKxTVB8w9ns7 lsyxokJsbtPd0bYyMcf7uWIdex4uPOv6pH/8u1xNPmA0Jb6V891RLYvn2KSXvyX6ukGQ npC+wDcGoTqXD7QMpn9OdRIzVoZhdBhXy+FRiJWFAbXTOAM3EvFRcjXeSR7kHJfuf6kW qjpehHH2X1zHfoa18AAuYXmQ12BYuBRmC5AiEbT3D+9XdS+3XxZanyCANeFHALAO+Yma ZnV8PH4uBXrkdEibtGc0tvDzr7lgk8IxrKmK78vjm/HTnv9dd3swCr0PJekToHNbOcSx KtgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375619; x=1783980419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=g5UZJPQX+RKDfv7IWxMkfC1RxiCXyXKO+7Td8ClJeRs=; b=RiQgO3hHm3raXynIGl6/pH8VuEFfLbps08NGSs08sTJYtQcikwS7oeJ0dI+ByHyLc9 SLIofHf3ogdfcDX8MHrP+8nDvfj0YV+v9yL9ZftjzAmonmhm+CsKXAfHvqckSfI5/AQy 0r9FoeufrmFb+F6gh+tMAbIEM4qsDIxVvvhhRXkB2xwWf5tB+JiGE6I9B31lf6QHpEe4 s8s+7imxki1rvDcwfhcphbOkJCY0Mw+B9FtaQ4Y1iLnemwLR3yImKjHLxzhjvfRyJpNN kw1ewUGUjaObgW6CN+c5/IG6g1nWmdD1RXkNpCFIsWSOvzDaNovH1/m63y/aoj7GSpPa x2cQ== X-Gm-Message-State: AOJu0YwmMNbvi+Cd+aYEo/XdhS9VMr00NN3Q17eFUa+VEc6KU5Zw/kjP BNlcbHCKa51cQxiBPb0hzaZcwn4lcOqZHJ7h4hy48tKKYjnAr32JsXAWNIYrNg== X-Gm-Gg: AfdE7cm8kDwdoVLzELXwGvIiKFBxzOT5eaq53DWdQfjKkOKJPu6LHetKjiuL/iqAdJ1 +Go6wnJWDvPYvey2MaU/sAtoqnh6gwSA+qloNRDUloalo0zsxWo4ixOg6QOUNCjArHaRxs9as4v hpXYwDF7nEHLeWCI0VOwTh5mONVUxpyuihxxZJSXTPEgoWn+K2rfyAQWup4RcltaEv2X7n3owuh +0ajPQgg1ysSQO7i2r5gCsfNkf1evLwSCWFHNj0lW6fKvL4nrLapNuqq0gTJuehqAol+/m7ULQR tMm3OqZaeZp1HoF1Fq30HaKMou16uloPSZitSeZ5i4Pvd0PrJdA7u24QeeFEzhRpz15796EtN+O BxL5LXKemz+5gBXY3UNYuDysO1pvrreGtY7yAFWxpQE07+24wAR381HjOdL/+RugLp2h3uIAOwq XmDrfQmeKmj6duE2pRPLJVHY+kMaz0GV0UAnGhYWoH+maQvJJ7cUJNmvn4zHjyKfSMpW4xWelsd im+lpMubqR7 X-Received: by 2002:a05:6a00:420c:b0:847:8250:2b2a with SMTP id d2e1a72fcca58-84826df6e2dmr2190693b3a.60.1783375619379; Mon, 06 Jul 2026 15:06:59 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.06.56 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:06:56 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 06/11] scripts/run-push-containers: push multiarch containers with skopeo-native Date: Mon, 6 Jul 2026 15:05:38 -0700 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:07:04 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4436 Support pushing multi-architecture container images (recipes inheriting meta-virtualization's oci-multiarch class) from the 'containers-library' job. These produce an OCI Image Index on disk rather than a single-arch image in a runtime store, so they are pushed with skopeo instead of vdkr/vpdmn (which only vimport one architecture). * scripts/run-push-containers: detect multiarch recipes per-recipe at runtime: 'bitbake -e' only contains OCI_MULTIARCH_OUTPUT when the recipe inherits oci-multiarch, and its value is the exact OCI layout path to push. Multiarch images are pushed with 'oe-run-native skopeo-native skopeo copy --all' direct from the OCI layout, after populating the skopeo-native recipe sysroot via 'bitbake skopeo-native -c addto_recipe_sysroot' (once per step, prepare_skopeo). The same tag set and registries are used for both paths; auth is passed to skopeo via --dest-authfile from CONTAINER_AUTH_CONFIG. * scripts/run-push-containers: defer the memres VM bring-up (kvm check, EXIT trap, restart, image rm) into a lazy start_vm() that only runs when the first single-arch recipe is pushed, so multiarch -only steps need neither /dev/kvm nor a VM boot. AI-Generated: Claude Cowork Opus 4.8 Signed-off-by: Tim Orling --- scripts/run-push-containers | 100 +++++++++++++++++++++++++++--------- 1 file changed, 77 insertions(+), 23 deletions(-) diff --git a/scripts/run-push-containers b/scripts/run-push-containers index 00d87ed..c13f493 100755 --- a/scripts/run-push-containers +++ b/scripts/run-push-containers @@ -58,7 +58,24 @@ utils.printheader("Pushing container images %s" % list(container_images.keys())) script = [ "set -e", "test -w /dev/kvm || { echo 'ERROR: /dev/kvm is not writable, cannot push containers'; exit 1; }", - # Always bring up a fresh memres VM in the foreground. + # Two push paths share this script: + # + # - Single-arch images (the common case) are imported into + # a vdkr/vpdmn memres VM and pushed from inside it. The + # VM bring-up is expensive and needs /dev/kvm, so it is + # wrapped in start_vm() and only runs when the first + # single-arch recipe is pushed. + # + # - Multi-arch images (recipes inheriting oci-multiarch) + # are pushed with skopeo-native straight from the OCI + # Image Index layout on disk — no VM, no /dev/kvm. + # skopeo's recipe sysroot is populated on first use by + # prepare_skopeo(). Detection is per-recipe at runtime: + # 'bitbake -e ' only contains OCI_MULTIARCH_OUTPUT + # when the recipe inherits oci-multiarch, and its value is + # the exact OCI layout path to push. + # + # Notes on the memres VM bring-up in start_vm(): # # 'memres status' only checks that the QEMU PID in daemon.pid # is alive (see daemon_is_running()/daemon_status() in @@ -92,8 +109,20 @@ script = [ # memres daemon is running, vcontainer-common.sh dispatches # login via '--daemon-interactive' and blocks reading the # password from stdin (see login case in vcontainer-common.sh). - "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime, - "%s-$(arch) --config %s memres restart /dev/null || true' EXIT" % runtime, + " %s-$(arch) --config %s memres restart X-Patchwork-Id: 91879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4558C43602 for ; Mon, 6 Jul 2026 22:07:04 +0000 (UTC) Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.161486.1783375624496164691 for ; Mon, 06 Jul 2026 15:07:04 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=RlQqWsOl; spf=pass (domain: gmail.com, ip: 209.85.210.172, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-842338c18e0so2875126b3a.1 for ; Mon, 06 Jul 2026 15:07:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375624; x=1783980424; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=vlTnvgs3WdajwKoonrkpzvKzJdjCAXsH32XvPJJn5xI=; b=RlQqWsOlnKsYIhhQ+zowlXBMXMh5w9iydq40lME8pmhJSWLfGZoPdGHNDo0ONybxe4 pRIljhGGi1oBXUeHQxIcbSuwrqKSX8snRKsnyjhFW3jCGqsqsctxKmZ40dq7y66O6qp/ +UskIQyDbIWtSLyILx8rr1KB7WROSlUrbd5n+CeusH7hF8Lrv6hMuT6o+BdC2VWF9ii6 LIrGzEyIkmmW2ozY5u+onwHzN/t3dRUIjgmmSz2wPbD7dsGf2mlZzzYQbhO4qBKynMrm Zy0V60e6SClCK4tCzCAttL2id+7f9HAUBaJAKJMmJluduuIFX2cnGedx7zcl/45pyQF1 Frxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375624; x=1783980424; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=vlTnvgs3WdajwKoonrkpzvKzJdjCAXsH32XvPJJn5xI=; b=iRrTNeSCkHi2PPcIGnfyimQHQORnJC11OXaiEC+cfRukvTX3gK0Ov4S806yhWNEhNh Q9nfKSf/MNBrwI74dS1D9ExieQL+IthiwG+OuTW9sIM0hSi4zgE+v1pDqu1VzRglNjFB T1PssHokuXIMGvoyWH1FNvDumUgwfTuSMRZPx8QUmRBtfqRAytW8g6oXdS8A35wZWKft sc31tLrZzCoUn6URbWRbVlIGzS6UNIp2iazoHL83uSZQI1o9WSgpC0z9EHyiS87xbBq/ tN8/yAQNBO4c78/eDqXyv2IwCtDb+o7vWkWk6b+Ve1g5QtLagJijDFOKXSxdKYpsu+8n v2AQ== X-Gm-Message-State: AOJu0Yxo/NbZMVx0HX2LWEKpmnYXKILpsJsQjWELHpnmDDmgekmhOZZ2 dmI7dxi/gl+o8HbXbqR+gz+Bo0yydCkqsuuZbIhMXEsFM9czUHt2GOB9ERI6VQ== X-Gm-Gg: AfdE7cnAHoYPpzcNoWuhu09lrQFzu2/ii1uT1tq+EE7Gd3iB/q8RXw7sKi3GBpibVZw IoZyUpUUlrMZ0/YZNNyECA4PiBLX0b6mDd5r0bh5Ycu9kGNd+lXL2MynScHWHHvaUPwK8yUMkaR oAyugH9saOVvOb/lrUx17fs1qFnY0704t20WpJ1c7a1q/JjZsbtrJCRGGIWg95JqOhlSOEDh9vq +ro+0M9TLHWMDB/+9KHuEm6Y5rxRKXsNi81XQevdUXTcS3UG/SigN6VBeBGfsSs/RjtEiAuQcAQ plgVNpKx1YPr0Ee0s1bPnJUh8+TRaimt3Q9xs9ViZcGkLsAq1cd1v+jfjfK8Eduo0zdym7vE4hK NpvlBgFPQYJyaHf/mN93w38IoxaIOGR/3bDT0lmEJlMIvNCrO/s0vnG0JKFS2dut+F1VxPGCi7Z YogBnwyjZmq3YutAJUo5OKsKFKfQLBtQCGRagB6xMl4wijxOcxcTtspSscehmYJI3iouBufZsWX +R9dhUGDz+A X-Received: by 2002:a05:6a00:399f:b0:845:d286:1fa5 with SMTP id d2e1a72fcca58-84826f0e4e9mr2409813b3a.52.1783375623772; Mon, 06 Jul 2026 15:07:03 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.07.00 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:07:01 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 07/11] containers-library: switch to multiarch; add alpine single-arch Date: Mon, 6 Jul 2026 15:05:39 -0700 Message-ID: <7725c52b1c0031a1578f5c3c2ba7bc72665b5c30.1783371914.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:07:04 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4437 Utilize the oci-multiarch detection in run-push-containers script, which uses 'skopeo-native copy' to 'push' the oci/ directory to the container registries. Add app-container-alpine as a single-arch container which demonstrates alpine-oci-base and ensures we are exercising the single-arch path in run-push-containers. Signed-off-by: Tim Orling --- config.json | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/config.json b/config.json index adabd57..878cca0 100644 --- a/config.json +++ b/config.json @@ -1915,39 +1915,45 @@ "_EXTRA_TAGS=\"$_PV_MAJOR $_PV_MAJOR_MINOR\"" ], "step1" : { - "shortname" : "Build 'base' container", - "BBTARGETS" : "container-base", - "CONTAINER_IMAGE_MAP" : {"container-base": "base"}, - "CONTAINER_VERSION_RECIPE" : "base-files" + "shortname" : "Build 'alpine' single-arch container", + "BBTARGETS" : "app-container-alpine", + "CONTAINER_IMAGE_MAP" : {"app-container-alpine": "alpine"}, + "CONTAINER_VERSION_RECIPE" : "alpine-oci-base" }, "step2" : { - "shortname" : "Build 'curl' container", - "BBTARGETS" : "app-container-curl", - "CONTAINER_IMAGE_MAP" : {"app-container-curl": "curl"}, - "CONTAINER_VERSION_RECIPE" : "curl" + "shortname" : "Build 'base' multiarch container", + "BBTARGETS" : "container-image-multiarch-container-base", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-container-base": "base"}, + "CONTAINER_VERSION_RECIPE" : "base-files" }, "step3" : { + "shortname" : "Build 'curl' multiarch container", + "BBTARGETS" : "container-image-multiarch-app-container-curl", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-curl": "curl"}, + "CONTAINER_VERSION_RECIPE" : "curl" + }, + "step4" : { "shortname" : "Build 'python' container", - "BBTARGETS" : "app-container-python", - "CONTAINER_IMAGE_MAP" : {"app-container-python": "python"}, + "BBTARGETS" : "container-image-multiarch-app-container-python", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-python": "python"}, "CONTAINER_VERSION_RECIPE" : "python3" }, - "step4" : { + "step5" : { "shortname" : "Build 'mosquitto' container", - "BBTARGETS" : "app-container-mosquitto", - "CONTAINER_IMAGE_MAP" : {"app-container-mosquitto": "mosquitto"}, + "BBTARGETS" : "container-image-multiarch-app-container-mosquitto", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-mosquitto": "mosquitto"}, "CONTAINER_VERSION_RECIPE" : "mosquitto" }, - "step5" : { + "step6" : { "shortname" : "Build 'valkey' container", - "BBTARGETS" : "app-container-valkey", - "CONTAINER_IMAGE_MAP" : {"app-container-valkey": "valkey"}, + "BBTARGETS" : "container-image-multiarch-app-container-valkey", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-valkey": "valkey"}, "CONTAINER_VERSION_RECIPE" : "valkey" }, - "step6" : { + "step7" : { "shortname" : "Build 'nginx' container", - "BBTARGETS" : "app-container-nginx", - "CONTAINER_IMAGE_MAP" : {"app-container-nginx": "nginx"}, + "BBTARGETS" : "container-image-multiarch-app-container-nginx", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-nginx": "nginx"}, "CONTAINER_VERSION_RECIPE" : "nginx" } } From patchwork Mon Jul 6 22:05:40 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD86AC43458 for ; Mon, 6 Jul 2026 22:07:14 +0000 (UTC) Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.161488.1783375628642154536 for ; Mon, 06 Jul 2026 15:07:08 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=maVJxAgB; spf=pass (domain: gmail.com, ip: 209.85.210.171, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-8471013fac2so3857171b3a.1 for ; Mon, 06 Jul 2026 15:07:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375628; x=1783980428; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=2tcSruuJO1rQHGCLoyUGX+qMkiVQJy0Gh7TBeqkW7tA=; b=maVJxAgBSJndLV0VthQr3SH7DyB7usHBm0I+CgMD1morSOxZiSGUn1XK3138dzDEyM j+pf15fJ+Wdg2S1SvuCcktUUpDYArPlC4eNl3A1HvlO6Q/uqAN5oL6Zcyunm535oc812 LCl7aVKT6bFHFzzJybZ4sS1uu1ltAQLjOgKVLHFGiB0uNJ6v5HTlIkEeF0p1I8cREpfU 3sK4prUNekhrp5RkgpLkohKWuQEHYKGWNfHfKTok2Mw0vR3ieRQyVKkadkD/FYeSiWLX BOcuTpBe5zKh3dcr8PcMIBm6hoF0d8g/ElAVIkM/S6VaelYDHoy9f51g+/bV0uC4fhYy Ppuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375628; x=1783980428; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=2tcSruuJO1rQHGCLoyUGX+qMkiVQJy0Gh7TBeqkW7tA=; b=J5i2LwSfPcYFLDKs96oFywwNhCg0yJs2hSW9nY+Y42+GVNDGCfiAlDg//YwR3snIrh hY/K8MJTwhsE6yMywvC4r1XejXft4BRQvf3VhSFmsX6Bq244K6w7i8QmENRi4LMwst3g s8FIpk7y+lD++ZiJ6ELyBDB+pDp3SbFur7WI4iyX1cl7/Pdpp2j0bjhK3XsxaWKnQIW3 kg5Uss9DhWiZcckVijtDxNiw5l80uE2yjREywcCbFDWOCPWCMvLoJ8cBGqVn1PogFusn cB4Doi+BDa/TsCyr8Rbt4TCFoS9TxHABPRZfg3QUvG/HWP7f3/d+TdVpvEcyayyhEvah Ijrg== X-Gm-Message-State: AOJu0YwoGjSPybwz3pNSZME08wb5XYEWDSr9PRx/TgDhXd6q8q/CDGcI C7XnemDEhWViCRiRD/F+MtSIim2YRKC4jEayRWnRlVcw+PxbXqyYCjh/eT1mcQ== X-Gm-Gg: AfdE7cmReK1HE9UhJZR2AVnrvl5t9Uevu6zYDy7zKSMD+rmbPKXG7FJSV/ePHrF4M4M FgnQLmDjbTMFSC46uQfwiKOP3TDxothCrRmETKjQArMbB84cbozjhrmkdP6etH1l9CIcRkb6Csa 2g1KAQ6MP/KzHLMD5GkDyXrssHl68cAtQwgl3hKXx6U7xhxZoRWoc8he2UXB5sH6jiV8M8n7U8A 35FHUhmMKWV8u9iT11I5iif6GBvZAMdoyzmykGJi1tGh2eAVZs68D+GcK+L7VBhp2Jv1mQAtr5m IINZuEV6n5zGeT+Kf20LxuuLK0Rsx6NiOCWqvRq2/0AlXE44GI99DZDNlpSS/id0R7uD50COedy UUtJYvcFi043pn+Q+8jJhUh+ltIBczIqUOYZ6CcBQg3sH5UJhgFsI26OZ0yGZDRaaKWPfyEDFEz w42S+1trgKj8/GBApEzyEHO+XxB+28gwF7iVmkLixWuICkAA3/Jvmzr4lxo870VEjIFKuxjH8zp g7+P+VsIM5m X-Received: by 2002:a05:6a00:e07:b0:845:d274:bf88 with SMTP id d2e1a72fcca58-84826d8637amr2453529b3a.49.1783375627722; Mon, 06 Jul 2026 15:07:07 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.07.05 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:07:05 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 08/11] run-push-containers: conditionally sign pushed containers with cosign Date: Mon, 6 Jul 2026 15:05:40 -0700 Message-ID: <79eda3320fba3266fcfb00710d46cc5949a506a0.1783371914.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:07:14 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4438 Introduce CONTAINER_COSIGN_KEY, a cosign private key path that gates signing. When unset (the default) no cosign sysroot is prepared and signing is skipped entirely. When set, every image that is successfully pushed is signed in place: - prepare_cosign() lazily populates the cosign-native sysroot (bitbake cosign-native -c addto_recipe_sysroot) and logs cosign in to each target registry. CONTAINER_REGISTRIES entries carry a namespace path (e.g. quay.io/ticotimo), but the auth file is keyed by the bare registry host and 'cosign login' only accepts a host authority, so each entry is stripped to its host ('${reg%%/*}') for both the credential lookup and the login: auths..auth is base64-decoded into user:password and the password is fed on stdin (--password-stdin). - sign_image() signs the image by digest with 'cosign sign --recursive --key' (which also covers every child manifest of a multi-arch index) so cosign does not warn about signing a mutable tag. The digest must be the one the tag resolves to: the multi-arch index digest is taken from 'skopeo copy --all --digestfile' at push time, and the single-arch manifest digest from 'skopeo inspect' (correct there, as there is no list). Signing 'skopeo inspect' on a multi-arch tag is avoided because without --raw it returns the host platform's child digest, which would sign one arch and leave the index unsigned. Signing is deduped per digest within the run, and a transparency-log conflict ('entry already exists' / HTTP 409, which also recurs on rebuilds that reproduce the same digest) is treated as success so signing stays idempotent rather than aborting the step. Signing runs in both push paths (multiarch skopeo-native and single-arch memres VM) and only after a successful push, so a missing/failed-build artefact is still skipped rather than signed. COSIGN_PASSWORD is exported (defaulting to empty) so cosign reads the passphrase from the environment instead of falling through to an interactive prompt (which dies with "inappropriate ioctl for device" under the autobuilder); an encrypted key requires the real value to be present in the build environment. Signed-off-by: Tim Orling --- config.json | 1 + scripts/run-push-containers | 115 +++++++++++++++++++++++++++++++++++- 2 files changed, 113 insertions(+), 3 deletions(-) diff --git a/config.json b/config.json index 878cca0..a1eee57 100644 --- a/config.json +++ b/config.json @@ -48,6 +48,7 @@ "CONTAINER_TAGS" : ["latest"], "CONTAINER_TAG_CMDS" : [], "CONTAINER_IMAGE_MAP" : {}, + "CONTAINER_COSIGN_KEY" : "", "extravars" : [ "SANITY_TESTED_DISTROS = ''", "BB_HASHSERVE = '${AUTOBUILDER_HASHSERV}'", diff --git a/scripts/run-push-containers b/scripts/run-push-containers index c13f493..9f6f2fa 100755 --- a/scripts/run-push-containers +++ b/scripts/run-push-containers @@ -15,6 +15,9 @@ # CONTAINER_TAG_CMDS - extra shell to populate _EXTRA_TAGS # CONTAINER_VERSION_RECIPE - recipe whose PV provides the version tag # CONTAINER_AUTH_CONFIG - registry auth file staged into the guest +# CONTAINER_COSIGN_KEY - cosign private key path; when set, each +# successfully pushed image is signed with +# cosign (otherwise signing is skipped) # import subprocess @@ -53,6 +56,10 @@ if not auth_config: else: auth_config = "${HOME}/.docker/config.json" +# Signing is conditional: only when CONTAINER_COSIGN_KEY points at a cosign +# private key. When unset, no cosign sysroot is prepared and no signing runs. +cosign_key = utils.getconfigvar("CONTAINER_COSIGN_KEY", ourconfig, args.target, args.stepnum) + utils.printheader("Pushing container images %s" % list(container_images.keys())) script = [ @@ -124,6 +131,99 @@ script = [ " _SKOPEO_READY=1", "}", ] + +# Cosign signing helpers, only emitted when a signing key is configured. +# +# prepare_cosign() is lazy like prepare_skopeo()/start_vm(): it populates the +# cosign-native sysroot once and logs cosign in to every target registry. The +# push paths authenticate via the auth file directly (--dest-authfile / +# --config), but cosign authenticates through its own credential store, so we +# must 'cosign login' explicitly. CONTAINER_REGISTRIES entries carry a +# namespace path (e.g. quay.io/ticotimo), but the auth file is keyed by the +# bare registry host and 'cosign login' only accepts a host authority (a path +# fails with "registries must be valid RFC 3986 URI authorities"). So we strip +# each entry to its host ('${reg%%/*}') for both the credential lookup and the +# login: read auths..auth from the auth file, base64-decode it into +# 'user:password', and feed the password on stdin (--password-stdin) so it +# never appears in the process table. +# +# COSIGN_PASSWORD is exported (defaulting to empty) so cosign reads the +# passphrase from the environment and never falls through to an interactive +# prompt under the autobuilder (no PTY) — without it cosign tries to read the +# passphrase from the terminal and dies with "inappropriate ioctl for device". +# The real value, if the key is encrypted, must be present in the build +# environment. +# +# sign_image() signs by digest ($2) so cosign does not warn about signing a +# mutable tag, while still pinning the index a consumer verifies. The digest +# MUST be the digest the tag resolves to: +# - multi-arch: the manifest-list/index digest, captured at push time from +# 'skopeo copy --all --digestfile' (the digest of the list it pushed). +# - single-arch: the lone manifest digest, which 'skopeo inspect' returns +# correctly (no list, so no platform ambiguity) — used when $2 is empty. +# 'cosign sign --recursive' then signs that digest and every child manifest. +# +# IMPORTANT: do NOT feed 'skopeo inspect' a multi-arch tag for the sign digest. +# Without --raw it resolves to the host platform's CHILD manifest, so signing +# image@ signs one architecture and leaves the index unsigned — +# why signed multi-arch images can still show as "Unsigned". Hence the index +# digest comes from --digestfile, not inspect. +if cosign_key: + login_block = """ for _reg in %s; do + _host=${_reg%%%%/*} + _creds=$(python3 -c 'import base64,json,os,sys +a=json.load(open(os.path.expandvars(sys.argv[1]))) +e=a.get("auths",{}).get(sys.argv[2]) or {} +t=e.get("auth") +sys.stdout.write(base64.b64decode(t).decode() if t else "")' "%s" "$_host") + if [ -z "$_creds" ]; then + echo "WARNING: no credentials for $_host in auth file, skipping cosign login" + continue + fi + _cuser=${_creds%%%%:*} + _cpass=${_creds#*:} + printf '%%s' "$_cpass" | oe-run-native cosign-native cosign login "$_host" -u "$_cuser" --password-stdin + done""" % (" ".join(registries), auth_config) + script += [ + "export COSIGN_PASSWORD=\"${COSIGN_PASSWORD:-}\"", + "_COSIGN_READY=0", + "prepare_cosign() {", + " if [ \"$_COSIGN_READY\" = 1 ]; then return 0; fi", + " bitbake cosign-native -c addto_recipe_sysroot", + login_block, + " _COSIGN_READY=1", + "}", + # Track digests already signed in this run so the same artefact is + # signed only once (see sign_image). + "_SIGNED_REFS=\"\"", + # $1 = /: just pushed; $2 = its digest (the index + # digest from 'skopeo copy --digestfile' on the multi-arch path). When + # $2 is empty (single-arch), resolve the lone manifest digest with + # 'skopeo inspect' — correct there since there is no list. oe-run-native + # prints 'Getting sysroot...' to stdout, so grep the digest out. + "sign_image() {", + " prepare_skopeo", + " prepare_cosign", + " _SIG_DIGEST=\"$2\"", + " if [ -z \"$_SIG_DIGEST\" ]; then _SIG_DIGEST=$(oe-run-native skopeo-native skopeo inspect --authfile %s --format '{{.Digest}}' docker://$1 | grep -oE 'sha256:[0-9a-f]{64}' | tail -n1); fi" % auth_config, + " if [ -z \"$_SIG_DIGEST\" ]; then echo \"WARNING: could not resolve digest for $1, skipping cosign sign\"; return 0; fi", + " _SIG_REF=\"${1%:*}@${_SIG_DIGEST}\"", + " case \" $_SIGNED_REFS \" in *\" $_SIG_REF \"*) return 0 ;; esac", + " _SIGNED_REFS=\"$_SIGNED_REFS $_SIG_REF\"", + # Sign by digest; --recursive also signs each child manifest. Treat a + # transparency-log conflict ('already exists' / HTTP 409, which also + # recurs on rebuilds that reproduce the same digest) as success so + # signing stays idempotent instead of aborting the step. + " _sign_out=$(oe-run-native cosign-native cosign sign --recursive --key %s \"$_SIG_REF\" 2>&1) || {" % cosign_key, + " case \"$_sign_out\" in", + " *\"already exists\"*|*createLogEntryConflict*) echo \"cosign: $_SIG_REF already in transparency log, treating as signed\" ;;", + " *) echo \"$_sign_out\" >&2; return 1 ;;", + " esac", + " }", + " echo \"$_sign_out\"", + "}", + ] + tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, args.stepnum) version_recipe = utils.getconfigvar("CONTAINER_VERSION_RECIPE", ourconfig, args.target, args.stepnum) for recipe, image in container_images.items(): @@ -205,12 +305,19 @@ for recipe, image in container_images.items(): script.append(" echo \"WARNING: %s did not build (no OCI image at $_OCI_MULTIARCH_OUTPUT), skipping push\"" % recipe) script.append(" else") script.append(" prepare_skopeo") + # --digestfile records the index digest skopeo pushed, so signing can pin + # it directly (see sign_image); only needed when signing is enabled. + digestfile = ' --digestfile "$_DGSTFILE"' if cosign_key else "" + if cosign_key: + script.append(" _DGSTFILE=$(mktemp)") # tiny tmp file, assumes autobuilder workdir is ephemeral for registry in registries: script += [ " for _tag in $_TAGS; do", - " oe-run-native skopeo-native skopeo copy --all --dest-authfile %s oci:${_OCI_MULTIARCH_OUTPUT} docker://%s/%s:${_tag}" % (auth_config, registry, image), - " done", + " oe-run-native skopeo-native skopeo copy --all%s --dest-authfile %s oci:${_OCI_MULTIARCH_OUTPUT} docker://%s/%s:${_tag}" % (digestfile, auth_config, registry, image), ] + if cosign_key: + script.append(" sign_image %s/%s:${_tag} \"$(cat \"$_DGSTFILE\")\"" % (registry, image)) + script.append(" done") script.append(" fi") script.append("else") # Single-arch: import the ${recipe}-latest-oci artefact into the memres VM @@ -225,8 +332,10 @@ for recipe, image in container_images.items(): " for _tag in $_TAGS; do", " %s-$(arch) vimport ${_DEPLOY_DIR_IMAGE}/%s-latest-oci %s/%s:${_tag}" % (runtime, recipe, registry, image), " %s-$(arch) push %s/%s:${_tag}" % (runtime, registry, image), - " done", ] + if cosign_key: + script.append(" sign_image %s/%s:${_tag}" % (registry, image)) + script.append(" done") script.append(" fi") script.append("fi") # Tear-down is handled by the EXIT trap installed above. From patchwork Mon Jul 6 22:05:41 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF47BC43458 for ; Mon, 6 Jul 2026 22:07:24 +0000 (UTC) Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.162443.1783375636044197810 for ; Mon, 06 Jul 2026 15:07:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=IkmHEHMf; spf=pass (domain: gmail.com, ip: 209.85.210.172, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-842338c18e0so2875225b3a.1 for ; Mon, 06 Jul 2026 15:07:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375635; x=1783980435; darn=lists.yoctoproject.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=CamkKx/oRJKQhapFWksRTOYDHnr9A2ItvVKsZ4QUceA=; b=IkmHEHMfgA+0Wt4ZpNWPM/2FOpfieznzfoyy290hcIda43HdG72iLgtQCp5tmbgg6n 6VqEfL31VL9RWTVpKSd36M3zvnUSRnXXk5ATEpnTCe/JQKi4XuAKboPixGb9Nlg1CJD2 Itj0fDVr2gK6ysTxe4XBwAhbyQmxTOQEoh/hBXN9pExtwLJv0gpws3pdDO8e6UwKbgjz rMi4KvLCIyPuCJj0DiA7w+otqElwo7QeW4vM4xt2vThLaLgkGPA6gBshUJpCNg3GvP94 Q/Bvza7qlONyHxwidEuv/UI7ITiZQi6MvDJZJ1C1rvNaVVMyIOFdY9hggwcGmgOi8YJk 9BxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375635; x=1783980435; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=CamkKx/oRJKQhapFWksRTOYDHnr9A2ItvVKsZ4QUceA=; b=i9tMWh279wvlJcT0VxyiCWhtQjrdYTLH/JwzG3mSvFIJQ9gXs5v+21PnZ3a2PzPOSY s5lxoCxjE4eTIuGRBU0l4x3PFb5PAuiXUtnwQOyJRp4txlV++K3db+zJzoT29rVo775t /voM+Rb6/5qgFGeQJRDFudwXrWNr+w8e3aOz7hW3sAPecxhK/rdIj0g6X070cXopo9m9 IUkyk4s+pm0uf82oSswghXOirQVaHFkBQ6/MnuvyTQNK7JMIs/QdBpT5VHLtoezne7wN W0V5X/RVlnYdNTRal7zgOMvo2S2zm4lCMLcb3oEfYTe5o83WLTFCP4F+AaCL2HelKGVN HHHg== X-Gm-Message-State: AOJu0Yy5/zP9lTSQkXjb/JWhoFhkPkS2dP4NqrIsto+ak4ZYtEJjTWFP eiSjMejuPc2wlxn9W6yWT+/EyVebQlfwrBsRsExwv5B2DkrRSuRG2tI/8I70TQ== X-Gm-Gg: AfdE7ck3IXn+utlYUVkSyyzgolRrJ2dpSvSQ4sG4PeFh2N6rgbmTbU09pDCKzieWv5L WGigGK8IG0zeCr8EbljVJcFS25m6HyM5/0gIb1KKGBh5/ilnosOjv2JZY8Evk8gXRQsBFlf8+MM b92gVD0vwSHDf62WT3BEGMNF76SC/GDcJV5q23glAFB9id4WKKwZCQOLdKHFVXKQgjK0Jl0gHR2 WDWKueukeby8hkMfIGe2LhPtUTrbuVKx3xXcpVoPyapGDdsHtOdrY6nGxWFV/S62lV6WNn2eOoL PuLsSB4tqWM8vbjJKF5+9vf0BkoQq7C/UgTVQBcV34XOvn0OVq7fgencrvVlbO1IpLaOmSgC56d MUDX0jxq7w8u+uh+pjcjicQxzbnjUZHby60V7wDgwL8rTM72+7SXbP5rtookknIn+VHo7E7rDoK cuWV775JY6BsN9lvrN+OCFyGP2fT4OaASafx+rh30zPo6I0addwsOn2GZm9uMRG9nryVT1E1JuT nxgFQdH31oi X-Received: by 2002:a05:6a00:3686:b0:842:5a8d:303a with SMTP id d2e1a72fcca58-84826e83eabmr2415677b3a.37.1783375633078; Mon, 06 Jul 2026 15:07:13 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.07.08 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:07:09 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 09/11] scripts/run-push-containers: add SPDX SBOM attestation Date: Mon, 6 Jul 2026 15:05:41 -0700 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:07:24 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4439 Attach a *.rootfs.spdx.json as an in-toto attestation to either the top-level container if it is single-arch or to each arch container image if it is multi-arch. Introduce CONTAINER_COSIGN_PUB which is the cosign public key path; when set, each attestation is verified with 'cosign verify-attestation' after it is attached Signed-off-by: Tim Orling --- scripts/run-push-containers | 106 +++++++++++++++++++++++++++++++++++- 1 file changed, 104 insertions(+), 2 deletions(-) diff --git a/scripts/run-push-containers b/scripts/run-push-containers index 9f6f2fa..debf83f 100755 --- a/scripts/run-push-containers +++ b/scripts/run-push-containers @@ -17,7 +17,14 @@ # CONTAINER_AUTH_CONFIG - registry auth file staged into the guest # CONTAINER_COSIGN_KEY - cosign private key path; when set, each # successfully pushed image is signed with -# cosign (otherwise signing is skipped) +# cosign (otherwise signing is skipped). Also +# gates SPDX SBOM attestation: each pushed image +# gets a 'cosign attest --type spdxjson' +# attestation of its SPDX SBOM (per-arch on the +# multi-arch path, top manifest on single-arch). +# CONTAINER_COSIGN_PUB - cosign public key path; when set, each +# attestation is verified with +# 'cosign verify-attestation' after it is made # import subprocess @@ -59,6 +66,9 @@ if not auth_config: # Signing is conditional: only when CONTAINER_COSIGN_KEY points at a cosign # private key. When unset, no cosign sysroot is prepared and no signing runs. cosign_key = utils.getconfigvar("CONTAINER_COSIGN_KEY", ourconfig, args.target, args.stepnum) +# Public key for verifying attestations; when unset, attestations are made but +# not verified. +cosign_pub = utils.getconfigvar("CONTAINER_COSIGN_PUB", ourconfig, args.target, args.stepnum) utils.printheader("Pushing container images %s" % list(container_images.keys())) @@ -193,6 +203,13 @@ sys.stdout.write(base64.b64decode(t).decode() if t else "")' "%s" "$_host") login_block, " _COSIGN_READY=1", "}", + # Resolve the lone manifest digest of a single-arch reference. Correct + # only when there is no manifest list (single-arch / child manifest); + # the multi-arch path passes explicit digests instead. oe-run-native + # prints 'Getting sysroot...' to stdout, so grep the digest out. + "resolve_digest() {", + " oe-run-native skopeo-native skopeo inspect --authfile %s --format '{{.Digest}}' docker://$1 | grep -oE 'sha256:[0-9a-f]{64}' | tail -n1" % auth_config, + "}", # Track digests already signed in this run so the same artefact is # signed only once (see sign_image). "_SIGNED_REFS=\"\"", @@ -205,7 +222,7 @@ sys.stdout.write(base64.b64decode(t).decode() if t else "")' "%s" "$_host") " prepare_skopeo", " prepare_cosign", " _SIG_DIGEST=\"$2\"", - " if [ -z \"$_SIG_DIGEST\" ]; then _SIG_DIGEST=$(oe-run-native skopeo-native skopeo inspect --authfile %s --format '{{.Digest}}' docker://$1 | grep -oE 'sha256:[0-9a-f]{64}' | tail -n1); fi" % auth_config, + " if [ -z \"$_SIG_DIGEST\" ]; then _SIG_DIGEST=$(resolve_digest $1); fi", " if [ -z \"$_SIG_DIGEST\" ]; then echo \"WARNING: could not resolve digest for $1, skipping cosign sign\"; return 0; fi", " _SIG_REF=\"${1%:*}@${_SIG_DIGEST}\"", " case \" $_SIGNED_REFS \" in *\" $_SIG_REF \"*) return 0 ;; esac", @@ -222,6 +239,46 @@ sys.stdout.write(base64.b64decode(t).decode() if t else "")' "%s" "$_host") " }", " echo \"$_sign_out\"", "}", + # Make sure an image's SPDX SBOM exists before we attest it. $1 is the + # expected spdx.json path, $2 the bitbake target that produces it + # (mc:: on the multi-arch path, plain single-arch). + # create_image_sbom_spdx runs by default for single-arch builds, so the + # file is normally already there; the multi-arch index recipe doesn't + # pull in the per-arch SBOM tasks, so we generate them on demand. + "ensure_spdx() {", + " if [ ! -e \"$1\" ]; then", + " echo \"SPDX SBOM $1 missing, generating: bitbake -c create_image_sbom_spdx $2\"", + " bitbake -c create_image_sbom_spdx $2 || true", + " fi", + " if [ ! -e \"$1\" ]; then echo \"WARNING: SPDX SBOM $1 still missing, skipping attestation\"; return 1; fi", + "}", + # Track refs already attested so the same digest is attested once. + "_ATTESTED_REFS=\"\"", + # Attest an SPDX SBOM against a digest-pinned image ref. $1 = + # /@sha256:...; $2 = spdx.json path. Idempotent: a + # transparency-log conflict on rebuilds is treated as success. When a + # public key is configured the attestation is verified afterwards. + "attest_image() {", + " prepare_skopeo", + " prepare_cosign", + " case \" $_ATTESTED_REFS \" in *\" $1 \"*) return 0 ;; esac", + " _ATTESTED_REFS=\"$_ATTESTED_REFS $1\"", + " _att_out=$(oe-run-native cosign-native cosign attest --key %s --type spdxjson --predicate \"$2\" \"$1\" 2>&1) || {" % cosign_key, + " case \"$_att_out\" in", + " *\"already exists\"*|*createLogEntryConflict*) echo \"cosign: attestation for $1 already in transparency log\" ;;", + " *) echo \"$_att_out\" >&2; return 1 ;;", + " esac", + " }", + " echo \"$_att_out\"", + ] + if cosign_pub: + script += [ + " oe-run-native cosign-native cosign verify-attestation --key %s --type spdxjson \"$1\" >/dev/null \\" % cosign_pub, + " && echo \"cosign: verified attestation for $1\" \\", + " || { echo \"ERROR: attestation verification failed for $1\" >&2; return 1; }", + ] + script += [ + "}", ] tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, args.stepnum) @@ -259,6 +316,13 @@ for recipe, image in container_images.items(): " *) _DISTRO_VERSION=\"$_DISTRO_VERSION_RAW\" ;;", "esac", "_DEPLOY_DIR_IMAGE=$(echo \"$_BBENV\" | awk -F'\"' '/^DEPLOY_DIR_IMAGE=/{ print $2; exit }')", + # MACHINE locates the single-arch SPDX SBOM (-.rootfs.spdx.json). + "_MACHINE=$(echo \"$_BBENV\" | awk -F'\"' '/^MACHINE=/{ print $2; exit }')", + # TOPDIR and the oci-multiarch plain vars drive per-arch SBOM + # attestation below; the latter two are empty for non-multiarch recipes. + "_TOPDIR=$(echo \"$_BBENV\" | awk -F'\"' '/^TOPDIR=/{ print $2; exit }')", + "_OCI_MULTIARCH_RECIPE=$(echo \"$_BBENV\" | awk -F'\"' '/^OCI_MULTIARCH_RECIPE=/{ print $2; exit }')", + "_OCI_MULTIARCH_PLATFORMS=$(echo \"$_BBENV\" | awk -F'\"' '/^OCI_MULTIARCH_PLATFORMS=/{ print $2; exit }')", # Only set (non-empty) when the recipe inherits # oci-multiarch; doubles as the multiarch marker and # the OCI layout path for skopeo below. @@ -318,6 +382,34 @@ for recipe, image in container_images.items(): if cosign_key: script.append(" sign_image %s/%s:${_tag} \"$(cat \"$_DGSTFILE\")\"" % (registry, image)) script.append(" done") + if cosign_key: + # Attest each child manifest with that arch's SPDX SBOM. The arch -> + # (multiconfig, machine) mapping lives in oci-multiarch.bbclass flags + # (OCI_MULTIARCH_MC/MACHINE[]); query them per platform with + # bitbake-getvar so an overridden mapping is still honoured. The child + # manifest digest is the per-arch source image's manifest digest, which + # the class copies verbatim into the index and skopeo copy --all pushes + # unchanged — so we attest @. + script.append(" for _plat in $_OCI_MULTIARCH_PLATFORMS; do") + script.append(" _mc=$(bitbake-getvar -q -r %s --value -f $_plat OCI_MULTIARCH_MC)" % recipe) + script.append(" _machine=$(bitbake-getvar -q -r %s --value -f $_plat OCI_MULTIARCH_MACHINE)" % recipe) + script.append(" if [ -z \"$_mc\" ] || [ -z \"$_machine\" ]; then echo \"WARNING: no mc/machine for platform $_plat, skipping attestation\"; continue; fi") + script.append(" _pdir=${_TOPDIR}/tmp-${_mc}/deploy/images/${_machine}") + script.append(" _mspdx=${_pdir}/${_OCI_MULTIARCH_RECIPE}-${_machine}.rootfs.spdx.json") + # Locate the per-arch source OCI layout (same name patterns the class + # searches) and read its single manifest digest = the child digest. + script.append(" _mdig=\"\"") + script.append(" for _oci in ${_OCI_MULTIARCH_RECIPE}-latest-oci ${_OCI_MULTIARCH_RECIPE}-${_machine}-latest-oci ${_OCI_MULTIARCH_RECIPE}-oci; do") + script.append(" if [ -e \"${_pdir}/${_oci}/index.json\" ]; then") + script.append(" _mdig=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))[\"manifests\"][0][\"digest\"])' \"${_pdir}/${_oci}/index.json\")") + script.append(" break") + script.append(" fi") + script.append(" done") + script.append(" if [ -z \"$_mdig\" ]; then echo \"WARNING: no source OCI index for $_plat under $_pdir, skipping attestation\"; continue; fi") + script.append(" ensure_spdx \"$_mspdx\" \"mc:${_mc}:${_OCI_MULTIARCH_RECIPE}\" || continue") + for registry in registries: + script.append(" attest_image %s/%s@${_mdig} \"$_mspdx\"" % (registry, image)) + script.append(" done") script.append(" fi") script.append("else") # Single-arch: import the ${recipe}-latest-oci artefact into the memres VM @@ -327,6 +419,12 @@ for recipe, image in container_images.items(): script.append(" echo \"WARNING: %s did not build (no OCI image at $_OCI_IMAGE), skipping push\"" % recipe) script.append(" else") script.append(" start_vm") + if cosign_key: + # create_image_sbom_spdx runs by default for single-arch image builds, + # so the SBOM is normally already deployed; ensure_spdx regenerates it + # only if absent. Single manifest -> attest the digest the tag resolves + # to (resolve_digest, also used by sign_image). + script.append(" _SPDX=${_DEPLOY_DIR_IMAGE}/%s-${_MACHINE}.rootfs.spdx.json" % recipe) for registry in registries: script += [ " for _tag in $_TAGS; do", @@ -335,6 +433,10 @@ for recipe, image in container_images.items(): ] if cosign_key: script.append(" sign_image %s/%s:${_tag}" % (registry, image)) + script.append(" if ensure_spdx \"$_SPDX\" \"%s\"; then" % recipe) + script.append(" _ADIG=$(resolve_digest %s/%s:${_tag})" % (registry, image)) + script.append(" [ -n \"$_ADIG\" ] && attest_image %s/%s@${_ADIG} \"$_SPDX\"" % (registry, image)) + script.append(" fi") script.append(" done") script.append(" fi") script.append("fi") From patchwork Mon Jul 6 22:05:42 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB475C43602 for ; Mon, 6 Jul 2026 22:07:24 +0000 (UTC) Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.161490.1783375638008241500 for ; Mon, 06 Jul 2026 15:07:18 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=FZ5CDe2Z; spf=pass (domain: gmail.com, ip: 209.85.210.177, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-84347ad88edso4793317b3a.1 for ; Mon, 06 Jul 2026 15:07:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375637; x=1783980437; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=E2FO7Mm6qMpb9g9o9CU8GObzrUu2x4qgPD5lv75UqCw=; b=FZ5CDe2ZFsAvKEs6FSHqwKnQJEnfflorJSYxoKUnL79FJp+BhPCidMpkisocdFh4ur 375S5CqyVvHGGIIibpjJKWt7NPsK/VnaLvhn/MrHF75KNNAx4EnGoKZuLcdMXVm7XCWz ermr2RA+QxHTLxVT/qrkmbsWL+QhncSDT6mhOIh0jTjaKOxAUq6FCCkZaSKJ13u67RN7 pmR8bdAP9OEA6JqNDGK6xmx90dgwNabwnzEhVxevR2qr/L3qqB/1TJsr2t1PsGTM6yjX 86yFWtqCaRKrxom9Vv3fpGpcx/6FcA8C6lceaHpu58KCQhsYP6a1kyCs2PyGW5OkPyS4 OTTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375637; x=1783980437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=E2FO7Mm6qMpb9g9o9CU8GObzrUu2x4qgPD5lv75UqCw=; b=A5k0SFCKppr0aExOrKgPKcMP1oyX5XV6iHIoGMLdXy0qncw/KqFK2EJKVV8fxZwxC6 s/ArdlZPx5sUyyYQUHla5GfP5uIIk90CbIRfE+E6gjMnIWoXrfj41Lv36x2qS5Avxu9I 462Z6IyYzpjqlOTNUrhviBNgNuzJ+FfWBslspG3i0st0oNB3qCqi0e/fPTEJFl7tOlcb e9KVbn3AtHH0lD8zobg/vpOHwSSKj27dH0up/GV76XI9CxEx3oTmSbrRaAOmNXjNS6j0 7EQ4W3+K/nmHSzvloef4eGOT/qcXV6zlvkz/8d4dh/NJc+yVsM7EvpVZRH6uP4K5aDBg jABA== X-Gm-Message-State: AOJu0YzJvCGCG76U0HXvgeMNxOxI8MJ9AIlSBShsKzUuuesVA29llzUz KSc6Px3W2/cYqGnm+NpwdAaVhiFSUuf6TDkrW/BAuMEXRaveb5CJ3kEs5Hh5uA== X-Gm-Gg: AfdE7ckl0CaHAFuWEyLA1WRwpMo8cOHThMLp/NEMEepBa1wvpeHBPi9ox4CaMqkaBNc edM+GAODexhn/qzB5DSGS3waZ+ph5rV7WHVTc8H7V3PuoL6nsQXqd1S640mLbbyvYT4pTRgDnAf yusFr4rO98V/GwU98gaoR91XAozyj9lhBy3utK7rgqd+IwzGr5NNq50ZzQ5pZCM1wyls9QyYP/Q Y/lNv02MEIJDW7sBbwYVARn5x7xLBdWQ51cjoj7+7TGdhUpogjy+F4WkTLCAxGZXAt0muzirb08 wbyCiRbfGkafB4VuLKmn72VxHT4eg48KUmwIXUsLw7JTFNV9MqhzM5x7NJ0kehaevyqF53os63Y oVCNHvb+QKEK6HFQD5VzoIdnjtQIh8w7yChB9ls1MZmvj75aO2qsuVfPvNX3MHx4cRQXEP7cBcL I8YD37LY5sfYDo88RBhF47obUlvzo0Nn2MBE3wEZh1aS+ZNVG10UxKs+p23o+kWwl6GJJnJ1cz2 ALjKeS1KVEq X-Received: by 2002:a05:6a21:3202:b0:3b3:216b:2743 with SMTP id adf61e73a8af0-3c08ec76eecmr3000721637.4.1783375637040; Mon, 06 Jul 2026 15:07:17 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.07.14 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:07:14 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 10/11] config.json: add -dev container builds to containers-library Date: Mon, 6 Jul 2026 15:05:42 -0700 Message-ID: <016a8a8640de4acf06b678e875e1c74577b0af4f.1783371914.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:07:24 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4440 For some of the containers, we want to provide a -dev version, denoted by '-dev' appended to the tag(s). This runs the container as UID '0' root user and adds a shell. Inspired by dhi.io (Docker Hardened Images). Signed-off-by: Tim Orling --- config.json | 59 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 50 insertions(+), 9 deletions(-) diff --git a/config.json b/config.json index a1eee57..118401a 100644 --- a/config.json +++ b/config.json @@ -1925,37 +1925,78 @@ "shortname" : "Build 'base' multiarch container", "BBTARGETS" : "container-image-multiarch-container-base", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-container-base": "base"}, - "CONTAINER_VERSION_RECIPE" : "base-files" + "CONTAINER_VERSION_RECIPE" : "base-files", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, "step3" : { "shortname" : "Build 'curl' multiarch container", "BBTARGETS" : "container-image-multiarch-app-container-curl", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-curl": "curl"}, - "CONTAINER_VERSION_RECIPE" : "curl" + "CONTAINER_VERSION_RECIPE" : "curl", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, "step4" : { + "shortname" : "Build 'curl' multiarch container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-curl", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-curl": "curl"}, + "CONTAINER_VERSION_RECIPE" : "curl", + "extravars" : ["PACKAGECONFIG:pn-app-container-curl = 'dev'"] + }, + "step5" : { "shortname" : "Build 'python' container", "BBTARGETS" : "container-image-multiarch-app-container-python", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-python": "python"}, - "CONTAINER_VERSION_RECIPE" : "python3" + "CONTAINER_VERSION_RECIPE" : "python3", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, - "step5" : { + "step6" : { + "shortname" : "Build 'python' container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-python", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-python": "python"}, + "CONTAINER_VERSION_RECIPE" : "python3", + "extravars" : ["PACKAGECONFIG:pn-app-container-python = 'dev'"] + }, + "step7" : { "shortname" : "Build 'mosquitto' container", "BBTARGETS" : "container-image-multiarch-app-container-mosquitto", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-mosquitto": "mosquitto"}, - "CONTAINER_VERSION_RECIPE" : "mosquitto" + "CONTAINER_VERSION_RECIPE" : "mosquitto", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, - "step6" : { + "step8" : { + "shortname" : "Build 'mosquitto' container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-mosquitto", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-mosquitto": "mosquitto"}, + "CONTAINER_VERSION_RECIPE" : "mosquitto", + "extravars" : ["PACKAGECONFIG:pn-app-container-mosquitto = 'dev'"] + }, + "step9" : { "shortname" : "Build 'valkey' container", "BBTARGETS" : "container-image-multiarch-app-container-valkey", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-valkey": "valkey"}, - "CONTAINER_VERSION_RECIPE" : "valkey" + "CONTAINER_VERSION_RECIPE" : "valkey", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, - "step7" : { + "step10" : { + "shortname" : "Build 'valkey' container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-valkey", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-valkey": "valkey"}, + "CONTAINER_VERSION_RECIPE" : "valkey", + "extravars" : ["PACKAGECONFIG:pn-app-container-valkey = 'dev'"] + }, + "step11" : { "shortname" : "Build 'nginx' container", "BBTARGETS" : "container-image-multiarch-app-container-nginx", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-nginx": "nginx"}, - "CONTAINER_VERSION_RECIPE" : "nginx" + "CONTAINER_VERSION_RECIPE" : "nginx", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] + }, + "step12" : { + "shortname" : "Build 'nginx' container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-nginx", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-nginx": "nginx"}, + "CONTAINER_VERSION_RECIPE" : "nginx", + "extravars" : ["PACKAGECONFIG:pn-app-container-nginx = 'dev'"] } } }, From patchwork Mon Jul 6 22:05:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 91883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E75E0C44500 for ; Mon, 6 Jul 2026 22:07:24 +0000 (UTC) Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.162445.1783375642618904001 for ; Mon, 06 Jul 2026 15:07:22 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=N50PWKxj; spf=pass (domain: gmail.com, ip: 209.85.215.170, mailfrom: ticotimo@gmail.com) Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-c9e30214d8fso2509447a12.3 for ; Mon, 06 Jul 2026 15:07:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783375642; x=1783980442; darn=lists.yoctoproject.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=7JfXDxZREHAPUlsc7o966yQlfqBpuufpNLSUeatE+H8=; b=N50PWKxj3GvpSFsOJQEYsmWGAGUbC7MRdg1sB+zaOtfwiwgF6aHPv6mRA8xM62iqrv r6CF+TAo/Pbp25qnrjP29Yn+ZmzIstX43qt2cbwXatGHl5eG7ahUq9Ok2KLtehBQJsE6 ub2Ks4Tf0fVHyEJlHU09VDa31LNPtNo4t32UaD0aO2VZhtdKl4knwC0BDAFP+OTX5hTt ztNUbJK/4mOfF/w7PmWhUhYbmqtOyW0fscIdwrpaiDp/bYCJ4LwqXcfMEtveoP1/IH+P qC02j0/EUOHWk2EXXNoe5uswE/doaPgXr4dLNTjMR72jnDV8NRZVawGfmXVcK+iKC/TN zqKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783375642; x=1783980442; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=7JfXDxZREHAPUlsc7o966yQlfqBpuufpNLSUeatE+H8=; b=Oqk0fsmRyBp4/JR5eYaHFgvK2cKkttrvY5cqxlegKE1Y0b4RtyJaMkacTj9mZWkbfL N6xK1tXDBJkEfMMiVNDoyeQEjCQOK3kemMkaPNNTfZ7fXvwu3V1dmp/9gGGVBl0wnTbH 5z1sfASRNtHWo5jOkqyDsTVWgV74+aQdJpYES1DRIaUJfcYdKXlQSTRrCDaXDeMxptPB JmM+QVfS9z/yXMRqbSkl7DUectPo4h3g3y+ekwoejXhLxnLj8cYuE9fmt+lBQvpykFJq Ati8igd1Bn5kHBoYJwIjohZYaHAPvkJXRq6js1DuCzP2JyFnmdBTFXPNNrZNrzo9s0VA L8cA== X-Gm-Message-State: AOJu0YxMYK2QfFYz+rizaII45j2lW0EJa8SOiMMLmCAGfAYXBiTlkOPV 45XwV7ctfjtH+xv3hNT8jkQ3lv5PZP6NC+ALMBMsKzRXypdcZ5DDyUIJoTO9+A== X-Gm-Gg: AfdE7clXBlrFrrb0RrnkvhN7YIku5gDjne999hnC0T7bBj+jNtUJgp33211aNFzWxsY FHJ4xVh+5e2eMWIgG+PJRctS3AOV1QOlZExeJuiFSsxlfCL0mqVL3BMAi6pMQdo7JnNCOGC2mcR Cvh0fMtJJbJdZAUbnCrwavpK5Idrwmh6J7j/2bEqCiP2Hb+nghpKDfZ8brVtk4f/o+7emQ5W8Gp 6XkCOoIFDTc6MT/HwbRDZarCgaFGu4Eq/ikVjHkc6YE/9xhuIUd9Umr9M56/rc6JpKPid0IzzQt UiXqwvQlXq2ItIfqKZH/CO62yFoK6DIoafWn3DH01sFYK/+bSnaC7/YkYugC3vXqQ3C3z/6dLyV EuBej14tojN7+HcIqkTXhjzBFgFJ8EwL2cKyfjcqwUOOzbQICyw+05Zkwu4xGDkgecp5P+KB6Hm 1q4l0+mgK5g6ni19+H9gcDRvwCDCliTFg6GJJh9ckZK0pF0yA+jggYpCpHOQ0s6nCck/vR9a5J0 E9qHdDEpv7m X-Received: by 2002:a05:6a21:a94:b0:3b1:cce5:9140 with SMTP id adf61e73a8af0-3c08ee4f340mr3007352637.33.1783375641864; Mon, 06 Jul 2026 15:07:21 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ca5af7d5b95sm84720a12.3.2026.07.06.15.07.18 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 06 Jul 2026 15:07:18 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v4 11/11] run-push-containers: optionally push -dev tagged containers Date: Mon, 6 Jul 2026 15:05:43 -0700 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 06 Jul 2026 22:07:24 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4441 For containers which set PACKAGECONFIG 'dev', push additional container images with '-dev' suffix to the tags. Inspired by dhi.io (Docker Hardened Images). Signed-off-by: Tim Orling --- scripts/run-push-containers | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/scripts/run-push-containers b/scripts/run-push-containers index debf83f..5dd35af 100755 --- a/scripts/run-push-containers +++ b/scripts/run-push-containers @@ -327,6 +327,20 @@ for recipe, image in container_images.items(): # oci-multiarch; doubles as the multiarch marker and # the OCI layout path for skopeo below. "_OCI_MULTIARCH_OUTPUT=$(echo \"$_BBENV\" | awk -F'\"' '/^OCI_MULTIARCH_OUTPUT=/{ print $2; exit }')", + # Drives the '-dev' tag suffix below (dhi.io convention: same image + # name, tag suffixed) for recipes built with PACKAGECONFIG:pn- = "dev". + # + # $recipe here is the BBTARGET from CONTAINER_IMAGE_MAP, which on the + # multiarch path is the container-image-multiarch wrapper (e.g. + # container-image-multiarch-app-container-nginx) — a different PN + # than app-container-nginx, so 'PACKAGECONFIG:pn-app-container-nginx' + # never touches its PACKAGECONFIG. OCI_MULTIARCH_RECIPE (already + # extracted above) is the underlying app recipe name and is what + # PACKAGECONFIG:pn- actually targets; fall back to $recipe + # itself for single-arch recipes (OCI_MULTIARCH_RECIPE empty there). + "_PC_RECIPE=\"${_OCI_MULTIARCH_RECIPE:-%s}\"" % recipe, + "_PC_BBENV=$(bitbake -e \"$_PC_RECIPE\" 2>/dev/null) || true", + "_PACKAGECONFIG=$(echo \"$_PC_BBENV\" | awk -F'\"' '/^PACKAGECONFIG=/{ print $2; exit }')", "_EXTRA_TAGS=\"\"", ] if version_recipe: @@ -348,6 +362,18 @@ for recipe, image in container_images.items(): script.append( "_TAGS=\"%s $_PV $_DISTRO_CODENAME yocto-$_DISTRO_VERSION $_EXTRA_TAGS\"" % " ".join(static_tags) ) + # 'dev' builds (PACKAGECONFIG:pn- = "dev") push under the same + # image name as production, with every tag suffixed '-dev' (dhi.io + # convention), instead of a separate image name/repo. + script += [ + "case \" $_PACKAGECONFIG \" in", + " *\" dev \"*)", + " _DEVTAGS=\"\"", + " for _t in $_TAGS; do _DEVTAGS=\"$_DEVTAGS ${_t}-dev\"; done", + " _TAGS=\"$_DEVTAGS\"", + " ;;", + "esac", + ] # Detect multiarch vs single-arch per recipe and push accordingly, # but only for an image whose build actually produced its OCI artefact. #