From patchwork Tue Jul 28 01:40:34 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 326E1C531D0 for ; Tue, 28 Jul 2026 01:41:17 +0000 (UTC) Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.2414.1785202876887186949 for ; Mon, 27 Jul 2026 18:41:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=D+WFHMnJ; spf=pass (domain: gmail.com, ip: 209.85.210.171, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-84874b52eabso4482927b3a.0 for ; Mon, 27 Jul 2026 18:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202876; x=1785807676; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=qa4bGU/EjmW63DrYG02BiMdBc07C3ZfwGhsG8XH0vaY=; b=D+WFHMnJZcPLlIajz2oAfZJ5FfpAFPs6aoRue2WyNL0Vx9RyJFtYqisSG6o1G1w+/8 c6AE3D5Q/q0FvpjS+BAVSXou095+j8zZV8WKvBzbkV7dGT/OEr1cs4Af3j+dfI9pEQnS rfwCoUMWwSVyDhSfvCVXpefAkFpsFrziawvuFUlcZzYCFCDy+cP45rX7qezQEGyN6Xg/ lbBayUO0huFDj1qKbnWqa3CJf/M+PsAI1pDrDLGnxerLgxDVbJ3YcyeyES0hVTjNxU2n I6hknlxQtDct8uS+kxtA1a1/2Lf8SwYhOr7QnviOBleSblz7ESq6B/P5kbruDWr5Ejky ofIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202876; x=1785807676; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=qa4bGU/EjmW63DrYG02BiMdBc07C3ZfwGhsG8XH0vaY=; b=GGvyVGUt9KG+pUhyG0z4YZho2DMlhN5S9A6K+1MD4tOLO8cDSkYxciKW9icIm9tInO MXyTy1VVJPXI9EefVJbEWe6XKKMdZARiw8NvmJCvp5aMlBgBGsHgSqjdb5TnPgJgPXY4 +gw8VC67yjEWwMCE/u6mtOxXg1ej0LbmLOBrhzavL1QMBoy9L/pGZeoHYfG3I8Etkuvr 7Q1Q/j8myemFazI2tvfnU9EpP+42jPuL6EV9FK0J1hZ2omNlgfzR37tvsgQxyi8kmQs+ hZWVRCRktVr+hROmHQYIp79bNUBqdCjKfHWw6eCLYb1XOoSt4MagdV3rsPU2NI9+CJOz 4wdw== X-Gm-Message-State: AOJu0YyRyFS8tXdzTsH/xUGOgT+AH+H8gbuQOkDnGra8Z2rrGze/YQJo 5FRJcGJM8oK0a6ObpYF9K15DWk0tq7i8PiGlDLhPs58w9pKFay93x5sUXrLxYw== X-Gm-Gg: AR+sD10gAVjx+plO+vMg6Qwg+FsacY0e6bHhGxilPBJqppsCjgODZE3KUFHvAYz5cgK AmIsnVAAovVQTYQVKr8O/BXnRzrprEFWrLSq3WsGsuYD/iml7gePdNWkpxSCK78pYrRu9NKN50q AtQy4mqQnZ/GfJ8rVFguBjmrCnZwRE7cCAZFi9KkI//lkqu3BpidCWI2FZ2DS8V5uk5ai+p8rGg ix413odoWzR6FcnPh48IIFkDv8YiQ2xJW/qIhn1y0KvDfT+3XG+WpQw3kreONNtS0QnjQo9CJKN hGU3VE4bKjuI5DWNCSD92vhB7FYaopXmAh03vOpAKtJ71IopByUrd74pVwoWzOKjhiEtSQIe9td 57GdW3Ss3SlQl73NsYS8A1uSNhuvwwR4Po1EOSiD+Ou70y543c3nCAbcci3aXrPQjH9DWJ2yVoF XA+PcO24T3kiTgP+J2LB8EYdFoMjuSyK5UV7AXJ7a9oMLiKiM5yupkf/rdZsnrXzKgmKt+P4Q+a Hj5kQCGCKH0 X-Received: by 2002:a05:6a00:3e20:b0:847:893f:2d0c with SMTP id d2e1a72fcca58-84e931b4d0dmr296360b3a.5.1785202876252; Mon, 27 Jul 2026 18:41:16 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.15 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:15 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 01/12] scripts: add vcontainer-tarball setup, integration, and publishing Date: Mon, 27 Jul 2026 18:40:34 -0700 Message-ID: <0de865ea7a35bb75d9c1cf35847bea1b0b5ff514.1785198322.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:17 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4540 Introduce the vcontainer-tarball SDK plumbing used by container build jobs. The vcontainer-tarball is a meta-virtualization-derived SDK (modelled after buildtools-tarball) that ships the container build toolchain so worker jobs do not need to rebuild it for every step. * scripts/utils.py: add setup_vcontainer_tarball(), and add an env_glob keyword argument to setup_tools_tarball() and enable_tools_tarball() so the vcontainer-tarball can source its specific environment-setup-ci file rather than the universal glob. * scripts/run-config: source the vcontainer-tarball environment for build-targets when "vcontainer" parameter is passed in, similar to "extratools" behavior. * scripts/publish-artefacts: publish the vcontainer-tarball artefact so downstream test jobs can fetch a stable SDK. AI-Generated: Claude Cowork Opus 4.8 Signed-off-by: Tim Orling --- scripts/publish-artefacts | 5 +++++ scripts/run-config | 11 +++++++++++ scripts/utils.py | 20 ++++++++++++++++---- 3 files changed, 32 insertions(+), 4 deletions(-) diff --git a/scripts/publish-artefacts b/scripts/publish-artefacts index e56e131..0e820e9 100755 --- a/scripts/publish-artefacts +++ b/scripts/publish-artefacts @@ -146,5 +146,10 @@ case "$target" in sha256sums $TMPDIR/deploy/images/qemux86-64 cp -R --no-dereference --preserve=links $TMPDIR/deploy/images/qemux86-64/*qemux86* $DEST/patchtest ;; + "vcontainer-tarball") + mkdir -p $DEST/vcontainer-tarball + sha256sums $TMPDIR/deploy/sdk + cp -R --no-dereference --preserve=links $TMPDIR/deploy/sdk/*vcontainer* $DEST/vcontainer-tarball + ;; esac diff --git a/scripts/run-config b/scripts/run-config index e896234..90a5996 100755 --- a/scripts/run-config +++ b/scripts/run-config @@ -153,6 +153,17 @@ else: if args.phase == "init" and args.stepname == "buildtools": sys.exit(0) +vcontainer = utils.getconfigvar("vcontainer", ourconfig, args.target) +if jcfg: + if vcontainer: + addentry("vcontainer", "Setup vcontainer tarball", "init") +elif vcontainer: + # vcontainer is opt-in per target via the "vcontainer" config variable, + # so this is a no-op for targets which don't set it + utils.setup_vcontainer_tarball(ourconfig, args.target, args.builddir + "/../vcontainer-tarball") + if args.phase == "init" and args.stepname == "vcontainer": + sys.exit(0) + extratools = utils.getconfigvar("extratools", ourconfig, args.target) if jcfg: if extratools: diff --git a/scripts/utils.py b/scripts/utils.py index a4dd12e..bddc715 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -456,8 +456,8 @@ def sha256_file(filename): pass return method.hexdigest() -def enable_tools_tarball(btdir, name): - btenv = glob.glob(btdir + "/environment-setup*") +def enable_tools_tarball(btdir, name, env_glob="/environment-setup*"): + btenv = glob.glob(btdir + env_glob) print("Using %s %s" % (name, btenv)) # We either parse or wrap all our execution calls, rock and a hard place :( with open(btenv[0], "r") as f: @@ -474,6 +474,18 @@ def enable_tools_tarball(btdir, name): if line in os.environ: del os.environ[line] +# Unlike buildtools (a host/worker property, keyed by worker name globs), +# the vcontainer-tarball is only needed by specific jobs (e.g. +# containers-library), so it is keyed off the target/builder via a +# per-target "vcontainer" config variable, following the extratools pattern. +def setup_vcontainer_tarball(ourconfig, target, vcdir, checkonly=False): + vctarball = getconfigvar("vcontainer", ourconfig, target) or None + + if checkonly: + return vctarball + + setup_tools_tarball(ourconfig, vcdir, vctarball, name="vcontainer-tarball", env_glob="/environment-setup-ci") + def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): bttarball = None if "buildtools" in ourconfig and workername: @@ -488,7 +500,7 @@ def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): setup_tools_tarball(ourconfig, btdir, bttarball) -def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): +def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools", env_glob="/environment-setup*"): btenv = None if bttarball: @@ -557,7 +569,7 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): if not os.path.exists(btdir): print("Extracting %s %s" % (name, bttarball)) subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) - enable_tools_tarball(btdir, name) + enable_tools_tarball(btdir, name, env_glob) def get_string_from_version(version, milestone=None, rc=None): """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists, From patchwork Tue Jul 28 01:40:35 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93652 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DD69C531D0 for ; Tue, 28 Jul 2026 01:41:27 +0000 (UTC) Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.2387.1785202879208383769 for ; Mon, 27 Jul 2026 18:41:19 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=aVnNUYxd; spf=pass (domain: gmail.com, ip: 209.85.210.180, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-8486672f03cso3251210b3a.0 for ; Mon, 27 Jul 2026 18:41:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202878; x=1785807678; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=iey3Ghxi2FM/GI7svCR+Mu4x1xYuTD6DsWll+0T6rBM=; b=aVnNUYxdfp5i9PWWL+oCpFGTjjdC5ixD+T6jRIa7ytIFIrNzYqrE93fQOdVOwAwolE fu3v+LgwmlPTRRmStHqS+c6YfREfhda51nmXlQl0lucBa2+hbeFOmG84zTEFwyrxyNZ5 cpyjd5eY1DPdOjfQD9S1PKkiUBxlvUlJZqHffD2SHciw1Pj3LuoY9wO5HL8p7gzO6XQ2 6GmT2fKUJspBSckClRzTrcj1wWomiKzdn/Tjej7MAkP6Ri9rTuMUSxU0p5xkADV73zp9 jo9frebzeM7XWFFVmEOkfAgwgK7WGKTZLZzV6ng9MbQreodlLhBIlsU1WOEEFvuYiBXF gDuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202878; x=1785807678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=iey3Ghxi2FM/GI7svCR+Mu4x1xYuTD6DsWll+0T6rBM=; b=AZaAjEQWUfJGu8zzDdgPv6Ilzc8cF6rsYeSkqqGg19WFXOCFwliGsJw9nM6gceIE7z rtxT1gXGvkgzj5E00Jra5cogyUhg1dUE376hf14jarcLhD7AruUIIjJo+T26u+vPs31/ l51zVq9ECvNTe9VhCd+kD4en71PIvL/Bfe3pRD5neBsbR3srpwJ44aWF0AoFW/YQJbFS ZJbiHIyeCyvs//zjQVVpUo+Ggzt3Dne1UInsQkkEkxsdSss8cE2dYjMLNL3P2Q3UEAu7 IWlwPSJ9aRk5eRx0XN+aC8JLakO8aHXROuvPUJCT0h21UUHCow1fF+pJUZRXYIk0pIcC 3LXg== X-Gm-Message-State: AOJu0YykgTQ4mqiMyVIjMVlGjIDz9Yeg9x3mIXv3Dk+9v3QYhcwLUHw2 TVQf388rDbHqVvhc7J7I/XY8ZuRgBb/4KKUetkqrFVRv3b9zYEOBuHxGreZvMA== X-Gm-Gg: AR+sD11Du/6UuVrCSJEJAONIQ2vt61whvcVUjhNsUH05WZ6clogzstZEolaQaf8nCf1 7uZ4qQxAz9HUPSHV6Sk4JyEBoAQzFZE7Z4r1J/fnWnLY/UVk0hJTRN9mOkJE5yyr73h+1Y7vXNC 71tYY4ljvqOw298zEuMB8r/G9YYYb+lmSYFlu2cgC7Vlnak6ilCNnqzHlN+bVsfeH0i3Lu+ns/A r9MxuzIY27iWzlYzR6xa65HBNT2N/rrSzKkfM+YXUkw4EgJHSrv/4OF72btUgK55EmP4O4z8TSs jGuTb0BxfPo6E6XWOuil0/TKcGXdgBBGkEgjxNPuW4aj6HKcnImCj73HQgzHAnBSgXpBqISfYJ1 xnGyrPEmw3JrReNVW3CKBoc1+3/GuZsYXvtDF70rYXcZTR0kFNbViXk1C32S02bK6pXCZw0+K9M zXfmwE06RI2uqYA/+sDbVTf6VAlsvgzes9RCSpLWM785qRbq5ERrI/nvAb8lpMbQJapYO2b+m0c aUZzaS03JwvzH8hosOyTfzMl8URJ9y9s05ix5qvijmDEuN90+OWlDdiYdaze2kbSLBBrw8= X-Received: by 2002:a05:6a00:6c93:b0:848:2f7a:2e5a with SMTP id d2e1a72fcca58-84e9340e985mr321809b3a.73.1785202878546; Mon, 27 Jul 2026 18:41:18 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.17 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:17 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 02/12] config.json: add vcontainer-tarball build target Date: Mon, 27 Jul 2026 18:40:35 -0700 Message-ID: <8017979c637a4cbcce5e061d2e72c6e02380cfd5.1785198322.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:27 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4541 Add the vcontainer-tarball build target which produces the meta-virtualization SDK used by downstream container build/test jobs. Modelled after the buildtools-tarball target. Following meta-virtualization/docs/build-profiles.md add "require conf/distro/include/meta-virt-host.conf" to extravars, which appends required DISTRO_FEATURES and sets appropriate BBMULTICONFIG. The target places the extravars in the per-step config, and includes a publish-artefacts step so the resulting SDK tarball is staged for reuse by container-tests and other consumers. Since meta-virtualization is the first layer removed by 'remove-layers', our BBMULTICONFIGs become invalid immediately after layer removal and throw an error. In scripts/run-config, EXTRACMDS runs after BBTARGETS, but before 'remove-layers'. Use 'sed' to remove the "require ...meta-virt-host.conf" line. The error in the 'remove-layers' step can be summarized as: bb.parse.ParseError: ParseError at /home/pokybuild/yocto-worker/vcontainer-tarball/build/layers/openembedded-core/meta/conf/bitbake.conf:824: Could not include required file conf/multiconfig/vruntime-aarch64.conf Signed-off-by: Tim Orling --- config.json | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/config.json b/config.json index 4d329e3..8a665cc 100644 --- a/config.json +++ b/config.json @@ -1830,6 +1830,31 @@ }, "toaster" : { "EXTRACMDS" : ["${SCRIPTSDIR}/run-toaster-tests ${HELPERBUILDDIR} ${HELPERBUILDDIR}/../layers/bitbake"] + }, + "vcontainer-tarball": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Build vcontainer-tarballs", + "BBTARGETS" : "vcontainer-tarball", + "extravars" : [ + "require conf/distro/include/meta-virt-host.conf", + "INIT_MANAGER = 'systemd'" + ], + "EXTRACMDS" : ["sed -i '/meta-virt-host.conf/d' ${HELPERBUILDDIR}/conf/auto.conf"] + }, + "step2" : { + "shortname" : "Publish vcontainer SDK for test reuse", + "EXTRACMDS" : [ + "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" + ] + } } }, "repo-defaults" : { From patchwork Tue Jul 28 01:40:36 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21F00C53219 for ; Tue, 28 Jul 2026 01:41:27 +0000 (UTC) Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.2389.1785202882154739179 for ; Mon, 27 Jul 2026 18:41:22 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=iOqVZrNF; spf=pass (domain: gmail.com, ip: 209.85.210.174, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-8485ef63b68so3381783b3a.1 for ; Mon, 27 Jul 2026 18:41:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202881; x=1785807681; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=T3yuCYSfZZAoGe7kGcPvrddiV6aYXo1dRAp4Q1JuwzU=; b=iOqVZrNFnImla7q8H74j7RYsQH4KG5G8nzSj5WvJqfkTL3jTReni8kYllZuZX74ZYz p08kJBNOll3nMzv+qs+c0NAwq0a6wWERd4vudHAdUWhFNSBskn5wa9jngO8FuxP5mROj mvEpRCeAZSt618LJd6KxrXLe9PJMhiHcjfexGUWZdPWtZk2Yfnln0nNanYvsUDIYno+N u1+LjTd1iwuubEkElHBdWQ4WYm/S3nSRb5vf4w7YY2YWp1RJT+3+cGgTS7MpcW2YFvky yRwNMSJ+MierTU6eghu/GXT4z1C6jnh9Pb0Q/sPppRi0Zz/Bl4a8FrPiuZ2AdMK+li7L 2Juw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202881; x=1785807681; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=T3yuCYSfZZAoGe7kGcPvrddiV6aYXo1dRAp4Q1JuwzU=; b=Cjs2cey1/q2YUInJ6qO+kUacrhHWJJ2fnNubXI7KpOBJtYD2ubbieg+8h7P8v41v6L W4WD48Nf3oIriGgZrDa+B4NhEMvtQQuOqOkjpjYlJPM7feAYiLr+/1G6j49OcE0V/B9I 3+Z8OWn8cOol5QZhOultxqJ7IsuECf82KzbnblADWydmlhdDPwE65LAqhxBycbJugK1R YlObaYSISO368qAQJbJWMuyamSt1QtiB0tNip4Zd9TJFA5bCxnlZJ+HfX0bnexOk8htx z7364jkCTydQwOJx4KeXCda1zhFO2znAwEziiiJkdA4uJbPSWdJUea6+BJpE3hubbWDQ +qbA== X-Gm-Message-State: AOJu0YxICS3sUUajaBFJD+sT68MqsWy84HIPWtfXtvd4DBlhhIOSaAay e2eYa/Qzo2Nf5DRjS6bSAB85lpneDnFi5CBCOmw9ONdCkpUnpe/7s52+YhvukQ== X-Gm-Gg: AR+sD11LKu6dK8dA5Q//ClQtPHIcl0jAjJXg69TxreF4RfMSVlhTnxOQYVe6dkvhaW/ Ch+0Ul/nA/H7NpL2/ToJHn2FSBKhj5UIlogr8c9WV7pl8uYAwSCqBjU1NCp9bMsFCLzRCbN85AS cwbwBXM2Ca5umhnwalQvPlZWni2Z5Pd31OcQr5EEmkqGR0ldwqjJNCMQMNEMbvLwF1c1ZQlHFtT +3dpKto9ZFVdjSvRCU9CWQocp/Wa7x3H78XukzF2Ozzx7DIXUWZvkGlTNpdj9kgcgeFvDYMmy7P f4IfOWxPCy2xjyt1MlD2Ac7n6upnelnZn5q3iLLK8hEk70yTHWAIV7WLsNS2AB2Sm6WM8gGLyA/ 3C/jIJDoLhtL+UOOoaul1AJMw91Pv0YUo3USz2rlLYwnfa0iKcUujo0/0WEKMnYBi032VKA+BS1 od9uZ2As7uM8AK/SJiNwyA57YUfR6jGbDz2ubzI4PNQofYmx6YCDvLmiX0s9BlQO8cJwQf3W38B aDKXFhtJZ8xypQst2vn6zXahqvg2Tpm/qKwaV6mh6hKprlZ8H9+j7ZObRko X-Received: by 2002:a05:6a00:cd3:b0:847:9151:3409 with SMTP id d2e1a72fcca58-84e9335fe7emr305291b3a.55.1785202881344; Mon, 27 Jul 2026 18:41:21 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.19 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:19 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 03/12] scripts: add run-vcontainer-tests for meta-virtualization Date: Mon, 27 Jul 2026 18:40:36 -0700 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:27 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4542 Add scripts/run-vcontainer-tests, the test runner used by the vcontainers-test job. It sources the vcontainer-tarball SDK, discovers the meta-virtualization pytest suite, and runs a configurable set of suites (vdkr, vpdmn, memres) against the checked-out layers. Suites can be selected per-step so the top-level 'vcontainer-tests' job runs the container engine agnostic tests: - tests/test_container_cross_install.py - tests/test_container_registry_script.py - tests/test_vcontainer_auth_config.py - tests/test_multiarch_oci.py - tests/test_multilayer_oci.py The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective suites (including memres for each container engine): - tests/test_vdkr.py - tests/test_vdkr_registry.py and - tests/test_vpdmn.py The purpose of this script is to test the just built vcontainer-tarball, so that it can be considered "known-good" and be published for use by container building jobs that need the vcontainer-tarball installed. For older hosts, with python < 3.10, we need buildtools for pytest 9.x in meta-virtualization/tests/requirements.txt. AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling --- config.json | 28 +++++ scripts/run-vcontainer-tests | 212 +++++++++++++++++++++++++++++++++++ 2 files changed, 240 insertions(+) create mode 100755 scripts/run-vcontainer-tests diff --git a/config.json b/config.json index 8a665cc..5a97674 100644 --- a/config.json +++ b/config.json @@ -1855,6 +1855,34 @@ "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" ] } + }, + "vcontainer-tests": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Run vcontainer pytest suite", + "EXTRACMDS" : [ + "${SCRIPTSDIR}/run-vcontainer-tests -s vcontainer -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}" + ] + }, + "step2" : { + "shortname" : "Run vdkr pytest suite", + "EXTRACMDS" : [ + "${SCRIPTSDIR}/run-vcontainer-tests -s vdkr -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}" + ] + }, + "step3" : { + "shortname" : "Run vpdmn pytest suite", + "EXTRACMDS" : [ + "${SCRIPTSDIR}/run-vcontainer-tests -s vpdmn -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}" + ] + } } }, "repo-defaults" : { diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests new file mode 100755 index 0000000..a7dbab3 --- /dev/null +++ b/scripts/run-vcontainer-tests @@ -0,0 +1,212 @@ +#!/bin/bash +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Run meta-virtualization pytest test suites against the vcontainer +# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake +# step. +# +# Usage: +# run-vcontainer-tests -s -b -m [options] +# +# Required: +# -s suite name: one of "vcontainer", "vdkr", "vpdmn" +# -b bitbake build directory (${BUILDDIR}) +# -m path to the meta-virtualization layer +# +# Optional: +# -S path to the vcontainer standalone SDK installer. The +# autobuilder -tests jobs share the SDK produced by the +# separate vcontainer-tarball builder. When unset, falls +# back to looking in /tmp/deploy/sdk. +# -e where to extract the standalone SDK tarball +# (default: /vcontainer-test-extracted) +# -i path to an OCI image directory (enables vdkr/vpdmn +# import tests) +# -a target architecture for vdkr/vpdmn tests (default: x86_64) +# -r directory to copy pytest artefacts (junit xml / log) to +# -k pytest marker filter (default excludes long running / +# infrastructure dependent tests) +# -h show this help and exit +# +# The script is intentionally conservative: any pytest tests that cannot run +# in the CI environment (those marked "slow", "network", "boot") are skipped +# so that the autobuilder step completes without needing network access. Those +# can be re-enabled via -k before invocation. +# +# It is assumed that /dev/kvm is writable by the CI user running the tests, +# since the performance is significantly faster with 'memres'. +# + +set -e +set -u +set -o pipefail +set -x + +usage() { + cat >&2 <<'EOF' +Usage: run-vcontainer-tests -s -b -m [options] + +Required: + -s vcontainer | vdkr | vpdmn + -b bitbake build directory + -m path to the meta-virtualization layer + +Optional: + -S path to the vcontainer standalone SDK installer + -e where to extract the standalone SDK tarball + -i path to an OCI image directory + -a target architecture (default: x86_64) + -r directory to copy pytest artefacts to + -k pytest marker filter + -h show this help and exit +EOF +} + +suite="" +builddir="" +metavirtdir="" +sdk_tarball="" +extract_dir="" +oci_image="" +arch="" +results_dir="" +marker_filter="not slow and not network and not boot and not incus and not k3s" + +while getopts ":s:b:m:S:e:i:a:r:k:h" opt; do + case "$opt" in + s) suite="$OPTARG" ;; + b) builddir="$OPTARG" ;; + m) metavirtdir="$OPTARG" ;; + S) sdk_tarball="$OPTARG" ;; + e) extract_dir="$OPTARG" ;; + i) oci_image="$OPTARG" ;; + a) arch="$OPTARG" ;; + r) results_dir="$OPTARG" ;; + k) marker_filter="$OPTARG" ;; + h) usage; exit 0 ;; + :) echo "ERROR: option -$OPTARG requires an argument" >&2; usage; exit 2 ;; + \?) echo "ERROR: unknown option -$OPTARG" >&2; usage; exit 2 ;; + esac +done + +if [ -z "$suite" ] || [ -z "$builddir" ] || [ -z "$metavirtdir" ]; then + echo "ERROR: -s, -b and -m are required" >&2 + usage + exit 2 +fi + +builddir=$(realpath "$builddir") +metavirtdir=$(realpath "$metavirtdir") + +if [ ! -d "$metavirtdir/tests" ]; then + echo "ERROR: meta-virtualization tests directory not found at $metavirtdir/tests" >&2 + exit 1 +fi + +# Locate the vcontainer standalone SDK tarball. Prefer an explicitly-provided +# SDK (-S), and fall back to looking in the local build's +# deploy/sdk directory when running stand-alone. +if [ -n "$sdk_tarball" ]; then + if [ ! -f "$sdk_tarball" ]; then + echo "ERROR: SDK installer '$sdk_tarball' is set but not a file" >&2 + exit 1 + fi +else + sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh" + if [ ! -f "$sdk_tarball" ]; then + # Try to find any matching tarball in case naming changed (e.g. versioned) + alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh 2>/dev/null | head -n1 || true) + if [ -n "$alt" ]; then + sdk_tarball="$alt" + else + echo "ERROR: vcontainer standalone SDK not found." >&2 + echo " Pass -S with an existing SDK installer, or" >&2 + echo " build vcontainer-tarball so $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2 + exit 1 + fi + fi +fi + +extract_dir="${extract_dir:-$builddir/vcontainer-test-extracted}" +rm -rf "$extract_dir" +mkdir -p "$(dirname "$extract_dir")" + +# Self-extracting installer (silent, -y agrees to license, -d picks dir) +"$sdk_tarball" -d "$extract_dir" -y + +# Prepare a Python venv so we don't pollute the worker's system packages. +python3 -m venv "$builddir/meta-virt-test-venv" +# shellcheck disable=SC1091 +source "$builddir/meta-virt-test-venv/bin/activate" +# Avoid warnings by upgrading pip; install pytest/pexpect into the venv via pip. +export PIP_DISABLE_PIP_VERSION_CHECK=1 +python3 -m pip install --quiet -r "$metavirtdir/tests/requirements.txt" + +# Per-suite test file selection. Uses -k/-m for fine-grained filtering and +# keeps the CLI small for logging clarity. +case "$suite" in + vdkr) + test_files=( + "tests/test_vdkr.py" + "tests/test_vdkr_registry.py" + ) + ;; + vpdmn) + test_files=( + "tests/test_vpdmn.py" + ) + ;; + vcontainer) + # Broad vcontainer/bbclass/tooling coverage that doesn't require the + # vdkr/vpdmn CLI harness to be running. + test_files=( + "tests/test_container_cross_install.py" + "tests/test_container_registry_script.py" + "tests/test_vcontainer_auth_config.py" + "tests/test_multiarch_oci.py" + "tests/test_multilayer_oci.py" + ) + ;; + *) + echo "ERROR: unknown suite '$suite' (expected vcontainer|vdkr|vpdmn)" >&2 + exit 2 + ;; +esac + +pytest_args=( + -v + --tb=short + -m "$marker_filter" + --vdkr-dir "$extract_dir" + --junitxml="$builddir/pytest-$suite-results.xml" +) + +# Allow tests that consume an OCI image (import/save/load) to find one. +if [ -n "$oci_image" ] && [ -d "$oci_image" ]; then + pytest_args+=(--oci-image "$oci_image") +fi + +# Pass architecture through when set (default is x86_64). +if [ -n "$arch" ]; then + pytest_args+=(--arch "$arch") +fi + +cd "$metavirtdir" +# Don't let a single failing test kill the whole step - collect the junit +# report, then surface the exit code via the junit file + exit status. +set +e +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}" +rc=$? +set -e + +# Copy artefacts to the results dir if one was provided. +if [ -n "$results_dir" ]; then + mkdir -p "$results_dir" + cp -f "$builddir/pytest-$suite-results.xml" "$results_dir/" 2>/dev/null || true + if [ -f /tmp/pytest-vcontainer.log ]; then + cp -f /tmp/pytest-vcontainer.log "$results_dir/pytest-$suite.log" || true + fi +fi + +exit $rc From patchwork Tue Jul 28 01:40:37 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15940C53209 for ; Tue, 28 Jul 2026 01:41:27 +0000 (UTC) Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.2418.1785202884721806782 for ; Mon, 27 Jul 2026 18:41:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=hR/gKekR; spf=pass (domain: gmail.com, ip: 209.85.210.173, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-84a4d8fd6ecso3387241b3a.1 for ; Mon, 27 Jul 2026 18:41:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202884; x=1785807684; darn=lists.yoctoproject.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=nWuk+FPjb3drsqOpTGVduyc+q08Wp7jSDLINYERmlTg=; b=hR/gKekRVIPW745W3cK8qf6CB8mWDhbr+v57Jkug1nEsOcrLdf3EVlKfwLs62hvaZL t+PosebBSEfc4qOeT8rkxmYYYmiVWomQppykIPSsKkp08wYYyP8C1R7A9bfVJtyormgy qUjUVixQZUzG5q404AWk70HUPNRn4fU0LH6Kg7Qz5+OiEojRwELw2/+LiIhS1hZQ3UzH dZHAlWFX1bKbNS0jy89/2WtAHlQvzkEmuOGPjARP8oUSwYxouOLK1RBSEYnTEdBLQjIz cxG1x6WRcjJ+zPoC/UB2fKDnM84RtJ58iYKDoXAZzbAGWOaE9cPMewp4UvFdFy5aqAfo miug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202884; x=1785807684; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=nWuk+FPjb3drsqOpTGVduyc+q08Wp7jSDLINYERmlTg=; b=f80qjxC42p+ikR2RzVDKf88WkgBRCrleoc4LhdZLXliWDQXdk9vS/lig9neX3GkLPN 5kEL+WV7QidhCLQMnVpHi5NFpGM5ak5VitdcxEfMoLTliiA62EgCrnUg0zLQ+iN7PMtk vOfgI7/d/hwkN5+dy1gvneXALXmLc791UVQ1YVGPqoq57uUVpAAtf7wlZJ/9KgP1rOUo 6Piu4o/LfxlMYP3b26+6kMXqFgfXm6dZtygKoGnW3Zqu7v+gt8h2dyKRBHya6kN8pJzd g/36HVNYB45RhIwmvjf9AiPz/aaGjPJLO3jUf6MMbUA5wy3MmXEkBECFrYOAb3VR/yTp 1yOg== X-Gm-Message-State: AOJu0YxTOpA4yD2M8iV1FMpk5M1ZRRm5/45MnniQDfxNFdRL7DcAoK2o /kd4DdiiloM/mB7Xo8gyDwLyK0YAMI1igOazE6NTcaFNLLFu7pmvT5y4Bivvaw== X-Gm-Gg: AR+sD10o6uqi8sNcCrvme3YMWaeZ/rtSAgUCOzVCBqfyIwcfgQg5s/o2s+JLsrMh37M rkCQeHahYNs6euXRP79vIMF1icPhwzLMzRYsH20pMRbII7D9QIBcrQghdcCs+QLp/t+Hwbi7tRN HoRtRZNDNBRt+JqE2iFc4u4GAUwCxcYvKI02OgL/V9GVK8zGcN83wgUHfcxImHPetTYx44Lo465 JtGJY0tgXrKcjQ0wuxdHNPtHJY9npnN48GR68AfPhw9oQFDoMa4mNkqCOlSrG6h8obTcsqpEQ3J PowTT8o5r18Azip66ScMhrrQahwQ9Uk9DxX5o4ygqOfBYjVaj52EL9FHrexN4vqhFDpY20l9WV7 YzUI5hZg3bzA/N2paXnfbdw14oTn1sqenX3Q/zBPfnGq9P2ajiA9xI6s6gW8V4sBWneEZ1sQNyP b1+ThYcB7GU08A7YDMirR1e7cxmX8tXDUo0NHriE5/kpBzYMSJ9GK0F9h0uo9SCMZqNXkzw8yv3 A== X-Received: by 2002:a05:6a00:84e:b0:848:2f84:f427 with SMTP id d2e1a72fcca58-84e93338b86mr294282b3a.64.1785202883839; Mon, 27 Jul 2026 18:41:23 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.22 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:22 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 04/12] scripts: add container registry push, auth, tagging, runtime selection Date: Mon, 27 Jul 2026 18:40:37 -0700 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:27 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4543 Add the push-containers infrastructure that drives the post-build steps for the 'containers-library' job. After each build step the runtime container store is harvested and pushed to one or more registries with derived per-step tags. * config.json: add CONTAINER_REGISTRIES, CONTAINER_AUTH_CONFIG, CONTAINER_RUNTIME, CONTAINER_TAG_CMDS, CONTAINER_VERSION_RECIPE and CONTAINER_IMAGE_MAP configuration knobs. * scripts/run-config: drive push-containers as a post-step action (by running scripts/run-push-containers). * scripts/run-push-containers: Tags are generated from recipe and distro metadata (yocto- tag uses major.minor on snapshots and full PV on releases) with CONTAINER_VERSION_RECIPE allowing a step to source PV from a different recipe than the image itself. * Skip pushing when building the container image failed * Registry auth is staged via .../config.json or podman .../auth.json using CONTAINER_AUTH_CONFIG, replacing an interactive login that could hang. CONTAINER_RUNTIME picks between vdkr (Docker-compatible) and vpdmn (Podman) runtimes. * Robustness: skip gracefully when no registries are configured, fix the OCI directory path, handle memres already running, and avoid hanging when memres has not yet come up. * The vcontainer-tarball bundles its own python3/site-packages; sourcing its environment-setup-ci puts them on PATH and shadows the buildbot-venv (e.g. its websockets<10 breaks bitbake's hashserv). It must never touch the build steps' PATH, so there is no job-wide "vcontainer" init step: the tarball is set up (downloaded/extracted on demand, cached) and its env sourced only in the synthesized steps that actually use its runtime wrappers -- push-containers (vpdmn/vdkr 'vimport' for single-arch; multiarch uses skopeo-native from the build). AI-Generated: Claude Cowork Opus 4.8 Signed-off-by: Tim Orling --- config.json | 5 + scripts/run-config | 24 +++-- scripts/run-push-containers | 180 ++++++++++++++++++++++++++++++++++++ 3 files changed, 201 insertions(+), 8 deletions(-) create mode 100755 scripts/run-push-containers diff --git a/config.json b/config.json index 5a97674..d9d0fdc 100644 --- a/config.json +++ b/config.json @@ -43,6 +43,11 @@ "BUILDINFOVARS" : ["INHERIT += 'image-buildinfo'", "IMAGE_BUILDINFO_VARS:append = ' IMAGE_BASENAME IMAGE_NAME'"], "WRITECONFIG" : true, "SENDERRORS" : true, + "CONTAINER_RUNTIME" : "vpdmn", + "CONTAINER_REGISTRIES" : [], + "CONTAINER_TAGS" : ["latest"], + "CONTAINER_TAG_CMDS" : [], + "CONTAINER_IMAGE_MAP" : {}, "extravars" : [ "SANITY_TESTED_DISTROS = ''", "BB_HASHSERVE = '${AUTOBUILDER_HASHSERV}'", diff --git a/scripts/run-config b/scripts/run-config index 90a5996..391db3e 100755 --- a/scripts/run-config +++ b/scripts/run-config @@ -154,15 +154,11 @@ else: sys.exit(0) vcontainer = utils.getconfigvar("vcontainer", ourconfig, args.target) -if jcfg: - if vcontainer: - addentry("vcontainer", "Setup vcontainer tarball", "init") -elif vcontainer: - # vcontainer is opt-in per target via the "vcontainer" config variable, - # so this is a no-op for targets which don't set it +# The vcontainer-tarball bundles its own python3/site-packages; avoid +# shadowing the buildbot-venv, except in steps that require 'vpdmn' or +# 'vkdr'. This avoids build failures on 'buildtools' workers. +if not jcfg and vcontainer and args.stepname in ("push-containers"): utils.setup_vcontainer_tarball(ourconfig, args.target, args.builddir + "/../vcontainer-tarball") - if args.phase == "init" and args.stepname == "vcontainer": - sys.exit(0) extratools = utils.getconfigvar("extratools", ourconfig, args.target) if jcfg: @@ -195,6 +191,8 @@ utils.mkdir(errordir) errorlogs = set() +push_containers = properties.get("push_containers", False) + def log_file_contents(filename, builddir, stepnum, stepname): logfile = logname(builddir, stepnum, stepname) with open(logfile, "a") as outf, open(filename, "r") as f: @@ -313,6 +311,16 @@ def handle_stepnum(stepnum): hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets)) bitbakecmd(args.builddir, "bitbake %s -k" % (sanitytargets), report, stepnum, args.stepname) + # Push container images to registries when push_containers is enabled. + # The push logic itself lives in scripts/run-push-containers. + container_images = utils.getconfigdict("CONTAINER_IMAGE_MAP", ourconfig, args.target, stepnum) + if container_images and push_containers: + if jcfg: + addstepentry("push-containers", "Push containers", shortdesc, desc, str(container_images), str(stepnum)) + elif args.stepname == "push-containers": + hp.printheader("Step %s/%s: Pushing container images %s" % (stepnum, maxsteps, list(container_images.keys()))) + bitbakecmd(args.builddir, "%s/run-push-containers %s %s" % (scriptsdir, args.target, stepnum), report, stepnum, args.stepname) + # Run any extra commands specified cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum) if jcfg: diff --git a/scripts/run-push-containers b/scripts/run-push-containers new file mode 100755 index 0000000..00d87ed --- /dev/null +++ b/scripts/run-push-containers @@ -0,0 +1,180 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Push container images built by a target step to the configured registries. +# +# Invoked by run-config for 'push-containers' steps with the OE build +# environment already sourced (bitbake and the vcontainer runtime wrappers +# must be on PATH). All configuration comes from the target/step config: +# +# CONTAINER_IMAGE_MAP - dict of image recipe -> registry image name +# CONTAINER_RUNTIME - vpdmn (default) or vdkr +# CONTAINER_REGISTRIES - list of registries to push to +# CONTAINER_TAGS - list of static tags (e.g. latest) +# CONTAINER_TAG_CMDS - extra shell to populate _EXTRA_TAGS +# CONTAINER_VERSION_RECIPE - recipe whose PV provides the version tag +# CONTAINER_AUTH_CONFIG - registry auth file staged into the guest +# + +import subprocess +import sys + +import utils + +parser = utils.ArgParser(description='Pushes container images for a target step to the configured registries.') + +parser.add_argument('target', + help="The target build name") +parser.add_argument('stepnum', + type=int, + help="The step number within the target") + +args = parser.parse_args() + +ourconfig = utils.loadconfig() + +container_images = utils.getconfigdict("CONTAINER_IMAGE_MAP", ourconfig, args.target, args.stepnum) +if not container_images: + print("No CONTAINER_IMAGE_MAP for %s step %s, nothing to push" % (args.target, args.stepnum)) + sys.exit(0) + +registries = utils.getconfiglist("CONTAINER_REGISTRIES", ourconfig, args.target, args.stepnum) +if not registries: + utils.printheader("push-containers skipped — CONTAINER_REGISTRIES is empty, no containers pushed") + sys.exit(0) + +runtime = utils.getconfigvar("CONTAINER_RUNTIME", ourconfig, args.target, args.stepnum) or "vpdmn" +static_tags = utils.getconfiglist("CONTAINER_TAGS", ourconfig, args.target, args.stepnum) +auth_config = utils.getconfigvar("CONTAINER_AUTH_CONFIG", ourconfig, args.target, args.stepnum) +if not auth_config: + if runtime == "vpdmn": + auth_config = "${HOME}/.config/containers/auth.json" + else: + auth_config = "${HOME}/.docker/config.json" + +utils.printheader("Pushing container images %s" % list(container_images.keys())) + +script = [ + "set -e", + "test -w /dev/kvm || { echo 'ERROR: /dev/kvm is not writable, cannot push containers'; exit 1; }", + # Always bring up a fresh memres VM in the foreground. + # + # 'memres status' only checks that the QEMU PID in daemon.pid + # is alive (see daemon_is_running()/daemon_status() in + # meta-virtualization's vrunner.sh); it returns 0 as soon as + # QEMU forks, so a hung/partially-booted VM from a previous + # run — or a VM in mid-boot — is reported as healthy. The + # subsequent 'login'/'vimport'/'push' commands then hang on + # the unresponsive daemon socket. + # + # 'memres restart' is synchronous: it does stop+start and + # runs a PING/PONG readiness probe against the daemon socket + # (120s timeout), exiting non-zero if the VM never answers. + # Running it in the foreground gives us a trustworthy ready + # signal via its exit code, so we can drop the status-poll + # loop entirely. + # + # Install an EXIT trap first so we always tear the daemon + # down, even if bitbake -e / vimport / push fails mid-step + # under 'set -e'. The trap is armed before the restart so + # a restart failure also triggers cleanup. + # + # Registry auth is staged into the guest at VM boot via + # the global '--config' flag — vrunner.sh's setup_auth_share() + # copies $AUTH_CONFIG onto a read-only 9p share, and + # vdkr-init.sh / vpdmn-init.sh's install_auth_config() + # installs it at /root/.docker/config.json (vdkr) or + # /run/containers/0/auth.json (vpdmn) inside the guest. + # Subsequent 'push' calls use those creds directly, so no + # explicit 'login' step is needed. Calling 'login' would + # actually hang under the autobuilder (no PTY): when the + # memres daemon is running, vcontainer-common.sh dispatches + # login via '--daemon-interactive' and blocks reading the + # password from stdin (see login case in vcontainer-common.sh). + "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime, + "%s-$(arch) --config %s memres restart ' suffix on AUTOREV/dev recipes — Docker + # reference format does not allow '+' in tags, and the + # base PV is what consumers expect. + # + # DISTRO_VERSION needs context-sensitive handling. Poky's + # DISTRO_VERSION resolves to '${PV}+snapshot-${METADATA_REVISION}' + # off a tag and just '${PV}' on a release tag. The '+' in + # the snapshot form is illegal in a Docker tag, but more + # importantly the patch level on a snapshot build (e.g. + # '6.0.99' between 6.0 and 6.1) is a moving target that + # doesn't correspond to any real release — only the + # major.minor line is meaningful. So: + # - snapshot build (DISTRO_VERSION contains '+') → tag + # with major.minor only, e.g. 'yocto-6.0'. + # - release-tag build (no '+') → tag with the full + # version, e.g. 'yocto-5.0.5' from the yocto-5.0.5 tag. + script += [ + "_BBENV=$(bitbake -e %s 2>/dev/null) || true" % recipe, + "_PV=$(echo \"$_BBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')", + "_DISTRO_CODENAME=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_CODENAME=/{ print $2; exit }')", + "_DISTRO_VERSION_RAW=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_VERSION=/{ print $2; exit }')", + "case \"$_DISTRO_VERSION_RAW\" in", + " *+*) _DISTRO_VERSION=$(echo \"${_DISTRO_VERSION_RAW%%+*}\" | cut -d. -f1,2) ;;", + " *) _DISTRO_VERSION=\"$_DISTRO_VERSION_RAW\" ;;", + "esac", + "_DEPLOY_DIR_IMAGE=$(echo \"$_BBENV\" | awk -F'\"' '/^DEPLOY_DIR_IMAGE=/{ print $2; exit }')", + "_EXTRA_TAGS=\"\"", + ] + if version_recipe: + # When the image recipe's PV is a wrapper-style + # placeholder (e.g. app-container-python_1.0.0.bb, + # whose 1.0.0 is meaningless to a downstream user), + # CONTAINER_VERSION_RECIPE points at the recipe whose + # PV is actually meaningful for the resulting tag — + # typically the language runtime or app being packaged + # (e.g. python3 -> 3.14.x). Override _PV from that + # recipe; image-recipe state still drives + # DEPLOY_DIR_IMAGE and DISTRO_* since those are + # environment-wide. + script += [ + "_VBBENV=$(bitbake -e %s 2>/dev/null) || true" % version_recipe, + "_PV=$(echo \"$_VBBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')", + ] + script += tag_cmds + script.append( + "_TAGS=\"%s $_PV $_DISTRO_CODENAME yocto-$_DISTRO_VERSION $_EXTRA_TAGS\"" % " ".join(static_tags) + ) + # Only push an image whose build actually produced an OCI artefact. + # build-targets runs 'bitbake ... -k', so a failed image build does + # not abort the build step or the other images, and the failure is + # already reported there. If we let the push proceed, 'vimport' of + # the missing ${recipe}-latest-oci would fail under 'set -e' and + # abort the whole push step — taking down the images that *did* + # build with it. So skip a missing image here (warn, don't fail) + # and let the successfully-built ones publish. + script += [ + "_OCI_IMAGE=${_DEPLOY_DIR_IMAGE}/%s-latest-oci" % recipe, + "if [ ! -e \"$_OCI_IMAGE\" ]; then", + " echo \"WARNING: %s did not build (no OCI image at $_OCI_IMAGE), skipping push\"" % recipe, + "else", + ] + for registry in registries: + # No per-registry 'login': credentials were staged into + # the guest by '--config' on 'memres restart' above. + script += [ + " for _tag in $_TAGS; do", + " %s-$(arch) vimport ${_DEPLOY_DIR_IMAGE}/%s-latest-oci %s/%s:${_tag}" % (runtime, recipe, registry, image), + " %s-$(arch) push %s/%s:${_tag}" % (runtime, registry, image), + " done", + ] + script.append("fi") +# Tear-down is handled by the EXIT trap installed above. +utils.flush() +sys.exit(subprocess.call(["/bin/bash", "-c", "\n".join(script)])) From patchwork Tue Jul 28 01:40:38 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 530AAC54F51 for ; Tue, 28 Jul 2026 01:41:37 +0000 (UTC) Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.2392.1785202887386744254 for ; Mon, 27 Jul 2026 18:41:27 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=Vrh53jBz; spf=pass (domain: gmail.com, ip: 209.85.210.181, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-8486ac3f347so454882b3a.1 for ; Mon, 27 Jul 2026 18:41:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202887; x=1785807687; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=HBMGTEVe/H1D+RkXoZgE7mcyNuCkXgjcjz+SKGcN2N4=; b=Vrh53jBzisYDw9twzfa+EsdCnWUp3sPs7iVzg7fRjBmZabQff+0vpnO5ObfwEObaA9 fnOcTNk2gn7ZTmS+89lDrXju7EZ1oqQrS5QiX3qh12VMrlojTLyQB6lBrPjfPjLzSWXg nDoU5g6o5/I0pwdHBBz9GPXzo/Vs4Bm3zpNJNS3hYvbLMFR8+8Btt3FcsI0OcaPDxrJc 5zq6AuZ0hYQcSointBjBoXN89etBNQg8KzV1nD1YEI+xrvbyOCkOoy72FIfdg/Amg6Me U2MSvZm3yg+Cz1I362lXm/T3Mn3hH8go/dk5P279vzVQWqkwAhUMEPbPSJgAJdQwXwCO OzRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202887; x=1785807687; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=HBMGTEVe/H1D+RkXoZgE7mcyNuCkXgjcjz+SKGcN2N4=; b=oMcg6ZxTS+KKcMbTu5R5IEsAirC7TVYftSD6e/Qcsu0kV7owsdS93TigaiBwwCywFx cP/pXsOwrhflWdu7wcyI9V4w1TVFyjnDs/tFZH3cyRea5okHrX7UzqqPz1wh5hi286Hj TtTR/hgCHi/yiap0XY8Nor9VC54JsZjpLyO6f9gTJv1ryJJGrXzt5maHFyRhW32X3fhj npSQp8GguiFsG9NWvGv3tQwb6g9LOmVvpC/sGjS/re8AtqCi3B1fecN3W1KiODYGAD/4 vBWLk/t1KxxI4br3sFO+qlTGruFaf7s2/V9N0zXuyjMiL0UvDLS6vDejoNrs1SPzWY7n h40Q== X-Gm-Message-State: AOJu0YyC/gF8uIOF69icd1yUDJZzJKywskDehFI6JESnAiY2MmUSs/rb UELY6iIR+3dzbfLnSBUfeCLnXQ44UdQTMYNZMUy7v0tLs9oAi8lB9t30kxI1GQ== X-Gm-Gg: AR+sD13vbOXxi3nx8HJdiIvvTei9IjYfXXecijCUPLbVIdwCZzGi2GIiMb+9crRhNsi JeuOV/P0dt8cKZSovLQbntfAMT7OUqwCWQkjfq2fTSQV3DxnvRhhH074mi9XGA+HHPTiVA0GvoD ejaRrc2VvUuCHsdHFDXJEmYC6odjMBpKAm4P4QuQ+XBmqUWqmIBmoeqoJmqAW8sokiNv8DF5Wqg +bIegvcATIT+3y/9+HIjvohMD1/0jFwOlgLAj4q9BUTF4nVb3ZwZ0HkET8P5s4IrzZvWy9PmTb8 xxuBo8gBVWsDsZX/QeOOjVE+0Y22SYNzhDyyk/i2D3x9O1MOZtRzkpJjIygXCUIOQ6xDR5vlbim wvI1GVob5QC8Cl0nKViOZgiwNsVxkHsgvjesseLEWtLaRYuBbCQhx3/qazIYYLvuRByUZtPz1SC A/7fdHUxMajh/WiEKlZgkEIBQNUSoZz/Q7dP3g1pGKX/pi4J9OqMPtRLCjrxMmuPAS0qwHDK+NX w== X-Received: by 2002:a05:6a00:3a14:b0:848:40f5:ff5e with SMTP id d2e1a72fcca58-84e9329ceaamr295762b3a.37.1785202886745; Mon, 27 Jul 2026 18:41:26 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.24 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:25 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 05/12] config.json: add 'containers-library' build job Date: Mon, 27 Jul 2026 18:40:38 -0700 Message-ID: <306a85f2070349bd3e21a0ec4c90bf70a6fe0bf3.1785198322.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:37 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4544 Add 'containers-library' build job that build container images on top of the vcontainer-tarball SDK: * original container images from the intial "container-cross-install" branch of meta-virtualization - container-base - app-container-curl * additional images modelled after docker.io/library/* - app-container-python - app-container-mosquitto - app-container-valkey - app-container-nginx * Tag containers with versions based on the recipe to which they are mapped, e.g. python:3, python:3.14, python:3.14.5. Also tag with DISTRO_CODENAME and DISTRO_VERSION, e.g. wrynose and yocto-6.0. * Similar to vcontainer-tarball job, follow the guidance in meta-virtualization/docs/build-profiles.md and add "require conf/distro/include/meta-virt-host.conf" to extravars. This adds the required DISTRO_FEATURES and BBMULTICONFIG (vruntime-*, container-*). It also fixes warnings due to BBMASKing, by setting the appropriate BBFILE_PATTERN_IGNORE_EMPTY on layers which are intentionally masked. Use 'sed' in EXTRACMDS to remove the added 'require' line from conf/auto.conf or else remove-layers step throws a file not found error. * Append 'usrmerge' to DISTRO_FEATURES and set INIT_MANAGER to systemd Signed-off-by: Tim Orling --- config.json | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/config.json b/config.json index d9d0fdc..8c7a91a 100644 --- a/config.json +++ b/config.json @@ -1888,6 +1888,65 @@ "${SCRIPTSDIR}/run-vcontainer-tests -s vpdmn -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}" ] } + }, + "containers-library": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-openembedded/meta-webserver", + "${BUILDDIR}/../meta-virtualization" + ], + "extravars" : [ + "require conf/distro/include/meta-virt-host.conf", + "DISTRO_FEATURES:append = ' usrmerge'", + "INIT_MANAGER = 'systemd'" + ], + "EXTRACMDS" : ["sed -i '/meta-virt-host.conf/d' ${HELPERBUILDDIR}/conf/auto.conf"], + "vcontainer" : "${VCONTAINER_TARBALL_URL}", + "CONTAINER_TAG_CMDS" : [ + "_PV_MAJOR=$(echo $_PV | cut -d. -f1)", + "_PV_MAJOR_MINOR=$(echo $_PV | cut -d. -f1,2)", + "_EXTRA_TAGS=\"$_PV_MAJOR $_PV_MAJOR_MINOR\"" + ], + "step1" : { + "shortname" : "Build 'base' container", + "BBTARGETS" : "container-base", + "CONTAINER_IMAGE_MAP" : {"container-base": "base"}, + "CONTAINER_VERSION_RECIPE" : "base-files" + }, + "step2" : { + "shortname" : "Build 'curl' container", + "BBTARGETS" : "app-container-curl", + "CONTAINER_IMAGE_MAP" : {"app-container-curl": "curl"}, + "CONTAINER_VERSION_RECIPE" : "curl" + }, + "step3" : { + "shortname" : "Build 'python' container", + "BBTARGETS" : "app-container-python", + "CONTAINER_IMAGE_MAP" : {"app-container-python": "python"}, + "CONTAINER_VERSION_RECIPE" : "python3" + }, + "step4" : { + "shortname" : "Build 'mosquitto' container", + "BBTARGETS" : "app-container-mosquitto", + "CONTAINER_IMAGE_MAP" : {"app-container-mosquitto": "mosquitto"}, + "CONTAINER_VERSION_RECIPE" : "mosquitto" + }, + "step5" : { + "shortname" : "Build 'valkey' container", + "BBTARGETS" : "app-container-valkey", + "CONTAINER_IMAGE_MAP" : {"app-container-valkey": "valkey"}, + "CONTAINER_VERSION_RECIPE" : "valkey" + }, + "step6" : { + "shortname" : "Build 'nginx' container", + "BBTARGETS" : "app-container-nginx", + "CONTAINER_IMAGE_MAP" : {"app-container-nginx": "nginx"}, + "CONTAINER_VERSION_RECIPE" : "nginx" + } } }, "repo-defaults" : { From patchwork Tue Jul 28 01:40:39 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93656 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48165C53219 for ; Tue, 28 Jul 2026 01:41:37 +0000 (UTC) Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.2419.1785202890813901490 for ; Mon, 27 Jul 2026 18:41:30 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=FnypSso5; spf=pass (domain: gmail.com, ip: 209.85.210.175, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-84e0688b859so2030745b3a.0 for ; Mon, 27 Jul 2026 18:41:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202890; x=1785807690; darn=lists.yoctoproject.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qR3lz2jblWLKLj9Xaztceu/mgLi95WzSkXeiyKIIqNk=; b=FnypSso5WbCFtMz7F8o2dbLKogIy9663DcGxbcvqnUYtS5WOf70Bbk2R9S35RszcG9 n8H+DtZHoRXhH2al48iGL3ayP5WPJujviJMIqMXMZsO8zOuCymDUpJLCMVnkiICVB5RT 5nIP4g1xW5yugy+iSG4/tD8X7+9v3tuBC1ZmN12G/NATX7t8DCu31mum1Ow6Q9BWEmfo 1ZB4WreTMLkQo/yJaw3+07TEXnjQa/9jSXKGzXVIDG6iLaQ/tb2n84IQt1rSp21nZEeM enPmepOnHs8/Zv367VlcxhuYSB1VTZV3YVIzzpNIpsjp2vsVn9YjiJpXYoh+7lGXsI9r X8nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202890; x=1785807690; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=qR3lz2jblWLKLj9Xaztceu/mgLi95WzSkXeiyKIIqNk=; b=LPYyBWRqJSFDaijnGtw5qVtJDu9gP39S2XDZl2zwPQiK3RIe2whf2KFnD7De8pCSud +/qSLEZn5o6ujZwnnmxBqYcWC5fX/SditHc6sjZmNEHslGC/P8fBOYgtwWRPxojvh88p zYeAVQDVyq0t4nVmJBhVLTIyQgxXcWdFS7o4pRHMeeULc/sQDfjWQv7TxMD3Aery0k+F HCE72jYW6WNk9YMD8bbGnrl9q2sWxxRbXvKUvy2IJZXUfkINEMzuw+ddgvT1srF9mt9x WW3hV634g7pUj+YKfJsmgq5cgTrc7FrhlBvKGNFXq2wZSW+ashGx7/K4p82htu1hgQYJ yvkg== X-Gm-Message-State: AOJu0YxTCaPZckaaKdtu64aHlknGE1d6T30XNDx7CkYj7jLFGuy9jsf5 v8D5KXm3gYpdpjsjYsXByBwhi1mJS7IDDDXis8ruWWVBj1cyDlTpf+vCfvR7iA== X-Gm-Gg: AR+sD10w4SN9GK3EnmwoQN/f0CvPGeZvIwu8ogM+5koPjEocDfoXlnr0tn/XFquVkxQ TrviHAiLoDYfBc0/sW+BKBe8IxZlismUmIKSGEEE97xczZsUYytITnhYA9guriS5Br3HDYJvpHk d+fdOOp5TqWL5e62DvVcPj9gj7gin0IovuT3p3epBICVomXMSQ840RX/fvOT+XXgQv3W2Ks13Uq c4fPQuwf/hDrQMUN49y10fkAoPiRl7f0ZCIQ8+z1YbYZYnjPpTCi7R+T2cHgxDk57Jo03U9P5zh sBZuI3f+S2y9bLwDHpzdjTiPuRw4cJSFD2JP5fBBr9+LH3dYwbs4S/GR0kMkzqbV/otaNH8na93 WWAX/+FeFaxLqsVd0besorAK0l3sNsVS/Deml5wRqZ788GwmCbR9W40UBVEHkMjFLXhlWYleNbx hklVQWLb0i88Sua8wApzTvjzhhgjXORPI6fExGQX7mmv3khfj4sGa/eJ8Al0c3dbft1ctIYvz0E gawLNc4dtGr X-Received: by 2002:a05:6a00:4292:b0:848:6895:b763 with SMTP id d2e1a72fcca58-84e9332e67bmr277552b3a.40.1785202890019; Mon, 27 Jul 2026 18:41:30 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.27 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:28 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 06/12] scripts/run-push-containers: push multiarch containers with skopeo-native Date: Mon, 27 Jul 2026 18:40:39 -0700 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:37 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4545 Support pushing multi-architecture container images (recipes inheriting meta-virtualization's oci-multiarch class) from the 'containers-library' job. These produce an OCI Image Index on disk rather than a single-arch image in a runtime store, so they are pushed with skopeo instead of vdkr/vpdmn (which only vimport one architecture). * scripts/run-push-containers: detect multiarch recipes per-recipe at runtime: 'bitbake -e' only contains OCI_MULTIARCH_OUTPUT when the recipe inherits oci-multiarch, and its value is the exact OCI layout path to push. Multiarch images are pushed with 'oe-run-native skopeo-native skopeo copy --all' direct from the OCI layout, after populating the skopeo-native recipe sysroot via 'bitbake skopeo-native -c addto_recipe_sysroot' (once per step, prepare_skopeo). The same tag set and registries are used for both paths; auth is passed to skopeo via --dest-authfile from CONTAINER_AUTH_CONFIG. * scripts/run-push-containers: defer the memres VM bring-up (kvm check, EXIT trap, restart, image rm) into a lazy start_vm() that only runs when the first single-arch recipe is pushed, so multiarch -only steps need neither /dev/kvm nor a VM boot. AI-Generated: Claude Cowork Opus 4.8 Signed-off-by: Tim Orling --- scripts/run-push-containers | 100 +++++++++++++++++++++++++++--------- 1 file changed, 77 insertions(+), 23 deletions(-) diff --git a/scripts/run-push-containers b/scripts/run-push-containers index 00d87ed..c13f493 100755 --- a/scripts/run-push-containers +++ b/scripts/run-push-containers @@ -58,7 +58,24 @@ utils.printheader("Pushing container images %s" % list(container_images.keys())) script = [ "set -e", "test -w /dev/kvm || { echo 'ERROR: /dev/kvm is not writable, cannot push containers'; exit 1; }", - # Always bring up a fresh memres VM in the foreground. + # Two push paths share this script: + # + # - Single-arch images (the common case) are imported into + # a vdkr/vpdmn memres VM and pushed from inside it. The + # VM bring-up is expensive and needs /dev/kvm, so it is + # wrapped in start_vm() and only runs when the first + # single-arch recipe is pushed. + # + # - Multi-arch images (recipes inheriting oci-multiarch) + # are pushed with skopeo-native straight from the OCI + # Image Index layout on disk — no VM, no /dev/kvm. + # skopeo's recipe sysroot is populated on first use by + # prepare_skopeo(). Detection is per-recipe at runtime: + # 'bitbake -e ' only contains OCI_MULTIARCH_OUTPUT + # when the recipe inherits oci-multiarch, and its value is + # the exact OCI layout path to push. + # + # Notes on the memres VM bring-up in start_vm(): # # 'memres status' only checks that the QEMU PID in daemon.pid # is alive (see daemon_is_running()/daemon_status() in @@ -92,8 +109,20 @@ script = [ # memres daemon is running, vcontainer-common.sh dispatches # login via '--daemon-interactive' and blocks reading the # password from stdin (see login case in vcontainer-common.sh). - "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime, - "%s-$(arch) --config %s memres restart /dev/null || true' EXIT" % runtime, + " %s-$(arch) --config %s memres restart X-Patchwork-Id: 93653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35642C531D0 for ; Tue, 28 Jul 2026 01:41:37 +0000 (UTC) Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.2420.1785202893818908489 for ; Mon, 27 Jul 2026 18:41:33 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=FfJ8OFVA; spf=pass (domain: gmail.com, ip: 209.85.210.180, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-84a4d8fd6ecso3387293b3a.1 for ; Mon, 27 Jul 2026 18:41:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202893; x=1785807693; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=OWLWgewJ11A3YU7XeX5oFwuluCumDIee0mj/jLpomng=; b=FfJ8OFVAuaXfY7zWqcQIpoy47WKr9ZN0aL/mp0c8kF7rpAbBbh5a0c5BI0bpJ6tlD7 WjlsAvSi9YATRmBFowa+2jRSCN7pNP0Dg3ik/zBownIMPW7NgDM2OlQpB/N1CV4jiLy4 MjJlvygUyrB1iUXQxjvNFCBHmL3K8EAGcaNVlUqPnp7x7IXAQm3auXOxo3ATaZVpZmUN Pl+1ajHrwx//Xwpf3Q9doSOlTfdt1f4SUOi6m2ap4qdkbAmtTsUuYHIlOP8off+pQo0z EUe6sp0MsJOTuYfAyl7z6TJuuHlsEDxAsdyIGg3cAusAZpMWnQEQTDwar5hCyYUhMnU+ Jh6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202893; x=1785807693; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=OWLWgewJ11A3YU7XeX5oFwuluCumDIee0mj/jLpomng=; b=mj3CmE3B9y097qpHN4Q737wFZ4ksFRUb5kRI7G8KQOYQkpCS+1PgLv5nPNyRXD8y4D hzZjJC9P0XIYSF9Z7oVAfDuMSgJPLQmu8Y8kEXG7OnieLaodA4GQq6YYoZup5I/eR2OK Y8cIO4AFusp3KWe6cmgL0xyxLDPjJPU4p7TDs2iomlRBv+4gsG2ZrZ2IOnfeqtDVE1Xk nCIVnaJqv1a4NQV57T0/vfGAlJr+x+ISUfe7EKnCFooCatZSnt2sQGdIOc7gerWzTgb6 UfgkLxrFq/LVaYDR+4l0rWH/0A/vS+O0KTnN/AA+nnZ0MArJSRiLzxcd71TrlZJi/oM4 Ibsg== X-Gm-Message-State: AOJu0YyWUcHWf4usKtlAAQfjC/zJik4kqCBr2vujIDYiifnfVfsWiDSK UBSZCqqQPye4Z+DmK5lUnADNFJ50UXqqSXJKKEJxeFNxhTOnFJeF0cTfHK5mYg== X-Gm-Gg: AR+sD1345fmv4Unk9UjewO2BGqrTgPSrPRfLcKeUNRWusbEyHVUjtuCqd6mUA+TXBUE x6wiwosv1omWEJ0CVAcBZ9Wy0ApG2e0XaP6mdO/cMFd60J2Hk3ltc0HxCwizMoazkv+c4L2G5xs vZYgftGrRk1bx9S54MrLejEGdiCOns0X6D8xQUkwlagGTDZ3t2nJG0bYA305bfk0ZmqcwVX6Gjf ml1sKe138isp/UXZvH3dG4aP7Elr9a8o6xnQRp3CBg8uQwu71TUBJ6hXRxJaWeyESp8URB4wys1 VYe0P3FNaAt7DpU3o5OsTpPMW9uBi2kgAD3Ewuxw2gNSF5uxRcf695JPxSpgipqXOOtTzH86p6U +jWeH8gOTBsXmLxtPmsAdP5+qRvM8iOjOEzutNILHdkzFQhXzNiawVMZ9Q0yASmzliABCljneFv KM9wbhXBd7VtO1OR8tEykcYMjvuz0IlvS+NwkJajf+tojUFevQSHZaX5uKQcf7qWvcSeUHmkFdP v250hlf4stW/4TjutXf2ppceiGPEscKcfPDc6rhHE3EsfAxiXrmp/VXTd2gDTgMui6wtgQ= X-Received: by 2002:a05:6a00:4f86:b0:848:5010:ad3a with SMTP id d2e1a72fcca58-84e933214f2mr322799b3a.58.1785202893181; Mon, 27 Jul 2026 18:41:33 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.31 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:31 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 07/12] containers-library: switch to multiarch Date: Mon, 27 Jul 2026 18:40:40 -0700 Message-ID: <00eb5ec16719f386599c76fd78fa8613ea2be3f9.1785198322.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:37 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4546 Utilize the oci-multiarch detection in run-push-containers script, which uses 'skopeo-native copy' to 'push' the oci/ directory to the container registries. This does mean we are no longer exercising the single-arch path in run-push-containers, but the previous commit is left in place to show how that is done if we decide to re-enable it. Signed-off-by: Tim Orling --- config.json | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/config.json b/config.json index 8c7a91a..d92c493 100644 --- a/config.json +++ b/config.json @@ -1912,39 +1912,39 @@ "_EXTRA_TAGS=\"$_PV_MAJOR $_PV_MAJOR_MINOR\"" ], "step1" : { - "shortname" : "Build 'base' container", - "BBTARGETS" : "container-base", - "CONTAINER_IMAGE_MAP" : {"container-base": "base"}, + "shortname" : "Build 'base' multiarch container", + "BBTARGETS" : "container-image-multiarch-container-base", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-container-base": "base"}, "CONTAINER_VERSION_RECIPE" : "base-files" }, "step2" : { - "shortname" : "Build 'curl' container", - "BBTARGETS" : "app-container-curl", - "CONTAINER_IMAGE_MAP" : {"app-container-curl": "curl"}, + "shortname" : "Build 'curl' multiarch container", + "BBTARGETS" : "container-image-multiarch-app-container-curl", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-curl": "curl"}, "CONTAINER_VERSION_RECIPE" : "curl" }, "step3" : { "shortname" : "Build 'python' container", - "BBTARGETS" : "app-container-python", - "CONTAINER_IMAGE_MAP" : {"app-container-python": "python"}, + "BBTARGETS" : "container-image-multiarch-app-container-python", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-python": "python"}, "CONTAINER_VERSION_RECIPE" : "python3" }, "step4" : { "shortname" : "Build 'mosquitto' container", - "BBTARGETS" : "app-container-mosquitto", - "CONTAINER_IMAGE_MAP" : {"app-container-mosquitto": "mosquitto"}, + "BBTARGETS" : "container-image-multiarch-app-container-mosquitto", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-mosquitto": "mosquitto"}, "CONTAINER_VERSION_RECIPE" : "mosquitto" }, "step5" : { "shortname" : "Build 'valkey' container", - "BBTARGETS" : "app-container-valkey", - "CONTAINER_IMAGE_MAP" : {"app-container-valkey": "valkey"}, + "BBTARGETS" : "container-image-multiarch-app-container-valkey", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-valkey": "valkey"}, "CONTAINER_VERSION_RECIPE" : "valkey" }, "step6" : { "shortname" : "Build 'nginx' container", - "BBTARGETS" : "app-container-nginx", - "CONTAINER_IMAGE_MAP" : {"app-container-nginx": "nginx"}, + "BBTARGETS" : "container-image-multiarch-app-container-nginx", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-nginx": "nginx"}, "CONTAINER_VERSION_RECIPE" : "nginx" } } From patchwork Tue Jul 28 01:40:41 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29124C53209 for ; Tue, 28 Jul 2026 01:41:37 +0000 (UTC) Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.2421.1785202896212741216 for ; Mon, 27 Jul 2026 18:41:36 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=AgiRSPgB; spf=pass (domain: gmail.com, ip: 209.85.210.172, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-84862b0d5aeso3350347b3a.2 for ; Mon, 27 Jul 2026 18:41:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202895; x=1785807695; darn=lists.yoctoproject.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=lqtnxB0yWJIqRvRFQ7Rt72fFtE9TJ2kHbkVQ4r8SHT8=; b=AgiRSPgBfXRpcxwdXWAF4c0N4ux1MHnwF4dP7RELGxWLAnM2z1DYLv3AD0o2dU7P6I aQKn67QZYANx6UfSPox7f6FYE7qIC6MimslMrqb30JxR44nFsuCOOCQkbBbhnPCDP/vv J7Bh2lc2Y6qEQERo1mtVrr7WTelHCUoUiGDb2TX9hWw9u6EjBBThwSC2NykC7AyYV7bS nobDmkRISveM2dT4i7z56WwuhU5mqFiuZOPeiaQIa6h5WjD1wVTu0S78wL1ZUighT6AI xgIsPfVCTaZOdfxfb1KFWXXw8OZDiiOfh+83mepI0JKFTZMcj94syRixebTNKsNaKkMy qHXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202895; x=1785807695; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=lqtnxB0yWJIqRvRFQ7Rt72fFtE9TJ2kHbkVQ4r8SHT8=; b=CbSYC140zaCoWrWo4NF/sERM1OyMYUZVi75LvUvj8rfE4GVxRyvH1SkomA6PcpHxoA RLp/Vr6esx9FEu94zSLlwQiVbgzEDLlaYwa6f8S7O9pQ3CMM5ha4XlqmqESQ3Zcr8PgH L3O0ShOFfUqmpxMkpErbEraHjmzUibTbOxoq/MtLpXLo7MYAESSwG7gs4YNch8f/jfHS 4t5O+nzKPEQpRGkQ4MxtW1huKh0vR+C6loB3ZO36FlWDjDYBe7NFM0MvC/tTWO62A7z5 zLcmSnyChX+sIwxyML+XVxSU3mfeiCHR3fgfBnjf5vY9yEF8loBQTVNZDStH+2rfJsXy HuZA== X-Gm-Message-State: AOJu0Ywpw9auJaVyhB4p2s9bBeCx7g2ERYUm5L9Yd+2BbdFIad0VMzkg fHX+1NI7HXEQDYpILy9e4z+fX5id6HhRfcdXH+7sw6jrMaVW6/B7Sk3kB+e9+A== X-Gm-Gg: AR+sD10aZ268YK/0Bvn3DIDCP0U93WRLuxQpg04W9Kpd7buNUif2dNFgAg279UOOL+0 G5lVC9QVEYHrcT5A6KhEgKTsLMc3tULF6Arlh5NO2oghPioimRnjuxayyP6cfS3Jc3wkk2Qqvi8 uoIMERqUs+nfzxybvsrtRs3B4ojxGjkZUlPHWmozq3eYBhTPQcUQnPHvBlc6rPDgDLdXdwDz9wJ uMge1WQ7MrVZRtfNro8bzC01p4flYfiXGJUY9eZVC6mdZfkLPusiZnyaiVMimjcMkCCLxAJ/4rb nEmlfr/u0OZ7sjZqEaQJJctVNz8v766DsOdvrXAGnU+qy9PmaUBvRm2tltdpW7Dfwk0m4I6jsnE 0xvQoVgL7fE4RKCIG4/5uw1iyl83WXeIDnwN3KKn4WEvdYWgTB+t7+AVR1I1mfZq4XKKV/r9r+X dp64fW1gEx1i6BJSiTXWHKXcoGWHRVbhV5ly+TrSMqWQ/rhbFOK4+c9PVna/qWxNR5d6OJULGJn JtUyT6bTt+c X-Received: by 2002:a05:6a00:3a1f:b0:845:bda6:574b with SMTP id d2e1a72fcca58-84e931f8e3cmr318001b3a.5.1785202895323; Mon, 27 Jul 2026 18:41:35 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.34 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:34 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 08/12] run-push-containers: conditionally sign pushed containers with cosign Date: Mon, 27 Jul 2026 18:40:41 -0700 Message-ID: <10f3fd7b6decfa230ce7b3c7c1c660112a42814f.1785198322.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:37 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4547 Introduce CONTAINER_COSIGN_KEY, a cosign private key path that gates signing. When unset (the default) no cosign sysroot is prepared and signing is skipped entirely. When set, every image that is successfully pushed is signed in place: - prepare_cosign() lazily populates the cosign-native sysroot (bitbake cosign-native -c addto_recipe_sysroot) and logs cosign in to each target registry. CONTAINER_REGISTRIES entries carry a namespace path (e.g. quay.io/ticotimo), but the auth file is keyed by the bare registry host and 'cosign login' only accepts a host authority, so each entry is stripped to its host ('${reg%%/*}') for both the credential lookup and the login: auths..auth is base64-decoded into user:password and the password is fed on stdin (--password-stdin). - sign_image() signs the image by digest with 'cosign sign --recursive --key' (which also covers every child manifest of a multi-arch index) so cosign does not warn about signing a mutable tag. The digest must be the one the tag resolves to: the multi-arch index digest is taken from 'skopeo copy --all --digestfile' at push time, and the single-arch manifest digest from 'skopeo inspect' (correct there, as there is no list). Signing 'skopeo inspect' on a multi-arch tag is avoided because without --raw it returns the host platform's child digest, which would sign one arch and leave the index unsigned. Signing is deduped per digest within the run, and a transparency-log conflict ('entry already exists' / HTTP 409, which also recurs on rebuilds that reproduce the same digest) is treated as success so signing stays idempotent rather than aborting the step. Signing runs in both push paths (multiarch skopeo-native and single-arch memres VM) and only after a successful push, so a missing/failed-build artefact is still skipped rather than signed. COSIGN_PASSWORD is exported (defaulting to empty) so cosign reads the passphrase from the environment instead of falling through to an interactive prompt (which dies with "inappropriate ioctl for device" under the autobuilder); an encrypted key requires the real value to be present in the build environment. Signed-off-by: Tim Orling --- config.json | 1 + scripts/run-push-containers | 115 +++++++++++++++++++++++++++++++++++- 2 files changed, 113 insertions(+), 3 deletions(-) diff --git a/config.json b/config.json index d92c493..00fc498 100644 --- a/config.json +++ b/config.json @@ -48,6 +48,7 @@ "CONTAINER_TAGS" : ["latest"], "CONTAINER_TAG_CMDS" : [], "CONTAINER_IMAGE_MAP" : {}, + "CONTAINER_COSIGN_KEY" : "", "extravars" : [ "SANITY_TESTED_DISTROS = ''", "BB_HASHSERVE = '${AUTOBUILDER_HASHSERV}'", diff --git a/scripts/run-push-containers b/scripts/run-push-containers index c13f493..9f6f2fa 100755 --- a/scripts/run-push-containers +++ b/scripts/run-push-containers @@ -15,6 +15,9 @@ # CONTAINER_TAG_CMDS - extra shell to populate _EXTRA_TAGS # CONTAINER_VERSION_RECIPE - recipe whose PV provides the version tag # CONTAINER_AUTH_CONFIG - registry auth file staged into the guest +# CONTAINER_COSIGN_KEY - cosign private key path; when set, each +# successfully pushed image is signed with +# cosign (otherwise signing is skipped) # import subprocess @@ -53,6 +56,10 @@ if not auth_config: else: auth_config = "${HOME}/.docker/config.json" +# Signing is conditional: only when CONTAINER_COSIGN_KEY points at a cosign +# private key. When unset, no cosign sysroot is prepared and no signing runs. +cosign_key = utils.getconfigvar("CONTAINER_COSIGN_KEY", ourconfig, args.target, args.stepnum) + utils.printheader("Pushing container images %s" % list(container_images.keys())) script = [ @@ -124,6 +131,99 @@ script = [ " _SKOPEO_READY=1", "}", ] + +# Cosign signing helpers, only emitted when a signing key is configured. +# +# prepare_cosign() is lazy like prepare_skopeo()/start_vm(): it populates the +# cosign-native sysroot once and logs cosign in to every target registry. The +# push paths authenticate via the auth file directly (--dest-authfile / +# --config), but cosign authenticates through its own credential store, so we +# must 'cosign login' explicitly. CONTAINER_REGISTRIES entries carry a +# namespace path (e.g. quay.io/ticotimo), but the auth file is keyed by the +# bare registry host and 'cosign login' only accepts a host authority (a path +# fails with "registries must be valid RFC 3986 URI authorities"). So we strip +# each entry to its host ('${reg%%/*}') for both the credential lookup and the +# login: read auths..auth from the auth file, base64-decode it into +# 'user:password', and feed the password on stdin (--password-stdin) so it +# never appears in the process table. +# +# COSIGN_PASSWORD is exported (defaulting to empty) so cosign reads the +# passphrase from the environment and never falls through to an interactive +# prompt under the autobuilder (no PTY) — without it cosign tries to read the +# passphrase from the terminal and dies with "inappropriate ioctl for device". +# The real value, if the key is encrypted, must be present in the build +# environment. +# +# sign_image() signs by digest ($2) so cosign does not warn about signing a +# mutable tag, while still pinning the index a consumer verifies. The digest +# MUST be the digest the tag resolves to: +# - multi-arch: the manifest-list/index digest, captured at push time from +# 'skopeo copy --all --digestfile' (the digest of the list it pushed). +# - single-arch: the lone manifest digest, which 'skopeo inspect' returns +# correctly (no list, so no platform ambiguity) — used when $2 is empty. +# 'cosign sign --recursive' then signs that digest and every child manifest. +# +# IMPORTANT: do NOT feed 'skopeo inspect' a multi-arch tag for the sign digest. +# Without --raw it resolves to the host platform's CHILD manifest, so signing +# image@ signs one architecture and leaves the index unsigned — +# why signed multi-arch images can still show as "Unsigned". Hence the index +# digest comes from --digestfile, not inspect. +if cosign_key: + login_block = """ for _reg in %s; do + _host=${_reg%%%%/*} + _creds=$(python3 -c 'import base64,json,os,sys +a=json.load(open(os.path.expandvars(sys.argv[1]))) +e=a.get("auths",{}).get(sys.argv[2]) or {} +t=e.get("auth") +sys.stdout.write(base64.b64decode(t).decode() if t else "")' "%s" "$_host") + if [ -z "$_creds" ]; then + echo "WARNING: no credentials for $_host in auth file, skipping cosign login" + continue + fi + _cuser=${_creds%%%%:*} + _cpass=${_creds#*:} + printf '%%s' "$_cpass" | oe-run-native cosign-native cosign login "$_host" -u "$_cuser" --password-stdin + done""" % (" ".join(registries), auth_config) + script += [ + "export COSIGN_PASSWORD=\"${COSIGN_PASSWORD:-}\"", + "_COSIGN_READY=0", + "prepare_cosign() {", + " if [ \"$_COSIGN_READY\" = 1 ]; then return 0; fi", + " bitbake cosign-native -c addto_recipe_sysroot", + login_block, + " _COSIGN_READY=1", + "}", + # Track digests already signed in this run so the same artefact is + # signed only once (see sign_image). + "_SIGNED_REFS=\"\"", + # $1 = /: just pushed; $2 = its digest (the index + # digest from 'skopeo copy --digestfile' on the multi-arch path). When + # $2 is empty (single-arch), resolve the lone manifest digest with + # 'skopeo inspect' — correct there since there is no list. oe-run-native + # prints 'Getting sysroot...' to stdout, so grep the digest out. + "sign_image() {", + " prepare_skopeo", + " prepare_cosign", + " _SIG_DIGEST=\"$2\"", + " if [ -z \"$_SIG_DIGEST\" ]; then _SIG_DIGEST=$(oe-run-native skopeo-native skopeo inspect --authfile %s --format '{{.Digest}}' docker://$1 | grep -oE 'sha256:[0-9a-f]{64}' | tail -n1); fi" % auth_config, + " if [ -z \"$_SIG_DIGEST\" ]; then echo \"WARNING: could not resolve digest for $1, skipping cosign sign\"; return 0; fi", + " _SIG_REF=\"${1%:*}@${_SIG_DIGEST}\"", + " case \" $_SIGNED_REFS \" in *\" $_SIG_REF \"*) return 0 ;; esac", + " _SIGNED_REFS=\"$_SIGNED_REFS $_SIG_REF\"", + # Sign by digest; --recursive also signs each child manifest. Treat a + # transparency-log conflict ('already exists' / HTTP 409, which also + # recurs on rebuilds that reproduce the same digest) as success so + # signing stays idempotent instead of aborting the step. + " _sign_out=$(oe-run-native cosign-native cosign sign --recursive --key %s \"$_SIG_REF\" 2>&1) || {" % cosign_key, + " case \"$_sign_out\" in", + " *\"already exists\"*|*createLogEntryConflict*) echo \"cosign: $_SIG_REF already in transparency log, treating as signed\" ;;", + " *) echo \"$_sign_out\" >&2; return 1 ;;", + " esac", + " }", + " echo \"$_sign_out\"", + "}", + ] + tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, args.stepnum) version_recipe = utils.getconfigvar("CONTAINER_VERSION_RECIPE", ourconfig, args.target, args.stepnum) for recipe, image in container_images.items(): @@ -205,12 +305,19 @@ for recipe, image in container_images.items(): script.append(" echo \"WARNING: %s did not build (no OCI image at $_OCI_MULTIARCH_OUTPUT), skipping push\"" % recipe) script.append(" else") script.append(" prepare_skopeo") + # --digestfile records the index digest skopeo pushed, so signing can pin + # it directly (see sign_image); only needed when signing is enabled. + digestfile = ' --digestfile "$_DGSTFILE"' if cosign_key else "" + if cosign_key: + script.append(" _DGSTFILE=$(mktemp)") # tiny tmp file, assumes autobuilder workdir is ephemeral for registry in registries: script += [ " for _tag in $_TAGS; do", - " oe-run-native skopeo-native skopeo copy --all --dest-authfile %s oci:${_OCI_MULTIARCH_OUTPUT} docker://%s/%s:${_tag}" % (auth_config, registry, image), - " done", + " oe-run-native skopeo-native skopeo copy --all%s --dest-authfile %s oci:${_OCI_MULTIARCH_OUTPUT} docker://%s/%s:${_tag}" % (digestfile, auth_config, registry, image), ] + if cosign_key: + script.append(" sign_image %s/%s:${_tag} \"$(cat \"$_DGSTFILE\")\"" % (registry, image)) + script.append(" done") script.append(" fi") script.append("else") # Single-arch: import the ${recipe}-latest-oci artefact into the memres VM @@ -225,8 +332,10 @@ for recipe, image in container_images.items(): " for _tag in $_TAGS; do", " %s-$(arch) vimport ${_DEPLOY_DIR_IMAGE}/%s-latest-oci %s/%s:${_tag}" % (runtime, recipe, registry, image), " %s-$(arch) push %s/%s:${_tag}" % (runtime, registry, image), - " done", ] + if cosign_key: + script.append(" sign_image %s/%s:${_tag}" % (registry, image)) + script.append(" done") script.append(" fi") script.append("fi") # Tear-down is handled by the EXIT trap installed above. From patchwork Tue Jul 28 01:40:42 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93660 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71DBDC54EFC for ; Tue, 28 Jul 2026 01:41:47 +0000 (UTC) Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.2422.1785202898743357332 for ; Mon, 27 Jul 2026 18:41:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=bnFHICky; spf=pass (domain: gmail.com, ip: 209.85.210.171, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-845c92bc464so2757268b3a.2 for ; Mon, 27 Jul 2026 18:41:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202898; x=1785807698; darn=lists.yoctoproject.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=KRvD6DUGxSSWc2HjzUy6NxaZaT5wg+mIu7aXGn+VKMs=; b=bnFHICkyc63Tgxv1GkOUGYf8OcJKUP/9NhZpocO7C8VlJuPd0OzhI0QdhfipWQtnzr GKVl5xBUXXBrEFrdJbryh4CJeCqBkMExAhJC0RJHZCBY+0ZjL7ilajNQDF53KVBfgV0j GhujOTNcSQbH8aGY49LFr6iUaOscGeKdbJfuXzpQcqQcLskNDd3XWziF4uNHsM2mvDOR 5Hp+3I67CJvthiZWY554oTqLJlqnPxihmde4ITOVOyIwBVgxuyVmg5qJbad448aaoxWT iK/JFbRh1wBd8DLkZAIO4jfvUGVLZ12aX4AQJcdCKh5A+sQEkrSJPqGUoHsTbZaubLle F6Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202898; x=1785807698; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=KRvD6DUGxSSWc2HjzUy6NxaZaT5wg+mIu7aXGn+VKMs=; b=ImdWBTVKYal92E9YqIlyP75aTAFQXA72H91VaXvwh6ezDQYhjdnvPQmv6AHqn/QnKo D77NhCdpZqvBNEldpgIgrhPO+wJsbBw5TtnS0dslTZT20kDy/mF53ArSbBBfyb6RRySx 8JjD4Qo/6SGtR47eytYN7H/QkmHkxUtAwS5DaaEX0dgSIEYWcFD8c8kgqXFsrVLjVNra fayrkr8S9Pg02HWVukzS1UhfxbZC0ER1StTzwdb4HgDTo88SmAtiazPEDPLAhJKxMSKv H4EYDOsVGpkJa82xP+nVvbsMh+B31SmLOnAu8Gl+QHhjzxNz/aNPsbKpX5TmTeZq25yk BnUg== X-Gm-Message-State: AOJu0YwhMIAV617D3d6wo84+v+/bx94kzqXfNYhubwgFTMDTYgARRcIi 0rNmjKNIIwe1XxP3dWuzk1ixceimcaUJBj2Z+UJOxEPejKanreEdMo9ZSm4Gdw== X-Gm-Gg: AR+sD13jSlzHAPMpU9cTwslZe093RBzicqMLHZct28C5G8rXcRWSARbVWYdigqkWe93 NK4S+adFxiDk+EjvH9zZ2KycHZ4JOcp7GxaJKLT17OQG4Ex4Ebb0/KUXuKTuFMSgO9sWFSjLc7g BRex8RAi5w7XKBWw/XCLla8AQBwDOKJXGUXXGc1mMPDgXDCs+UG1hsxGu/FVOQ2v/PGKtY/RrIG i46KhD0WLwfyGQglvMIIP7KBMJr5jCT605qP0eUPfQLZj3YGzkw6GvS0zGAkjRQsYDh5Sfxaer7 7uRKa7P3eeOMHfSWnaSnG1m98McPxu258zwOafgpbMW+AwEN8nu8A2cP9EppQ9eu+UD+Vnh1qzY FWGRc9vPY2yW1h6xYfC8Kg9FcnZkYCS7nzAlWYAf4Bmp3xRJRjgn40FPMJQnYENVbgZVSqJT2yj RCxuIICCPMXWLwM4/nbMNTE9kf49FRFFDbV638DF9a6+s4O5Izj448RPuaY4K+4eUWCF2cSHrpr g== X-Received: by 2002:a05:6a00:3992:b0:848:7f8a:8d0e with SMTP id d2e1a72fcca58-84e932c5f3emr288181b3a.55.1785202897961; Mon, 27 Jul 2026 18:41:37 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.36 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:36 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 09/12] scripts/run-push-containers: add SPDX SBOM attestation Date: Mon, 27 Jul 2026 18:40:42 -0700 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:47 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4548 Attach a *.rootfs.spdx.json as an in-toto attestation to either the top-level container if it is single-arch or to each arch container image if it is multi-arch. Introduce CONTAINER_COSIGN_PUB which is the cosign public key path; when set, each attestation is verified with 'cosign verify-attestation' after it is attached Signed-off-by: Tim Orling --- scripts/run-push-containers | 106 +++++++++++++++++++++++++++++++++++- 1 file changed, 104 insertions(+), 2 deletions(-) diff --git a/scripts/run-push-containers b/scripts/run-push-containers index 9f6f2fa..debf83f 100755 --- a/scripts/run-push-containers +++ b/scripts/run-push-containers @@ -17,7 +17,14 @@ # CONTAINER_AUTH_CONFIG - registry auth file staged into the guest # CONTAINER_COSIGN_KEY - cosign private key path; when set, each # successfully pushed image is signed with -# cosign (otherwise signing is skipped) +# cosign (otherwise signing is skipped). Also +# gates SPDX SBOM attestation: each pushed image +# gets a 'cosign attest --type spdxjson' +# attestation of its SPDX SBOM (per-arch on the +# multi-arch path, top manifest on single-arch). +# CONTAINER_COSIGN_PUB - cosign public key path; when set, each +# attestation is verified with +# 'cosign verify-attestation' after it is made # import subprocess @@ -59,6 +66,9 @@ if not auth_config: # Signing is conditional: only when CONTAINER_COSIGN_KEY points at a cosign # private key. When unset, no cosign sysroot is prepared and no signing runs. cosign_key = utils.getconfigvar("CONTAINER_COSIGN_KEY", ourconfig, args.target, args.stepnum) +# Public key for verifying attestations; when unset, attestations are made but +# not verified. +cosign_pub = utils.getconfigvar("CONTAINER_COSIGN_PUB", ourconfig, args.target, args.stepnum) utils.printheader("Pushing container images %s" % list(container_images.keys())) @@ -193,6 +203,13 @@ sys.stdout.write(base64.b64decode(t).decode() if t else "")' "%s" "$_host") login_block, " _COSIGN_READY=1", "}", + # Resolve the lone manifest digest of a single-arch reference. Correct + # only when there is no manifest list (single-arch / child manifest); + # the multi-arch path passes explicit digests instead. oe-run-native + # prints 'Getting sysroot...' to stdout, so grep the digest out. + "resolve_digest() {", + " oe-run-native skopeo-native skopeo inspect --authfile %s --format '{{.Digest}}' docker://$1 | grep -oE 'sha256:[0-9a-f]{64}' | tail -n1" % auth_config, + "}", # Track digests already signed in this run so the same artefact is # signed only once (see sign_image). "_SIGNED_REFS=\"\"", @@ -205,7 +222,7 @@ sys.stdout.write(base64.b64decode(t).decode() if t else "")' "%s" "$_host") " prepare_skopeo", " prepare_cosign", " _SIG_DIGEST=\"$2\"", - " if [ -z \"$_SIG_DIGEST\" ]; then _SIG_DIGEST=$(oe-run-native skopeo-native skopeo inspect --authfile %s --format '{{.Digest}}' docker://$1 | grep -oE 'sha256:[0-9a-f]{64}' | tail -n1); fi" % auth_config, + " if [ -z \"$_SIG_DIGEST\" ]; then _SIG_DIGEST=$(resolve_digest $1); fi", " if [ -z \"$_SIG_DIGEST\" ]; then echo \"WARNING: could not resolve digest for $1, skipping cosign sign\"; return 0; fi", " _SIG_REF=\"${1%:*}@${_SIG_DIGEST}\"", " case \" $_SIGNED_REFS \" in *\" $_SIG_REF \"*) return 0 ;; esac", @@ -222,6 +239,46 @@ sys.stdout.write(base64.b64decode(t).decode() if t else "")' "%s" "$_host") " }", " echo \"$_sign_out\"", "}", + # Make sure an image's SPDX SBOM exists before we attest it. $1 is the + # expected spdx.json path, $2 the bitbake target that produces it + # (mc:: on the multi-arch path, plain single-arch). + # create_image_sbom_spdx runs by default for single-arch builds, so the + # file is normally already there; the multi-arch index recipe doesn't + # pull in the per-arch SBOM tasks, so we generate them on demand. + "ensure_spdx() {", + " if [ ! -e \"$1\" ]; then", + " echo \"SPDX SBOM $1 missing, generating: bitbake -c create_image_sbom_spdx $2\"", + " bitbake -c create_image_sbom_spdx $2 || true", + " fi", + " if [ ! -e \"$1\" ]; then echo \"WARNING: SPDX SBOM $1 still missing, skipping attestation\"; return 1; fi", + "}", + # Track refs already attested so the same digest is attested once. + "_ATTESTED_REFS=\"\"", + # Attest an SPDX SBOM against a digest-pinned image ref. $1 = + # /@sha256:...; $2 = spdx.json path. Idempotent: a + # transparency-log conflict on rebuilds is treated as success. When a + # public key is configured the attestation is verified afterwards. + "attest_image() {", + " prepare_skopeo", + " prepare_cosign", + " case \" $_ATTESTED_REFS \" in *\" $1 \"*) return 0 ;; esac", + " _ATTESTED_REFS=\"$_ATTESTED_REFS $1\"", + " _att_out=$(oe-run-native cosign-native cosign attest --key %s --type spdxjson --predicate \"$2\" \"$1\" 2>&1) || {" % cosign_key, + " case \"$_att_out\" in", + " *\"already exists\"*|*createLogEntryConflict*) echo \"cosign: attestation for $1 already in transparency log\" ;;", + " *) echo \"$_att_out\" >&2; return 1 ;;", + " esac", + " }", + " echo \"$_att_out\"", + ] + if cosign_pub: + script += [ + " oe-run-native cosign-native cosign verify-attestation --key %s --type spdxjson \"$1\" >/dev/null \\" % cosign_pub, + " && echo \"cosign: verified attestation for $1\" \\", + " || { echo \"ERROR: attestation verification failed for $1\" >&2; return 1; }", + ] + script += [ + "}", ] tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, args.stepnum) @@ -259,6 +316,13 @@ for recipe, image in container_images.items(): " *) _DISTRO_VERSION=\"$_DISTRO_VERSION_RAW\" ;;", "esac", "_DEPLOY_DIR_IMAGE=$(echo \"$_BBENV\" | awk -F'\"' '/^DEPLOY_DIR_IMAGE=/{ print $2; exit }')", + # MACHINE locates the single-arch SPDX SBOM (-.rootfs.spdx.json). + "_MACHINE=$(echo \"$_BBENV\" | awk -F'\"' '/^MACHINE=/{ print $2; exit }')", + # TOPDIR and the oci-multiarch plain vars drive per-arch SBOM + # attestation below; the latter two are empty for non-multiarch recipes. + "_TOPDIR=$(echo \"$_BBENV\" | awk -F'\"' '/^TOPDIR=/{ print $2; exit }')", + "_OCI_MULTIARCH_RECIPE=$(echo \"$_BBENV\" | awk -F'\"' '/^OCI_MULTIARCH_RECIPE=/{ print $2; exit }')", + "_OCI_MULTIARCH_PLATFORMS=$(echo \"$_BBENV\" | awk -F'\"' '/^OCI_MULTIARCH_PLATFORMS=/{ print $2; exit }')", # Only set (non-empty) when the recipe inherits # oci-multiarch; doubles as the multiarch marker and # the OCI layout path for skopeo below. @@ -318,6 +382,34 @@ for recipe, image in container_images.items(): if cosign_key: script.append(" sign_image %s/%s:${_tag} \"$(cat \"$_DGSTFILE\")\"" % (registry, image)) script.append(" done") + if cosign_key: + # Attest each child manifest with that arch's SPDX SBOM. The arch -> + # (multiconfig, machine) mapping lives in oci-multiarch.bbclass flags + # (OCI_MULTIARCH_MC/MACHINE[]); query them per platform with + # bitbake-getvar so an overridden mapping is still honoured. The child + # manifest digest is the per-arch source image's manifest digest, which + # the class copies verbatim into the index and skopeo copy --all pushes + # unchanged — so we attest @. + script.append(" for _plat in $_OCI_MULTIARCH_PLATFORMS; do") + script.append(" _mc=$(bitbake-getvar -q -r %s --value -f $_plat OCI_MULTIARCH_MC)" % recipe) + script.append(" _machine=$(bitbake-getvar -q -r %s --value -f $_plat OCI_MULTIARCH_MACHINE)" % recipe) + script.append(" if [ -z \"$_mc\" ] || [ -z \"$_machine\" ]; then echo \"WARNING: no mc/machine for platform $_plat, skipping attestation\"; continue; fi") + script.append(" _pdir=${_TOPDIR}/tmp-${_mc}/deploy/images/${_machine}") + script.append(" _mspdx=${_pdir}/${_OCI_MULTIARCH_RECIPE}-${_machine}.rootfs.spdx.json") + # Locate the per-arch source OCI layout (same name patterns the class + # searches) and read its single manifest digest = the child digest. + script.append(" _mdig=\"\"") + script.append(" for _oci in ${_OCI_MULTIARCH_RECIPE}-latest-oci ${_OCI_MULTIARCH_RECIPE}-${_machine}-latest-oci ${_OCI_MULTIARCH_RECIPE}-oci; do") + script.append(" if [ -e \"${_pdir}/${_oci}/index.json\" ]; then") + script.append(" _mdig=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))[\"manifests\"][0][\"digest\"])' \"${_pdir}/${_oci}/index.json\")") + script.append(" break") + script.append(" fi") + script.append(" done") + script.append(" if [ -z \"$_mdig\" ]; then echo \"WARNING: no source OCI index for $_plat under $_pdir, skipping attestation\"; continue; fi") + script.append(" ensure_spdx \"$_mspdx\" \"mc:${_mc}:${_OCI_MULTIARCH_RECIPE}\" || continue") + for registry in registries: + script.append(" attest_image %s/%s@${_mdig} \"$_mspdx\"" % (registry, image)) + script.append(" done") script.append(" fi") script.append("else") # Single-arch: import the ${recipe}-latest-oci artefact into the memres VM @@ -327,6 +419,12 @@ for recipe, image in container_images.items(): script.append(" echo \"WARNING: %s did not build (no OCI image at $_OCI_IMAGE), skipping push\"" % recipe) script.append(" else") script.append(" start_vm") + if cosign_key: + # create_image_sbom_spdx runs by default for single-arch image builds, + # so the SBOM is normally already deployed; ensure_spdx regenerates it + # only if absent. Single manifest -> attest the digest the tag resolves + # to (resolve_digest, also used by sign_image). + script.append(" _SPDX=${_DEPLOY_DIR_IMAGE}/%s-${_MACHINE}.rootfs.spdx.json" % recipe) for registry in registries: script += [ " for _tag in $_TAGS; do", @@ -335,6 +433,10 @@ for recipe, image in container_images.items(): ] if cosign_key: script.append(" sign_image %s/%s:${_tag}" % (registry, image)) + script.append(" if ensure_spdx \"$_SPDX\" \"%s\"; then" % recipe) + script.append(" _ADIG=$(resolve_digest %s/%s:${_tag})" % (registry, image)) + script.append(" [ -n \"$_ADIG\" ] && attest_image %s/%s@${_ADIG} \"$_SPDX\"" % (registry, image)) + script.append(" fi") script.append(" done") script.append(" fi") script.append("fi") From patchwork Tue Jul 28 01:40:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63FBDC53209 for ; Tue, 28 Jul 2026 01:41:47 +0000 (UTC) Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.2423.1785202901355165821 for ; Mon, 27 Jul 2026 18:41:41 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=Ywv4jufR; spf=pass (domain: gmail.com, ip: 209.85.210.179, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-8423f236418so2538883b3a.1 for ; Mon, 27 Jul 2026 18:41:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202901; x=1785807701; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=yaMhRIeZDmDZcvUzreSIEIweTn0GounFk13YcooEGGE=; b=Ywv4jufRWKcA5/XvYgZVrL/TWvCJfej6gC+bc7LoTZ50pgiOh2xM79fwiS8lizZl+B D2FZg1bpkW7Bq+ScVqjdXHC4RS94yi1gH/FutQm9Dw2Pu9/LhPgrloOw8BX8W2ltOn4n UHFk0CTdnnKxtkBglYZB8vHYQWuQ8+WyRhBFd7/hqNYFdvYJFLTpn3udriRuHHNIuPkn LaxUTByvx6L3MNdmsGmF0Zlv99DMR5Z7OEcLSZoDOSq8OwB/2F75k54I6oKy7eb9pzmp ptdQqmJBuDOmXVt6jdIf5TUkL+rH0Acjs4gMcrF0Ihot2a7ieASJTZKcsEcB34sTDaYp dVgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202901; x=1785807701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=yaMhRIeZDmDZcvUzreSIEIweTn0GounFk13YcooEGGE=; b=ecyrycf5e5a2tgIoToATfw3xAN/bfxEN+nFCMrz6uyZYaSs/7cYbsn8GUiEh4ZF31V J2kgcVTyW/tpSXj3n25jhhCfuhPwGB/h2Rnlo9XaSQkB7H1ToJWf/jjuV1qgeaB7wFgS kNcFxtQXTDesiOFxjAIYAVBtEX1IxI4IWkEcwjTmgJiB04XzVTEaOqmbTQS1BR4vqQHr TZ+KD2lK9Wtk3gqaja5oShAdquczo1HQq/wPPijmZFNkKZTdztt8X8kfxkOg1kaQkb5G FwSsTwcysRtQYewV/iEkoiRrUTkmx06z/UWWEAsEs9LFD+IpS+Pjk7H0oRfPkR2IhF2M Uy+g== X-Gm-Message-State: AOJu0YwRCwlJ02Lstp/pbUnDggLllgLVo3fSIuN6OZg5u4vybuB/uIUz d+Zby4rmtYLUwLPWFgGl7MjCDbaccBfG2C78m65TBd5uW9s8haV3yh3Kelb66Q== X-Gm-Gg: AR+sD103MolNUf5TQu7CqQ8i27UCtwHxggUrg4Jg6CtkS8dPC9Tcjh6e7Ae/G40E3ro 7/M6Z4FfVuxj9CCgfIRv/SGdatsON1cGm7q6aDEFcxT1bLICEZz1r0ItIRk7xhCL/Q92cv145jJ 3RUFvUVu4PXgwqANnnwGftgnOpWNG1GPTdTBHruks0fRNi7nZtAsi9MJn2Dc4pgvsQA414xhv/z TiGBTaqbYsFbQptgxSmuLsk8MZCzuPES0rxHl7LiKasMWqPcYFaS43AOztuRaT3jbfuRnmw8f/c zOJ29GqpZ497ETruQydkUQeJdl48UgRqa4j+uQJm3C9NxYBtW1LFaYAKDKR9kVA+RFC8RmoEaO6 s4qwKgVDQRWaGUK69BxFVZRtpTr80iI5RB5jrvkxsTgTVa1vw3fSTq0cZBiNInNuD5uLhjj2kyd Ca6eQ2LjppFE5o0BAFtPAsIJmoScQslbO4co/Nd9+5tw5VRvzQYcUK+mIG9CqHqQv3FGjMl/jY6 A== X-Received: by 2002:a05:6a00:13a3:b0:848:2f74:1d66 with SMTP id d2e1a72fcca58-84e9330f06fmr320125b3a.76.1785202900684; Mon, 27 Jul 2026 18:41:40 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.38 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:39 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 10/12] config.json: add -dev container builds to containers-library Date: Mon, 27 Jul 2026 18:40:43 -0700 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:47 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4549 For some of the containers, we want to provide a -dev version, denoted by '-dev' appended to the tag(s). This runs the container as UID '0' root user and adds a shell. Inspired by dhi.io (Docker Hardened Images). Signed-off-by: Tim Orling --- config.json | 59 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 50 insertions(+), 9 deletions(-) diff --git a/config.json b/config.json index 00fc498..8c23c2b 100644 --- a/config.json +++ b/config.json @@ -1916,37 +1916,78 @@ "shortname" : "Build 'base' multiarch container", "BBTARGETS" : "container-image-multiarch-container-base", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-container-base": "base"}, - "CONTAINER_VERSION_RECIPE" : "base-files" + "CONTAINER_VERSION_RECIPE" : "base-files", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, "step2" : { "shortname" : "Build 'curl' multiarch container", "BBTARGETS" : "container-image-multiarch-app-container-curl", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-curl": "curl"}, - "CONTAINER_VERSION_RECIPE" : "curl" + "CONTAINER_VERSION_RECIPE" : "curl", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, "step3" : { + "shortname" : "Build 'curl' multiarch container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-curl", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-curl": "curl"}, + "CONTAINER_VERSION_RECIPE" : "curl", + "extravars" : ["PACKAGECONFIG:pn-app-container-curl = 'dev'"] + }, + "step4" : { "shortname" : "Build 'python' container", "BBTARGETS" : "container-image-multiarch-app-container-python", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-python": "python"}, - "CONTAINER_VERSION_RECIPE" : "python3" + "CONTAINER_VERSION_RECIPE" : "python3", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, - "step4" : { + "step5" : { + "shortname" : "Build 'python' container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-python", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-python": "python"}, + "CONTAINER_VERSION_RECIPE" : "python3", + "extravars" : ["PACKAGECONFIG:pn-app-container-python = 'dev'"] + }, + "step6" : { "shortname" : "Build 'mosquitto' container", "BBTARGETS" : "container-image-multiarch-app-container-mosquitto", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-mosquitto": "mosquitto"}, - "CONTAINER_VERSION_RECIPE" : "mosquitto" + "CONTAINER_VERSION_RECIPE" : "mosquitto", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, - "step5" : { + "step7" : { + "shortname" : "Build 'mosquitto' container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-mosquitto", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-mosquitto": "mosquitto"}, + "CONTAINER_VERSION_RECIPE" : "mosquitto", + "extravars" : ["PACKAGECONFIG:pn-app-container-mosquitto = 'dev'"] + }, + "step8" : { "shortname" : "Build 'valkey' container", "BBTARGETS" : "container-image-multiarch-app-container-valkey", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-valkey": "valkey"}, - "CONTAINER_VERSION_RECIPE" : "valkey" + "CONTAINER_VERSION_RECIPE" : "valkey", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] }, - "step6" : { + "step9" : { + "shortname" : "Build 'valkey' container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-valkey", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-valkey": "valkey"}, + "CONTAINER_VERSION_RECIPE" : "valkey", + "extravars" : ["PACKAGECONFIG:pn-app-container-valkey = 'dev'"] + }, + "step10" : { "shortname" : "Build 'nginx' container", "BBTARGETS" : "container-image-multiarch-app-container-nginx", "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-nginx": "nginx"}, - "CONTAINER_VERSION_RECIPE" : "nginx" + "CONTAINER_VERSION_RECIPE" : "nginx", + "extravars" : ["PACKAGE_EXTRA_ARCHS:append = ' container-dummy-provides'"] + }, + "step11" : { + "shortname" : "Build 'nginx' container (dev)", + "BBTARGETS" : "container-image-multiarch-app-container-nginx", + "CONTAINER_IMAGE_MAP" : {"container-image-multiarch-app-container-nginx": "nginx"}, + "CONTAINER_VERSION_RECIPE" : "nginx", + "extravars" : ["PACKAGECONFIG:pn-app-container-nginx = 'dev'"] } } }, From patchwork Tue Jul 28 01:40:44 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93658 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5898BC53219 for ; Tue, 28 Jul 2026 01:41:47 +0000 (UTC) Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.2397.1785202904180994443 for ; Mon, 27 Jul 2026 18:41:44 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=f3rgXmOb; spf=pass (domain: gmail.com, ip: 209.85.210.176, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-84861fc51f5so2599705b3a.1 for ; Mon, 27 Jul 2026 18:41:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202903; x=1785807703; darn=lists.yoctoproject.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=2kZqLcIhUdmdBGX6T1eXHtWfAhJvixU83AG8CnGFqYE=; b=f3rgXmObcHU7ogrjXpBQuvWo4fNyW6zZNOq3o3xkFrgu/O3UXgfVD5BxYz0TP+OE2D s3gjUd0bhtUzlwb1BoboNQbmBjLxsqetTtaLknhLnHIhPUsoqZ5l4hzZdgw8FnmtbAsz YJkGwhXm9q09CBli6LiZU/+M+ai0tKuEp4vBx/y0cuhl2FvNdzTSlZvAJPrHfhZpeZ7s HmqKVCBnASiS/bfdgecdTUVDfIFCnFQYQgPuL3RlpUy+UVvbiXVTDazgjifvRhnMYUG7 OmTmgX7Ltr6PiNTNYLWcWBb0vPxZ+93FuW3MhObM+1RWt+3cu+tQVt5otMuY3h3h7Z+E 8WQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202903; x=1785807703; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=2kZqLcIhUdmdBGX6T1eXHtWfAhJvixU83AG8CnGFqYE=; b=gZ8Dq44zTmL+1qiBHmpWDfG9QpN1xCCD/7bqBvD/kZDqLT9FntdeLsMc69qmsP+3WE IaH1Hy7Y+YrUkXlNEm5Ka9X4CmiurwybshZE8Wg5+CKU/RErmdoZj5aIsGqjU/nFoqMq +SRFjjQr3N7n7JNyJ8bkpvav/EZRKRT2X5o1DWz4PwDl5uC5rdmbtG16W2DyGqvqY7VL 2LnBRbnXL/WIZgg/wz/PAyyQl+wwKmJhG0XJsbpO8xK389glrn4CPLpT6sbPl2dXPXHU KInFedfkR9tM7EugxqEbrLB50txYvuVXxnNKROJg8QZ55EOMuglY+9qJ5CXQSgMgh+5u MyxA== X-Gm-Message-State: AOJu0YyzsGYZpQIH0+9i8YDLeLPPGxB2pBAHlqB8CviUs/ZKIo2dYE7p FnG5ELHK6SE+4gHRVc175eP04cHVNF8tqqXI3VsjH4VOqQNPfMpL2CjLq+5dyg== X-Gm-Gg: AR+sD13bB+EyIH81TapJm2Qnk8LQzwAMDEeBO1jjfayyav5byEG2KlCmHsgh72xlWsY ftPCkwOVBv+eAqmthMzSm3FaMLjIwXyRMPJGHvZpksfbaZqNh0/hgsGavWSQbT79fOAW2id8Tem jHtxM80waSrPWfGcuYb6C1RPOZaLZhN5p1Daak7xefrKZKng1gJWrH6QSDPTuCn/IYLcInM8KS3 DJA5toQ7wUpl6M/+wHZt4qf7c8GiJpr45AecSrzpotvdNEtX0Clfg9493pqLJnO4BdLqQGhR2Pg 1HFZ66yTYPjevnv+EJzSUnmeW85u8o4KpKkWf7IB0/D0KH2k/j2C8Ztr/ZzezKIfKTfeI2m20vR 0htcfuOKHgK9CMmUPDLjK11krSGLqxW5FQ0z0obOURik3FGmjOKz5vqv8HH3aUXqy9Sc07nYUsr QxGDp7jRXd8WZK0tkaW8pa9G0Ba9IZXVIHhnnIMUNwI3tcmesyIEraiPDXuSfvLOeUzcGIVJUws cIpmH/2acDk X-Received: by 2002:a05:6a00:10c6:b0:847:83bd:6671 with SMTP id d2e1a72fcca58-84e93225c16mr255668b3a.19.1785202903514; Mon, 27 Jul 2026 18:41:43 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.41 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:42 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 11/12] run-push-containers: optionally push -dev tagged containers Date: Mon, 27 Jul 2026 18:40:44 -0700 Message-ID: <1fc1f15f2f6c7b0c370cf2218f2b4d3d460347b9.1785198322.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:47 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4550 For containers which set PACKAGECONFIG 'dev', push additional container images with '-dev' suffix to the tags. Inspired by dhi.io (Docker Hardened Images). Signed-off-by: Tim Orling --- scripts/run-push-containers | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/scripts/run-push-containers b/scripts/run-push-containers index debf83f..5dd35af 100755 --- a/scripts/run-push-containers +++ b/scripts/run-push-containers @@ -327,6 +327,20 @@ for recipe, image in container_images.items(): # oci-multiarch; doubles as the multiarch marker and # the OCI layout path for skopeo below. "_OCI_MULTIARCH_OUTPUT=$(echo \"$_BBENV\" | awk -F'\"' '/^OCI_MULTIARCH_OUTPUT=/{ print $2; exit }')", + # Drives the '-dev' tag suffix below (dhi.io convention: same image + # name, tag suffixed) for recipes built with PACKAGECONFIG:pn- = "dev". + # + # $recipe here is the BBTARGET from CONTAINER_IMAGE_MAP, which on the + # multiarch path is the container-image-multiarch wrapper (e.g. + # container-image-multiarch-app-container-nginx) — a different PN + # than app-container-nginx, so 'PACKAGECONFIG:pn-app-container-nginx' + # never touches its PACKAGECONFIG. OCI_MULTIARCH_RECIPE (already + # extracted above) is the underlying app recipe name and is what + # PACKAGECONFIG:pn- actually targets; fall back to $recipe + # itself for single-arch recipes (OCI_MULTIARCH_RECIPE empty there). + "_PC_RECIPE=\"${_OCI_MULTIARCH_RECIPE:-%s}\"" % recipe, + "_PC_BBENV=$(bitbake -e \"$_PC_RECIPE\" 2>/dev/null) || true", + "_PACKAGECONFIG=$(echo \"$_PC_BBENV\" | awk -F'\"' '/^PACKAGECONFIG=/{ print $2; exit }')", "_EXTRA_TAGS=\"\"", ] if version_recipe: @@ -348,6 +362,18 @@ for recipe, image in container_images.items(): script.append( "_TAGS=\"%s $_PV $_DISTRO_CODENAME yocto-$_DISTRO_VERSION $_EXTRA_TAGS\"" % " ".join(static_tags) ) + # 'dev' builds (PACKAGECONFIG:pn- = "dev") push under the same + # image name as production, with every tag suffixed '-dev' (dhi.io + # convention), instead of a separate image name/repo. + script += [ + "case \" $_PACKAGECONFIG \" in", + " *\" dev \"*)", + " _DEVTAGS=\"\"", + " for _t in $_TAGS; do _DEVTAGS=\"$_DEVTAGS ${_t}-dev\"; done", + " _TAGS=\"$_DEVTAGS\"", + " ;;", + "esac", + ] # Detect multiarch vs single-arch per recipe and push accordingly, # but only for an image whose build actually produced its OCI artefact. # From patchwork Tue Jul 28 01:40:45 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 93657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B958C531D0 for ; Tue, 28 Jul 2026 01:41:47 +0000 (UTC) Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.2426.1785202906799628438 for ; Mon, 27 Jul 2026 18:41:46 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=ZNn6MYnD; spf=pass (domain: gmail.com, ip: 209.85.210.181, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-84e507b079dso1943219b3a.0 for ; Mon, 27 Jul 2026 18:41:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785202906; x=1785807706; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=4b9t2kE795ANKe1mv5hsNIyxoSO+MCakNXX2BaiSt04=; b=ZNn6MYnDDmquUWiEyZirzr3X2DoGwdRhQvgONBkSxtPSdoWJuawPymaMxo3W974T0D 5jTiDh6nvKIE+xp4T8zwffEjFxwgsP6Dnjp/XdM31BO3udJku0D8+ssIrgv7bj67BMh6 pdwpA8fKE1BuFTVNdLhWu0UvebPTdnzmBiDrNiGShAtFLlyXDcFKUAIm57UdBP4B0ih7 RbJuJ7C36BOB1XwhHsuKZktuva409iTkcZB3+0VQFJdgWsZXNjbqgtxQqlrzLjmkhNTI +Oi5an7TWxZoicpvCxyWScCV4Qw4dLQOiGgKODij1CQ3yvnE9vrSxdOjQ44R9T+viB2d tFrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785202906; x=1785807706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=4b9t2kE795ANKe1mv5hsNIyxoSO+MCakNXX2BaiSt04=; b=hAkCiY4xzPePcnWAlzbGgsxu9uP6hlRflbfa/FTdTdiaDLs+XSAVlmyul/7dXnfJvL vfl7FZJBARbGqF93bZSWv2T5rM4yVk0E6DXL537Tys+kxVTT5s5l/d9aRPSBFNWpOvWz bPURuvE9DxKpvlJ8uYbHey8hX66R5J0E7ARj67HziuJfh5lG7+Af82GiZ26raW16P8ZS hS1rt/f9vZaXtGI1eBJfaqmxz78FQz5qiuRSDlKFmbvUMHY616HmXdiLeoV2eHu1F1R0 7k0Xn8todmRaRFIVn8aHK/CRuWOsd8YOjXufA+PT/AyYSMJkTO15vQro7tkyLVRJz2ec Bm1w== X-Gm-Message-State: AOJu0YwxVw4BcfMr0kVqD0WivqG4AU+OjI32ondug8Gh/NaNDLernfu7 sdzcJMMyozNyRdDV2XbGmbSyYnOVln2IoMdxbnlxk0DJlshkYETPA0HvOB8i8w== X-Gm-Gg: AR+sD10t3u6/cn+8hQ9gVR6A9lrPk5XAh338wFGCqN1p2zQDF4s2ZzPHbP2NGd1QvGI mXmh7HT0gnbrpBPQzC8VvEHQFE9trctCorfnkVfQHjZlUVSLvogz6Q0ch2N/ufjNT6qZQJLHF0Q CpV5JHpC6IDEMHNSKBuIcceOXry88GvsX6CNlvLNlbmJftd3+vlt66x0/vOYAfatW3QxbTnm+QV fCJVxK4phKieL/CJ+uCC8RghwTaCflMHbs9rcNwWNXkCO+JkUbj8PIDgPkDD2Px3BpMRXE9K+SS x39CSs24zePjRA6gaPIToIsNr7GNUrI2aG45wrrjFANtgtl9PxvB6pxLCiCLkBeZCXNVBhdiHu0 JVIpwkFugbiBX7NVcQUc3v0N7lBzj/EqBkzYVNR2XQw5TX2a0VhPJfsZ/XNvdXiQ1Wl/TqZmNok dUdzuPxC0GVXe/bbxjJwpqkZw7PUI4O9aUra7u/LdzqmMCogC1VDbjPH4GviPvKb2WrNuFGr/bA A== X-Received: by 2002:aa7:8883:0:b0:847:84a1:9782 with SMTP id d2e1a72fcca58-84e93315f3fmr329749b3a.26.1785202906155; Mon, 27 Jul 2026 18:41:46 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5344e577sm3661537b3a.58.2026.07.27.18.41.45 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Jul 2026 18:41:45 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v5 12/12] config.json: set VCONTAINER_TARBALL_URL Date: Mon, 27 Jul 2026 18:40:45 -0700 Message-ID: <7ff37632f6b606ce17dabf815a8b30cd7e93a02f.1785198322.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 Jul 2026 01:41:47 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4551 For now, set VCONTAINER_TARBALL_URL to .../pub/vcontainer-tarball-latest/... which is where the vcontainer-tarball job stages the under-test most recent build, until we decide if we want to use a release build. Signed-off-by: Tim Orling --- config.json | 2 ++ 1 file changed, 2 insertions(+) diff --git a/config.json b/config.json index 8c23c2b..9a14c5b 100644 --- a/config.json +++ b/config.json @@ -12,6 +12,8 @@ "BUILDTOOLS_MAKE_URL" : "https://downloads.yoctoproject.org/releases/yocto/yocto-5.0.4/buildtools/x86_64-buildtools-make-nativesdk-standalone-5.0.4.sh;sha256=2eb7a6c013113f4fdb87e800167606dc13af1bd8bbc1c9e2443b7be37fefd124", "EXTRATOOLS_URL" : "https://downloads.yoctoproject.org/tools/buildtools/x86_64-buildtools-imagemagick-nativesdk-standalone-4.3+snapshot-5f2ba20f203114db9a3b11264467f8c23a05041d.sh;sha256=9cbff3a7cf524bdfa7779dce8afaf3453114d8017918d2927f723ea38a36ebdc", + "VCONTAINER_TARBALL_URL" : "${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh", + "REPO_STASH_DIR" : "${BASE_HOMEDIR}/git/mirror", "TRASH_DIR" : "${BASE_HOMEDIR}/git/trash",