From patchwork Mon Jun 1 23:18:00 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 336DDCD5BD1 for ; Mon, 1 Jun 2026 23:18:29 +0000 (UTC) Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.12804.1780355904591640026 for ; Mon, 01 Jun 2026 16:18:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=YSYSQKae; spf=pass (domain: konsulko.com, ip: 209.85.214.176, mailfrom: tim.orling@konsulko.com) Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2bf1f074a12so38650525ad.0 for ; Mon, 01 Jun 2026 16:18:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355904; x=1780960704; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=SBIw3hIjvJrIbgvIBjSdTntx3ahE30r50oyPmZupKrM=; b=YSYSQKaeJqw6NTqM5Gqb93BGlwJqoow2EO1BDW8i2WP6KgpWgDgS0TMHbcgNQFYgV0 dlQUZRM1ylNRnN1/QFhnhURAWlImI2zE3pf013F6JWNUWGYJX70AeIukeDiI540SK5SH KBZRsqGyD2cEgzJZveQAbRNJ3QZmhv8kZyIqU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355904; x=1780960704; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=SBIw3hIjvJrIbgvIBjSdTntx3ahE30r50oyPmZupKrM=; b=SV3f7B/bE5sxrQehcp8Tu4Sxb83lDh3lSJ47NN49oFx73E4blh52ec4PTbzgT2MBGX DCdTDawz25rhylBYC+UBVpmrPKchagWdumMadPok5pJqH+e7usjocx36AW9Kdk+hDCM8 eGkeNRL0bww2QIxcuha/7FqP1dj2CAvY7Aclzi2obG/CAfB0/9rJtPW9at01pyw8LkbF 9LTNhfJMnGW9mX9cn7E0qBvup94wl2BwSObQe1hLEe2/YsZQOUqwiI+RmnMEEjN9FCr0 zN/uAx7KjqchInuxaXm+p0CGkNd4ymx1zj4350PmhZvYpwl3FtSKPblQ0KkJKFlTViuc 38DQ== X-Gm-Message-State: AOJu0YxSCzoV24u1Zf4jTAxH173LAq4uUVNEeqZZyALKTATfVcWD8DQ5 X5I9m74oR4Ax1ZNmKC12SiKFx0nu5PBGZr6vFGAYtvJsP9HOw1T5G25CqfBCUEKfdEcur8e7PkV tN1fd X-Gm-Gg: Acq92OF31877NXXKU8PVMxtUGPevYboZtOThMh6fCGbmME9lW5szBpK/89/9C9qqBjJ n0LlHq10RHB71JbLCWDbDMT8bepvoBZheFOipUotCxPTZ8OIVQ3YbNQWhAWcwfJV7kiLTXpqOj0 POM8epg9l4WUF4np7J2Fck8jqGl6/bR8f7jMseKdhqiZPvSu10cPwvw0OJwZ3bLLBYXRpQ4dVLX wRba3ugLPQ9LmjkAbvJBuJM2SbYrgvaGoJHsfEiw/s7KWz6sujlK/+kMFLX8XSyKbOSh435kNdE hZxEGWJpHSOQmuuc8CfUt86jj4+sv39LFJAduFEh7u+i6dLDcBshUyOR9WO305pp5pMvlk3cQuL qsbNF3GEyeyu5mxLOKnl+9AG442EC0lxxAxd/qxXdsTjQ8QcFGh6UbY7N3bbK1AsnUGjRRxlnfq WiIxvPXkSFo+RKQSEeyjkbNVA4B77Jp65Hbz6BinJuULIzDeX9mDVulLgJX/lkP3JSNBUwJaBc7 g== X-Received: by 2002:a17:903:17c8:b0:2bf:356f:4e17 with SMTP id d9443c01a7336-2bf367dbb52mr146935235ad.13.1780355903954; Mon, 01 Jun 2026 16:18:23 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2bf23b001a8sm117627545ad.40.2026.06.01.16.18.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:23 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated Date: Mon, 1 Jun 2026 16:18:00 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:29 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4098 From: Tim Orling Previously the entire download/cache-validation block in setup_tools_tarball() was guarded by `if not os.path.exists(btdir)`. Once the extraction directory existed from a prior build, every subsequent call was a no-op: the cached tarball was never re-validated, so a freshly-published SDK (e.g. vcontainer-tarball-latest) was silently ignored and the stale btdir kept being used. Fix by moving the lock/download block outside the btdir existence guard so cache validation always runs. Track tarball_updated to know when the cached download was actually replaced, then remove the stale btdir when True so the fresh tarball is re-extracted. Also add an mtime-based staleness check for local-path (cp) sources so that a newer source file automatically invalidates the cached copy. AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- scripts/utils.py | 64 +++++++++++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 23 deletions(-) diff --git a/scripts/utils.py b/scripts/utils.py index 88842f0..87acad6 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -481,31 +481,49 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): if ";" in bttarball: bttarball, sha256 = bttarball.split(";") btdir = os.path.abspath(btdir) + btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball) + btlock = btdlpath + ".lock" + if not os.path.exists(os.path.dirname(btdlpath)): + os.makedirs(os.path.dirname(btdlpath), exist_ok=True) + # Always run the cache-validation / download step so that a freshly + # published tarball is detected even when btdir already exists from a + # previous build. tarball_updated is set to True whenever the cached + # download is replaced, which triggers removal of the stale btdir. + tarball_updated = False + while True: + try: + with open(btlock, 'a+') as lf: + fileno = lf.fileno() + fcntl.flock(fileno, fcntl.LOCK_EX) + if sha256 and os.path.exists(btdlpath): + dl_sha256 = sha256_file(btdlpath) + if dl_sha256 != sha256: + os.unlink(btdlpath) + elif bttarball.startswith("/") and os.path.exists(btdlpath): + # For local-path sources (e.g. vcontainer-tarball-latest) + # invalidate the cached copy when the source is newer so + # that a freshly-published tarball is always picked up. + if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath): + os.unlink(btdlpath) + os.unlink(btdlpath) + if not os.path.exists(btdlpath): + if bttarball.startswith("/"): + subprocess.check_call(["cp", bttarball, btdlpath]) + else: + subprocess.check_call(["wget", "-O", btdlpath, bttarball]) + os.chmod(btdlpath, 0o775) + tarball_updated = True + break + except OSError: + # We raced with someone else, try again + pass + # If the underlying tarball changed, remove any stale extraction + # directory so it is re-extracted below. + if tarball_updated and os.path.exists(btdir): + print("Removing stale %s extraction at %s" % (name, btdir)) + subprocess.check_call(["rm", "-rf", btdir]) if not os.path.exists(btdir): - btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball) print("Extracting %s %s" % (name, bttarball)) - btlock = btdlpath + ".lock" - if not os.path.exists(os.path.dirname(btdlpath)): - os.makedirs(os.path.dirname(btdlpath), exist_ok=True) - while True: - try: - with open(btlock, 'a+') as lf: - fileno = lf.fileno() - fcntl.flock(fileno, fcntl.LOCK_EX) - if sha256 and os.path.exists(btdlpath): - dl_sha256 = sha256_file(btdlpath) - if dl_sha256 != sha256: - os.unlink(btdlpath) - if not os.path.exists(btdlpath): - if bttarball.startswith("/"): - subprocess.check_call(["cp", bttarball, btdlpath]) - else: - subprocess.check_call(["wget", "-O", btdlpath, bttarball]) - os.chmod(btdlpath, 0o775) - break - except OSError: - # We raced with someone else, try again - pass subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) enable_tools_tarball(btdir, name) From patchwork Mon Jun 1 23:18:01 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CE6DCD6E55 for ; Mon, 1 Jun 2026 23:18:29 +0000 (UTC) Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.12710.1780355906572078617 for ; Mon, 01 Jun 2026 16:18:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=fXaBVEBf; spf=pass (domain: konsulko.com, ip: 209.85.214.171, mailfrom: tim.orling@konsulko.com) Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2bf114b0cf9so25473295ad.2 for ; Mon, 01 Jun 2026 16:18:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355906; x=1780960706; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Nh4bcp7Dv4HSkseIi709lWKX5+tSq1zA9PLL4tWYeAM=; b=fXaBVEBfmatW+1f61rjD3Abm8H+kQcje3V8BnOCj2PRN0Qhaq0IoqowndbMENhkv8M gknSp/fbljTORZeymkotHd2xrogaF2veVM3kGxW7QesgwdIcduGQqYCJStJ6TjyCS5s4 0N8EAZZ6YlqypbH41Q/XgPwfAXBOX0TeK67ro= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355906; x=1780960706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=Nh4bcp7Dv4HSkseIi709lWKX5+tSq1zA9PLL4tWYeAM=; b=XrFN/PaltrHXQAdvtD/TuS6RmDn6OI5yMt4zF62WBe0tnDyDO2SRsSsSiWrhh8eCFt BCK5weC4Za8HWQACe2S2f/k+KBgoe1HmveZseTyQ8AjsksuwgDdqa5KKu3xCXaKIv2Hq UEK6KpZT6PQro2e3gHVo8ZK7dM/HCdsVpmiNlPPUMpj+hjHIm/2hMN6/4cxCnlxktLEF clSr7ec0SrzXkeIop9GZOFwLrFxCm1AWc1sNGYhO9J/QTxQlDzJ3vQUIO1IgQuShDRRH 69ZXlIDpGBRsoaTEE0/OUnnutbRJ8wOZZYjMNm/Lu2WZb/uW0Qmt4rO92WyTp3N74FGm 5bkA== X-Gm-Message-State: AOJu0YxcwvkKjC6dbZNw9HtTRLchVSjC3dWcCVs+wV+ppF2hkNY5X0gC XgbPT/SIGD843sUYGTCGsSmR5cGGt5aeJBUwZ9EIXTKFtx5qOs1H1IkYLpiUMXnxWk6ginUTG61 obF1t X-Gm-Gg: Acq92OHB/22xTRCV9tGFw+hGssRI6DS741JiyoGX/YJSsqTn1I47VF8hX1MKuvjO6lV 4OudAo9OO/xPm/39F7O3yxNS/tHrFMyVoknm9+UTZxLzPVpM8W49VW1pn+77t1Pdz3ZLMRhnHmL ZZ5ldoIWY+D3YdBY9jIOlh4kotDU0gaUEmCNb4EEK8/9f7KivEQi18GmB6Clkxn55dy/YGhO7ed R9yCpksGNKX2MLQUv9qL8sFpCkmZxBx9UuMcfT42fF0TxAshWSoIvWBBzd0WSbcRfU/+xy8fdWg 7A5syDiJ/c526sdwgM10jM03XX9QnbSY2BqTFE7/9/J1F9sZgnW3qxGRWxAC91Fy398haTFWd1R YarYAoQVo1Jz2k9K7mrVJDPkrWL8OB1cf8EpwpeoehOFW/H9Gz0bpcHRWmlnWQYeKoOl3AiVyXw hax+XdTCbs0LdUJzQY6PCQ3a3ZxamV2hSnEyOqRgJNKuzmtBHLTIZtLz6bqRq5EY1ZR2NBxhkXq KKqmlBM2bR4 X-Received: by 2002:a17:903:3884:b0:2bf:33a9:bf4a with SMTP id d9443c01a7336-2bf368a3d92mr159245785ad.38.1780355905965; Mon, 01 Jun 2026 16:18:25 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2c0ce622c26sm68104205ad.67.2026.06.01.16.18.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:25 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 02/10] scripts/utils: warn and force re-download for HTTPS sources without SHA256 Date: Mon, 1 Jun 2026 16:18:01 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:29 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4099 From: Tim Orling Without a SHA256 checksum there is no way to verify that a cached HTTPS download is still current. Rather than silently reusing a potentially stale copy, delete the cached file and force a re-download each run, and emit a clear WARNING telling the operator how to avoid the overhead (by appending ;sha256= to the URL in their config). AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- scripts/utils.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/scripts/utils.py b/scripts/utils.py index 87acad6..ea905d9 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -505,6 +505,14 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): # that a freshly-published tarball is always picked up. if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath): os.unlink(btdlpath) + elif not bttarball.startswith("/") and os.path.exists(btdlpath): + # HTTPS/FTP source with no SHA256: there is no way to + # verify the cached copy is current, so force a + # re-download every run. Add a sha256= suffix to + # the URL in your config to avoid this. + print("WARNING: no SHA256 provided for %s source %s; " + "forcing re-download to avoid using a stale cached copy" + % (name, bttarball)) os.unlink(btdlpath) if not os.path.exists(btdlpath): if bttarball.startswith("/"): From patchwork Mon Jun 1 23:18:02 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 274DECD6E57 for ; Mon, 1 Jun 2026 23:18:29 +0000 (UTC) Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.12711.1780355908591917515 for ; Mon, 01 Jun 2026 16:18:28 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=JgsDifeJ; spf=pass (domain: konsulko.com, ip: 209.85.215.178, mailfrom: tim.orling@konsulko.com) Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-c8585ecdd71so953532a12.0 for ; Mon, 01 Jun 2026 16:18:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355908; x=1780960708; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=ehfwuZgaXhLcgzLPWhIEpy0u++Bs/zTf491pbLjBbNM=; b=JgsDifeJ9WrhlnHBUPP4lQPgVfp3eGoiXFbRrz/K0kbh9g5eDXdoVNqh3TwZ/gckBJ mB5dghsJgsus9S/ydg9agnssdZe46HSHHSzIXVaDG152pftbxKhRnVqUIjYRv5TscUuf GXyHg2aPYePU+6gvkRdjdOY62vEcE4aBKK2lA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355908; x=1780960708; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=ehfwuZgaXhLcgzLPWhIEpy0u++Bs/zTf491pbLjBbNM=; b=Jpwp0IRtAsrQzSg7ld5qlaVcaLdU2punsZpxxBQXzAXe0jCXeR1MN3YztFxXPU/y0a rsCFiH1HQFVrJaQX8p5D33b7krWC3HkBiJbD+HWYb/7CX+fbUey4wgcP6HMAwsH3RwRv 3VjTvLCYsm+cxonH3pHg5Mmz9W25gZNeGLEdPswDXaHfMpQjG1IrTyWfHqlv05EZIKsx O4ZUys0h+xy4CR7U4tksT4SPdaarjuFY5usBzBrvBDauVNuTMqajgcDg9QIOx1M6toYE ZNzRVYUpMhV6KUt5BTpACDcPFAYmn3cmNldo3KbK7jL49U+QF6ZFiLAORgWHxeOXD/ex eG1Q== X-Gm-Message-State: AOJu0YymqG04emMT6XXOBXIbRBUCuthbFu7vdkTK0UYGAhroeKC8mwMm 585yrIqX/tT7Apiho6dwOIS+q76nmeENI3v2JwjtysRe1iQaQvnyUdw3j65ZDVdFsc+3aKQJyVs wJ75d X-Gm-Gg: Acq92OHShK8zHrysPjBmwTxkfP2SSN2pFBVSPAWSUPY8Lq3Shg1aCsX8nj624L6JODL Iog+om8f+5N5Wl12UlXSwEOo8Zh1hwL58KUngVLCxbln7zJM6f3+NP//a+nM2jaKZRre7o3PUem JsIpHzb1ZqPByjlfpbxKqjhtaq526SgAa0WE09iymuamJiKIzKqt0bBSQPfGo5viNonJItpvmbV eT1glYUNcRSbnFHjpCxkKupqIptQTosdzwB2Py8osIPCBX6GSnctgG06/2HVoxQYnr/TTbRa9eW i8GvUGcHuH/3XBBX/LJ8Vl5p4Ym3goI3kJBXrWpizH4PyrhQqUY0762/NQLUDIT1jKZ/4QR2+ZJ C5LFNEkmiPHtjTcTX2HmYrZpuldxh92ubwwzyCZ4GKrh6YDnLHnQijFIzA379kdEiTxsRyyTlGq AQXVmVOQLehm6XYxeJLLNmT0EdLmKTV816QuOSZmEkl69ykuT+Y+C3U5Q1VxMtOWjXKcWUPpLev Q== X-Received: by 2002:a05:6a21:6f8b:b0:34f:14d6:15f5 with SMTP id adf61e73a8af0-3b427f2fa22mr14659566637.29.1780355908018; Mon, 01 Jun 2026 16:18:28 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-c85772993c9sm12550525a12.15.2026.06.01.16.18.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:27 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 03/10] config.json: add sha256sum for BUILDTOOLS URLs Date: Mon, 1 Jun 2026 16:18:02 -0700 Message-ID: <5fc50caa59790443a710534af53b206af8be8edb.1780354513.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:29 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4100 From: Tim Orling setup_tools_tarball() now checks the sha256sum of cached tools tarballs to determine if an update is available and needs to be re-downloaded. To prevent unnecessary downloads, add ;sha256= to the BUILDTOOLS URLs. extratools/EXTRATOOLS_URL uses the same utils.setup_tools_tarball(), so it also needs the ;sha256=. Signed-off-by: Tim Orling --- config.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/config.json b/config.json index b27fee9..7df4271 100644 --- a/config.json +++ b/config.json @@ -7,10 +7,10 @@ "BUILD_HISTORY_REPO" : "ssh://git@push.yoctoproject.org/poky-buildhistory", "BUILD_HISTORY_FORKPUSH" : {"openembedded-core-contrib:ross/mut" : "openembedded-core:master", "openembedded-core-contrib:abelloni/master-next": "openembedded-core:master", "openembedded-core:master-next" : "openembedded-core:master"}, - "BUILDTOOLS_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.1.sh", - "BUILDTOOLS_ARM_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/aarch64-buildtools-extended-nativesdk-standalone-5.1.sh", - "BUILDTOOLS_MAKE_URL" : "https://downloads.yoctoproject.org/releases/yocto/yocto-5.0.4/buildtools/x86_64-buildtools-make-nativesdk-standalone-5.0.4.sh", - "EXTRATOOLS_URL" : "https://downloads.yoctoproject.org/tools/buildtools/x86_64-buildtools-imagemagick-nativesdk-standalone-4.3+snapshot-5f2ba20f203114db9a3b11264467f8c23a05041d.sh", + "BUILDTOOLS_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.1.sh;sha256=5af9d92898af17fcc2fca4d07607a59f41b1c39a0a4ff058f64d834b98ec7fd5", + "BUILDTOOLS_ARM_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/aarch64-buildtools-extended-nativesdk-standalone-5.1.sh;sha256=8074d582a60a5135fd5606b8326e749185d26f51aca27570447831310bcac187", + "BUILDTOOLS_MAKE_URL" : "https://downloads.yoctoproject.org/releases/yocto/yocto-5.0.4/buildtools/x86_64-buildtools-make-nativesdk-standalone-5.0.4.sh;sha256=2eb7a6c013113f4fdb87e800167606dc13af1bd8bbc1c9e2443b7be37fefd124", + "EXTRATOOLS_URL" : "https://downloads.yoctoproject.org/tools/buildtools/x86_64-buildtools-imagemagick-nativesdk-standalone-4.3+snapshot-5f2ba20f203114db9a3b11264467f8c23a05041d.sh;sha256=9cbff3a7cf524bdfa7779dce8afaf3453114d8017918d2927f723ea38a36ebdc", "REPO_STASH_DIR" : "${BASE_HOMEDIR}/git/mirror", "TRASH_DIR" : "${BASE_HOMEDIR}/git/trash", From patchwork Mon Jun 1 23:18:03 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B264CD6E65 for ; Mon, 1 Jun 2026 23:18:39 +0000 (UTC) Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.12713.1780355910656147910 for ; Mon, 01 Jun 2026 16:18:30 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=lKS7b3mc; spf=pass (domain: konsulko.com, ip: 209.85.216.42, mailfrom: tim.orling@konsulko.com) Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-36b8e1760ccso2719562a91.0 for ; Mon, 01 Jun 2026 16:18:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355910; x=1780960710; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Wup+ARkC5NxdlM6S9euHh+cIF99+71JfQqhwZqaIX50=; b=lKS7b3mcs4l8xgSuILkDliqRDAOq9B5jDh2anB7ZzulJC/f5hCqtXp25rKIYH5OX1y 9aiPIZ5Do5mftUjFhG2EqO8Mns2UQTqu2LRXeH2/slITAC+4WT3DE90hYCYaEh9nhzJE EvssM8lRpFrsYcikUKDhWRX+Wo4UmyufQ40XY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355910; x=1780960710; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=Wup+ARkC5NxdlM6S9euHh+cIF99+71JfQqhwZqaIX50=; b=ary48PMy4RdnV5q+LE86eQJO5lbFhLp2640t9lLYFu9u+cONfjrADhXr1MUMrARaON s/iit1k9ExVL3edRzzsjQukBWG5PjrKmjZO5nukPS5TLTjgeOo7NihRLxUdFxxFoz4/l RFF52vyLASMB2FM27x8p3CQCJwsWApcjPerSYZMvruk8NA+6v8cvzMVmowItm71hJEwY WVjmVW90f1j1ehlliEt3ndaK2t1lMkMsX2POS6ii1Rcdr1bXUwLggOkFa1oknhKeOVFm 18bdUTlQDx9XqoPcE3H8qPsVXs21SJ5QZiKBTPJ6fmGsB/waqJyR27m1Axhvh6/U1sLY SZJw== X-Gm-Message-State: AOJu0YxswUs+xeC9bhHp61fpV263a32S6RAiXKds1A9LcdtjFjklsHhw 7DXbpIhxjE+/trdqdn+nsq2cPM7B3ER2RlK2320rWEZ5FdjMHzNyNhxsTrJ48gOueldAitfsx// 4iImB X-Gm-Gg: Acq92OEZ+kOzh5/2wdFFeLi7M5ptqG7dx1wPTVr1s/SPEcTUD0Sb/2AgBaR5h2HJlq8 qYpDOOOAjO0Zs66QX/JEBvm1Vh4j3UMNBGfw/Dts5E/MxifhQrCtAKVCQKH60kM9S/NxjzcRzfn S2XczW/ZqueEQwyCrvT5/R4/sqFRgbTjMZHpiaBQED6sAVcp5ez99JjkvDF2+//oHTOnS+FfNEV IQHb4BUlscsUeR1SbbhnMptlAZ2Pih5JuTPxnLyEMKhAirQOnwfWJdUg/hGFKLmRM7ngaozemOV gRPyO7hoNR1QRwIDN8mV1vj7GKq0FJYDPXgTd3rVJZfHrMcjGDUmK+BzrOcT54PK+NkP4eIjwPC d20z6dtfg7QsXjzZeIjg12iGHBS+DbctBKxhk5LjJp+jyO6mtXJSp1oypQovTRxvwN58TIZ2kqp mAkl3HRv0sMFexkuxUjAKgRqmeqmG8rWDhKir0WFVOQILIo9bUYrT12WOKR7cS8XgZVdEQy60T7 g== X-Received: by 2002:a17:90b:17ce:b0:36b:71e6:3de3 with SMTP id 98e67ed59e1d1-36c4fef326amr12660452a91.3.1780355910105; Mon, 01 Jun 2026 16:18:30 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-36dd9751ea1sm700191a91.12.2026.06.01.16.18.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:29 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 04/10] scripts/utils: add getconfigdict() for dict-type config values Date: Mon, 1 Jun 2026 16:18:03 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:39 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4101 From: Tim Orling Parallel to getconfiglist(), but for JSON object values. The merge priority is defaults < target-level < step-level so that more-specific keys win: a step can override individual entries in a target-level dict without replacing the whole thing, and both levels refine the defaults. Used by the upcoming CONTAINER_IMAGES support, where each entry maps a Yocto recipe name (the on-disk OCI path stem) to an image name (the name pushed to the container registry). AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- scripts/utils.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/scripts/utils.py b/scripts/utils.py index ea905d9..112ebc2 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -112,6 +112,21 @@ def getconfiglist(name, config, target, stepnum): ret.extend(config['defaults'][name]) return expandresult(ret, config) +# Get a build configuration dict, merging defaults < target < step so that +# more-specific entries win (step-level keys override target-level, which +# override defaults). +def getconfigdict(name, config, target, stepnum): + ret = {} + step = "step" + str(stepnum) + if name in config['defaults']: + ret.update(config['defaults'][name]) + if target in config['overrides']: + if name in config['overrides'][target]: + ret.update(config['overrides'][target][name]) + if step in config['overrides'][target] and name in config['overrides'][target][step]: + ret.update(config['overrides'][target][step][name]) + return expandresult(ret, config) + # Return only unique configuration values (identified with '=' in them) def getconfiglistfilter(name, config, target, stepnum): def merge(main, newvals): From patchwork Mon Jun 1 23:18:04 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B024CD6E61 for ; Mon, 1 Jun 2026 23:18:39 +0000 (UTC) Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.12714.1780355912813146290 for ; Mon, 01 Jun 2026 16:18:32 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=VBK5U8UA; spf=pass (domain: konsulko.com, ip: 209.85.215.170, mailfrom: tim.orling@konsulko.com) Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-c8584bbbf2cso1704539a12.3 for ; Mon, 01 Jun 2026 16:18:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355912; x=1780960712; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=LfEDyXtR2H5UTOnUnmi38QGbL6XG1SWvmAgTnwagwjc=; b=VBK5U8UACbvWKecA9YTE0hS4ELSNdSF44BK0UYRnCT5utE56OslrJCZHqo4MGMBhr0 6rWqyJHksSNI7SMIkzTW4E5C7lwra39rkGafNaYbbHmy1ltVUHAd3CapgFxkhxQLd8tc kOlSrh/rRG0Mc9OteRQKgzo1efZGzKsi3eh3I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355912; x=1780960712; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=LfEDyXtR2H5UTOnUnmi38QGbL6XG1SWvmAgTnwagwjc=; b=cgYYWuFMbA9Q3b4r64cmltuVHS5e2MOfgjeKlhVIrgtYcrSggwPe/awHYKUQ3wmM7G yI5r6fb9Qohqr6W1oe4/c15bFrtzH2FD92gL2Dbv22nNOHcE8H41vDC2j2shEV7dUxZp vE2EstqIeN6mPb/JO9b5XqJ8M1kuLjm4wx6C9/hubsM+LmsHZaf4pGDrMYfiuad3nCVp RMsHqF/wVDtRwr7arw3CE0UCPntS4fk+u/d6+19el8Z5z50PPVCWZQGq8dD6jT9H3nCS SBgwH6H2pyCDV1veX8ZJ77DqB7h94BhKbN0cj3Qf8Q5cBsa97g9awuoMBINKeubKnHE3 wfgw== X-Gm-Message-State: AOJu0YzMrMhZfcLmWBCN2p2xrbNLA1meMIvqhcIKHL4dk8DBip77Xz7e 0Myk9p3vH9v+uxjE51G4aVXn135xrUIxFIi1gcoOhPUbRkIiALoNsxyDpXN75UsI6hrO8DLGUcD EPk9M X-Gm-Gg: Acq92OHmaNA42UGobwgjLmm1dLlJW2oCOtJ2XrMpdOzDgRZ16s373stj3r8FEaqCMNM vwPKXw/M6XlZL6kBEwWkJz56n5yfpvX++tBk+LmhqW4eFscTn6od4Nsjw+fEouT5z+Y8fg4D1wR oweQrMhaMA8inO9GesixcFLhixRKqv2nNXbP0zxEXWX2iTll8i8fp1KAcy/enTCCfouLWxOZyeN WeFcB1FIa6XsmJOE14G7JHj84Qho0VWpI6b/azkUt7ds2Y27Flopj2Jtbcc/6lGPmmPsUg0eTVI eEaRviTRN9orrzvBWC0XQSJDltBWIPkKgK54sNgq7rySncqcZmN3f/qPg9472y1RJ2ImGzgM3/Y ClCdyC0KHM1OXypnnX7LoJe5OUo9shSkr/oqreNGhnPFKsf59EcikLefJSuK75H4TfgvYYHccKq fqh1m5XbmU7iOiTkLQc2BDzUTZ7gHH4q4C+3YJDsD/vVtNYTQdBPNJ5sUoD7es2cGm5caEsJGQm Q== X-Received: by 2002:a05:6a21:512:b0:3b2:86a4:b01c with SMTP id adf61e73a8af0-3b427f2b31fmr13983966637.26.1780355912092; Mon, 01 Jun 2026 16:18:32 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-c85771a7c24sm12021678a12.2.2026.06.01.16.18.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:31 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing Date: Mon, 1 Jun 2026 16:18:04 -0700 Message-ID: <2cf468a3da7c7db15ee42c14404a71aaacab0f28.1780354513.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:39 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4102 From: Tim Orling Introduce the vcontainer-tarball SDK plumbing used by container build jobs. The vcontainer-tarball is a meta-virtualization-derived SDK (modelled after buildtools-tarball) that ships the container build toolchain so worker jobs do not need to rebuild it for every step. * scripts/utils.py: add setup_vcontainer_tarball(), and add an env_glob keyword argument to setup_tools_tarball() and enable_tools_tarball() so the vcontainer-tarball can source its specific environment-setup-ci file rather than the universal glob. * scripts/run-config: source the vcontainer-tarball environment for build-targets / cmds / test-targets / plain-cmds steps, gated by a new NOVCONTAINER step variable so individual steps (such as the dashboard indexing step) can opt out independently of NOBUILDTOOLS. * scripts/shared-repo-unpack: invoke setup_vcontainer_tarball so workers extract the SDK during unpack. * scripts/publish-artefacts: publish the vcontainer-tarball artefact so downstream test jobs can fetch a stable SDK. AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling --- config.json | 1 + scripts/publish-artefacts | 5 +++++ scripts/run-config | 19 +++++++++++++++++++ scripts/shared-repo-unpack | 1 + scripts/utils.py | 22 ++++++++++++++++++---- 5 files changed, 44 insertions(+), 4 deletions(-) diff --git a/config.json b/config.json index 7df4271..16d8a04 100644 --- a/config.json +++ b/config.json @@ -1433,6 +1433,7 @@ "step3" : { "shortname" : "Populate/update dashboard site", "NOBUILDTOOLS" : true, + "NOVCONTAINER" : true, "EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index ${HELPERBUILDDIR}/../"] } }, diff --git a/scripts/publish-artefacts b/scripts/publish-artefacts index e56e131..0e820e9 100755 --- a/scripts/publish-artefacts +++ b/scripts/publish-artefacts @@ -146,5 +146,10 @@ case "$target" in sha256sums $TMPDIR/deploy/images/qemux86-64 cp -R --no-dereference --preserve=links $TMPDIR/deploy/images/qemux86-64/*qemux86* $DEST/patchtest ;; + "vcontainer-tarball") + mkdir -p $DEST/vcontainer-tarball + sha256sums $TMPDIR/deploy/sdk + cp -R --no-dereference --preserve=links $TMPDIR/deploy/sdk/*vcontainer* $DEST/vcontainer-tarball + ;; esac diff --git a/scripts/run-config b/scripts/run-config index e896234..0f5a26a 100755 --- a/scripts/run-config +++ b/scripts/run-config @@ -153,6 +153,25 @@ else: if args.phase == "init" and args.stepname == "buildtools": sys.exit(0) +if jcfg: + vcontainer = utils.setup_vcontainer_tarball(ourconfig, args.workername, None, checkonly=True) + if vcontainer: + addentry("vcontainer", "Setup vcontainer tarball", "init") +else: + # If we're executing a specific step, check whether vcontainer is disabled for it + vcontainer = True + if args.stepname in ("build-targets", "cmds", "test-targets", "plain-cmds"): + try: + vcontainer = not utils.getconfigvar("NOVCONTAINER", ourconfig, args.target, int(args.phase)) + except ValueError: + # Not an integer step phase + pass + + if vcontainer: + utils.setup_vcontainer_tarball(ourconfig, args.workername, args.builddir + "/../vcontainer-tarball") + if args.phase == "init" and args.stepname == "vcontainer": + sys.exit(0) + extratools = utils.getconfigvar("extratools", ourconfig, args.target) if jcfg: if extratools: diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack index 797dec6..869b214 100755 --- a/scripts/shared-repo-unpack +++ b/scripts/shared-repo-unpack @@ -77,6 +77,7 @@ for repo in sorted(repos.keys()): utils.flush() utils.setup_buildtools_tarball(ourconfig, args.workername, args.abworkdir + "/buildtools") +utils.setup_vcontainer_tarball(ourconfig, args.workername, args.abworkdir + "/vcontainer-tarball") if "bitbake" not in repos: sys.exit(0) diff --git a/scripts/utils.py b/scripts/utils.py index 112ebc2..b020a7b 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -456,8 +456,8 @@ def sha256_file(filename): pass return method.hexdigest() -def enable_tools_tarball(btdir, name): - btenv = glob.glob(btdir + "/environment-setup*") +def enable_tools_tarball(btdir, name, env_glob="/environment-setup*"): + btenv = glob.glob(btdir + env_glob) print("Using %s %s" % (name, btenv)) # We either parse or wrap all our execution calls, rock and a hard place :( with open(btenv[0], "r") as f: @@ -474,6 +474,20 @@ def enable_tools_tarball(btdir, name): if line in os.environ: del os.environ[line] +def setup_vcontainer_tarball(ourconfig, workername, vcdir, checkonly=False): + vctarball = None + if "vcontainer" in ourconfig and workername: + vccfg = getconfig("vcontainer", ourconfig) + for entry in vccfg: + if fnmatch.fnmatch(workername, entry): + vctarball = vccfg[entry] + break + + if checkonly: + return vctarball + + setup_tools_tarball(ourconfig, vcdir, vctarball, name="vcontainer-tarball", env_glob="/environment-setup-ci") + def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): bttarball = None if "buildtools" in ourconfig and workername: @@ -488,7 +502,7 @@ def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): setup_tools_tarball(ourconfig, btdir, bttarball) -def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): +def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools", env_glob="/environment-setup*"): btenv = None if bttarball: @@ -548,7 +562,7 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): if not os.path.exists(btdir): print("Extracting %s %s" % (name, bttarball)) subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) - enable_tools_tarball(btdir, name) + enable_tools_tarball(btdir, name, env_glob) def get_string_from_version(version, milestone=None, rc=None): """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists, From patchwork Mon Jun 1 23:18:05 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89120 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45674CD6E55 for ; Mon, 1 Jun 2026 23:18:39 +0000 (UTC) Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.12715.1780355914917257488 for ; Mon, 01 Jun 2026 16:18:34 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=U1XNlBWp; spf=pass (domain: konsulko.com, ip: 209.85.216.41, mailfrom: tim.orling@konsulko.com) Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-36b8d414666so2896179a91.3 for ; Mon, 01 Jun 2026 16:18:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355914; x=1780960714; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=fR/tiEwkmI9AuzyCs9P34K1Bt3u8bKLCLuR4F330iHQ=; b=U1XNlBWpyvZTI5h4/tOp+YDviGwwPcito9imGLyszcRZLnDaB1ngB/5GGW5tvpaIHH 6CJd2EHhz8ulBsRAGCkINMybE+pXKmgtfyzLs31le2mlm/kydA1Zh2MPe0JRYSRtNT+P vWxMFvrJzUhM/+RJJT9aMr9GDLgoELDlaT+p8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355914; x=1780960714; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=fR/tiEwkmI9AuzyCs9P34K1Bt3u8bKLCLuR4F330iHQ=; b=WXyeojJ2B80BEwoDOoUYBHE2//BmOh+q8CW4FStshHmkz9OP+wxJxhDBK+ghPT9B// RltDOTpml5hKrd1+yUdvqnmfzokR8ExaW/kYx/ANbrSMZVsVgYt9zLxBB92PkISUVKlu aNeHig8mss1lNLvxVz7g7dStsEq1Hn2QoMB+Z66YAKxM948Ye1iODJLWGA0b3SBOPEV+ iSMeJgtCjofQ0JizS5xYssosd4kCFzyUAilmrVmNj5YOUOz9EtNe++aRsYQz/w6Vz32X L+Jj6CXw4M2+IzeAHNzPoTso6U/dxfMNKWijxZh1ZmDPWCqSLgTxUHNUflTSUO+XA4CX y21Q== X-Gm-Message-State: AOJu0YzMbp/lFdoBIYLeys5T5+L5Mqm4QW/AljXqWrm7rXkOq3HxGgTC 7lkCfrI3JQFHR6xqjLMW+9AgN6wKRsqRh2YtNFE97qfFYwsNFJ4zc0aF2omHRFkFqEXandqFti9 GsrHW X-Gm-Gg: Acq92OHFXQ3L1PFu1ZGj8cNALCbuxd52Mgsc/SQBSRj/uvdoZaFtYEsURWjeNX2Gb7S ifu53UYAVLZEEhvyEdYnE2Fmz0UuLgEdtIGnsdc5Wiu9MCUNQhoNsXXXWtj52wOsV5CwyCqe513 YRT4VGZJ941WjpBgfQ1UQ3lvOg5tFJK5JWpaZkmUOJRlq/5Wdor6wegU2GwpeY6uw/OMNADrHxR deTYwdmyRCT3VtXfMAWoOpsum2WCrUi3dj01o2j0IOiGT/og84Dg6Pco+0lZaYPfpACno7f+Q1u yCGwmfs9q0Y5kju+rrFN5/gitm/LQfbUCD5tPQ/Ccp65TgNFcte2ohmjS4OFeNj9y3SZ0hYwK8w E8znWWRua4pgNV9em002SHIDcYsGCixOTFFN2yw+BpaRtEoNaLWH9AVNn7Nx53ybKAM4i682M+o qFzaPoISC3XgUAz2JopE/RfnUheyZiGWSOj2Wva4NcR6iDX3m6CSE7/578pkbUKHXaDduOhVbwA w== X-Received: by 2002:a17:90b:4a91:b0:36c:689c:a4c9 with SMTP id 98e67ed59e1d1-36c689cb2efmr12331985a91.21.1780355914334; Mon, 01 Jun 2026 16:18:34 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-36bbe19dc07sm6848325a91.5.2026.06.01.16.18.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:33 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 06/10] config.json: add vcontainer-tarball build target Date: Mon, 1 Jun 2026 16:18:05 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:39 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4103 From: Tim Orling Add the vcontainer-tarball build target which produces the meta-virtualization SDK used by downstream container build/test jobs. Modelled after the buildtools-tarball target. The target uses EXTRACMDS to reset BBMULTICONFIG for the worker shell, places the extravars in the per-step config, and includes a publish-artefacts step so the resulting SDK tarball is staged for reuse by container-tests and other consumers. Since meta-virtualization is the first layer removed by 'remove-layers', our BBMULTICONFIGs become invalid immediately after layer removal and throw an error. In scripts/run-config, EXTRACMDS runs after BBTARGETS, but before 'remove-layers'. Use 'sed' to reset BBMULTICONFIG to ''. The error in the 'remove-layers' step can be summarized as: bb.parse.ParseError: ParseError at /home/pokybuild/yocto-worker/vcontainer-tarball/build/layers/openembedded-core/meta/conf/bitbake.conf:824: Could not include required file conf/multiconfig/vruntime-aarch64.conf Signed-off-by: Tim Orling --- config.json | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/config.json b/config.json index 16d8a04..7206c41 100644 --- a/config.json +++ b/config.json @@ -1864,6 +1864,32 @@ }, "toaster" : { "EXTRACMDS" : ["${SCRIPTSDIR}/run-toaster-tests ${HELPERBUILDDIR} ${HELPERBUILDDIR}/../layers/bitbake"] + }, + "vcontainer-tarball": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Build vcontainer-tarballs", + "BBTARGETS" : "vcontainer-tarball", + "extravars" : [ + "DISTRO_FEATURES:append = ' virtualization vcontainer'", + "BBMULTICONFIG = 'vruntime-aarch64 vruntime-x86-64'", + "INIT_MANAGER = 'systemd'" + ], + "EXTRACMDS" : ["sed -i '/vruntime-aarch64 vruntime-x86-64/d' ${HELPERBUILDDIR}/conf/auto.conf"] + }, + "step2" : { + "shortname" : "Publish vcontainer SDK for test reuse", + "EXTRACMDS" : [ + "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" + ] + } } }, "repo-defaults" : { From patchwork Mon Jun 1 23:18:06 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89121 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51534CD5BD1 for ; Mon, 1 Jun 2026 23:18:39 +0000 (UTC) Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.12717.1780355918068990977 for ; Mon, 01 Jun 2026 16:18:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=n8Qse1hW; spf=pass (domain: konsulko.com, ip: 209.85.210.179, mailfrom: tim.orling@konsulko.com) Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-8423f52af13so1218676b3a.2 for ; Mon, 01 Jun 2026 16:18:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355917; x=1780960717; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=c3YxBLJtmIQ3m4N+t4kJmwp56pZxSysUKkEjicX0XAo=; b=n8Qse1hWYv/OpK6StXwILGSkd7bvJBwOYnU55mrQdfRVPx8m/Cw2H4h1pHSPdIhILx UE5tG14zGgNSxoSrxdSQJ6pGGuJPDDNjzPCNAVmt/ce5OSQWRdq0W7q2Z/RtfQdwi4Nz 7VeLK7PMQpuSt1L5UPXocafKxac8pw+4cIbtY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355917; x=1780960717; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=c3YxBLJtmIQ3m4N+t4kJmwp56pZxSysUKkEjicX0XAo=; b=egIHnlMnywrdshaaGEtD6simrAPrmoZp8kTon3VBCYCFX2qYQJQhz+iUv3RkqkcUpA Iih5ev2mNcsZjD6JGttpxmShnMrmxSTTaBvDZML/3ABTq89Q61TlIeKT6RaK1UOkS4WS m4FJ1hqforuMYGFJrLlMcZEZJFE5EfsyroetvFa98cBnzGimGr4BvznXDyxOT4WlGpeJ 592x7FpCoMAwN93ZjoRRmadV/3lSgOfWxGUZFMOp7dMFNpghKig/Oe2XIvZoG0DRYnIQ Ees8z2Rvxr3ukTKL1QYBQuFu0y+EI4wKweELowTxZiPF/Q/rq0BvYhtBplGCe+Jmdp+W zXVQ== X-Gm-Message-State: AOJu0YwMRA7/8Cn1FvBTTrBPPCBvXyO1TxXkK4vK6CZZcOhb3zt4mpgc 2ssgcFiloQbLiyQUSVkWjxWffNWAQO4ZjabDjk532an9ggU+Ca9DQpXPFv8ruZkyzL1bIvPo9QW OTzh2 X-Gm-Gg: Acq92OHymcF5gbublAaKhuNrVwIkMNq9VFAODpmio+ZKSvoju/CbCra+8q+37V3gKAB gf7/wQLwZenU5Sze6/ZjXsR5Rm2u95X+nYjKPV7OhHsUSWRH6whJXDa1J4VfrxUK14Lib/qdqhx PPFmWB2D7kmHvCYZMeTSDSgNR/d4u1djC5Ffom+D47/l0JJAtn+oB9twQFVOob6p+g71FV5v6Y+ +j5rQvWN4SE6zlxFKD8dIVmCV8AhSNR0gWKZllyD7V/UCUsVX2LizJ1vEcoiA4zaT6JKzow4HB7 q4Alk8Of3Lamo5sDD0knHEot9rr5VPqKNfJbz+BGLoM2qCOkiuNLkHdwLvMRFy1yR0G5FUH718C JPLUdy0Qo+EgzLBXLEOzSjmjt2cLsUbrUVvWZNlvSMIzqtKiqtYecMe6hBKH3//9LtcFCfY4569 TiLVyKDf9uJnhVOaLhjYM1yCk0D4PU7IJvf5bXFFAa05ZZWZ8G3udc0rIdRqRMIXgVbk90RWJvi d57jnc7LbTt X-Received: by 2002:a05:6a00:2450:b0:82c:6b46:271d with SMTP id d2e1a72fcca58-842257240cfmr12072730b3a.48.1780355917365; Mon, 01 Jun 2026 16:18:37 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84214ce85d2sm11069030b3a.51.2026.06.01.16.18.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:36 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 07/10] scripts: add run-vcontainer-tests for meta-virtualization Date: Mon, 1 Jun 2026 16:18:06 -0700 Message-ID: <1b74cb9ed9d6f1865141964682958160f4041d44.1780354513.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:39 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4104 From: Tim Orling Add scripts/run-vcontainer-tests, the test runner used by the vcontainer test jobs. It sources the vcontainer-tarball SDK, discovers the meta-virtualization pytest suite, and runs a configurable set of suites (vdkr, vpdmn, memres) against the checked-out layers. Suites can be selected per-step so the top-level 'vcontainer-tests' job runs the container engine agnostic tests: - tests/test_container_cross_install.py - tests/test_container_registry_script.py - tests/test_vcontainer_auth_config.py - tests/test_multiarch_oci.py - tests/test_multilayer_oci.py The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective suites (including memres for each container engine): - tests/test_vdkr.py - tests/test_vdkr_registry.py and - tests/test_vpdmn.py AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling --- config.json | 34 ++++++++ scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++ 2 files changed, 198 insertions(+) create mode 100755 scripts/run-vcontainer-tests diff --git a/config.json b/config.json index 7206c41..79a9d10 100644 --- a/config.json +++ b/config.json @@ -1890,6 +1890,40 @@ "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" ] } + }, + "vcontainer-tests": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Run vcontainer pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + }, + "step2" : { + "shortname" : "Run vdkr pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + }, + "step3" : { + "shortname" : "Run vpdmn pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + } } }, "repo-defaults" : { diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests new file mode 100755 index 0000000..1394c7c --- /dev/null +++ b/scripts/run-vcontainer-tests @@ -0,0 +1,164 @@ +#!/bin/bash +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Run meta-virtualization pytest test suites against the vcontainer +# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake +# step. +# +# Arguments: +# $1 - suite name: one of "vcontainer", "vdkr", "vpdmn" +# $2 - bitbake build directory (${BUILDDIR}) +# $3 - path to the meta-virtualization layer +# +# Optional environment variables: +# RESULTS_DIR - directory to copy pytest artefacts (junit xml / log) to +# VCONTAINER_EXTRACT_DIR - where to extract the standalone SDK tarball +# (default: ${builddir}/vcontainer-test-extracted) +# TEST_OCI_IMAGE - path to an OCI image directory (enables vdkr/vpdmn +# import tests) +# VDKR_ARCH - target architecture for vdkr/vpdmn tests (default: x86_64) +# +# The script is intentionally conservative: any pytest tests that cannot run +# in the CI environment (those marked "slow", "network", "boot") are skipped +# so that the autobuilder step completes without needing network access. Those +# can be re-enabled by exporting META_VIRT_PYTEST_MARKERS before invocation. +# +# It is assumed that /dev/kvm is writable by the CI user running the tests, +# since the performance is significantly faster with 'memres'. +# + +set -e +set -u +set -o pipefail +set -x + +if [ $# -lt 3 ]; then + echo "Usage: $0 " >&2 + echo " suite: vcontainer | vdkr | vpdmn" >&2 + exit 2 +fi + +suite="$1" +builddir=$(realpath "$2") +metavirtdir=$(realpath "$3") + +if [ ! -d "$metavirtdir/tests" ]; then + echo "ERROR: meta-virtualization tests directory not found at $metavirtdir/tests" >&2 + exit 1 +fi + +# Locate the vcontainer standalone SDK tarball. Prefer an externally-built +# SDK passed via VCONTAINER_SDK (the autobuilder -tests jobs share the SDK +# produced by the separate vcontainer-tarball builder), and fall back to +# looking in the local build's deploy/sdk directory when running stand-alone. +sdk_tarball="" +if [ -n "${VCONTAINER_SDK:-}" ]; then + if [ -f "$VCONTAINER_SDK" ]; then + sdk_tarball="$VCONTAINER_SDK" + else + echo "ERROR: VCONTAINER_SDK=$VCONTAINER_SDK is set but not a file" >&2 + exit 1 + fi +fi +if [ -z "$sdk_tarball" ]; then + sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh" + if [ ! -f "$sdk_tarball" ]; then + # Try to find any matching tarball in case naming changed (e.g. versioned) + alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh 2>/dev/null | head -n1 || true) + if [ -n "$alt" ]; then + sdk_tarball="$alt" + else + echo "ERROR: vcontainer standalone SDK not found." >&2 + echo " Set VCONTAINER_SDK to an existing SDK installer, or" >&2 + echo " build vcontainer-tarball so $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2 + exit 1 + fi + fi +fi + +extract_dir="${VCONTAINER_EXTRACT_DIR:-$builddir/vcontainer-test-extracted}" +rm -rf "$extract_dir" +mkdir -p "$(dirname "$extract_dir")" + +# Self-extracting installer (silent, -y agrees to license, -d picks dir) +"$sdk_tarball" -d "$extract_dir" -y + +# Prepare a Python venv so we don't pollute the worker's system packages. +python3 -m venv "$builddir/meta-virt-test-venv" +# shellcheck disable=SC1091 +source "$builddir/meta-virt-test-venv/bin/activate" +# Avoid warnings by upgrading pip; install pytest/pexpect into the venv via pip. +python3 -m pip install --quiet --upgrade pip setuptools wheel +python3 -m pip install --quiet --upgrade pytest pytest-timeout pexpect + +# Default marker filter excludes long running / infrastructure dependent tests. +marker_filter="${META_VIRT_PYTEST_MARKERS:-not slow and not network and not boot and not incus and not k3s}" + +# Per-suite test file selection. Uses -k/-m for fine-grained filtering and +# keeps the CLI small for logging clarity. +case "$suite" in + vdkr) + test_files=( + "tests/test_vdkr.py" + "tests/test_vdkr_registry.py" + ) + ;; + vpdmn) + test_files=( + "tests/test_vpdmn.py" + ) + ;; + vcontainer) + # Broad vcontainer/bbclass/tooling coverage that doesn't require the + # vdkr/vpdmn CLI harness to be running. + test_files=( + "tests/test_container_cross_install.py" + "tests/test_container_registry_script.py" + "tests/test_vcontainer_auth_config.py" + "tests/test_multiarch_oci.py" + "tests/test_multilayer_oci.py" + ) + ;; + *) + echo "ERROR: unknown suite '$suite' (expected vcontainer|vdkr|vpdmn)" >&2 + exit 2 + ;; +esac + +pytest_args=( + -v + --tb=short + -m "$marker_filter" + --vdkr-dir "$extract_dir" + --junitxml="$builddir/pytest-$suite-results.xml" +) + +# Allow tests that consume an OCI image (import/save/load) to find one. +if [ -n "${TEST_OCI_IMAGE:-}" ] && [ -d "${TEST_OCI_IMAGE}" ]; then + pytest_args+=(--oci-image "$TEST_OCI_IMAGE") +fi + +# Pass architecture through when set in the environment (default is x86_64). +if [ -n "${VDKR_ARCH:-}" ]; then + pytest_args+=(--arch "$VDKR_ARCH") +fi + +cd "$metavirtdir" +# Don't let a single failing test kill the whole step - collect the junit +# report, then surface the exit code via the junit file + exit status. +set +e +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}" +rc=$? +set -e + +# Copy artefacts to the results dir if one was provided. +if [ -n "${RESULTS_DIR:-}" ]; then + mkdir -p "$RESULTS_DIR" + cp -f "$builddir/pytest-$suite-results.xml" "$RESULTS_DIR/" 2>/dev/null || true + if [ -f /tmp/pytest-vcontainer.log ]; then + cp -f /tmp/pytest-vcontainer.log "$RESULTS_DIR/pytest-$suite.log" || true + fi +fi + +exit $rc From patchwork Mon Jun 1 23:18:07 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89126 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79325CD5BD1 for ; Mon, 1 Jun 2026 23:18:49 +0000 (UTC) Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.12810.1780355920404827819 for ; Mon, 01 Jun 2026 16:18:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=bmcL3V8r; spf=pass (domain: konsulko.com, ip: 209.85.214.173, mailfrom: tim.orling@konsulko.com) Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2bf114b0cf9so25473875ad.2 for ; Mon, 01 Jun 2026 16:18:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355920; x=1780960720; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=R1bVo+m2+SeQAfcXST6hum76BOjKggBED5KLaEPA0bM=; b=bmcL3V8r1WCpIdjESpuCWGow0KynQUtwLHUZF8+0dqhDV3BWoMte+SaU35W3xLA20B es7knuyRwAKxACcAp/bLzxquanJXRLY6qSWt/NStt04jK5W6snmax/HoOKdPwlLgPhzn /JEkvWAelbsg9MRw4rn4upAN/uEudN7J22R+A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355920; x=1780960720; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=R1bVo+m2+SeQAfcXST6hum76BOjKggBED5KLaEPA0bM=; b=dk17qvPF6qqIBZqgFXos+nfNQIzyFJNJuz7ZMITGNsmVycU/qGz+h9nSBzyyuYKxDs k5TlpQfLYMCyxnmCB5r7Q/XOCWep9VHYiegdHHg7LWg8LWcR4NDQUfyLdm//iOfedWsF lhsM4jz1djyguf9d+srRVuoruxWPWQZBOqXMhriO/3QxwX8by46/yeBzPnfctI6Pqg8L aaLLmkjVkITEhAvwlM6Lx1iUrQDFwhDPjWj2xWhWGwRtu/dzVfd6SU9l5p5fNfSwesAj WHp3gSmjHXsrBUaM34TQ6mu4l9t8Kg07Z0LULuLPLxf4fgPAYKpxu7OZSPWpXYEKODuW WsOQ== X-Gm-Message-State: AOJu0YxDZE31Y/Gbub+oe3TaAEdwOjA7SYnhDwGSUiTNqYYUOfx/zuF3 PHBGaSn4VMNcAG59ObnlIJEEe9KeEloVr3WC5FH9wKk9gUCqXsM7+PxBKfNp/kwCUjo3vcKV6w6 IIqPI X-Gm-Gg: Acq92OG6K7vQMcvy5Kt0304t9DSj40nxIntNJvMJI7McTlRFajwVupFOPx/qaTqh1im lFnJMVcowz9AhRoxaikh2q+Ds+44CkeLtBuFwxIFa9zNUD0uxzA6++rN6w9Oydr18Dq+2lrjJPS y7g1ONvjfqxMkDgtlshNDVwRkKN2o/f8yshl59WMFMXf1hFTuIlnkc3ZBjA+BRI1+vFahGKrXrq rMj4m48GPdgOTyuBpiXBj0xL+oPxXhmjbdjmwHxA5or3hgmcSvabn+cR/ZjHRW/27C2yDI7fKoG oLZp2maWj88NStwgP34eY0lZmQlL9xfQxsE+kBZUIGfk7lbGAOt0v67hJWWv0jp9JmjMzL3PdsS Y5qWloof78YhtX06byikBSmqIaf8kF+ng2MyfCCzdVHdmd2OpwbK4FMU6kyYVLUlyF4W51655hJ IyApfOgCPgPM2hL2EH33dclGevLq7l/y4I32eKWgWv/mySvJ6g/0/XNgs2TE7Hbf2Po/ZG5yCci pxDkwKwGDVB X-Received: by 2002:a17:903:8c8:b0:2bd:606d:b342 with SMTP id d9443c01a7336-2bf36833abfmr137072665ad.26.1780355919784; Mon, 01 Jun 2026 16:18:39 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2bf23b01727sm120388715ad.53.2026.06.01.16.18.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:39 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 08/10] scripts: add container registry push, auth, tagging, runtime selection Date: Mon, 1 Jun 2026 16:18:07 -0700 Message-ID: <31f010d5e7dbc1e2307dfb00f06a9b6380a19c5c.1780354513.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:49 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4105 From: Tim Orling Add the push-containers infrastructure that drives the post-build steps for the 'containers-library' job. After each build step the runtime container store is harvested and pushed to one or more registries with derived per-step tags. * config.json: add CONTAINER_REGISTRIES, CONTAINER_AUTH_CONFIG, CONTAINER_RUNTIME, CONTAINER_TAG_CMDS, CONTAINER_VERSION_RECIPE and CONTAINER_IMAGE_MAP configuration knobs. * scripts/run-config: drive push-containers as a post-step action. Tags are generated from recipe and distro metadata (yocto- tag uses major.minor on snapshots and full PV on releases) with CONTAINER_VERSION_RECIPE allowing a step to source PV from a different recipe than the image itself. * Registry auth is staged via .../config.json or podman .../auth.json using CONTAINER_AUTH_CONFIG, replacing an interactive login that could hang. CONTAINER_RUNTIME picks between vdkr (Docker-compatible) and vpdmn (Podman) runtimes. * Robustness: skip gracefully when no registries are configured, fix the OCI directory path, handle memres already running, and avoid hanging when memres has not yet come up. AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling --- config.json | 5 ++ scripts/run-config | 129 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+) diff --git a/config.json b/config.json index 79a9d10..9f85a7d 100644 --- a/config.json +++ b/config.json @@ -43,6 +43,11 @@ "BUILDINFOVARS" : ["INHERIT += 'image-buildinfo'", "IMAGE_BUILDINFO_VARS:append = ' IMAGE_BASENAME IMAGE_NAME'"], "WRITECONFIG" : true, "SENDERRORS" : true, + "CONTAINER_RUNTIME" : "vpdmn", + "CONTAINER_REGISTRIES" : [], + "CONTAINER_TAGS" : ["latest"], + "CONTAINER_TAG_CMDS" : [], + "CONTAINER_IMAGE_MAP" : {}, "extravars" : [ "SANITY_TESTED_DISTROS = ''", "BB_HASHSERVE = '${AUTOBUILDER_HASHSERV}'", diff --git a/scripts/run-config b/scripts/run-config index 0f5a26a..0fe0385 100755 --- a/scripts/run-config +++ b/scripts/run-config @@ -203,6 +203,8 @@ utils.mkdir(errordir) errorlogs = set() +push_containers = properties.get("push_containers", False) + def log_file_contents(filename, builddir, stepnum, stepname): logfile = logname(builddir, stepnum, stepname) with open(logfile, "a") as outf, open(filename, "r") as f: @@ -321,6 +323,133 @@ def handle_stepnum(stepnum): hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets)) bitbakecmd(args.builddir, "bitbake %s -k" % (sanitytargets), report, stepnum, args.stepname) + # Push container images to registries when push_containers is enabled + container_images = utils.getconfigdict("CONTAINER_IMAGE_MAP", ourconfig, args.target, stepnum) + if container_images and push_containers: + if jcfg: + addstepentry("push-containers", "Push containers", shortdesc, desc, str(container_images), str(stepnum)) + elif args.stepname == "push-containers": + runtime = utils.getconfigvar("CONTAINER_RUNTIME", ourconfig, args.target, stepnum) or "vpdmn" + registries = utils.getconfiglist("CONTAINER_REGISTRIES", ourconfig, args.target, stepnum) + if not registries: + hp.printheader("Step %s/%s: push-containers skipped — CONTAINER_REGISTRIES is empty, no containers pushed" % (stepnum, maxsteps)) + else: + static_tags = utils.getconfiglist("CONTAINER_TAGS", ourconfig, args.target, stepnum) + auth_config = utils.getconfigvar("CONTAINER_AUTH_CONFIG", ourconfig, args.target, stepnum) + if not auth_config: + if runtime == "vpdmn": + auth_config = "${HOME}/.config/containers/auth.json" + else: + auth_config = "${HOME}/.docker/config.json" + hp.printheader("Step %s/%s: Pushing container images %s" % (stepnum, maxsteps, list(container_images.keys()))) + script = [ + "set -e", + "test -w /dev/kvm || { echo 'ERROR: /dev/kvm is not writable, cannot push containers'; exit 1; }", + # Always bring up a fresh memres VM in the foreground. + # + # 'memres status' only checks that the QEMU PID in daemon.pid + # is alive (see daemon_is_running()/daemon_status() in + # meta-virtualization's vrunner.sh); it returns 0 as soon as + # QEMU forks, so a hung/partially-booted VM from a previous + # run — or a VM in mid-boot — is reported as healthy. The + # subsequent 'login'/'vimport'/'push' commands then hang on + # the unresponsive daemon socket. + # + # 'memres restart' is synchronous: it does stop+start and + # runs a PING/PONG readiness probe against the daemon socket + # (120s timeout), exiting non-zero if the VM never answers. + # Running it in the foreground gives us a trustworthy ready + # signal via its exit code, so we can drop the status-poll + # loop entirely. + # + # Install an EXIT trap first so we always tear the daemon + # down, even if bitbake -e / vimport / push fails mid-step + # under 'set -e'. The trap is armed before the restart so + # a restart failure also triggers cleanup. + # + # Registry auth is staged into the guest at VM boot via + # the global '--config' flag — vrunner.sh's setup_auth_share() + # copies $AUTH_CONFIG onto a read-only 9p share, and + # vdkr-init.sh / vpdmn-init.sh's install_auth_config() + # installs it at /root/.docker/config.json (vdkr) or + # /run/containers/0/auth.json (vpdmn) inside the guest. + # Subsequent 'push' calls use those creds directly, so no + # explicit 'login' step is needed. Calling 'login' would + # actually hang under the autobuilder (no PTY): when the + # memres daemon is running, vcontainer-common.sh dispatches + # login via '--daemon-interactive' and blocks reading the + # password from stdin (see login case in vcontainer-common.sh). + "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime, + "%s-$(arch) --config %s memres restart ' suffix on AUTOREV/dev recipes — Docker + # reference format does not allow '+' in tags, and the + # base PV is what consumers expect. + # + # DISTRO_VERSION needs context-sensitive handling. Poky's + # DISTRO_VERSION resolves to '${PV}+snapshot-${METADATA_REVISION}' + # off a tag and just '${PV}' on a release tag. The '+' in + # the snapshot form is illegal in a Docker tag, but more + # importantly the patch level on a snapshot build (e.g. + # '6.0.99' between 6.0 and 6.1) is a moving target that + # doesn't correspond to any real release — only the + # major.minor line is meaningful. So: + # - snapshot build (DISTRO_VERSION contains '+') → tag + # with major.minor only, e.g. 'yocto-6.0'. + # - release-tag build (no '+') → tag with the full + # version, e.g. 'yocto-5.0.5' from the yocto-5.0.5 tag. + script += [ + "_BBENV=$(bitbake -e %s 2>/dev/null) || true" % recipe, + "_PV=$(echo \"$_BBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')", + "_DISTRO_CODENAME=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_CODENAME=/{ print $2; exit }')", + "_DISTRO_VERSION_RAW=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_VERSION=/{ print $2; exit }')", + "case \"$_DISTRO_VERSION_RAW\" in", + " *+*) _DISTRO_VERSION=$(echo \"${_DISTRO_VERSION_RAW%%+*}\" | cut -d. -f1,2) ;;", + " *) _DISTRO_VERSION=\"$_DISTRO_VERSION_RAW\" ;;", + "esac", + "_DEPLOY_DIR_IMAGE=$(echo \"$_BBENV\" | awk -F'\"' '/^DEPLOY_DIR_IMAGE=/{ print $2; exit }')", + "_EXTRA_TAGS=\"\"", + ] + if version_recipe: + # When the image recipe's PV is a wrapper-style + # placeholder (e.g. app-container-python_1.0.0.bb, + # whose 1.0.0 is meaningless to a downstream user), + # CONTAINER_VERSION_RECIPE points at the recipe whose + # PV is actually meaningful for the resulting tag — + # typically the language runtime or app being packaged + # (e.g. python3 -> 3.14.x). Override _PV from that + # recipe; image-recipe state still drives + # DEPLOY_DIR_IMAGE and DISTRO_* since those are + # environment-wide. + script += [ + "_VBBENV=$(bitbake -e %s 2>/dev/null) || true" % version_recipe, + "_PV=$(echo \"$_VBBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')", + ] + script += tag_cmds + script.append( + "_TAGS=\"%s $_PV $_DISTRO_CODENAME yocto-$_DISTRO_VERSION $_EXTRA_TAGS\"" % " ".join(static_tags) + ) + for registry in registries: + # No per-registry 'login': credentials were staged into + # the guest by '--config' on 'memres restart' above. + script += [ + "for _tag in $_TAGS; do", + " %s-$(arch) vimport ${_DEPLOY_DIR_IMAGE}/%s-latest-oci %s/%s:${_tag}" % (runtime, recipe, registry, image), + " %s-$(arch) push %s/%s:${_tag}" % (runtime, registry, image), + "done", + ] + # Tear-down is handled by the EXIT trap installed above. + bitbakecmd(args.builddir, "\n".join(script), report, stepnum, args.stepname) + # Run any extra commands specified cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum) if jcfg: From patchwork Mon Jun 1 23:18:08 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D42FCD6E55 for ; Mon, 1 Jun 2026 23:18:49 +0000 (UTC) Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.12720.1780355923133472940 for ; Mon, 01 Jun 2026 16:18:43 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=ndSPBWc7; spf=pass (domain: konsulko.com, ip: 209.85.214.175, mailfrom: tim.orling@konsulko.com) Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2bf36a6905cso18203025ad.3 for ; Mon, 01 Jun 2026 16:18:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355922; x=1780960722; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=oZaYgB5sxJQCR6N/shrfavaj7wwDBCDDp0HvhRjyLdw=; b=ndSPBWc7R8aTo5Z7kwIhlvXERSdOX9PwDucw/mCHtWVz0M9JezEwdu4lXXHuFSR53s BxUCvPHvt0zKiVc10mGgM1I3pDhMbjOAeLfb7tObYYWigRrQFEYofsxC8kGeBRN52N5X YvoEzBXUgnd6oSUTxOGVcWn759yeSJPhJZ1dQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355922; x=1780960722; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=oZaYgB5sxJQCR6N/shrfavaj7wwDBCDDp0HvhRjyLdw=; b=kTuZ0nloSpmYUIZiTC07yREu7MdoIKs1lyir8KSROMQuT16hE8nAXD2kwDBJCJQTqF gJg34acaDiftVN0AHj4Oj7n8QPFwKnDkI1PHl2nilPgCk9EfWW2/YtKaKn0GnH3iNFjn VICXGlTGTLy+bgfLPscKJZdS3dH56aDNz9aea43vRYMCzPzwH4Crj6mlcDM93b4+0rud JdaiMgmSvsEA2B4/H2bAItIBp0Vai51U+qkcuy0mtSr4UiNDHRWb/kHLDndpUQ6uS7Pd KoW/+BJWbQvqmdBp29oiWXQfbvq310Ltl+vdUYSM5j2o+Dd/cNRQ23wXwrSUAW64Ruy1 IgZA== X-Gm-Message-State: AOJu0YyvgtWj5ziVsq7B5pGg/dV3+imPBBSaQWSapZ98ctGqBej8iNDQ IpnYpBkyOKHxhvDE7eBOo/Ah10jgj6U1QF/MjobxVqmO1CZopeCVCNxDRUQiyMjtVcPagIBhMnM Ebm9V X-Gm-Gg: Acq92OFo5T+oE9irtSPIhEzDNBx+R9VsH0m5yHxVlhSzZq4xABPeV1vERksnu4JEfLZ TY2rytsY5ENbm1Hc1fz48nlCfm5pSz92BhnlAfXru2T17RW+V+tJoxJFY8H+XdDCON9cG4v2xhH PBLKXlGb+AonU9zFthr+x0GXy1B0gIt6pcGR9QfG7CC1igyelu4glTwI7YYLLL8batSeXVgMngd gOncm0pWL1517LXSt8AK8ZbQUmchS9hnt8lyfXx0B71h5WiQEJ3D+QG//YwW6X1w7a0fCwUDcAM 1gK7u6RSpmvXFVXd5S2lwC61159l0wrndDVzpQOH8p4bZ5stajQCGJ62t1KxeBUpnzxjM9RM+Mr Qm54QEGHUxWRMq7qXoan63PHoGDbhZrHGrwUz2DFFbI7oVewtR5GuB0QbkXL5SNs9D/i5B0ljmQ hw+FjskUyOmfeYDeHuFo7HzhI4xsK6mO18rsRXx5d9oTmgTQYftnqNsrI/vMIyBSEtUcUagtG6L Z4AyU94p3Hw X-Received: by 2002:a17:902:c949:b0:2be:1c3c:72bb with SMTP id d9443c01a7336-2bf36860a25mr157784465ad.33.1780355922442; Mon, 01 Jun 2026 16:18:42 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2bf23a00a53sm145613935ad.27.2026.06.01.16.18.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:42 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 09/10] scripts/run-config: push_containers workaround Date: Mon, 1 Jun 2026 16:18:08 -0700 Message-ID: <242884114dd9bbeeac8bfca7dee5df4e19454f0f.1780354513.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:49 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4106 From: Tim Orling Workaround for: Error: reading blob sha256:: file integrity checksum failed for "" The first time the blob (e.g. bin/bash.bash) is copied, it succeeds. All subsequent tries fail. Might possibly be an issue with sstate or a recent change to vcontainer-common to allow for multiarch containers. Signed-off-by: Tim Orling --- scripts/run-config | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/run-config b/scripts/run-config index 0fe0385..03d921d 100755 --- a/scripts/run-config +++ b/scripts/run-config @@ -381,6 +381,8 @@ def handle_stepnum(stepnum): # password from stdin (see login case in vcontainer-common.sh). "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime, "%s-$(arch) --config %s memres restart : file integrity checksum failed for ""' + "%s-$(arch) image rm --all 2>/dev/null" % (runtime), ] tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, stepnum) version_recipe = utils.getconfigvar("CONTAINER_VERSION_RECIPE", ourconfig, args.target, stepnum) From patchwork Mon Jun 1 23:18:09 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 89125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D46CCD6E57 for ; Mon, 1 Jun 2026 23:18:49 +0000 (UTC) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.12812.1780355925338435754 for ; Mon, 01 Jun 2026 16:18:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=WKvx22JF; spf=pass (domain: konsulko.com, ip: 209.85.216.43, mailfrom: tim.orling@konsulko.com) Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-36c68964315so1385867a91.2 for ; Mon, 01 Jun 2026 16:18:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1780355925; x=1780960725; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=oyRTlKvOn47aHDOgL6NxLR76/UbRCfIJZZmSnkj19xo=; b=WKvx22JF+sXjt1BDxV1uln2WHOwWDCTXO5rORri1G8/FXoRJPqQz+0s/L9RJmh/Q1p OxbbTHB102i/+pFN0H6g2n8aD3+QBGk7hPCs2ZUq9RMlbbBHWeTkNowex2f23cFmsi+9 o7jFyg1WFQYTvkS18X936E5Fe9de7XfVnxm9E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780355925; x=1780960725; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=oyRTlKvOn47aHDOgL6NxLR76/UbRCfIJZZmSnkj19xo=; b=aJ5BONKck7WaLlbiMxnTRyhsRDbYkyoRcvKu2gK+bas8AeRIPgCUgG7z9rYGd2/7bs eTYrPvzRMDO+Zu+xQ/8ZG7oHJXyuX4dO9eJmKSQXzR3Flc/Co4+S6pw51pzwODxw+VNY YiYfIiV+gjRJQaC41qKGqcNgyRGB29KEtLrEWhVNGUmH4ac6LgxiaT66M9+m4yTghRz6 1WUdPFyeP3yZ1kAZn2c/U1/hcrjYpxTXSitxnH94h1xd1fZ7DUzXV7qT9JJH7ErSdsOz I8Twn8IkmhQtj8K6HRNw1ajnlMZ9aIIoCCYvs3fLGvpIM8xyO7tXAUflg3xyxfz0M2HM w8gw== X-Gm-Message-State: AOJu0YzywKxmcDcF/ZZuX3uoCWLd6eG4nX14g2lkWK2ras016Lj1nMHZ 0rcwdAx1UsYzQqd8Qql2j6f+9hSk3ftgUL4LiD29KK9Ic8HNa+6IXhvA1025fsD5dqnu+LTiIXb ajEEJ X-Gm-Gg: Acq92OFXtJaOiEhbz4hY8MDIufZl/190gmDnNHiC2Sk0FVLd2voVBzBlxiPqB1YCn4y bBgRNu/6Z4mNJKxYIWHirfcmR/TR7MDMk1MBdr1z5YImR3rs9kAYDFB+JQE4o6GXWWZ9JPY+ECj qnGW2W38gpXpkCSKa5DAK9KjsabF1d3SnJa5fXJKyDkPKcnXhMuPJn4sLJZmBdRr+UNBeb1WF2s JC1ok1lBiUMC3Uh7C/wrdrWk9/TkNWSKtJWhPjgoBS2mKlaG/yvitreECOUU+ErPPoQJ7PZLmY0 ceufVCEfGsNcp6GAwI0pbqKkCTvuIC4m01HHNKQ3l+lsiT3W3zwjtuqwxSnEs7hU9B53wo4wPki Kwc/BMZTIzWuNLEdeP3CCzUHy4RDuXUfHr1MBbMRf6LSCRdlQinW9MJd3+wVyuII5fVV+QqzhQo dOxwREvnd/kM6iNq9a1wDSEE5PM4AG3VBTZvqB42ZvGTv/J67xKn73wCoIAR90Sa1HftVJDQazA Q== X-Received: by 2002:a17:903:2bce:b0:2bf:128d:f7ff with SMTP id d9443c01a7336-2bf367d2311mr143953035ad.16.1780355924718; Mon, 01 Jun 2026 16:18:44 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2bf239fd84fsm117901465ad.23.2026.06.01.16.18.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2026 16:18:44 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH v2 10/10] config.json: add 'containers-library' build job Date: Mon, 1 Jun 2026 16:18:09 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 01 Jun 2026 23:18:49 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4107 From: Tim Orling Add 'containers-library' build job that build container images on top of the vcontainer-tarball SDK: * original container images from the intial "container-cross-install" branch of meta-virtualization - container-base - app-container-curl * additional images modelled after docker.io/library/* - app-container-python - app-container-mosquitto - app-container-valkey - app-container-nginx * Tag containers with versions based on the recipe to which they are mapped, e.g. python:3, python:3.14, python:3.14.5. Signed-off-by: Tim Orling --- config.json | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/config.json b/config.json index 9f85a7d..dae55d6 100644 --- a/config.json +++ b/config.json @@ -1929,6 +1929,61 @@ "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" ] } + }, + "containers-library": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-openembedded/meta-webserver", + "${BUILDDIR}/../meta-virtualization" + ], + "extravars" : [ + "DISTRO_FEATURES:append = ' virtualization vcontainer'" + ], + "CONTAINER_TAG_CMDS" : [ + "_PV_MAJOR=$(echo $_PV | cut -d. -f1)", + "_PV_MAJOR_MINOR=$(echo $_PV | cut -d. -f1,2)", + "_EXTRA_TAGS=\"$_PV_MAJOR $_PV_MAJOR_MINOR\"" + ], + "step1" : { + "shortname" : "Build 'base' container", + "BBTARGETS" : "container-base", + "CONTAINER_IMAGE_MAP" : {"container-base": "base"}, + "CONTAINER_VERSION_RECIPE" : "base-files" + }, + "step2" : { + "shortname" : "Build 'curl' container", + "BBTARGETS" : "app-container-curl", + "CONTAINER_IMAGE_MAP" : {"app-container-curl": "curl"}, + "CONTAINER_VERSION_RECIPE" : "curl" + }, + "step3" : { + "shortname" : "Build 'python' container", + "BBTARGETS" : "app-container-python", + "CONTAINER_IMAGE_MAP" : {"app-container-python": "python"}, + "CONTAINER_VERSION_RECIPE" : "python3" + }, + "step4" : { + "shortname" : "Build 'mosquitto' container", + "BBTARGETS" : "app-container-mosquitto", + "CONTAINER_IMAGE_MAP" : {"app-container-mosquitto": "mosquitto"}, + "CONTAINER_VERSION_RECIPE" : "mosquitto" + }, + "step5" : { + "shortname" : "Build 'valkey' container", + "BBTARGETS" : "app-container-valkey", + "CONTAINER_IMAGE_MAP" : {"app-container-valkey": "valkey"}, + "CONTAINER_VERSION_RECIPE" : "valkey" + }, + "step6" : { + "shortname" : "Build 'nginx' container", + "BBTARGETS" : "app-container-nginx", + "CONTAINER_IMAGE_MAP" : {"app-container-nginx": "nginx"}, + "CONTAINER_VERSION_RECIPE" : "nginx" + } } }, "repo-defaults" : {