From patchwork Fri May 8 02:00:24 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11ED8CD37A7 for ; Fri, 8 May 2026 02:00:54 +0000 (UTC) Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.4654.1778205649238987498 for ; Thu, 07 May 2026 19:00:49 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=FgJjSL3r; spf=pass (domain: konsulko.com, ip: 209.85.215.180, mailfrom: tim.orling@konsulko.com) Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-c6dd5b01e14so623830a12.0 for ; Thu, 07 May 2026 19:00:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205648; x=1778810448; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=SBIw3hIjvJrIbgvIBjSdTntx3ahE30r50oyPmZupKrM=; b=FgJjSL3ruj9od+FirZRoHsGudITjA2/tH93xleuLvR3tdmsF3uTfYdLIzLz2ZY7uqT /CKBMCtcdKhGs+zl4IupPMbQb8+cZUSuatGKzOBmzfnhoDC2WcxcNg3fSdkcneoBrp4I WDEF1VysGM9Mw7amp2589yHz4sK69ztPEYknk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205648; x=1778810448; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=SBIw3hIjvJrIbgvIBjSdTntx3ahE30r50oyPmZupKrM=; b=OxChMkNkTvjuaogqNsxJMEuDz1tcbukpKsWiZ2a8SO+92YtDSc7xWRqe3LbZIoOWrN 116PSu3c+/YgCjA2hS43y29wOMGdrxcTdVDAte0QTr93ZkKiCJ8oJd81aCJvcvO/GS9R /UlFktqhfhvzhBXoknJMoFDUNMYfgSCGQccZEkaJDaAWLRrM1+epcTC6uWmi6PX01aR9 akHl0z0VoPUDXaiNkRXAlEd/2MYmQh/m5qWPxGvjI6il1ZmGYxdOx74m2u8LsCq3zoqD TBJM0i5IXhBu6+7RfSGo1RGu0Q/1AMxQ+JY9ASJKe5gQt9xpthM3agi2g1mtXC9LVJzo 3xiQ== X-Gm-Message-State: AOJu0Yx+7jCXd3SkV2fpNOxRzxxK/KkPwJ29r7rkfH5ucQRkEdCrupCW q2Aj4sozpRHWG8cRr9QZ2hQk8DbmD83nwkwQX1qZcOBqv7VDN71s14t4Ltlv0rmxQNAPdfLCpmY /hm1/ X-Gm-Gg: AeBDietV7cTcYR4jnnWhk2qkFZ2mIIqo71vLR5q2N3HBT3tPFlLnKeoBs5q9i4zvRi+ hMfAhgiX0X62a6ce6a1JhDDX7aXzvO8D+Qy1yZbSSLzxCRWscE1Cl7vTRDoGlMlzlMkxjkih2K4 Fyy7sxkQS/pGyZ9nIIkdVCAe+5azbtmYZgNPrpIUV1/GARR+alxbcgrLAdbABiCsSZEQUoKXeGw 6VpBj13WBReFylLY6hJZ2NqU00oyzzJNCNKXwk1ECsKJXPiJ2sWNu9g4phKNsL+Wa1bxJ54sTL1 FJxqqZXwmyPE8rnsve1wSpR6agIpTB6zLuNg0NQF1sp+dBFViCcU5TyCv8i0NSbjd+wfjDfaBlB Zlj40sVJG9C4/xeVxCehEVGXHVsTNh+S3L756iLIKrNgeNvWnxjBA3sBMR5Rs5yndjgEckee04f JEwUpGL5tIqJ/PqGVJK09TqhkdvyYVh+GewS4+zqAS3OejPm06evlJkT6MY5AUDnI= X-Received: by 2002:a05:6a00:b8b:b0:82f:8696:5a75 with SMTP id d2e1a72fcca58-83a5d09b11dmr10020397b3a.28.1778205648331; Thu, 07 May 2026 19:00:48 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-83965d36a12sm9343236b3a.27.2026.05.07.19.00.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:00:47 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 01/11] scripts/utils: fix stale extraction dir when tarball is updated Date: Thu, 7 May 2026 19:00:24 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:00:54 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3940 From: Tim Orling Previously the entire download/cache-validation block in setup_tools_tarball() was guarded by `if not os.path.exists(btdir)`. Once the extraction directory existed from a prior build, every subsequent call was a no-op: the cached tarball was never re-validated, so a freshly-published SDK (e.g. vcontainer-tarball-latest) was silently ignored and the stale btdir kept being used. Fix by moving the lock/download block outside the btdir existence guard so cache validation always runs. Track tarball_updated to know when the cached download was actually replaced, then remove the stale btdir when True so the fresh tarball is re-extracted. Also add an mtime-based staleness check for local-path (cp) sources so that a newer source file automatically invalidates the cached copy. AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- scripts/utils.py | 64 +++++++++++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 23 deletions(-) diff --git a/scripts/utils.py b/scripts/utils.py index 88842f0..87acad6 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -481,31 +481,49 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): if ";" in bttarball: bttarball, sha256 = bttarball.split(";") btdir = os.path.abspath(btdir) + btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball) + btlock = btdlpath + ".lock" + if not os.path.exists(os.path.dirname(btdlpath)): + os.makedirs(os.path.dirname(btdlpath), exist_ok=True) + # Always run the cache-validation / download step so that a freshly + # published tarball is detected even when btdir already exists from a + # previous build. tarball_updated is set to True whenever the cached + # download is replaced, which triggers removal of the stale btdir. + tarball_updated = False + while True: + try: + with open(btlock, 'a+') as lf: + fileno = lf.fileno() + fcntl.flock(fileno, fcntl.LOCK_EX) + if sha256 and os.path.exists(btdlpath): + dl_sha256 = sha256_file(btdlpath) + if dl_sha256 != sha256: + os.unlink(btdlpath) + elif bttarball.startswith("/") and os.path.exists(btdlpath): + # For local-path sources (e.g. vcontainer-tarball-latest) + # invalidate the cached copy when the source is newer so + # that a freshly-published tarball is always picked up. + if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath): + os.unlink(btdlpath) + os.unlink(btdlpath) + if not os.path.exists(btdlpath): + if bttarball.startswith("/"): + subprocess.check_call(["cp", bttarball, btdlpath]) + else: + subprocess.check_call(["wget", "-O", btdlpath, bttarball]) + os.chmod(btdlpath, 0o775) + tarball_updated = True + break + except OSError: + # We raced with someone else, try again + pass + # If the underlying tarball changed, remove any stale extraction + # directory so it is re-extracted below. + if tarball_updated and os.path.exists(btdir): + print("Removing stale %s extraction at %s" % (name, btdir)) + subprocess.check_call(["rm", "-rf", btdir]) if not os.path.exists(btdir): - btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball) print("Extracting %s %s" % (name, bttarball)) - btlock = btdlpath + ".lock" - if not os.path.exists(os.path.dirname(btdlpath)): - os.makedirs(os.path.dirname(btdlpath), exist_ok=True) - while True: - try: - with open(btlock, 'a+') as lf: - fileno = lf.fileno() - fcntl.flock(fileno, fcntl.LOCK_EX) - if sha256 and os.path.exists(btdlpath): - dl_sha256 = sha256_file(btdlpath) - if dl_sha256 != sha256: - os.unlink(btdlpath) - if not os.path.exists(btdlpath): - if bttarball.startswith("/"): - subprocess.check_call(["cp", bttarball, btdlpath]) - else: - subprocess.check_call(["wget", "-O", btdlpath, bttarball]) - os.chmod(btdlpath, 0o775) - break - except OSError: - # We raced with someone else, try again - pass subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) enable_tools_tarball(btdir, name) From patchwork Fri May 8 02:00:25 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DD4BCD37AD for ; Fri, 8 May 2026 02:00:54 +0000 (UTC) Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.4505.1778205651381398038 for ; Thu, 07 May 2026 19:00:51 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=JSGOjal+; spf=pass (domain: konsulko.com, ip: 209.85.210.180, mailfrom: tim.orling@konsulko.com) Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-82fbdd60b64so1057621b3a.3 for ; Thu, 07 May 2026 19:00:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205650; x=1778810450; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Nh4bcp7Dv4HSkseIi709lWKX5+tSq1zA9PLL4tWYeAM=; b=JSGOjal+ho+TXTUJHddldQnl4xWWmmuZjh+1TPEdjaAAsos09XK2lDAfWmIsrHTT7O /2n97+7RX5sxeia5HI0vWqOqbxTd5bKQGQfE6RktHPYJzvj7gpHBltioI98ioXiTr+TK Xm0bCM+Xq2vQHpf2Ez+O5Ixx7CNkerPEaZLik= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205650; x=1778810450; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=Nh4bcp7Dv4HSkseIi709lWKX5+tSq1zA9PLL4tWYeAM=; b=ZoisIs2/omxIcwMATjl5jg//+L0DgR7Nqjl5nQ0sjW5cpLzxQj5FoMaBZQTFa4VSeJ 6a8zuvvKzNMN5TmMtH4Yjjbn9akiheQLVbhI+rdyxiC8k9dIKiKeEnKtDtTi2WF0AN9g C8uGTRksv/SVEG53ntqZ1jXWc9T6oXJnzVXbwAFbOq0TkSkeltt7cYwdR0HaNZO1iNIU jMBD6AzNd6AUPphapj/c0ER8BUH8hS1mt7Nt2xA6XiMv8936vKzE+zpHiNqSbfWzQQN+ RjPm87j8CnOH3cbHKCUg2m5SpdnDeIIKoy0yZ0jTSOzREyNz1NICdG2fkQFYUgeitBLR Yv5w== X-Gm-Message-State: AOJu0YzMX5RYul5N67qSNseqQk4opbxrX+PHDAQDLCQSzWnUSrZAbtmQ PNm3AZschfQyPudZWgvc/4g+i1jd+k3yTx2n9BevI79JCHzX3DLUMMYHjv0rhD5WAVNqtvazMRs oijgF X-Gm-Gg: AeBDieux+bHKgz6oYhT/2BH7RxZPMwOtB5ellxxqHysv5Y4UWfzGFhDeTTPPcs8LptB aeFxD17D3jh/RtAJ3itVjo7B2ROGNRzolOGkok52803efESZorpGwP+WXgOtdOjrMnbYdTJ/TqH uJbMu8rauCVSubwUydAD1M9yaIrSK1xt7nXMrbGEpN1n+2nZ/7o5onjIOCMnGn6eB/Dkv+YPfWe 0lL9UFna+9ozzZNgztHKO3EnsWmvaOyqswErzC599/GBe1WjYItOHVKJKQS9N8i9yKoHqGIp1SX lB7GHMOwWmd8r627aIlk/YyY+cC4nnBZtsD5MiX1g7aZ7EvSN5JuaFgrNDTjJsFQiuh5zdgAWiM AG/sD7Dt+fyS1A19/qZnPYKivI3KLeFsOqz31OcVJVBZzGUhooFjWaKOFWBEotM0DdKuHsqrUUj giobXl/qpBR9KWI95fVVpWWekKdDrYGeWvJ5OtPLKVmozNeUAd5YIZaMr8Pb2jLAvewQ0A1L08T A== X-Received: by 2002:a05:6a00:cc1:b0:82f:9e98:1356 with SMTP id d2e1a72fcca58-83a5c4bdb96mr10871278b3a.20.1778205650436; Thu, 07 May 2026 19:00:50 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-83965d34872sm10352726b3a.22.2026.05.07.19.00.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:00:50 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 02/11] scripts/utils: warn and force re-download for HTTPS sources without SHA256 Date: Thu, 7 May 2026 19:00:25 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:00:54 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3941 From: Tim Orling Without a SHA256 checksum there is no way to verify that a cached HTTPS download is still current. Rather than silently reusing a potentially stale copy, delete the cached file and force a re-download each run, and emit a clear WARNING telling the operator how to avoid the overhead (by appending ;sha256= to the URL in their config). AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- scripts/utils.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/scripts/utils.py b/scripts/utils.py index 87acad6..ea905d9 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -505,6 +505,14 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): # that a freshly-published tarball is always picked up. if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath): os.unlink(btdlpath) + elif not bttarball.startswith("/") and os.path.exists(btdlpath): + # HTTPS/FTP source with no SHA256: there is no way to + # verify the cached copy is current, so force a + # re-download every run. Add a sha256= suffix to + # the URL in your config to avoid this. + print("WARNING: no SHA256 provided for %s source %s; " + "forcing re-download to avoid using a stale cached copy" + % (name, bttarball)) os.unlink(btdlpath) if not os.path.exists(btdlpath): if bttarball.startswith("/"): From patchwork Fri May 8 02:00:26 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5380CD3427 for ; Fri, 8 May 2026 02:00:53 +0000 (UTC) Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.4655.1778205653439823037 for ; Thu, 07 May 2026 19:00:53 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=BWbBsNiE; spf=pass (domain: konsulko.com, ip: 209.85.214.179, mailfrom: tim.orling@konsulko.com) Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2bab2548e8bso7122575ad.0 for ; Thu, 07 May 2026 19:00:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205653; x=1778810453; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=dGpJGGdx2pP961hlODhNEoWrZCG6FZrq0blRiEQh8qQ=; b=BWbBsNiEW2o+wMMkCrwZrSn4niubmb5FI5ryIEYDvKz8wszfQQbRsckyXCWHRmGJqr ox1B9eTtlOaWi79kPcjHhiXKgsTdw9+F3rd/OYCRlayMVhVze6zHZAP3DdO/u3dSjAE1 x87c24Rg0XvDoWTBZG9ipJL6R7yP1P8yO0rsU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205653; x=1778810453; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=dGpJGGdx2pP961hlODhNEoWrZCG6FZrq0blRiEQh8qQ=; b=msoITmaqfpTF2YP3epIEhoK7BzivtHXSUimFq4tAKPlLRoN9q0KNxO07tpdigS2tvr JUoKMcUyhuzKBYansfjYiJZCn3bj8sKtpl+LMY0r9f+qYiCF9szmIT0pAi/xFb/NYgwE acEuhE1kmbhl0S4hGngkKP5OTzPxv0W135+HOeaJeT44xUO4sOX1JL5og7+o83CtMC9l XTY5x32l1OFpY/I1nmeyHBxjOdTUdxPn4x24S2lqefsezXDaDDzvwGyO0H4aUxJNuYzt YSzMAeJj/sMz0PRLAf3uGtGM1Dj5GDsTGNDNL6pqXBo2Yj4u4qCGEXn+AY0WYZM6ebMC 4JqQ== X-Gm-Message-State: AOJu0Yzi/K6RWB6wotdhRZ0+gdm+In8a5L0uJUNY09QjffoN+Ul7mEHX inJfv9du4N502Jdhg/RophYymGWYzzFYtYIea/b1EOTzJY6pdGT/XfKTisV5sYlgsIn+zNgmtue njsdm X-Gm-Gg: Acq92OFyU/rdCXFBeC+JNV+7dG73e3YPZOVeMqQonEJ1JTNdHzT4WHSjU+/rgOVNLo+ nSLQFvB7duXyiILrO24tnhB0WTFPwy2qY85fFUA/PYblhrG+yLHrV6QVxnogMCqc2Q1IpBk/iAa QzoZa2/MWk3kPkQSy+2qLuGc9HJg7T+KUHUo8zmqBD6FkuqG+LJRnJ2QrzUnsKXBsESpkTEbSfU dUqfKzB4gkMotaAFYYS4HhyZ/mL4mfpLhmpWGohRK4c4nymGOll1TQRrUHC1GSM+sHPGQCKdhxk wA8gGRd7qtQ/a6jTGV0OI6bxH823ntAY7ukBVODNjThDmtpNLlXIh5jv7LsD+4+0hUojt1Oox85 2npR+woJHzI9RlK7QOu6UB9PUmfl1Wc1oFWxWYDASbQI8M1BB9dGREHHKVzEjt9dwSYHECDG6M1 8+Brzs/lxNPHGqHXBBUGKzGEjByS6yTAuGSX9hBt1TT9/q0Uydj+hd13oPZnor9D+uaByabu9YA w== X-Received: by 2002:a17:903:48f:b0:2ba:4f37:d3a7 with SMTP id d9443c01a7336-2ba794c4db4mr82636975ad.27.1778205652537; Thu, 07 May 2026 19:00:52 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1d27083sm1910185ad.13.2026.05.07.19.00.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:00:52 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 03/11] config.json: add sha256sum for BUILDTOOLS URLs Date: Thu, 7 May 2026 19:00:26 -0700 Message-ID: <2ca072fcdf500ca956d45749f302518e17f7d3b8.1778202125.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:00:53 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3942 From: Tim Orling setup_tools_tarball() now checks the sha256sum of cached tools tarballs to determine if an update is available and needs to be re-downloaded. To prevent unnecessary downloads, add ;sha256= to the BUILDTOOLS URLs. extratools/EXTRATOOLS_URL uses the same utils.setup_tools_tarball(), so it also needs the ;sha256=. Signed-off-by: Tim Orling --- config.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/config.json b/config.json index f0e220e..8d42bbe 100644 --- a/config.json +++ b/config.json @@ -7,10 +7,10 @@ "BUILD_HISTORY_REPO" : "ssh://git@push.yoctoproject.org/poky-buildhistory", "BUILD_HISTORY_FORKPUSH" : {"openembedded-core-contrib:ross/mut" : "openembedded-core:master", "openembedded-core-contrib:abelloni/master-next": "openembedded-core:master", "openembedded-core:master-next" : "openembedded-core:master"}, - "BUILDTOOLS_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.1.sh", - "BUILDTOOLS_ARM_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/aarch64-buildtools-extended-nativesdk-standalone-5.1.sh", - "BUILDTOOLS_MAKE_URL" : "https://downloads.yoctoproject.org/releases/yocto/yocto-5.0.4/buildtools/x86_64-buildtools-make-nativesdk-standalone-5.0.4.sh", - "EXTRATOOLS_URL" : "https://downloads.yoctoproject.org/tools/buildtools/x86_64-buildtools-imagemagick-nativesdk-standalone-4.3+snapshot-5f2ba20f203114db9a3b11264467f8c23a05041d.sh", + "BUILDTOOLS_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.1.sh;sha256=5af9d92898af17fcc2fca4d07607a59f41b1c39a0a4ff058f64d834b98ec7fd5", + "BUILDTOOLS_ARM_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/aarch64-buildtools-extended-nativesdk-standalone-5.1.sh;sha256=8074d582a60a5135fd5606b8326e749185d26f51aca27570447831310bcac187", + "BUILDTOOLS_MAKE_URL" : "https://downloads.yoctoproject.org/releases/yocto/yocto-5.0.4/buildtools/x86_64-buildtools-make-nativesdk-standalone-5.0.4.sh;sha256=2eb7a6c013113f4fdb87e800167606dc13af1bd8bbc1c9e2443b7be37fefd124", + "EXTRATOOLS_URL" : "https://downloads.yoctoproject.org/tools/buildtools/x86_64-buildtools-imagemagick-nativesdk-standalone-4.3+snapshot-5f2ba20f203114db9a3b11264467f8c23a05041d.sh;sha256=9cbff3a7cf524bdfa7779dce8afaf3453114d8017918d2927f723ea38a36ebdc", "REPO_STASH_DIR" : "${BASE_HOMEDIR}/git/mirror", "TRASH_DIR" : "${BASE_HOMEDIR}/git/trash", From patchwork Fri May 8 02:00:27 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FBB0CD3427 for ; Fri, 8 May 2026 02:01:04 +0000 (UTC) Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.4656.1778205655459610112 for ; Thu, 07 May 2026 19:00:55 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=lAsA5VEW; spf=pass (domain: konsulko.com, ip: 209.85.214.175, mailfrom: tim.orling@konsulko.com) Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2ba0714574fso8848485ad.2 for ; Thu, 07 May 2026 19:00:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205654; x=1778810454; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Wup+ARkC5NxdlM6S9euHh+cIF99+71JfQqhwZqaIX50=; b=lAsA5VEWtcfdnUS7ZUdvMltau1ukZ8+kHJgS9hKdHD4vTmGJNucCSFCY5CrII4Zhz6 vmr1lwHAcyauEePzP4btlEOMvPbuf09QCgbFNGTL/QF4XpD5mHwgVX3SC/Z0LWZr7v8f y53/XAWLfAC5lcSycN2faWHFDKBahmP9yX6bk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205654; x=1778810454; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=Wup+ARkC5NxdlM6S9euHh+cIF99+71JfQqhwZqaIX50=; b=Vq8BwlLJM8sgDljjhuEAVIs4q0elDrN86zYXdMLODhmZfQ2RGmnIhvT188Kh4koTRm d3VsSVf9IWT+L+GG2x7ceOfR/9uafISCqwHXpgPcblEbt9f9odyH9FlGUFZBEzkS667D csKtWXY5hPJwfY+G75AHs9PKETGvITdJfGz+TSgjkAxb0mQB3SqQ8lfYZZ9yiFN6yz5O ZzaFTwN+Szp+cvBpn8AhHStTvtxZdjZGKDKcO3K7YPXKOcolkmg+lAZrHPpOQGr9LoSe GVMo0IG43AB6cHembnsa641kQgnxkBz/F9A8rBh/Oqkxjn4ZZqtY2QUMBmzAM/AvPZxS AvpQ== X-Gm-Message-State: AOJu0YzYok3J0uCrmJPRh3yWuvXFRzArOG+1CaCnh8TH5va5VKCS/uy7 h0QqJmNosM7U85DXolK60z6RokJYNfDpmd+Da2KUcvGXvCJskKKw9V5IJQG3uU3cthQAYYEMx1h MeBMX X-Gm-Gg: Acq92OFdyMQUQzIDdC2FSli9kWdy9+f6RwiN7+jLGsORjgw7kw/txpmdJPYPN1pDQwR MolhV8vNSmcenNI9/a1Eu52snMHlPomxrIg4LfxbGUGiOCaydsoM7KLDh/fTEOG4SsQZrx93BlY +Vuetajp8BclFtOosmg/ZhSNKweDtziDuByTGSWGuoDT0MSSTGoPsyKqfWiqXa3vbMcMcmE3rav wKb/hEncqylnL1eScZKNGN9ezhXwcdo9RMqpqQzFJnVZ7m1wixuH/37LibBFn+TCV6DfUb76kxE hQLudD0t9rkhnQLsDq7yC/c10/MmPJULSI9F/cZDkc/vT66TJt8Xp2EnJcXPi2xyUqu+VDFj2aR XRmIGwJYrLdefjTp8nuJcFYEwCNZb+2vH8vEG3r6Rh1h7TekRhlTCx831pyBHtrNo+HLLOCVJ+0 bEleyryyuiVuvcPMi8icJz0hx8dW761Fmce+ADXvecNZ0DdYLSjF4ozFknq67yRxk= X-Received: by 2002:a17:903:b46:b0:2b2:eb9d:1648 with SMTP id d9443c01a7336-2ba79c25ad9mr107017305ad.37.1778205654359; Thu, 07 May 2026 19:00:54 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1d4050dsm1888045ad.31.2026.05.07.19.00.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:00:54 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 04/11] scripts/utils: add getconfigdict() for dict-type config values Date: Thu, 7 May 2026 19:00:27 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:01:04 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3943 From: Tim Orling Parallel to getconfiglist(), but for JSON object values. The merge priority is defaults < target-level < step-level so that more-specific keys win: a step can override individual entries in a target-level dict without replacing the whole thing, and both levels refine the defaults. Used by the upcoming CONTAINER_IMAGES support, where each entry maps a Yocto recipe name (the on-disk OCI path stem) to an image name (the name pushed to the container registry). AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- scripts/utils.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/scripts/utils.py b/scripts/utils.py index ea905d9..112ebc2 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -112,6 +112,21 @@ def getconfiglist(name, config, target, stepnum): ret.extend(config['defaults'][name]) return expandresult(ret, config) +# Get a build configuration dict, merging defaults < target < step so that +# more-specific entries win (step-level keys override target-level, which +# override defaults). +def getconfigdict(name, config, target, stepnum): + ret = {} + step = "step" + str(stepnum) + if name in config['defaults']: + ret.update(config['defaults'][name]) + if target in config['overrides']: + if name in config['overrides'][target]: + ret.update(config['overrides'][target][name]) + if step in config['overrides'][target] and name in config['overrides'][target][step]: + ret.update(config['overrides'][target][step][name]) + return expandresult(ret, config) + # Return only unique configuration values (identified with '=' in them) def getconfiglistfilter(name, config, target, stepnum): def merge(main, newvals): From patchwork Fri May 8 02:00:28 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3738ACD37A7 for ; Fri, 8 May 2026 02:01:04 +0000 (UTC) Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.4506.1778205657434994842 for ; Thu, 07 May 2026 19:00:57 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=Ct2jFXuL; spf=pass (domain: konsulko.com, ip: 209.85.210.173, mailfrom: tim.orling@konsulko.com) Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-82748257f5fso1626794b3a.1 for ; Thu, 07 May 2026 19:00:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205657; x=1778810457; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=s0gnjesXBkRgihvb4pMTCD07X8E5Q82+b+cm+vmuixY=; b=Ct2jFXuL9cgGZs5VFGLq71IOpPl6sIreCTIu5GZaBhpGwY03b4UJhsE6RoghUIm7c7 keuNLnZuoumL8E2emZZXZcVTCgDJp9yDR32sh4bFSq9gAqaQ1gZUaEiFAjZKbfzqbd6b AYKqDvS+OJsleU+qyiUQei5Wa79QvRCxZn/oQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205657; x=1778810457; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=s0gnjesXBkRgihvb4pMTCD07X8E5Q82+b+cm+vmuixY=; b=Nzp1qZCQzsaEB+GE1FsuGIxoEV+X0xgW2COkSma5BZo9+Wx5DbJfX9p+/vCj5jJqzs putfeLy7PlMcdRmJZmqcPZ85qihV/U+7NEOO8Uke2rmqH8FzPkunQfI2Qi+NI1yARxlg qRyDOOCPd9Xgnw1oyXYzUzgxZismK5SVWYyNp+CfvlgMhM+4EDp6rno8O19lYPZt22QQ TA+uCtjwXoWW/0uaQSbqG7Y7ooh2EDi+cmoKSnLb6jHaipe5+dGn9JDqizaCcCeA1IBw PkKV0Ii1r2AiIK8RQusuDgmSvRXNOu9ExbqgQfdhAsGmIAr1pQ5nSQPs8kALo0HZzUqg sQTw== X-Gm-Message-State: AOJu0YwD0kajcfpPtNoqOHjkCoJYfpYunlpHCWo5QREOCROFK7nwzYwY 0q+LZiESFIk46lYy4TF/B62jQ/e4wT1a3da2fOKhWwtxZg6UXiGslYunI4NXToIVPn8YgS6SFvo B2xQB X-Gm-Gg: AeBDietrDX42xdR2rZjVDNb5I2Xbc5QGs2p56YofroVDNsgLuGjI6JCTW8ewm6gpURD 0mhzoj81DkilkvHBlzu2FwNFJNzCO4dPDlWFyGsY+LhEH+JsUgQwTs7IpzOVrNKEcKrx6W1mpHt B6ohlXzdNYzxtJGcRiRHyMZG/XGDbY9CTVa34FxoHeL5A6M/2dGh0mTYuyLhGzC75A+D67sYE+r jZY2XKGDlXndQi/jtKnLpNHHknmcUAvhoNy2aEBYYadHyvqv96cEaQV9084op4/Yg0shYkZ49jq 1r3Mekrprx6bpbHA14Yp0G9ldamCh0LsPfB5QWQaMl3DDkXWXhdQRL6xL7WgsYc1/23Czf9RraO MAAhSCxTWpo2EYrC5zwS2TWppmDyShuUpiGPGCvfR7Pf4s86WfcEDaO+B3iZlk3Db3t6ugqzXk5 b+eEUvMyl2dQ5N10Xr+68oYgSibBPvnEjz3mBcZbVSOdIsBOyQu+t7U53eTHN3b5c= X-Received: by 2002:a05:6a00:aa8d:b0:81f:5acb:55fc with SMTP id d2e1a72fcca58-83bb6bce110mr4333850b3a.10.1778205656543; Thu, 07 May 2026 19:00:56 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-8396563f049sm11904845b3a.7.2026.05.07.19.00.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:00:56 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 05/11] scripts: add vcontainer-tarball setup, integration, and publishing Date: Thu, 7 May 2026 19:00:28 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:01:04 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3944 From: Tim Orling Introduce the vcontainer-tarball SDK plumbing used by container build jobs. The vcontainer-tarball is a meta-virtualization-derived SDK (modelled after buildtools-tarball) that ships the container build toolchain so worker jobs do not need to rebuild it for every step. * scripts/utils.py: add setup_vcontainer_tarball(), and add an env_glob keyword argument to setup_tools_tarball() and enable_tools_tarball() so the vcontainer-tarball can source its specific environment-setup-ci file rather than the universal glob. * scripts/run-config: source the vcontainer-tarball environment for build-targets / cmds / test-targets / plain-cmds steps, gated by a new NOVCONTAINER step variable so individual steps (such as the dashboard indexing step) can opt out independently of NOBUILDTOOLS. * scripts/shared-repo-unpack: invoke setup_vcontainer_tarball so workers extract the SDK during unpack. * scripts/publish-artefacts: publish the vcontainer-tarball artefact so downstream test jobs can fetch a stable SDK. AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling --- config.json | 1 + scripts/publish-artefacts | 5 +++++ scripts/run-config | 19 +++++++++++++++++++ scripts/shared-repo-unpack | 1 + scripts/utils.py | 22 ++++++++++++++++++---- 5 files changed, 44 insertions(+), 4 deletions(-) diff --git a/config.json b/config.json index 8d42bbe..ab0ad7c 100644 --- a/config.json +++ b/config.json @@ -1433,6 +1433,7 @@ "step3" : { "shortname" : "Populate/update dashboard site", "NOBUILDTOOLS" : true, + "NOVCONTAINER" : true, "EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index ${HELPERBUILDDIR}/../"] } }, diff --git a/scripts/publish-artefacts b/scripts/publish-artefacts index e56e131..0e820e9 100755 --- a/scripts/publish-artefacts +++ b/scripts/publish-artefacts @@ -146,5 +146,10 @@ case "$target" in sha256sums $TMPDIR/deploy/images/qemux86-64 cp -R --no-dereference --preserve=links $TMPDIR/deploy/images/qemux86-64/*qemux86* $DEST/patchtest ;; + "vcontainer-tarball") + mkdir -p $DEST/vcontainer-tarball + sha256sums $TMPDIR/deploy/sdk + cp -R --no-dereference --preserve=links $TMPDIR/deploy/sdk/*vcontainer* $DEST/vcontainer-tarball + ;; esac diff --git a/scripts/run-config b/scripts/run-config index e896234..0f5a26a 100755 --- a/scripts/run-config +++ b/scripts/run-config @@ -153,6 +153,25 @@ else: if args.phase == "init" and args.stepname == "buildtools": sys.exit(0) +if jcfg: + vcontainer = utils.setup_vcontainer_tarball(ourconfig, args.workername, None, checkonly=True) + if vcontainer: + addentry("vcontainer", "Setup vcontainer tarball", "init") +else: + # If we're executing a specific step, check whether vcontainer is disabled for it + vcontainer = True + if args.stepname in ("build-targets", "cmds", "test-targets", "plain-cmds"): + try: + vcontainer = not utils.getconfigvar("NOVCONTAINER", ourconfig, args.target, int(args.phase)) + except ValueError: + # Not an integer step phase + pass + + if vcontainer: + utils.setup_vcontainer_tarball(ourconfig, args.workername, args.builddir + "/../vcontainer-tarball") + if args.phase == "init" and args.stepname == "vcontainer": + sys.exit(0) + extratools = utils.getconfigvar("extratools", ourconfig, args.target) if jcfg: if extratools: diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack index 797dec6..869b214 100755 --- a/scripts/shared-repo-unpack +++ b/scripts/shared-repo-unpack @@ -77,6 +77,7 @@ for repo in sorted(repos.keys()): utils.flush() utils.setup_buildtools_tarball(ourconfig, args.workername, args.abworkdir + "/buildtools") +utils.setup_vcontainer_tarball(ourconfig, args.workername, args.abworkdir + "/vcontainer-tarball") if "bitbake" not in repos: sys.exit(0) diff --git a/scripts/utils.py b/scripts/utils.py index 112ebc2..b020a7b 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -456,8 +456,8 @@ def sha256_file(filename): pass return method.hexdigest() -def enable_tools_tarball(btdir, name): - btenv = glob.glob(btdir + "/environment-setup*") +def enable_tools_tarball(btdir, name, env_glob="/environment-setup*"): + btenv = glob.glob(btdir + env_glob) print("Using %s %s" % (name, btenv)) # We either parse or wrap all our execution calls, rock and a hard place :( with open(btenv[0], "r") as f: @@ -474,6 +474,20 @@ def enable_tools_tarball(btdir, name): if line in os.environ: del os.environ[line] +def setup_vcontainer_tarball(ourconfig, workername, vcdir, checkonly=False): + vctarball = None + if "vcontainer" in ourconfig and workername: + vccfg = getconfig("vcontainer", ourconfig) + for entry in vccfg: + if fnmatch.fnmatch(workername, entry): + vctarball = vccfg[entry] + break + + if checkonly: + return vctarball + + setup_tools_tarball(ourconfig, vcdir, vctarball, name="vcontainer-tarball", env_glob="/environment-setup-ci") + def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): bttarball = None if "buildtools" in ourconfig and workername: @@ -488,7 +502,7 @@ def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): setup_tools_tarball(ourconfig, btdir, bttarball) -def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): +def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools", env_glob="/environment-setup*"): btenv = None if bttarball: @@ -548,7 +562,7 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): if not os.path.exists(btdir): print("Extracting %s %s" % (name, bttarball)) subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) - enable_tools_tarball(btdir, name) + enable_tools_tarball(btdir, name, env_glob) def get_string_from_version(version, milestone=None, rc=None): """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists, From patchwork Fri May 8 02:00:29 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BB42CD37AD for ; Fri, 8 May 2026 02:01:04 +0000 (UTC) Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.4657.1778205659515921101 for ; Thu, 07 May 2026 19:00:59 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=Ky886oWu; spf=pass (domain: konsulko.com, ip: 209.85.216.41, mailfrom: tim.orling@konsulko.com) Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-3658ff4d34dso1011737a91.3 for ; Thu, 07 May 2026 19:00:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205658; x=1778810458; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=RevNAg63UntrgGhEbeJlSDWjtYKRFpzF9WJml6ixDsc=; b=Ky886oWuz5IANirq0j7n/c1SvhHnypTzfNsTYXT6n94o1oTupU70eXa7iOL2SpMeGK TDCBwC93ALilVSPxNCD3Npxgj3uKhjR2/zU2js/6lm7e1jJuQ3heKM+ZMtUBfOMozN4e IirR8sQRDV9oU96UAJVXpi6i7EbmtxrNKVa38= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205658; x=1778810458; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=RevNAg63UntrgGhEbeJlSDWjtYKRFpzF9WJml6ixDsc=; b=i/JWufVFQ1/PzmnkG/LGkcZXze83eeby5bd5f0ya3pGD+lnxbid+ThHaedxK+1lq0B RHOSiDcEC1fhdl9ucpnzzPtvs74Ii45Fy2EXGGSVkPJpFWKthR/rkZggUcA48zCBUq5a 3EX5K00t/CjxCqkNHlDzIfjPfQoVfwKjwUuDLlKcf8OgFCtcGkaQ1vIh5EGsUplBit7e A6yM7beFAxiTJaEgSO9kbmE4TH0BVFMH1kFTlKpQ8r0KT8gSP+eGj0Gdea2mdKzEC1D/ NQSFKIaBHIlZzai46QiPkt1FrCuJOpll2LIyU4RbipZvKF++AsdwbI4ln2YN3R+cog3t DKdg== X-Gm-Message-State: AOJu0Yz3JS7TiYAzx1DRbJV02n1OW+IlRbGMB4il/Apm8/lfyvrWoYUv ADDlguPkf7WLzjw/JES73j51fFTzaAyYclP/iAR/eyHvNFWeZbqc6g4l9JZCCnuNXLWvxfQldlJ /cbtU X-Gm-Gg: Acq92OE05/X8gZqgTKoeEINczOCQ6jqI7M7YHn3DLIZBg1tgfpNE4FHEtsi9vJ4LS0P DAS0jweUVfUwtjZTqGfngvg6AkyuJbYs70o6mmQKLRbu4YE1GDD/PUFrMZWSaP1HVp4ptBkNM0m owaIOHk5cOmF4SX/nTTmRj80lsuwZEn+BwuBJyQgLo0SdZ06hZbxcpAxOliQQnO6Q/UOrhN6Yoi CTHb/VMbz/yHRWSJtrmyGNXl0F1SyOxyBF+o5U29n26JKnjJpDzhNVHi6kOCB6tyBG0S7HutyqL HelA/I3m9KLM5ebR1Nm494BIXaYUwcfk+Z2HKxfYXCXCZMG/EvjKogflv1k5iDA8kc3meDJaWQC P9L5HWZT/0bPTIJJNGluf8+gJCF4es1l3nNBycXljkoAQnmX8fr/Hnylkogo6BhSGEgpE43GvIv RsZ06W+KeT918XYBZeqICKPb7YbBxeOYOPXELaCGGG2gWKs0HJo6bJiMXVt1lGEPwyf+s/xQD5y w== X-Received: by 2002:a17:90b:3a0f:b0:361:472e:3be6 with SMTP id 98e67ed59e1d1-365ac79e167mr10090501a91.26.1778205658488; Thu, 07 May 2026 19:00:58 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3664da42c37sm233896a91.6.2026.05.07.19.00.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:00:58 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 06/11] config.json: add vcontainer-tarball build target Date: Thu, 7 May 2026 19:00:29 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:01:04 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3945 From: Tim Orling Add the vcontainer-tarball build target which produces the meta-virtualization SDK used by downstream container build/test jobs. Modelled after the buildtools-tarball target. The target uses EXTRACMDS to reset BBMULTICONFIG for the worker shell, places the extravars in the per-step config, and includes a publish-artefacts step so the resulting SDK tarball is staged for reuse by container-tests and other consumers. Signed-off-by: Tim Orling --- config.json | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/config.json b/config.json index ab0ad7c..9e6898d 100644 --- a/config.json +++ b/config.json @@ -1869,6 +1869,32 @@ }, "toaster" : { "EXTRACMDS" : ["${SCRIPTSDIR}/run-toaster-tests ${HELPERBUILDDIR} ${HELPERBUILDDIR}/../layers/bitbake"] + }, + "vcontainer-tarball": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Build vcontainer-tarballs", + "BBTARGETS" : "vcontainer-tarball", + "extravars" : [ + "DISTRO_FEATURES:append = ' virtualization vcontainer'", + "BBMULTICONFIG = 'vruntime-aarch64 vruntime-x86-64'", + "INIT_MANAGER = 'systemd'" + ], + "EXTRACMDS" : ["sed -i '/vruntime-aarch64 vruntime-x86-64/d' ${HELPERBUILDDIR}/conf/auto.conf"] + }, + "step2" : { + "shortname" : "Publish vcontainer SDK for test reuse", + "EXTRAPLAINCMDS" : [ + "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" + ] + } } }, "repo-defaults" : { From patchwork Fri May 8 02:00:30 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87652 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24EADCD343F for ; Fri, 8 May 2026 02:01:04 +0000 (UTC) Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.4660.1778205661344432292 for ; Thu, 07 May 2026 19:01:01 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=M1t75Uzk; spf=pass (domain: konsulko.com, ip: 209.85.214.174, mailfrom: tim.orling@konsulko.com) Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2b788a98557so11127175ad.2 for ; Thu, 07 May 2026 19:01:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205660; x=1778810460; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=H9NJtCAEJvuu08gLcRGkO7F1iIExljVavKS4xNeQ7io=; b=M1t75UzkWbrYQAGS+owFt0V4s+CZJRgi8+ocMhFlaVNFGIyPPPbuHgQKpJeZS33LY8 C6fh48sYQZkje5ZDgnkeTVOFYYbmjYXnCZAfvk89Nz2qs3v5QmsZqmveXvHtQwolVa78 O/EDS4KKwMTMmmiaeRtRBkjUVAHR+7UwULD18= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205660; x=1778810460; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=H9NJtCAEJvuu08gLcRGkO7F1iIExljVavKS4xNeQ7io=; b=h68I3vvmcs0/nvB8Yl3xPUvPz0g5x9EUMvWFXjtooIyvj8ZQ1knm0UfQUZwElDvuHg AzcYtCGw8DGRlJVMMsRjtS8CS4BxibrWvJOa+EC0NfIQjna457Yg5QKH2V4LHibzyrJS qnvCN/PUTFvF+zGuo8x5tL5b4y3dTJn3MrGb8jZkH7k8xK63OgskddDwHbbYYGiPee7y MLdL5aLzrfUcgEfbLf6WWJHfJMnoCnqh7hzpWAkXelyYpmk7aLguTNA5SBLo6YFBwDhG 4NnTchbsQyHBce4oz4uYuGJfypZ85riWsYH9iD4b9set1KxASKet2Zl/3P5IwcnIYSnL coRA== X-Gm-Message-State: AOJu0Yy7ilv8Ximl4OJJA3tDI8TudkPAr/eW96dphvorP1m7fuPYPKv5 9acqEnIRKlsu5Mspn5GpItRrB9UhS40FoWRtlOcRJDyvkrzzIXK2zItkKeSTwCtKBYm9YFuJdcd OBPNr X-Gm-Gg: Acq92OFAdGgHf4YJyIkwHWdKXMYgi98OUCOATuQ3vXMx7YwSma6r7gbs3biaBCpN5d9 dPWNV2phSG0tXF+WpXTaL9Ee/t8VtpaKsfBM1CavC4GLMAwhUIonDGpgXiFtK45Q7zX7Bvh8enu uoEhbHde8Uh2ChdgbaEsm85Ut0NyCE6Bcr1i8GrF7iVOy/An5zMcjfWqfjXVjjzidp6txeFA1n1 Q2IbzVktuen4h8H/kElfWZ4+RDlK8p7d4e6eYjZr7tMNSjQj+REhVNLExqk86VAwbawbwfMThze E622f6T0t8iObcxlJ6xpzVh949k9l22KxYDg+rc/GGt5BvsSN2CkAjgnjyVwnRy4IpMPsIHx1SD XRZsRk0vRZI9iehjOR0DyUeNV9i2F+V6RQoflGys112ndfCeszpWaZs6GQFek3ENncejX9DVoPp M216ST+L9yS88f+DoQ0rMDjaylQA6TUNxKjVy+BNbSozUsCUd+GfbU8nDgW+PcrIdnoB8f2BYy5 A== X-Received: by 2002:a17:902:c946:b0:2ba:7610:3f21 with SMTP id d9443c01a7336-2baf0cf312bmr7313395ad.5.1778205660230; Thu, 07 May 2026 19:01:00 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1d269fcsm2141945ad.15.2026.05.07.19.00.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:00:59 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 07/11] config.json: add 'containers-' build jobs Date: Thu, 7 May 2026 19:00:30 -0700 Message-ID: <1c444ecb2272a384e77a786f7ca32767f791ce34.1778202125.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:01:04 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3946 From: Tim Orling Add 'containers-' build jobs that build container images on top of the vcontainer-tarball SDK: * containers-meta-virt: the original app-container-* images that ship in meta-virtualization itself (e.g. app-container-curl). * containers-library: a layer for additional images modelled after docker.io/library/*, sourced from meta-yocto-containers-demo Signed-off-by: Tim Orling --- config.json | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/config.json b/config.json index 9e6898d..86c1ffb 100644 --- a/config.json +++ b/config.json @@ -1895,6 +1895,49 @@ "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" ] } + }, + "containers-meta-virt": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Build 'base' container", + "BBTARGETS" : "container-base", + "extravars" : [ + "DISTRO_FEATURES:append = ' virtualization vcontainer'" + ] + }, + "step2" : { + "shortname" : "Build 'curl' container", + "BBTARGETS" : "app-container-curl", + "extravars" : [ + "DISTRO_FEATURES:append = ' virtualization vcontainer'" + ] + } + }, + "containers-library": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization", "meta-yocto-containers-demo"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-openembedded/meta-webserver", + "${BUILDDIR}/../meta-virtualization", + "${BUILDDIR}/../meta-yocto-containers-demo" + ], + "extravars" : [ + "DISTRO_FEATURES:append = ' virtualization vcontainer'" + ], + "step1" : { + "shortname" : "Build 'python' container", + "BBTARGETS" : "app-container-python" + } } }, "repo-defaults" : { @@ -2018,6 +2061,12 @@ "revision" : "HEAD", "no-layer-add" : true }, + "meta-yocto-containers-demo" : { + "url" : "https://github.com/moto-timo/meta-yocto-containers-demo.git", + "branch" : "master", + "revision" : "HEAD", + "no-layer-add" : true + }, "auto-upgrade-helper" : { "url" : "git://git.yoctoproject.org/auto-upgrade-helper", "branch" : "master", From patchwork Fri May 8 02:00:31 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87658 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F538CD37AD for ; Fri, 8 May 2026 02:01:14 +0000 (UTC) Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.4511.1778205664373536050 for ; Thu, 07 May 2026 19:01:04 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=T62VhsvH; spf=pass (domain: konsulko.com, ip: 209.85.214.172, mailfrom: tim.orling@konsulko.com) Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2ad617d5b80so8619205ad.1 for ; Thu, 07 May 2026 19:01:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205663; x=1778810463; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=lHMHMqUFw3Mr58TmHJ1eNwHZJQZt/gX7bY7S8Vl0ghA=; b=T62VhsvH0a4hZmU8m+sTNrPwJzbw1uaTfqxy3daQtKzWKOKKTEZkLjDQ8X0+J3JjNR a9DAQkmziSddn7nAlg1h2l9B0vxwsq+TwwMJQLYhWYYcBd0ws1n+w4l8z/i34iz2zk+A fsMpAW8HSUwaZpHT/gjhf6cEk8NR5NzQR6hxY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205663; x=1778810463; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=lHMHMqUFw3Mr58TmHJ1eNwHZJQZt/gX7bY7S8Vl0ghA=; b=hCsC7leVSiEPh9thui+4pmt6NSODNwMrip7CFKtP8QCPo4UWTaT4VJ5S8Rp0Oz8eGX QW5qMwjE66aO4WCbhbGQGECcUR2VgQDerQLK13Y+8RPlg3X0xzD2YBnLLfjXGKSXcJG6 jTnvUhqFtJO1X2Oe9TR5hMlC2No8/QhhjDcAsuLYC9XFr5+V8arGbhbgd6peermNBex1 aPD7ODLJGdAbUjAjSKTltdSmnIReE30Cg4lJ9tR/urOdMt9CcTvayKbNyjf2J42F72/2 T63hzE9vE9ifDChXCNMk8VvYpFp6Vk/exO/yxKaQA84cMiDE0Ux2DvsxRGFmEfoygAmw 4R7A== X-Gm-Message-State: AOJu0YwStU8U5A8XQLb2Nd5k3kQSQGMhAjsBan21yGz4+HvMk2YFW5zm tXvyopGSisMog/DJMpsI8XIc7Mz++Ow69Pcljz9N7A/lbpk8rWZvhRY0E5Xw5ki5hqHH/pyo0sh E2aef X-Gm-Gg: Acq92OHd6aVwBR/fBqgwwaDi4GBNSjhOx+4oXE3QTjabpw9Hz68JsIBg/5/wf8atU52 IAVJ+l4ytTrx/K1x3hNy7L7QGujeYlhruUb5XgCMMcz50cHzHvI/H6fl+eWIR+afbwBHRH4cWEM pS+5mKpZ0D8vlRqa8mxjuWVVgSLQuHEe2H1PL7Xe2L9wX25SPpOCrGfrK0Pxf6cmhE7A1qZ1RyV sJvBaGqV6DEQQQvhHxmnZdKOHGii7u4JIeXsIurXl0CS1ju5Fqjhr+zifd9PNLRxvIzQAQs+kCF iDQB58L81d3698rgvAa+fYpZ4TvwFeZqPmJ5yYmBIwXbOsZTo7cgshDgV/z332OW/Obgt2Ixlsy dlfTfAW38MDJRGKRd32gaaUp5iZ0vyhmpJ3u362wHHS4Qh3i20mcRc/u7ChNWiNNuXfrXukv5wz xvb6uS2QLCahpSyYvXpeXKSzvaxUKaQt2QAv4hEJv43+XFtUY6XcRABm/1Xl+K5M4= X-Received: by 2002:a17:903:3c05:b0:2ba:1202:4fa8 with SMTP id d9443c01a7336-2baf0dd5625mr6771605ad.20.1778205661966; Thu, 07 May 2026 19:01:01 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1d27254sm1936285ad.4.2026.05.07.19.01.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:01:01 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 08/11] scripts: add run-vcontainer-tests for meta-virtualization Date: Thu, 7 May 2026 19:00:31 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:01:14 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3947 From: Tim Orling Add scripts/run-vcontainer-tests, the test runner used by the vcontainer test jobs. It sources the vcontainer-tarball SDK, discovers the meta-virtualization pytest suite, and runs a configurable set of suites (vdkr, vpdmn, memres) against the checked-out layers. Suites can be selected per-step so the top-level 'vcontainer-tests' job runs the container engine agnostic tests: - tests/test_container_cross_install.py - tests/test_container_registry_script.py - tests/test_vcontainer_auth_config.py - tests/test_multiarch_oci.py - tests/test_multilayer_oci.py The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective suites (including memres for each container engine): - tests/test_vdkr.py - tests/test_vdkr_registry.py and - tests/test_vpdmn.py AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling --- scripts/run-vcontainer-tests | 165 +++++++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100755 scripts/run-vcontainer-tests diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests new file mode 100755 index 0000000..cbb5544 --- /dev/null +++ b/scripts/run-vcontainer-tests @@ -0,0 +1,165 @@ +#!/bin/bash +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Run meta-virtualization pytest test suites against the vcontainer +# standalone SDK (vdkr/vpdmn) that was built by the previous bitbake +# step. +# +# Arguments: +# $1 - suite name: one of "vcontainer", "vdkr", "vpdmn" +# $2 - bitbake build directory (${BUILDDIR}) +# $3 - path to the meta-virtualization layer +# +# Optional environment variables: +# RESULTS_DIR - directory to copy pytest artefacts (junit xml / log) to +# VCONTAINER_EXTRACT_DIR - where to extract the standalone SDK tarball +# (default: ${builddir}/vcontainer-test-extracted) +# TEST_OCI_IMAGE - path to an OCI image directory (enables vdkr/vpdmn +# import tests) +# VDKR_ARCH - target architecture for vdkr/vpdmn tests (default: x86_64) +# +# The script is intentionally conservative: any pytest tests that cannot run +# in the CI environment (those marked "slow", "network", "boot") are skipped +# are skipped so that the autobuilder step completes without needing network +# access. Those can be re-enabled by exporting META_VIRT_PYTEST_MARKERS +# before invocation. +# +# It is assumed that /dev/kvm is writable by the CI user running the tests, +# since the performance is significantly faster with 'memres'. +# + +set -e +set -u +set -o pipefail +set -x + +if [ $# -lt 3 ]; then + echo "Usage: $0 " >&2 + echo " suite: vcontainer | vdkr | vpdmn" >&2 + exit 2 +fi + +suite="$1" +builddir=$(realpath "$2") +metavirtdir=$(realpath "$3") + +if [ ! -d "$metavirtdir/tests" ]; then + echo "ERROR: meta-virtualization tests directory not found at $metavirtdir/tests" >&2 + exit 1 +fi + +# Locate the vcontainer standalone SDK tarball. Prefer an externally-built +# SDK passed via VCONTAINER_SDK (the autobuilder -tests jobs share the SDK +# produced by the separate vcontainer-tarball builder), and fall back to +# looking in the local build's deploy/sdk directory when running stand-alone. +sdk_tarball="" +if [ -n "${VCONTAINER_SDK:-}" ]; then + if [ -f "$VCONTAINER_SDK" ]; then + sdk_tarball="$VCONTAINER_SDK" + else + echo "ERROR: VCONTAINER_SDK=$VCONTAINER_SDK is set but not a file" >&2 + exit 1 + fi +fi +if [ -z "$sdk_tarball" ]; then + sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh" + if [ ! -f "$sdk_tarball" ]; then + # Try to find any matching tarball in case naming changed (e.g. versioned) + alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh 2>/dev/null | head -n1 || true) + if [ -n "$alt" ]; then + sdk_tarball="$alt" + else + echo "ERROR: vcontainer standalone SDK not found." >&2 + echo " Set VCONTAINER_SDK to an existing SDK installer, or" >&2 + echo " build vcontainer-tarball so $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2 + exit 1 + fi + fi +fi + +extract_dir="${VCONTAINER_EXTRACT_DIR:-$builddir/vcontainer-test-extracted}" +rm -rf "$extract_dir" +mkdir -p "$(dirname "$extract_dir")" + +# Self-extracting installer (silent, -y agrees to license, -d picks dir) +"$sdk_tarball" -d "$extract_dir" -y + +# Prepare a Python venv so we don't pollute the worker's system packages. +python3 -m venv "$builddir/meta-virt-test-venv" +# shellcheck disable=SC1091 +source "$builddir/meta-virt-test-venv/bin/activate" +# Avoid warnings by upgrading pip; install pytest/pexpect into the venv via pip. +python3 -m pip install --quiet --upgrade pip setuptools wheel +python3 -m pip install --quiet --upgrade pytest pytest-timeout pexpect + +# Default marker filter excludes long running / infrastructure dependent tests. +marker_filter="${META_VIRT_PYTEST_MARKERS:-not slow and not network and not boot and not incus and not k3s}" + +# Per-suite test file selection. Uses -k/-m for fine-grained filtering and +# keeps the CLI small for logging clarity. +case "$suite" in + vdkr) + test_files=( + "tests/test_vdkr.py" + "tests/test_vdkr_registry.py" + ) + ;; + vpdmn) + test_files=( + "tests/test_vpdmn.py" + ) + ;; + vcontainer) + # Broad vcontainer/bbclass/tooling coverage that doesn't require the + # vdkr/vpdmn CLI harness to be running. + test_files=( + "tests/test_container_cross_install.py" + "tests/test_container_registry_script.py" + "tests/test_vcontainer_auth_config.py" + "tests/test_multiarch_oci.py" + "tests/test_multilayer_oci.py" + ) + ;; + *) + echo "ERROR: unknown suite '$suite' (expected vcontainer|vdkr|vpdmn)" >&2 + exit 2 + ;; +esac + +pytest_args=( + -v + --tb=short + -m "$marker_filter" + --vdkr-dir "$extract_dir" + --junitxml="$builddir/pytest-$suite-results.xml" +) + +# Allow tests that consume an OCI image (import/save/load) to find one. +if [ -n "${TEST_OCI_IMAGE:-}" ] && [ -d "${TEST_OCI_IMAGE}" ]; then + pytest_args+=(--oci-image "$TEST_OCI_IMAGE") +fi + +# Pass architecture through when set in the environment (default is x86_64). +if [ -n "${VDKR_ARCH:-}" ]; then + pytest_args+=(--arch "$VDKR_ARCH") +fi + +cd "$metavirtdir" +# Don't let a single failing test kill the whole step - collect the junit +# report, then surface the exit code via the junit file + exit status. +set +e +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}" +rc=$? +set -e + +# Copy artefacts to the results dir if one was provided. +if [ -n "${RESULTS_DIR:-}" ]; then + mkdir -p "$RESULTS_DIR" + cp -f "$builddir/pytest-$suite-results.xml" "$RESULTS_DIR/" 2>/dev/null || true + if [ -f /tmp/pytest-vcontainer.log ]; then + cp -f /tmp/pytest-vcontainer.log "$RESULTS_DIR/pytest-$suite.log" || true + fi +fi + +exit $rc From patchwork Fri May 8 02:00:32 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61EE0CD3427 for ; Fri, 8 May 2026 02:01:14 +0000 (UTC) Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.4663.1778205664818438544 for ; Thu, 07 May 2026 19:01:04 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=RycwRQqr; spf=pass (domain: konsulko.com, ip: 209.85.210.181, mailfrom: tim.orling@konsulko.com) Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-838d0b7c950so963810b3a.3 for ; Thu, 07 May 2026 19:01:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205664; x=1778810464; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Ek0wk7Jhu8BEqFxB1TRIiJokVIKU2hYC5xqez6fDbs0=; b=RycwRQqrgk0rTV/pMlPudurn0RziqDqeWQW5o1wabF6fSOyVK4MKBh9Bv0IqsG99tC ZsbCmD5yqMTUQ+WiF32RvxC+hGY+HD7XHhGSDF3TBzxZrsm9yTkUXKokR1GOuO+kC1CR SJiSPsVsHJ7BFZccCAUjUTY0TeqJoI18Ts12E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205664; x=1778810464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=Ek0wk7Jhu8BEqFxB1TRIiJokVIKU2hYC5xqez6fDbs0=; b=Zc69lWHmILMYjUWQBnXN+Q6FQqxaGUCj2AE/DL6web0mHeRFLD8QCKZjYwcI9Eo690 gP5jepj0C3JpqleElil39a87dfjNhDV8TsLFaH85nosdWxKorNEVwwCPYrlBRjW7Gmey T/ni2E+GGQYPTkrb/i5GqepdfZZMmkzxSvCyqo67v5mcy195EAr+GYy1Z5e//toekAD6 qH9jugySxsU2COJzPJ0vfCnNd23sEi3LwYGU8929N989SEurjb8y4TkBXgRQIiluNov/ f3VRhFxfdXG3EgX3J1XXJJpEuwKpQFXOoNjIdL+Sd/7DjrP8HbBpuQcmmwcCqz5fZIhW JDzA== X-Gm-Message-State: AOJu0Yz1hagRgqnDDz5tPeL7rw2IVcVGVNkkCTlKjzSSRApjQr/XlNGo 1O4i48G6MNwYlSbdnd59L0myf2Dv2fZeA30ryCrocc9eRKYuZgS5Y0c5y75BtKmqIVx/OF6B+sW XjxoH X-Gm-Gg: AeBDieumAmsEqNJPEyj7HB3ChtrzPLTH/rboMY6sI6ncG4QHxl5gphCdamJBMk6KTkH 8skTABkvYG3kbr3GdJhnE4KjNRxZtzG4WC8OJQgCXulD5gIeUKHpdgJ4bcjp7JWPLkibotejqQt VV00bJOroHmHyda95UX25fnyeZ31sD7jVjmmfPb/MgYIsGNbuzLw0CMsL/R0jvDO7koGRmU8Bxt SNsgsW/xer7aBAi9SYz/ydXu57qoAFzuSX5tKt9uG7se9FZgdmxGEayjv5UXAWtDkazASPYtCkK TNfOMEh/pneaCJv1bXtdWuFxVu6ao2HyJjCPUZbnNdnemFm7yb9RfN46qPMOon62VkufWukTlLr bPHBBCLi/rHe67/xTipFYFFo/Wd+0gJvNaCR7HOBXH2XY9bHhO14x5IARuIipcAtdMl1ahHCmPW ZcCuIvuT20LYk1z5jFf8c8gvpBSoaieJ0i3IS1+qtcriyTvqo+EhKkwjD4Pl3d7lpZXkhAxwx72 A== X-Received: by 2002:a05:6a00:4097:b0:82f:425b:4c27 with SMTP id d2e1a72fcca58-83a5bfbf782mr11566694b3a.14.1778205663745; Thu, 07 May 2026 19:01:03 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-83965d35b3bsm9526633b3a.24.2026.05.07.19.01.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:01:03 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 09/11] config.json: add vcontainer-tests, vdkr-tests, vpdmn-tests jobs Date: Thu, 7 May 2026 19:00:32 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:01:14 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3948 From: Tim Orling Add three pytest job overrides that run the meta-virtualization test suites against the vcontainer standalone SDK published by the vcontainer-tarball builder. Each job runs the new scripts/run-vcontainer-tests helper via EXTRACMDS (we need BUILDDIR variable so EXTRAPLAINCMDS is not sufficient) with both NOBUILDTOOLS and NOVCONTAINER set, so the workers don't spend time setting up either tarball: the test runner gets the published SDK installer through VCONTAINER_SDK and manages its own pytest venv. vcontainer-tests covers the broader bbclass/tooling tests (and so brings in meta-openembedded layers it transitively exercises); vdkr-tests and vpdmn-tests cover their respective CLI harnesses. AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling --- config.json | 42 +++++++++++++++++++++++++++++++++++- scripts/run-vcontainer-tests | 7 +++--- 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/config.json b/config.json index 86c1ffb..dda5b12 100644 --- a/config.json +++ b/config.json @@ -1891,7 +1891,7 @@ }, "step2" : { "shortname" : "Publish vcontainer SDK for test reuse", - "EXTRAPLAINCMDS" : [ + "EXTRACMDS" : [ "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" ] } @@ -1938,6 +1938,46 @@ "shortname" : "Build 'python' container", "BBTARGETS" : "app-container-python" } + }, + "vcontainer-tests": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Run vcontainer pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + } + }, + "vdkr-tests": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "step1" : { + "shortname" : "Run vdkr pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + } + }, + "vpdmn-tests": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "step1" : { + "shortname" : "Run vpdmn pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + } } }, "repo-defaults" : { diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests index cbb5544..1394c7c 100755 --- a/scripts/run-vcontainer-tests +++ b/scripts/run-vcontainer-tests @@ -3,7 +3,7 @@ # SPDX-License-Identifier: GPL-2.0-only # # Run meta-virtualization pytest test suites against the vcontainer -# standalone SDK (vdkr/vpdmn) that was built by the previous bitbake +# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake # step. # # Arguments: @@ -21,9 +21,8 @@ # # The script is intentionally conservative: any pytest tests that cannot run # in the CI environment (those marked "slow", "network", "boot") are skipped -# are skipped so that the autobuilder step completes without needing network -# access. Those can be re-enabled by exporting META_VIRT_PYTEST_MARKERS -# before invocation. +# so that the autobuilder step completes without needing network access. Those +# can be re-enabled by exporting META_VIRT_PYTEST_MARKERS before invocation. # # It is assumed that /dev/kvm is writable by the CI user running the tests, # since the performance is significantly faster with 'memres'. From patchwork Fri May 8 02:00:33 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E57BCD37A7 for ; Fri, 8 May 2026 02:01:14 +0000 (UTC) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.4665.1778205666591546791 for ; Thu, 07 May 2026 19:01:06 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=KicGi0u1; spf=pass (domain: konsulko.com, ip: 209.85.214.180, mailfrom: tim.orling@konsulko.com) Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2ba4a1a0325so10226015ad.0 for ; Thu, 07 May 2026 19:01:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205666; x=1778810466; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=8zoUlgS46/uIUQJfoLuWdhmaV6qcAEnnd0l06KjPCjQ=; b=KicGi0u1nW/EJdFnLVLfaAP8RgM0Dq1TwKC6n8ehD1fin8Nl3od6TL2aAWzrMOpUBK /B36NSslITM6EY4gjh7wQwltD1JNjYwgQugPzFaBZ7WxGi2bWU8XAbPmWIc1Yi3NQmSs lLpAIuEerS1h9UUuACJ9G4cMwCbMX2S7XfHDg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205666; x=1778810466; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=8zoUlgS46/uIUQJfoLuWdhmaV6qcAEnnd0l06KjPCjQ=; b=ZVbO0gN/tGkvl6NDp5i52RuAmEXaodI0nOJ2b6pIYiCfECxPLq2KlvfNF3QtQpe/6n nG4CfAOjlmJc2LKc1CxWDt4rA3VMTCxKVBAxV8ofplYUI7bhC2EPWYNQc+BYFNmBm7f7 NbZt/Z/+z3kL8bNUbyUwZOhu/NuLDl85FpaxRDoteTdKCr41e4QNY1xE70QcvqXWEXeo bPGsscgB9Odr6oRHk4XbseJ3g+62UjI3IC6u+vX5nisWR5i1RPWnxdHgFGNe2W9rqnNO hvdHcPJBt8K3brWkV6A7VEQNFCLC8hf6AzxM9S6G3p87DDfWKHwZdf7BI3QOPcFJpRtF 9u9w== X-Gm-Message-State: AOJu0Yxb4vLjJvnAGcL8tjuG/Og+j7RyjC0omgid4lzlJhFts1wOOwdr G5RrW+Qnsmdoielf3lxpBB9QfxaxLTCdoHgikhEimMCQPpZHfZaWZLPN9FedWwzirbAKCE8866A OBs1n X-Gm-Gg: Acq92OHSM6mho4LeyGwRKidcGAkPymeteKVOEUFzr0mebKp4d3norMi9/jJQWcjAi86 +W6cUIdJMGG2WkVhgaxzrgWmQ+Kb4oVT//W0yctRuAEHY9r9vhFh7xIKjHo2ph9g8e/ZYBR9S+S YBH83ngvoglABaRyEuxCk09h+4+pL3Y0H05WvAAW4c11gjiDugzlnYHWa2bwc0j3wYzSurP+usp NV3wfaO14lQxZ2DfjMg6nlpR5Y/cOIJQokNXb7+zySWwNt2/PJPsCQcPgz7DhsYV+fOMKhuXuJU dPJefku8Z9+0BntXsHWBWtjF66o+2cXfAeWnoZCyD1soquS75FaVw4/yw15ulB0m6TmQj/0UjVK 2scbDOIa+hz1IABy+kdMnadUBr35LVjBBr0Syi6olVCBZOyiBg9xNs08k8ir4t5cmty93icJene eNc1Dtx4zjS9E+dP3ZlJVyTGouuZ1N/bBpxvX4QgorQ5R1LrYii5EJIE5znBESsysAt9xHU0xMO w== X-Received: by 2002:a17:902:cf4c:b0:2b2:5840:809c with SMTP id d9443c01a7336-2ba78b49e00mr105427505ad.1.1778205665489; Thu, 07 May 2026 19:01:05 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1d4050dsm1891695ad.31.2026.05.07.19.01.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:01:05 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 10/11] scripts: add container registry push, auth, tagging, runtime selection Date: Thu, 7 May 2026 19:00:33 -0700 Message-ID: <94817f61d81b462e073e28f48cc71152f666eaeb.1778202125.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:01:14 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3949 From: Tim Orling Add the push-containers infrastructure that drives the post-build steps for the 'containers-' jobs. After each build step the runtime container store is harvested and pushed to one or more registries with derived per-step tags. * config.json: add CONTAINER_REGISTRIES, CONTAINER_AUTH_CONFIG, CONTAINER_RUNTIME, CONTAINER_TAG_CMDS and CONTAINER_VERSION_RECIPE configuration knobs. Tag app-container-python with python3 PV via CONTAINER_VERSION_RECIPE. * scripts/run-config: drive push-containers as a post-step action. Tags are generated from recipe and distro metadata (yocto- tag uses major.minor on snapshots and full PV on releases) with CONTAINER_VERSION_RECIPE allowing a step to source PV from a different recipe than the image itself. * Registry auth is staged via .../config.json or podman .../auth.json using CONTAINER_AUTH_CONFIG, replacing an interactive login that could hang. CONTAINER_RUNTIME picks between vdkr (Docker-compatible) and vpdmn (Podman) runtimes. * Robustness: skip gracefully when no registries are configured, fix the OCI directory path, handle memres already running, and avoid hanging when memres has not yet come up. AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling --- config.json | 17 +++++- scripts/run-config | 128 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 143 insertions(+), 2 deletions(-) diff --git a/config.json b/config.json index dda5b12..7cdf91a 100644 --- a/config.json +++ b/config.json @@ -1,4 +1,4 @@ -{ + { "BASE_HOMEDIR" : "/home/pokybuild", "BASE_SHAREDDIR" : "/srv/autobuilder/autobuilder.yocto.io", "BASE_PUBLISHDIR" : "/srv/autobuilder/downloads.yoctoproject.org", @@ -40,6 +40,10 @@ "SDKEXTRAS" : ["SSTATE_MIRRORS += '\\", "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH'", "BB_HASHSERVE = 'auto'", "BB_HASHSERVE_UPSTREAM = '${AUTOBUILDER_HASHSERV}'"], "BUILDINFO" : false, "BUILDHISTORY" : false, + "CONTAINER_RUNTIME" : "vdkr", + "CONTAINER_REGISTRIES" : [], + "CONTAINER_TAGS" : ["latest"], + "CONTAINER_TAG_CMDS" : [], "BUILDINFOVARS" : ["INHERIT += 'image-buildinfo'", "IMAGE_BUILDINFO_VARS:append = ' IMAGE_BASENAME IMAGE_NAME'"], "WRITECONFIG" : true, "SENDERRORS" : true, @@ -1908,6 +1912,7 @@ "step1" : { "shortname" : "Build 'base' container", "BBTARGETS" : "container-base", + "CONTAINER_IMAGES" : {"container-base": "base"}, "extravars" : [ "DISTRO_FEATURES:append = ' virtualization vcontainer'" ] @@ -1915,6 +1920,7 @@ "step2" : { "shortname" : "Build 'curl' container", "BBTARGETS" : "app-container-curl", + "CONTAINER_IMAGES" : {"app-container-curl": "curl"}, "extravars" : [ "DISTRO_FEATURES:append = ' virtualization vcontainer'" ] @@ -1934,9 +1940,16 @@ "extravars" : [ "DISTRO_FEATURES:append = ' virtualization vcontainer'" ], + "CONTAINER_TAG_CMDS" : [ + "_PV_MAJOR=$(echo $_PV | cut -d. -f1)", + "_PV_MAJOR_MINOR=$(echo $_PV | cut -d. -f1,2)", + "_EXTRA_TAGS=\"$_PV_MAJOR $_PV_MAJOR_MINOR\"" + ], "step1" : { "shortname" : "Build 'python' container", - "BBTARGETS" : "app-container-python" + "BBTARGETS" : "app-container-python", + "CONTAINER_IMAGES" : {"app-container-python": "python"}, + "CONTAINER_VERSION_RECIPE" : "python3" } }, "vcontainer-tests": { diff --git a/scripts/run-config b/scripts/run-config index 0f5a26a..48e0b85 100755 --- a/scripts/run-config +++ b/scripts/run-config @@ -198,6 +198,7 @@ utils.mkdir(args.builddir) revision = "unknown" report = utils.ErrorReport(ourconfig, args.target, args.builddir, properties['branch_oecore'], revision) +push_containers = properties.get("push_containers", False) errordir = utils.errorreportdir(args.builddir) utils.mkdir(errordir) @@ -321,6 +322,133 @@ def handle_stepnum(stepnum): hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets)) bitbakecmd(args.builddir, "bitbake %s -k" % (sanitytargets), report, stepnum, args.stepname) + # Push container images to registries when push_containers is enabled + container_images = utils.getconfigdict("CONTAINER_IMAGES", ourconfig, args.target, stepnum) + if container_images and push_containers: + if jcfg: + addstepentry("push-containers", "Push containers", shortdesc, desc, str(container_images), str(stepnum)) + elif args.stepname == "push-containers": + runtime = utils.getconfigvar("CONTAINER_RUNTIME", ourconfig, args.target, stepnum) or "vdkr" + registries = utils.getconfiglist("CONTAINER_REGISTRIES", ourconfig, args.target, stepnum) + if not registries: + hp.printheader("Step %s/%s: push-containers skipped — CONTAINER_REGISTRIES is empty, no containers pushed" % (stepnum, maxsteps)) + else: + static_tags = utils.getconfiglist("CONTAINER_TAGS", ourconfig, args.target, stepnum) + auth_config = utils.getconfigvar("CONTAINER_AUTH_CONFIG", ourconfig, args.target, stepnum) + if not auth_config: + if runtime == "vpdmn": + auth_config = "${HOME}/.config/containers/auth.json" + else: + auth_config = "${HOME}/.docker/config.json" + hp.printheader("Step %s/%s: Pushing container images %s" % (stepnum, maxsteps, list(container_images.keys()))) + script = [ + "set -e", + "test -w /dev/kvm || { echo 'ERROR: /dev/kvm is not writable, cannot push containers'; exit 1; }", + # Always bring up a fresh memres VM in the foreground. + # + # 'memres status' only checks that the QEMU PID in daemon.pid + # is alive (see daemon_is_running()/daemon_status() in + # meta-virtualization's vrunner.sh); it returns 0 as soon as + # QEMU forks, so a hung/partially-booted VM from a previous + # run — or a VM in mid-boot — is reported as healthy. The + # subsequent 'login'/'vimport'/'push' commands then hang on + # the unresponsive daemon socket. + # + # 'memres restart' is synchronous: it does stop+start and + # runs a PING/PONG readiness probe against the daemon socket + # (120s timeout), exiting non-zero if the VM never answers. + # Running it in the foreground gives us a trustworthy ready + # signal via its exit code, so we can drop the status-poll + # loop entirely. + # + # Install an EXIT trap first so we always tear the daemon + # down, even if bitbake -e / vimport / push fails mid-step + # under 'set -e'. The trap is armed before the restart so + # a restart failure also triggers cleanup. + # + # Registry auth is staged into the guest at VM boot via + # the global '--config' flag — vrunner.sh's setup_auth_share() + # copies $AUTH_CONFIG onto a read-only 9p share, and + # vdkr-init.sh / vpdmn-init.sh's install_auth_config() + # installs it at /root/.docker/config.json (vdkr) or + # /run/containers/0/auth.json (vpdmn) inside the guest. + # Subsequent 'push' calls use those creds directly, so no + # explicit 'login' step is needed. Calling 'login' would + # actually hang under the autobuilder (no PTY): when the + # memres daemon is running, vcontainer-common.sh dispatches + # login via '--daemon-interactive' and blocks reading the + # password from stdin (see login case in vcontainer-common.sh). + "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime, + "%s-$(arch) --config %s memres restart ' suffix on AUTOREV/dev recipes — Docker + # reference format does not allow '+' in tags, and the + # base PV is what consumers expect. + # + # DISTRO_VERSION needs context-sensitive handling. Poky's + # DISTRO_VERSION resolves to '${PV}+snapshot-${METADATA_REVISION}' + # off a tag and just '${PV}' on a release tag. The '+' in + # the snapshot form is illegal in a Docker tag, but more + # importantly the patch level on a snapshot build (e.g. + # '6.0.99' between 6.0 and 6.1) is a moving target that + # doesn't correspond to any real release — only the + # major.minor line is meaningful. So: + # - snapshot build (DISTRO_VERSION contains '+') → tag + # with major.minor only, e.g. 'yocto-6.0'. + # - release-tag build (no '+') → tag with the full + # version, e.g. 'yocto-5.0.5' from the yocto-5.0.5 tag. + script += [ + "_BBENV=$(bitbake -e %s 2>/dev/null) || true" % recipe, + "_PV=$(echo \"$_BBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')", + "_DISTRO_CODENAME=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_CODENAME=/{ print $2; exit }')", + "_DISTRO_VERSION_RAW=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_VERSION=/{ print $2; exit }')", + "case \"$_DISTRO_VERSION_RAW\" in", + " *+*) _DISTRO_VERSION=$(echo \"${_DISTRO_VERSION_RAW%%+*}\" | cut -d. -f1,2) ;;", + " *) _DISTRO_VERSION=\"$_DISTRO_VERSION_RAW\" ;;", + "esac", + "_DEPLOY_DIR_IMAGE=$(echo \"$_BBENV\" | awk -F'\"' '/^DEPLOY_DIR_IMAGE=/{ print $2; exit }')", + "_EXTRA_TAGS=\"\"", + ] + if version_recipe: + # When the image recipe's PV is a wrapper-style + # placeholder (e.g. app-container-python_1.0.0.bb, + # whose 1.0.0 is meaningless to a downstream user), + # CONTAINER_VERSION_RECIPE points at the recipe whose + # PV is actually meaningful for the resulting tag — + # typically the language runtime or app being packaged + # (e.g. python3 -> 3.14.x). Override _PV from that + # recipe; image-recipe state still drives + # DEPLOY_DIR_IMAGE and DISTRO_* since those are + # environment-wide. + script += [ + "_VBBENV=$(bitbake -e %s 2>/dev/null) || true" % version_recipe, + "_PV=$(echo \"$_VBBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')", + ] + script += tag_cmds + script.append( + "_TAGS=\"%s $_PV $_DISTRO_CODENAME yocto-$_DISTRO_VERSION $_EXTRA_TAGS\"" % " ".join(static_tags) + ) + for registry in registries: + # No per-registry 'login': credentials were staged into + # the guest by '--config' on 'memres restart' above. + script += [ + "for _tag in $_TAGS; do", + " %s-$(arch) vimport ${_DEPLOY_DIR_IMAGE}/%s-latest-oci %s/%s:${_tag}" % (runtime, recipe, registry, image), + " %s-$(arch) push %s/%s:${_tag}" % (runtime, registry, image), + "done", + ] + # Tear-down is handled by the EXIT trap installed above. + bitbakecmd(args.builddir, "\n".join(script), report, stepnum, args.stepname) + # Run any extra commands specified cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum) if jcfg: From patchwork Fri May 8 02:00:34 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87656 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51FFDCD343F for ; Fri, 8 May 2026 02:01:14 +0000 (UTC) Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.4512.1778205668116700911 for ; Thu, 07 May 2026 19:01:08 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@konsulko.com header.s=google header.b=CheiWr83; spf=pass (domain: konsulko.com, ip: 209.85.215.170, mailfrom: tim.orling@konsulko.com) Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-c801912c903so641177a12.0 for ; Thu, 07 May 2026 19:01:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1778205667; x=1778810467; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=ebl3YGTub9MrTIEoegIDvC3JC2jHUfps/hvTT6lZxpw=; b=CheiWr83nqcpnIjAz1vewu3LeIGlSvWGFpOtdNRurOoJBjljLj9zdEDqKn+zjfobOa hUEOmlpb/0J4oDjYp91/94M8dhGqJzRsLT49Dt6e9gDoYahrl7dk4J6OuouWT684qEl7 Ce7KJ0z/pUt4io3Wyzs/h6QJf1hNBvBWaVFHg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778205667; x=1778810467; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=ebl3YGTub9MrTIEoegIDvC3JC2jHUfps/hvTT6lZxpw=; b=hXNGwKsDdfOQq/82aKZrcwW7lwtj4aiqQXvKPfn+qPqv2bvBO7o+fSZmFgL1nfRCEd 42C19mwF1RFcUVDksGSUahutVCH6/eNGOukXMTmhbvuQl8KnXo2w34oNXicuscUR1IQH zGvbKxNBkKGhvZdwp8aN4jcf3chcHk3tTmyXyOLexqlR2vStM6KSfHemTJxGGaye1GIJ 83350fXIP24ctJpD8Lg2BH5E9lvAwPFY1FHbL6l3D/342y7+smifsOSrf7kmYeSU/34S M5vfO9LkWzcvBixvmGuYfpLI1obZiaN3/SC3VCyOu8PUn2neY5JXgWyf2p2p2LaRy9v5 ltpA== X-Gm-Message-State: AOJu0Yxjp3iK9ADbZ8vBzuM1QTCbjZIKCUxS08RE/MTczGiFNL5vAfRJ 58TjpGEcIGYZ91Ub6h9FLJsbx3lWO+h+KymQOJZqWF52lWZQTGmrzOybC+M8RoQrdcUe5DyNknP zMrQq X-Gm-Gg: AeBDieth42cYHA3if0fXKXBO06S8pGIRLZa9bVXWJGcEdeNVxcjnxhib/T7BS+IPMLk ZczuftwX9POfeYFJLM+x+ab2ev01f0xxw9B+Lp4zg6MIm9YYGxfAQpFHWZRVkdJbUOauRVimN9p FsjE00mi0zw/AnLCUWdfR2WN3KDy70X4xc/J1V4OiCHFZC7pT1EaS/UzIumy+gC4RSPkejJA2aJ nkWR+PfxAnJQ0F4SRQuSLWSJXfKNcUylqfuZAjQ7agPrbCEFblxH/zUhH2D98Nb5kX/+BQsKE+n Yrci94m95QCrmac/cWbpogU9AaZwB+cucfpjNhpF6NCqn26vHh0fEGRJd5wKimIyp71VQe9Dquf r3kc61Q2xKPZs3qdw+hhTUFoYOi0KYjezrWOit410Vr445VjO0njgq1O9SJ5xHCL++FrrPJeZmE J1RsIvQuG5ia4W8I4rzpb++WupwjA0Zao5gM10MM5WWUg3iaqDPgL2tg6J8W6TguGB6q/dvKwj1 y0ZWOK4yU0f X-Received: by 2002:a05:6a20:7292:b0:3a3:33bd:319b with SMTP id adf61e73a8af0-3aa5a99f71emr11485161637.39.1778205667288; Thu, 07 May 2026 19:01:07 -0700 (PDT) Received: from localhost (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-c8267735eccsm132092a12.32.2026.05.07.19.01.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2026 19:01:06 -0700 (PDT) From: tim.orling@konsulko.com To: yocto-patches@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][PATCH 11/11] config.json: containers-library: add mosquitto and valkey demos Date: Thu, 7 May 2026 19:00:34 -0700 Message-ID: <601c281165450b2be14de46bb9601e82229ca5f6.1778202125.git.tim.orling@konsulko.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 May 2026 02:01:14 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3950 From: Tim Orling Extend containers-library with two additional demo images sourced from meta-yocto-containers-demo: * app-container-mosquitto: the Eclipse Mosquitto MQTT broker. * app-container-valkey: Valkey (Redis-compatible) key/value datastore. Signed-off-by: Tim Orling --- config.json | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/config.json b/config.json index 7cdf91a..a9a2e9c 100644 --- a/config.json +++ b/config.json @@ -1950,6 +1950,18 @@ "BBTARGETS" : "app-container-python", "CONTAINER_IMAGES" : {"app-container-python": "python"}, "CONTAINER_VERSION_RECIPE" : "python3" + }, + "step2" : { + "shortname" : "Build 'mosquitto' container", + "BBTARGETS" : "app-container-mosquitto", + "CONTAINER_IMAGES" : {"app-container-mosquitto": "mosquitto"}, + "CONTAINER_VERSION_RECIPE" : "mosquitto" + }, + "step3" : { + "shortname" : "Build 'valkey' container", + "BBTARGETS" : "app-container-valkey", + "CONTAINER_IMAGES" : {"app-container-valkey": "valkey"}, + "CONTAINER_VERSION_RECIPE" : "valkey" } }, "vcontainer-tests": {