From patchwork Fri Jun 27 13:49:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99C2EC77B7F for ; Fri, 27 Jun 2025 13:49:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.14176.1751032190268941940 for ; Fri, 27 Jun 2025 06:49:50 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8B2721007 for ; Fri, 27 Jun 2025 06:49:32 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6AF243F66E for ; Fri, 27 Jun 2025 06:49:49 -0700 (PDT) From: Ross Burton To: meta-arm@lists.yoctoproject.org Subject: [PATCH][kirkstone 1/6] CI: add CI_CLEAN_REPOS variable to allow cleaning the repo reference cache Date: Fri, 27 Jun 2025 14:49:41 +0100 Message-ID: <20250627134946.154438-1-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:57 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/6585 If the repository reference directory gets corrupted it's not easy to wipe it, so add a variable CI_CLEAN_REPOS that if set in the pipeline will clean the clones and re-fetch them. Also, stop the fetch from detaching during the garbage collection, just in case it was a long-running GC that got killed that caused the corruption in the first place. Signed-off-by: Ross Burton Signed-off-by: Jon Mason --- .gitlab-ci.yml | 3 ++- ci/update-repos | 8 +++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 5aad0300d0..fd6cbd57c3 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -66,7 +66,8 @@ stages: # -# Prep stage, update repositories once +# Prep stage, update repositories once. +# Set the CI variable CI_CLEAN_REPOS=1 to refetch the respositories from scratch # update-repos: extends: .setup diff --git a/ci/update-repos b/ci/update-repos index d1b6c026ff..5ac083121a 100755 --- a/ci/update-repos +++ b/ci/update-repos @@ -4,6 +4,7 @@ import sys import os +import shutil import subprocess import pathlib @@ -35,9 +36,14 @@ if __name__ == "__main__": for repo in repositories: repodir = base_repodir / repo_shortname(repo) + + if "CI_CLEAN_REPOS" in os.environ: + print("Cleaning %s..." % repo) + shutil.rmtree(repodir, ignore_errors=True) + if repodir.exists(): print("Updating %s..." % repo) - subprocess.run(["git", "-C", repodir, "fetch"], check=True) + subprocess.run(["git", "-C", repodir, "-c", "gc.autoDetach=false", "fetch"], check=True) else: print("Cloning %s..." % repo) subprocess.run(["git", "clone", "--bare", repo, repodir], check=True) From patchwork Fri Jun 27 13:49:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6100C83026 for ; Fri, 27 Jun 2025 13:49:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.14177.1751032191019767204 for ; Fri, 27 Jun 2025 06:49:51 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4D01B1007 for ; Fri, 27 Jun 2025 06:49:33 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2C8AA3F66E for ; Fri, 27 Jun 2025 06:49:50 -0700 (PDT) From: Ross Burton To: meta-arm@lists.yoctoproject.org Subject: [PATCH][kirkstone 2/6] CI: clean up variable definitions Date: Fri, 27 Jun 2025 14:49:42 +0100 Message-ID: <20250627134946.154438-2-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134946.154438-1-ross.burton@arm.com> References: <20250627134946.154438-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:57 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/6586 Mostly taken from master, rationalise the variable definitions and stop hardcoding /persist/. Users of this CI will have to ensure that they've set CACHE_DIR if they want anything actually cached. Signed-off-by: Ross Burton --- .gitlab-ci.yml | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index fd6cbd57c3..9c9eb74191 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -7,6 +7,10 @@ variables: # by default FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY: 0 FF_KUBERNETES_HONOR_ENTRYPOINT: 1 + # The directory to use as the persistent cache (the root for DL_DIR, + # SSTATE_DIR, etc). The default is the build tree which will not be + # persistent, so this should be set in the runner. + CACHE_DIR: $CI_PROJECT_DIR stages: - prep @@ -20,11 +24,12 @@ stages: interruptible: true variables: KAS_WORK_DIR: $CI_PROJECT_DIR/work - KAS_REPO_REF_DIR: $CI_BUILDS_DIR/persist/repos - SSTATE_DIR: $CI_BUILDS_DIR/persist/sstate - DL_DIR: $CI_BUILDS_DIR/persist/downloads + KAS_BUILD_DIR: $KAS_WORK_DIR/build + KAS_REPO_REF_DIR: "" + SSTATE_DIR: $CACHE_DIR/sstate + DL_DIR: $CACHE_DIR/downloads BB_LOGCONFIG: $CI_PROJECT_DIR/ci/logging.yml - TOOLCHAIN_DIR: $CI_BUILDS_DIR/persist/toolchains + TOOLCHAIN_DIR: $CACHE_DIR/toolchains IMAGE_DIR: $CI_PROJECT_DIR/work/build/tmp/deploy/images TOOLCHAIN_LINK_DIR: $CI_PROJECT_DIR/work/build/toolchains before_script: From patchwork Fri Jun 27 13:49:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93C24C7EE31 for ; Fri, 27 Jun 2025 13:49:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.14154.1751032191817583382 for ; Fri, 27 Jun 2025 06:49:51 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 171C51007 for ; Fri, 27 Jun 2025 06:49:34 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E27513F66E for ; Fri, 27 Jun 2025 06:49:50 -0700 (PDT) From: Ross Burton To: meta-arm@lists.yoctoproject.org Subject: [PATCH][kirkstone 3/6] CI: remove obsolete installation of python3-subunit Date: Fri, 27 Jun 2025 14:49:43 +0100 Message-ID: <20250627134946.154438-3-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134946.154438-1-ross.burton@arm.com> References: <20250627134946.154438-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:57 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/6587 This is in the Kas image from 3.2, so this is now obsolete. Signed-off-by: Ross Burton --- .gitlab-ci.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 9c9eb74191..2a07ea08be 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -40,8 +40,6 @@ stages: - mkdir --verbose --parents $KAS_WORK_DIR $KAS_REPO_REF_DIR $SSTATE_DIR $DL_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR # Must do this here, as it's the only way to make sure the toolchain is installed on the same builder - ./ci/get-binary-toolchains $DL_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR - # This can be removed with Kas 3.2 - - sudo apt-get update && sudo apt-get install --yes python3-subunit # Generalised fragment to do a Kas build .build: From patchwork Fri Jun 27 13:49:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 985A3C7EE32 for ; Fri, 27 Jun 2025 13:49:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.14155.1751032192461974232 for ; Fri, 27 Jun 2025 06:49:52 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D51AA1007 for ; Fri, 27 Jun 2025 06:49:34 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AC3893F66E for ; Fri, 27 Jun 2025 06:49:51 -0700 (PDT) From: Ross Burton To: meta-arm@lists.yoctoproject.org Subject: [PATCH][kirkstone 4/6] CI: don't run update-repos if KAS_REPO_REF_DIR isn't set Date: Fri, 27 Jun 2025 14:49:44 +0100 Message-ID: <20250627134946.154438-4-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134946.154438-1-ross.burton@arm.com> References: <20250627134946.154438-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:57 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/6588 Mostly taken from master. Signed-off-by: Ross Burton --- .gitlab-ci.yml | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 2a07ea08be..ef75a45bba 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -76,8 +76,16 @@ update-repos: extends: .setup stage: prep script: - - flock --verbose --timeout 60 $KAS_REPO_REF_DIR ./ci/update-repos + - | + exit_code=0 + if [ -n "$KAS_REPO_REF_DIR" ]; then + flock --verbose --timeout 60 $KAS_REPO_REF_DIR --command ./ci/update-repos || exit_code=$? + # Exit now if that failed, unless the status was 128 (fetch failed) + test $exit_code != 0 -a $exit_code != 128 && exit 1 + fi + + exit $exit_code # # Build stage, the actual build jobs # From patchwork Fri Jun 27 13:49:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A47C7C7EE3A for ; Fri, 27 Jun 2025 13:49:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.14178.1751032193233779641 for ; Fri, 27 Jun 2025 06:49:53 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A08011007 for ; Fri, 27 Jun 2025 06:49:35 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 761E63F66E for ; Fri, 27 Jun 2025 06:49:52 -0700 (PDT) From: Ross Burton To: meta-arm@lists.yoctoproject.org Subject: [PATCH][kirkstone 5/6] CI: show all variables in update-repos Date: Fri, 27 Jun 2025 14:49:45 +0100 Message-ID: <20250627134946.154438-5-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134946.154438-1-ross.burton@arm.com> References: <20250627134946.154438-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:57 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/6589 This makes it easier to debug path usage. Signed-off-by: Ross Burton --- .gitlab-ci.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index ef75a45bba..91e92ae2ec 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -33,9 +33,6 @@ stages: IMAGE_DIR: $CI_PROJECT_DIR/work/build/tmp/deploy/images TOOLCHAIN_LINK_DIR: $CI_PROJECT_DIR/work/build/toolchains before_script: - - echo KAS_WORK_DIR = $KAS_WORK_DIR - - echo SSTATE_DIR = $SSTATE_DIR - - echo DL_DIR = $DL_DIR - rm -rf $KAS_WORK_DIR - mkdir --verbose --parents $KAS_WORK_DIR $KAS_REPO_REF_DIR $SSTATE_DIR $DL_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR # Must do this here, as it's the only way to make sure the toolchain is installed on the same builder @@ -79,6 +76,8 @@ update-repos: - | exit_code=0 + printenv + if [ -n "$KAS_REPO_REF_DIR" ]; then flock --verbose --timeout 60 $KAS_REPO_REF_DIR --command ./ci/update-repos || exit_code=$? # Exit now if that failed, unless the status was 128 (fetch failed) From patchwork Fri Jun 27 13:49:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF015C83010 for ; Fri, 27 Jun 2025 13:49:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.14158.1751032194009062241 for ; Fri, 27 Jun 2025 06:49:54 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 693651007 for ; Fri, 27 Jun 2025 06:49:36 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 40AE53F66E for ; Fri, 27 Jun 2025 06:49:53 -0700 (PDT) From: Ross Burton To: meta-arm@lists.yoctoproject.org Subject: [PATCH][kirkstone 6/6] CI: fix meta-virtualization URL Date: Fri, 27 Jun 2025 14:49:46 +0100 Message-ID: <20250627134946.154438-6-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134946.154438-1-ross.burton@arm.com> References: <20250627134946.154438-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:57 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/6590 Use https: so that Kas premirroring can work. Signed-off-by: Ross Burton --- ci/meta-virtualization.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/meta-virtualization.yml b/ci/meta-virtualization.yml index 1cd0e21a89..157e21e2a7 100644 --- a/ci/meta-virtualization.yml +++ b/ci/meta-virtualization.yml @@ -5,4 +5,4 @@ header: repos: meta-virtualization: - url: git://git.yoctoproject.org/meta-virtualization + url: https://git.yoctoproject.org/meta-virtualization