[yocto-autobuilder-helper,v2,07/10] scripts: add run-vcontainer-tests for meta-virtualization

From: Tim Orling <tim.orling@konsulko.com>

Add scripts/run-vcontainer-tests, the test runner used by the
vcontainer test jobs. It sources the vcontainer-tarball SDK,
discovers the meta-virtualization pytest suite, and runs a
configurable set of suites (vdkr, vpdmn, memres) against the
checked-out layers. Suites can be selected per-step so the
top-level 'vcontainer-tests' job runs the container engine
agnostic tests:

- tests/test_container_cross_install.py
- tests/test_container_registry_script.py
- tests/test_vcontainer_auth_config.py
- tests/test_multiarch_oci.py
- tests/test_multilayer_oci.py

The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective
suites (including memres for each container engine):

- tests/test_vdkr.py
- tests/test_vdkr_registry.py

and

- tests/test_vpdmn.py

AI-Generated: Claude Cowork Opus 4.7
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 config.json                  |  34 ++++++++
 scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++
 2 files changed, 198 insertions(+)
 create mode 100755 scripts/run-vcontainer-tests

On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
wrote:
> From: Tim Orling <tim.orling@konsulko.com>
> 
> Add scripts/run-vcontainer-tests, the test runner used by the
> vcontainer test jobs. It sources the vcontainer-tarball SDK,
> discovers the meta-virtualization pytest suite, and runs a
> configurable set of suites (vdkr, vpdmn, memres) against the
> checked-out layers. Suites can be selected per-step so the
> top-level 'vcontainer-tests' job runs the container engine
> agnostic tests:
> 
> - tests/test_container_cross_install.py
> - tests/test_container_registry_script.py
> - tests/test_vcontainer_auth_config.py
> - tests/test_multiarch_oci.py
> - tests/test_multilayer_oci.py
> 
> The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective
> suites (including memres for each container engine):
> 
> - tests/test_vdkr.py
> - tests/test_vdkr_registry.py
> 
> and
> 
> - tests/test_vpdmn.py
> 
> AI-Generated: Claude Cowork Opus 4.7
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
>  config.json                  |  34 ++++++++
>  scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++
>  2 files changed, 198 insertions(+)
>  create mode 100755 scripts/run-vcontainer-tests
> 
> diff --git a/config.json b/config.json
> index 7206c41..79a9d10 100644
> --- a/config.json
> +++ b/config.json
> @@ -1890,6 +1890,40 @@
>                      "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh"
>                  ]
>              }
> +        },
> +        "vcontainer-tests": {
> +            "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"],
> +            "ADDLAYER" : [
> +                "${BUILDDIR}/../meta-openembedded/meta-oe",
> +                "${BUILDDIR}/../meta-openembedded/meta-python",
> +                "${BUILDDIR}/../meta-openembedded/meta-networking",
> +                "${BUILDDIR}/../meta-openembedded/meta-filesystems",
> +                "${BUILDDIR}/../meta-virtualization"
> +            ],
> +            "step1" : {
> +                "shortname" : "Run vcontainer pytest suite",
> +                "NOBUILDTOOLS" : 1,
> +                "NOVCONTAINER" : 1,
> +                "EXTRACMDS" : [
> +                    "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> +                ]
> +            },
> +            "step2" : {
> +                "shortname" : "Run vdkr pytest suite",
> +                "NOBUILDTOOLS" : 1,
> +                "NOVCONTAINER" : 1,
> +                "EXTRACMDS" : [
> +                    "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> +                ]
> +            },
> +            "step3" : {
> +                "shortname" : "Run vpdmn pytest suite",
> +                "NOBUILDTOOLS" : 1,
> +                "NOVCONTAINER" : 1,
> +                "EXTRACMDS" : [
> +                    "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> +                ]
> +            }
>          }
>      },
>      "repo-defaults" : {
> diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests
> new file mode 100755
> index 0000000..1394c7c
> --- /dev/null
> +++ b/scripts/run-vcontainer-tests
> @@ -0,0 +1,164 @@
> +#!/bin/bash
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +# Run meta-virtualization pytest test suites against the vcontainer
> +# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake
> +# step.
> +#
> +# Arguments:
> +#   $1 - suite name: one of "vcontainer", "vdkr", "vpdmn"
> +#   $2 - bitbake build directory (${BUILDDIR})
> +#   $3 - path to the meta-virtualization layer
> +#
> +# Optional environment variables:
> +#   RESULTS_DIR     - directory to copy pytest artefacts (junit xml / log) to
> +#   VCONTAINER_EXTRACT_DIR - where to extract the standalone SDK tarball
> +#                     (default: ${builddir}/vcontainer-test-extracted)
> +#   TEST_OCI_IMAGE  - path to an OCI image directory (enables vdkr/vpdmn
> +#                     import tests)
> +#   VDKR_ARCH       - target architecture for vdkr/vpdmn tests (default: x86_64)

Taking some arguments on the command line and others via environment
variables is confusing. I wonder if should use getopts to parse arguments.

> +#
> +# The script is intentionally conservative: any pytest tests that cannot run
> +# in the CI environment (those marked "slow", "network", "boot") are skipped
> +# so that the autobuilder step completes without needing network access. Those
> +# can be re-enabled by exporting META_VIRT_PYTEST_MARKERS before invocation.
> +#
> +# It is assumed that /dev/kvm is writable by the CI user running the tests,
> +# since the performance is significantly faster with 'memres'.
> +#
> +
> +set -e
> +set -u
> +set -o pipefail
> +set -x
> +
> +if [ $# -lt 3 ]; then
> +    echo "Usage: $0 <suite> <builddir> <meta-virtualization-dir>" >&2
> +    echo "  suite: vcontainer | vdkr | vpdmn" >&2
> +    exit 2
> +fi
> +
> +suite="$1"
> +builddir=$(realpath "$2")
> +metavirtdir=$(realpath "$3")
> +
> +if [ ! -d "$metavirtdir/tests" ]; then
> +    echo "ERROR: meta-virtualization tests directory not found at $metavirtdir/tests" >&2
> +    exit 1
> +fi
> +
> +# Locate the vcontainer standalone SDK tarball. Prefer an externally-built
> +# SDK passed via VCONTAINER_SDK (the autobuilder -tests jobs share the SDK
> +# produced by the separate vcontainer-tarball builder), and fall back to
> +# looking in the local build's deploy/sdk directory when running stand-alone.
> +sdk_tarball=""
> +if [ -n "${VCONTAINER_SDK:-}" ]; then
> +    if [ -f "$VCONTAINER_SDK" ]; then
> +        sdk_tarball="$VCONTAINER_SDK"
> +    else
> +        echo "ERROR: VCONTAINER_SDK=$VCONTAINER_SDK is set but not a file" >&2
> +        exit 1
> +    fi
> +fi
> +if [ -z "$sdk_tarball" ]; then
> +    sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh"
> +    if [ ! -f "$sdk_tarball" ]; then
> +        # Try to find any matching tarball in case naming changed (e.g. versioned)
> +        alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh 2>/dev/null | head -n1 || true)
> +        if [ -n "$alt" ]; then
> +            sdk_tarball="$alt"
> +        else
> +            echo "ERROR: vcontainer standalone SDK not found." >&2
> +            echo "       Set VCONTAINER_SDK to an existing SDK installer, or" >&2
> +            echo "       build vcontainer-tarball so $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2
> +            exit 1
> +        fi
> +    fi
> +fi
> +
> +extract_dir="${VCONTAINER_EXTRACT_DIR:-$builddir/vcontainer-test-extracted}"
> +rm -rf "$extract_dir"
> +mkdir -p "$(dirname "$extract_dir")"
> +
> +# Self-extracting installer (silent, -y agrees to license, -d picks dir)
> +"$sdk_tarball" -d "$extract_dir" -y

Patch 5 in this series added support for setting up the vcontainer
tarball for a step. I think the commit message in this patch needs to
explain why we can't use that in the steps that call
run-vcontainer-tests.

> +
> +# Prepare a Python venv so we don't pollute the worker's system packages.
> +python3 -m venv "$builddir/meta-virt-test-venv"
> +# shellcheck disable=SC1091
> +source "$builddir/meta-virt-test-venv/bin/activate"

Note that this will shadow the venv we use when running test jobs on the
autobuilder workers. That may be ok, but we need to be careful.

> +# Avoid warnings by upgrading pip; install pytest/pexpect into the venv via pip.
> +python3 -m pip install --quiet --upgrade pip setuptools wheel
> +python3 -m pip install --quiet --upgrade pytest pytest-timeout pexpect

Do we want to automatically install the latest versions of these tools,
via the network, in each build? It's better to pin versions to ensure
that upgrades are planned rather than automagical.

Also, should we store this list of dependencies in a requirements.txt
file somewhere? Perhaps tests/requirements.txt in meta-virtualization.

> +
> +# Default marker filter excludes long running / infrastructure dependent tests.
> +marker_filter="${META_VIRT_PYTEST_MARKERS:-not slow and not network and not boot and not incus and not k3s}"

Would it be better to have a positive marker, e.g. 'autobuilder', to
mark tests that we do want to run? That would allow us to maintain the
filtering solely in meta-virtualization instead of needing updates here
as well when things change.

> +
> +# Per-suite test file selection. Uses -k/-m for fine-grained filtering and
> +# keeps the CLI small for logging clarity.
> +case "$suite" in
> +    vdkr)
> +        test_files=(
> +            "tests/test_vdkr.py"
> +            "tests/test_vdkr_registry.py"
> +        )
> +        ;;
> +    vpdmn)
> +        test_files=(
> +            "tests/test_vpdmn.py"
> +        )
> +        ;;
> +    vcontainer)
> +        # Broad vcontainer/bbclass/tooling coverage that doesn't require the
> +        # vdkr/vpdmn CLI harness to be running.
> +        test_files=(
> +            "tests/test_container_cross_install.py"
> +            "tests/test_container_registry_script.py"
> +            "tests/test_vcontainer_auth_config.py"
> +            "tests/test_multiarch_oci.py"
> +            "tests/test_multilayer_oci.py"
> +        )
> +        ;;
> +    *)
> +        echo "ERROR: unknown suite '$suite' (expected vcontainer|vdkr|vpdmn)" >&2
> +        exit 2
> +        ;;
> +esac
> +
> +pytest_args=(
> +    -v
> +    --tb=short
> +    -m "$marker_filter"
> +    --vdkr-dir "$extract_dir"
> +    --junitxml="$builddir/pytest-$suite-results.xml"
> +)
> +
> +# Allow tests that consume an OCI image (import/save/load) to find one.
> +if [ -n "${TEST_OCI_IMAGE:-}" ] && [ -d "${TEST_OCI_IMAGE}" ]; then
> +    pytest_args+=(--oci-image "$TEST_OCI_IMAGE")
> +fi
> +
> +# Pass architecture through when set in the environment (default is x86_64).
> +if [ -n "${VDKR_ARCH:-}" ]; then
> +    pytest_args+=(--arch "$VDKR_ARCH")
> +fi
> +
> +cd "$metavirtdir"
> +# Don't let a single failing test kill the whole step - collect the junit
> +# report, then surface the exit code via the junit file + exit status.
> +set +e
> +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}"
> +rc=$?
> +set -e
> +
> +# Copy artefacts to the results dir if one was provided.
> +if [ -n "${RESULTS_DIR:-}" ]; then
> +    mkdir -p "$RESULTS_DIR"
> +    cp -f "$builddir/pytest-$suite-results.xml" "$RESULTS_DIR/" 2>/dev/null || true
> +    if [ -f /tmp/pytest-vcontainer.log ]; then
> +        cp -f /tmp/pytest-vcontainer.log "$RESULTS_DIR/pytest-$suite.log" || true
> +    fi
> +fi
> +
> +exit $rc

On Tue, Jun 2, 2026 at 3:50 AM Paul Barker via lists.yoctoproject.org <paul=
pbarker.dev@lists.yoctoproject.org> wrote:

> On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
> wrote:
> > From: Tim Orling <tim.orling@konsulko.com>
> >
> > Add scripts/run-vcontainer-tests, the test runner used by the
> > vcontainer test jobs. It sources the vcontainer-tarball SDK,
> > discovers the meta-virtualization pytest suite, and runs a
> > configurable set of suites (vdkr, vpdmn, memres) against the
> > checked-out layers. Suites can be selected per-step so the
> > top-level 'vcontainer-tests' job runs the container engine
> > agnostic tests:
> >
> > - tests/test_container_cross_install.py
> > - tests/test_container_registry_script.py
> > - tests/test_vcontainer_auth_config.py
> > - tests/test_multiarch_oci.py
> > - tests/test_multilayer_oci.py
> >
> > The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective
> > suites (including memres for each container engine):
> >
> > - tests/test_vdkr.py
> > - tests/test_vdkr_registry.py
> >
> > and
> >
> > - tests/test_vpdmn.py
> >
> > AI-Generated: Claude Cowork Opus 4.7
> > Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> > ---
> >  config.json                  |  34 ++++++++
> >  scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++
> >  2 files changed, 198 insertions(+)
> >  create mode 100755 scripts/run-vcontainer-tests
> >
> > diff --git a/config.json b/config.json
> > index 7206c41..79a9d10 100644
> > --- a/config.json
> > +++ b/config.json
> > @@ -1890,6 +1890,40 @@
> >                      "install -d
> ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755
> ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh
> ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/
> vcontainer-standalone.sh.new && mv -f
> ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/
> vcontainer-standalone.sh.new
> ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh"
> >                  ]
> >              }
> > +        },
> > +        "vcontainer-tests": {
> > +            "NEEDREPOS" : ["bitbake", "meta-openembedded",
> "meta-virtualization"],
> > +            "ADDLAYER" : [
> > +                "${BUILDDIR}/../meta-openembedded/meta-oe",
> > +                "${BUILDDIR}/../meta-openembedded/meta-python",
> > +                "${BUILDDIR}/../meta-openembedded/meta-networking",
> > +                "${BUILDDIR}/../meta-openembedded/meta-filesystems",
> > +                "${BUILDDIR}/../meta-virtualization"
> > +            ],
> > +            "step1" : {
> > +                "shortname" : "Run vcontainer pytest suite",
> > +                "NOBUILDTOOLS" : 1,
> > +                "NOVCONTAINER" : 1,
> > +                "EXTRACMDS" : [
> > +
> "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh
> RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests
> vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> > +                ]
> > +            },
> > +            "step2" : {
> > +                "shortname" : "Run vdkr pytest suite",
> > +                "NOBUILDTOOLS" : 1,
> > +                "NOVCONTAINER" : 1,
> > +                "EXTRACMDS" : [
> > +
> "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh
> RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr
> ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> > +                ]
> > +            },
> > +            "step3" : {
> > +                "shortname" : "Run vpdmn pytest suite",
> > +                "NOBUILDTOOLS" : 1,
> > +                "NOVCONTAINER" : 1,
> > +                "EXTRACMDS" : [
> > +
> "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh
> RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn
> ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> > +                ]
> > +            }
> >          }
> >      },
> >      "repo-defaults" : {
> > diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests
> > new file mode 100755
> > index 0000000..1394c7c
> > --- /dev/null
> > +++ b/scripts/run-vcontainer-tests
> > @@ -0,0 +1,164 @@
> > +#!/bin/bash
> > +#
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +#
> > +# Run meta-virtualization pytest test suites against the vcontainer
> > +# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake
> > +# step.
> > +#
> > +# Arguments:
> > +#   $1 - suite name: one of "vcontainer", "vdkr", "vpdmn"
> > +#   $2 - bitbake build directory (${BUILDDIR})
> > +#   $3 - path to the meta-virtualization layer
> > +#
> > +# Optional environment variables:
> > +#   RESULTS_DIR     - directory to copy pytest artefacts (junit xml /
> log) to
> > +#   VCONTAINER_EXTRACT_DIR - where to extract the standalone SDK tarball
> > +#                     (default: ${builddir}/vcontainer-test-extracted)
> > +#   TEST_OCI_IMAGE  - path to an OCI image directory (enables vdkr/vpdmn
> > +#                     import tests)
> > +#   VDKR_ARCH       - target architecture for vdkr/vpdmn tests
> (default: x86_64)
>
> Taking some arguments on the command line and others via environment
> variables is confusing. I wonder if should use getopts to parse arguments.
>

This is addressed in v3.


> > +#
> > +# The script is intentionally conservative: any pytest tests that
> cannot run
> > +# in the CI environment (those marked "slow", "network", "boot") are
> skipped
> > +# so that the autobuilder step completes without needing network
> access. Those
> > +# can be re-enabled by exporting META_VIRT_PYTEST_MARKERS before
> invocation.
> > +#
> > +# It is assumed that /dev/kvm is writable by the CI user running the
> tests,
> > +# since the performance is significantly faster with 'memres'.
> > +#
> > +
> > +set -e
> > +set -u
> > +set -o pipefail
> > +set -x
> > +
> > +if [ $# -lt 3 ]; then
> > +    echo "Usage: $0 <suite> <builddir> <meta-virtualization-dir>" >&2
> > +    echo "  suite: vcontainer | vdkr | vpdmn" >&2
> > +    exit 2
> > +fi
> > +
> > +suite="$1"
> > +builddir=$(realpath "$2")
> > +metavirtdir=$(realpath "$3")
> > +
> > +if [ ! -d "$metavirtdir/tests" ]; then
> > +    echo "ERROR: meta-virtualization tests directory not found at
> $metavirtdir/tests" >&2
> > +    exit 1
> > +fi
> > +
> > +# Locate the vcontainer standalone SDK tarball. Prefer an
> externally-built
> > +# SDK passed via VCONTAINER_SDK (the autobuilder -tests jobs share the
> SDK
> > +# produced by the separate vcontainer-tarball builder), and fall back to
> > +# looking in the local build's deploy/sdk directory when running
> stand-alone.
> > +sdk_tarball=""
> > +if [ -n "${VCONTAINER_SDK:-}" ]; then
> > +    if [ -f "$VCONTAINER_SDK" ]; then
> > +        sdk_tarball="$VCONTAINER_SDK"
> > +    else
> > +        echo "ERROR: VCONTAINER_SDK=$VCONTAINER_SDK is set but not a
> file" >&2
> > +        exit 1
> > +    fi
> > +fi
> > +if [ -z "$sdk_tarball" ]; then
> > +    sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh"
> > +    if [ ! -f "$sdk_tarball" ]; then
> > +        # Try to find any matching tarball in case naming changed (e.g.
> versioned)
> > +        alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh
> 2>/dev/null | head -n1 || true)
> > +        if [ -n "$alt" ]; then
> > +            sdk_tarball="$alt"
> > +        else
> > +            echo "ERROR: vcontainer standalone SDK not found." >&2
> > +            echo "       Set VCONTAINER_SDK to an existing SDK
> installer, or" >&2
> > +            echo "       build vcontainer-tarball so
> $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2
> > +            exit 1
> > +        fi
> > +    fi
> > +fi
> > +
> >
> +extract_dir="${VCONTAINER_EXTRACT_DIR:-$builddir/vcontainer-test-extracted}"
> > +rm -rf "$extract_dir"
> > +mkdir -p "$(dirname "$extract_dir")"
> > +
> > +# Self-extracting installer (silent, -y agrees to license, -d picks dir)
> > +"$sdk_tarball" -d "$extract_dir" -y
>
> Patch 5 in this series added support for setting up the vcontainer
> tarball for a step. I think the commit message in this patch needs to
> explain why we can't use that in the steps that call
> run-vcontainer-tests.
>
> Added information related to this context to v3.


> > +
> > +# Prepare a Python venv so we don't pollute the worker's system
> packages.
> > +python3 -m venv "$builddir/meta-virt-test-venv"
> > +# shellcheck disable=SC1091
> > +source "$builddir/meta-virt-test-venv/bin/activate"
>
> Note that this will shadow the venv we use when running test jobs on the
> autobuilder workers. That may be ok, but we need to be careful.
>

This venv is very very specifically for only the meta-virtualization tests.
Other venvs in
autobuilder workers as far as I can tell do not have 'pip' and do not
install 'pytest' nor
the other requirements.


> > +# Avoid warnings by upgrading pip; install pytest/pexpect into the venv
> via pip.
> > +python3 -m pip install --quiet --upgrade pip setuptools wheel
> > +python3 -m pip install --quiet --upgrade pytest pytest-timeout pexpect
>
> Do we want to automatically install the latest versions of these tools,
> via the network, in each build? It's better to pin versions to ensure
> that upgrades are planned rather than automagical.
>
>
In v3, I altered the pip, setuptools, wheel upgrade to
exporting  PIP_DISABLE_PIP_VERSION_CHECK=1
to silence the pip upgrade warning.


> Also, should we store this list of dependencies in a requirements.txt
> file somewhere? Perhaps tests/requirements.txt in meta-virtualization.
>

In v3, I changed the installation of pytest, pytest-timeout and pexpect to
be instead from a
meta-virtualization/tests/requirements.txt file (pending).


> > +
> > +# Default marker filter excludes long running / infrastructure
> dependent tests.
> > +marker_filter="${META_VIRT_PYTEST_MARKERS:-not slow and not network and
> not boot and not incus and not k3s}"
>
> Would it be better to have a positive marker, e.g. 'autobuilder', to
> mark tests that we do want to run? That would allow us to maintain the
> filtering solely in meta-virtualization instead of needing updates here
> as well when things change.
>
>
Making CI related changes in meta-virtualization is STRONGLY rejected by
the maintainer of meta-virtualization and you will have to convince the
upstream layer maintainer.

This pattern was lifted directly from the meta-virtualization README and
existing examples.


> > +
> > +# Per-suite test file selection. Uses -k/-m for fine-grained filtering
> and
> > +# keeps the CLI small for logging clarity.
> > +case "$suite" in
> > +    vdkr)
> > +        test_files=(
> > +            "tests/test_vdkr.py"
> > +            "tests/test_vdkr_registry.py"
> > +        )
> > +        ;;
> > +    vpdmn)
> > +        test_files=(
> > +            "tests/test_vpdmn.py"
> > +        )
> > +        ;;
> > +    vcontainer)
> > +        # Broad vcontainer/bbclass/tooling coverage that doesn't
> require the
> > +        # vdkr/vpdmn CLI harness to be running.
> > +        test_files=(
> > +            "tests/test_container_cross_install.py"
> > +            "tests/test_container_registry_script.py"
> > +            "tests/test_vcontainer_auth_config.py"
> > +            "tests/test_multiarch_oci.py"
> > +            "tests/test_multilayer_oci.py"
> > +        )
> > +        ;;
> > +    *)
> > +        echo "ERROR: unknown suite '$suite' (expected
> vcontainer|vdkr|vpdmn)" >&2
> > +        exit 2
> > +        ;;
> > +esac
> > +
> > +pytest_args=(
> > +    -v
> > +    --tb=short
> > +    -m "$marker_filter"
> > +    --vdkr-dir "$extract_dir"
> > +    --junitxml="$builddir/pytest-$suite-results.xml"
> > +)
> > +
> > +# Allow tests that consume an OCI image (import/save/load) to find one.
> > +if [ -n "${TEST_OCI_IMAGE:-}" ] && [ -d "${TEST_OCI_IMAGE}" ]; then
> > +    pytest_args+=(--oci-image "$TEST_OCI_IMAGE")
> > +fi
> > +
> > +# Pass architecture through when set in the environment (default is
> x86_64).
> > +if [ -n "${VDKR_ARCH:-}" ]; then
> > +    pytest_args+=(--arch "$VDKR_ARCH")
> > +fi
> > +
> > +cd "$metavirtdir"
> > +# Don't let a single failing test kill the whole step - collect the
> junit
> > +# report, then surface the exit code via the junit file + exit status.
> > +set +e
> > +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}"
> > +rc=$?
> > +set -e
> > +
> > +# Copy artefacts to the results dir if one was provided.
> > +if [ -n "${RESULTS_DIR:-}" ]; then
> > +    mkdir -p "$RESULTS_DIR"
> > +    cp -f "$builddir/pytest-$suite-results.xml" "$RESULTS_DIR/"
> 2>/dev/null || true
> > +    if [ -f /tmp/pytest-vcontainer.log ]; then
> > +        cp -f /tmp/pytest-vcontainer.log
> "$RESULTS_DIR/pytest-$suite.log" || true
> > +    fi
> > +fi
> > +
> > +exit $rc
>
> --
> Paul Barker
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#4111):
> https://lists.yoctoproject.org/g/yocto-patches/message/4111
> Mute This Topic: https://lists.yoctoproject.org/mt/119603246/924729
> Group Owner: yocto-patches+owner@lists.yoctoproject.org
> Unsubscribe:
> https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
> [ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>