| Message ID | 20260701074030.1090807-5-twoerner@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | tests: standalone test-suite framework plus the first unit test | expand |
On Wed, 2026-07-01 at 03:40 -0400, Trevor Woerner via lists.yoctoproject.org wrote: > Knowing which lines a test run actually exercised is the difference > between "the suite is green" and "the suite is green and we know what > it touched". This commit wires branch coverage of the wic source into > the runner, kept entirely opt-in so a plain run stays fast and quiet. > > pyproject.toml gains coverage and pytest-cov in the tests extra, so > "pip install -e .[tests]" pulls in what the new flags need. > > run-tests.sh gains two options: > > - --coverage measures branch coverage of src/wic during the run and > prints a terminal report listing the lines that were missed; > > - --html [DIR] additionally writes a browsable HTML report (default > htmlcov/, or DIR if given) and implies --coverage. > > If --coverage is requested but pytest-cov is not installed the runner > fails loudly with the install command rather than running without the > measurement it was asked for. > > The coverage data file and the HTML report directory are build > artifacts, so .gitignore learns to ignore .coverage and htmlcov/. > > AI-Generated: codex/claude-opus 4.7 (xhigh) > Signed-off-by: Trevor Woerner <twoerner@gmail.com> Ok, now the run-tests.sh script starts to make sense. Have you considered something like `just` (https://github.com/casey/just) as a command runner? It's much simpler to maintain a `justfile` than a shell script. Using claude (opus 4.8) to convert run-tests.sh (as it is at the end of this series of patches) to a much simpler justfile, I get: # List the available recipes. default: @just --list # Run the test suite. Extra args pass through to pytest (e.g. `just test -k filemap`). test *args="tests": pytest {{args}} # Run the suite with a terminal branch-coverage report for wic. coverage *args="tests": pytest {{args}} --cov=wic --cov-branch --cov-report=term-missing # Run the suite with an HTML coverage report (default dir: htmlcov/). coverage-html dir="htmlcov" *args="tests": pytest {{args}} --cov=wic --cov-branch --cov-report=term-missing --cov-report=html:{{dir}} # Lint the test suite; held to a clean bar, so this must report nothing. lint-tests: ruff check tests # Lint src/ (not yet ruff-clean, so this is a preview report, not a gate). lint: lint-tests ruff check src Feel free to keep the shell script if you want to, but I think the above is easier to maintain :) Best regards,
diff --git a/.gitignore b/.gitignore index 07992096c0fb..534c49538091 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,7 @@ # pytest cache /.pytest_cache/ + +# coverage data and reports +/.coverage +/htmlcov/ diff --git a/pyproject.toml b/pyproject.toml index d660e39007c4..ece2757bb686 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -24,6 +24,8 @@ Repository = "https://git.yoctoproject.org/wic" [project.optional-dependencies] tests = [ "pytest >= 7.0", + "coverage >= 7.0", + "pytest-cov >= 4.0", ] [project.scripts] diff --git a/tests/run-tests.sh b/tests/run-tests.sh index 9db6d50338d6..a483da6a63a6 100755 --- a/tests/run-tests.sh +++ b/tests/run-tests.sh @@ -13,9 +13,13 @@ set -euo pipefail usage() { cat <<'USAGE' Usage: - tests/run-tests.sh [pytest args] + tests/run-tests.sh [--coverage] [--html [DIR]] [pytest args] Options: + --coverage also measure branch coverage of src/wic and print a + terminal report listing the lines that were missed + --html [DIR] also write an HTML coverage report (default dir: + htmlcov/); implies --coverage -h, --help show this help and exit Anything else is passed straight through to pytest (for example a @@ -24,6 +28,9 @@ tests/ is run. Examples: tests/run-tests.sh # whole suite + tests/run-tests.sh --coverage # + terminal coverage report + tests/run-tests.sh --html # + HTML report in htmlcov/ + tests/run-tests.sh --html /tmp/cov # + HTML report in /tmp/cov tests/run-tests.sh -k filemap -v # pass args through to pytest tests/run-tests.sh tests/unit # a single tier or file @@ -34,9 +41,25 @@ USAGE REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" PY="${PYTHON:-python3}" +coverage=0 +html=0 +html_dir="htmlcov" pytest_args=() while [ $# -gt 0 ]; do case "$1" in + --coverage) + coverage=1 + ;; + --html) + coverage=1 + html=1 + # Optional directory argument: consume the next token only if + # it is not another option or the pytest pass-through marker. + case "${2:-}" in + ""|-*|--) ;; + *) html_dir="$2"; shift ;; + esac + ;; -h|--help) usage exit 0 @@ -72,4 +95,18 @@ if [ ${#pytest_args[@]} -eq 0 ]; then pytest_args=("tests") fi +if [ "$coverage" -eq 1 ]; then + if ! "$PY" -c "import pytest_cov" >/dev/null 2>&1; then + echo "error: coverage requested but pytest-cov is not installed." >&2 + echo " run: $PY -m pip install -e \".[tests]\"" >&2 + exit 1 + fi + cov_args=(--cov=wic --cov-branch --cov-report=term-missing) + if [ "$html" -eq 1 ]; then + cov_args+=(--cov-report="html:${html_dir}") + echo "HTML coverage report: ${html_dir}/index.html" >&2 + fi + exec "$PY" -m pytest "${pytest_args[@]}" "${cov_args[@]}" +fi + exec "$PY" -m pytest "${pytest_args[@]}"
Knowing which lines a test run actually exercised is the difference between "the suite is green" and "the suite is green and we know what it touched". This commit wires branch coverage of the wic source into the runner, kept entirely opt-in so a plain run stays fast and quiet. pyproject.toml gains coverage and pytest-cov in the tests extra, so "pip install -e .[tests]" pulls in what the new flags need. run-tests.sh gains two options: - --coverage measures branch coverage of src/wic during the run and prints a terminal report listing the lines that were missed; - --html [DIR] additionally writes a browsable HTML report (default htmlcov/, or DIR if given) and implies --coverage. If --coverage is requested but pytest-cov is not installed the runner fails loudly with the install command rather than running without the measurement it was asked for. The coverage data file and the HTML report directory are build artifacts, so .gitignore learns to ignore .coverage and htmlcov/. AI-Generated: codex/claude-opus 4.7 (xhigh) Signed-off-by: Trevor Woerner <twoerner@gmail.com> --- changes in v3: - no change in this revision. changes in v2: - v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it. --- .gitignore | 4 ++++ pyproject.toml | 2 ++ tests/run-tests.sh | 39 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 44 insertions(+), 1 deletion(-)