[wic,v3,07/10] tests/docs: add the test-authoring guide

A suite is only as good as the tests added to it over time, so it needs
a guide that says where a new test goes, what one looks like, and how
to choose its inputs and assertions. This commit adds
tests/docs/authoring.md.

The guide covers:

  - where a test goes: one test_<area>.py per wic module under
    tests/unit/, extending an existing file or starting a new one;

  - the shape of a test: a small worked example, a preference for
    parametrize over in-test loops so each input/output pair is its own
    named case, and use of the tmp_path fixture for scratch files;

  - asserting the correct behaviour rather than whatever wic currently
    does, so a test never bakes in a bug; when a test exposes a defect,
    the source fix lands in the same change so the test passes;

  - probing the boundaries: a checklist of input classes worth covering
    for numbers, strings, paths, types/shape, and state, each asserting
    a specific value or exception rather than merely "it did not
    crash";

  - keeping the assertion strong: never weaken an assertion to get
    green;

  - how to run just the test you wrote, with and without coverage.

AI-Generated: codex/claude-opus 4.7 (xhigh)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks
  the work into a reviewable series, and this patch is one step of it.
---
 tests/docs/authoring.md | 124 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)
 create mode 100644 tests/docs/authoring.md

On Wed, 2026-07-01 at 03:40 -0400, Trevor Woerner via
lists.yoctoproject.org wrote:
> A suite is only as good as the tests added to it over time, so it needs
> a guide that says where a new test goes, what one looks like, and how
> to choose its inputs and assertions. This commit adds
> tests/docs/authoring.md.
> 
> The guide covers:
> 
>   - where a test goes: one test_<area>.py per wic module under
>     tests/unit/, extending an existing file or starting a new one;
> 
>   - the shape of a test: a small worked example, a preference for
>     parametrize over in-test loops so each input/output pair is its own
>     named case, and use of the tmp_path fixture for scratch files;
> 
>   - asserting the correct behaviour rather than whatever wic currently
>     does, so a test never bakes in a bug; when a test exposes a defect,
>     the source fix lands in the same change so the test passes;
> 
>   - probing the boundaries: a checklist of input classes worth covering
>     for numbers, strings, paths, types/shape, and state, each asserting
>     a specific value or exception rather than merely "it did not
>     crash";
> 
>   - keeping the assertion strong: never weaken an assertion to get
>     green;
> 
>   - how to run just the test you wrote, with and without coverage.
> 
> AI-Generated: codex/claude-opus 4.7 (xhigh)
> Signed-off-by: Trevor Woerner <twoerner@gmail.com>

Is this here to help human test authors, or to help Claude? A lot of it
seems very generic, e.g. suggesting which boundaries to test is not
specific to wic.

If it's to guide the AI agent, then say that explicitly and reference it
from a top-level AGENTS.md file. I'm not a big fan of this stuff
littering repositories, but if you're going to be using AI code
generation heavily then if may be needed.

> ---
> changes in v3:
> - no change in this revision.
> changes in v2:
> - v1 submitted the entire test suite as a single commit; v2 breaks
>   the work into a reviewable series, and this patch is one step of it.
> ---
>  tests/docs/authoring.md | 124 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 124 insertions(+)
>  create mode 100644 tests/docs/authoring.md
> 
> diff --git a/tests/docs/authoring.md b/tests/docs/authoring.md
> new file mode 100644
> index 000000000000..c60cceb2d32d
> --- /dev/null
> +++ b/tests/docs/authoring.md
> @@ -0,0 +1,124 @@
> +# Authoring a unit test
> +
> +Unit tests live under `tests/unit/`, one module per area of wic, named
> +`test_<area>.py`. They import the wic module under test directly and
> +run in-process, so they need no host tools and no build environment.
> +
> +## Contents
> +
> +- [Where a test goes](#where-a-test-goes)
> +- [Shape of a test](#shape-of-a-test)
> +- [Assert the correct behaviour](#assert-the-correct-behaviour)
> +- [Probe the boundaries](#probe-the-boundaries)
> +- [Keep the assertion strong](#keep-the-assertion-strong)
> +- [Running what you wrote](#running-what-you-wrote)
> +
> +## Where a test goes
> +
> +Group tests by the wic module they exercise, one `test_<area>.py` per
> +module. Start a new file when you begin covering a module that has no
> +file yet; otherwise extend the existing one.
> +
> +## Shape of a test
> +
> +```python
> +import pytest
> +
> +from wic.ksparser import sizetype
> +
> +
> +class TestSizetype:
> +    # sizetype(default, size_in_bytes=False) returns a parser f(arg);
> +    # with default "M", a bare number is read as mebibytes-in-KiB.
> +    def test_plain_value_uses_default_unit(self):
> +        parse = sizetype("M")
> +        assert parse("100") == 100 * 1024
> +
> +    @pytest.mark.parametrize("arg,expected", [
> +        ("1K", 1),
> +        ("1G", 1 * 1024 * 1024),
> +        ("0", 0),
> +    ])
> +    def test_suffixes(self, arg, expected):
> +        assert sizetype("M")(arg) == expected
> +```
> +
> +Prefer `@pytest.mark.parametrize` to express each input/output pair as
> +its own case rather than looping inside one test, so a single bad
> +value shows up as one named failure.
> +
> +Use `pytest`'s `tmp_path` fixture for any scratch files; a passing
> +test's scratch directory is removed automatically (see the retention
> +policy in `pyproject.toml`), and a failing one is kept for inspection.
> +
> +## Assert the correct behaviour
> +
> +Every assertion states the behaviour wic is *expected* to provide,
> +never the behaviour it currently happens to have. A test that locks in
> +a wrong result is worse than no test: it gives false confidence and it
> +turns red exactly when someone repairs the code.
> +
> +When a test exposes a defect, the fix to the wic source lands in the
> +same change as the test, so the test asserts the correct behaviour and
> +passes. Do not commit a test that asserts a known-wrong result.
> +
> +Wrong -- bakes in the bug:
> +
> +```python
> +# value lost its comma; asserting the broken output
> +assert result["file"] == "s3://bucket/a"
> +```
> +
> +Right -- asserts the correct output (and the fix lands with it):
> +
> +```python
> +assert result["file"] == "s3://bucket/a,b.img"
> +```
> +
> +## Probe the boundaries
> +
> +For a parameter, probe the boundary and just past it rather than only
> +the comfortable middle of its range. The classes worth covering:
> +
> +- numbers: empty, zero, negative, the maximum and one beyond it,
> +  non-power-of-two and non-multiple-of-1024/1000 values, primes,
> +  fractional values where an integer is assumed, `0` versus `0.0`
> +  versus `"0"`, off-by-one at a block or sector boundary, and very
> +  large magnitudes near `2**31` and `2**63`
> +- strings: the empty string, whitespace only, embedded spaces, tabs,
> +  newlines and CRLF endings, non-ASCII characters, shell
> +  metacharacters in a value that becomes part of a command line,
> +  leading-dash values that look like options, over-delimited or
> +  malformed forms (stray commas, doubled or missing separators), and
> +  very long values
> +- paths: traversal (`../`), doubled slashes, a trailing slash or its
> +  absence, `.` and `..` components, broken or looping symlinks, and a
> +  non-existent path
> +- types and shape: each wrong type the parameter might receive (a
> +  string where an integer is expected, a list where a string is
> +  expected, `None`, and so on), the wrong argument count, and
> +  duplicate keys or entries
> +- state: a second call that must not see stale cached data, a
> +  `cache=False` path that must evict, calling an operation twice, and
> +  finalising a half-constructed object
> +
> +Each case asserts a specific outcome -- an exact value, or a specific
> +exception -- rather than merely "it did not crash." A test that only
> +checks for the absence of an exception will pass against badly wrong
> +output.
> +
> +## Keep the assertion strong
> +
> +If a test is hard to make pass, the answer is in the code or in
> +understanding the correct behaviour, never in weakening the assertion
> +to something vague enough to pass. Loosening an assertion to get green
> +is how a suite quietly stops catching regressions.
> +
> +## Running what you wrote
> +
> +```bash
> +tests/run-tests.sh tests/unit/test_<area>.py -v
> +tests/run-tests.sh --coverage tests/unit/test_<area>.py
> +```
> +
> +A new test should leave the suite green, with zero failures.