diff mbox series

[wic,v2,7/9] tests/docs: add the test-authoring guide

Message ID 20260630160612.1005451-8-twoerner@gmail.com
State New
Headers show
Series tests: standalone test-suite framework plus the first unit test | expand

Commit Message

Trevor Woerner June 30, 2026, 4:06 p.m. UTC
A suite is only as good as the tests added to it over time, so it needs
a guide that says where a new test goes, what one looks like, and how
to choose its inputs and assertions. This commit adds
tests/docs/authoring.md.

The guide covers:

  - where a test goes: one test_<area>.py per wic module under
    tests/unit/, extending an existing file or starting a new one;

  - the shape of a test: a small worked example, a preference for
    parametrize over in-test loops so each input/output pair is its own
    named case, and use of the tmp_path fixture for scratch files;

  - asserting the correct behaviour rather than whatever wic currently
    does, so a test never bakes in a bug; when a test exposes a defect,
    the source fix lands in the same change so the test passes;

  - probing the boundaries: a checklist of input classes worth covering
    for numbers, strings, paths, types/shape, and state, each asserting
    a specific value or exception rather than merely "it did not
    crash";

  - keeping the assertion strong: never weaken an assertion to get
    green;

  - how to run just the test you wrote, with and without coverage.

AI-Generated: codex/claude-opus 4.7 (xhigh)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks
  the work into a reviewable series, and this patch is one step of it.
---
 tests/docs/authoring.md | 124 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)
 create mode 100644 tests/docs/authoring.md
diff mbox series

Patch

diff --git a/tests/docs/authoring.md b/tests/docs/authoring.md
new file mode 100644
index 000000000000..c60cceb2d32d
--- /dev/null
+++ b/tests/docs/authoring.md
@@ -0,0 +1,124 @@ 
+# Authoring a unit test
+
+Unit tests live under `tests/unit/`, one module per area of wic, named
+`test_<area>.py`. They import the wic module under test directly and
+run in-process, so they need no host tools and no build environment.
+
+## Contents
+
+- [Where a test goes](#where-a-test-goes)
+- [Shape of a test](#shape-of-a-test)
+- [Assert the correct behaviour](#assert-the-correct-behaviour)
+- [Probe the boundaries](#probe-the-boundaries)
+- [Keep the assertion strong](#keep-the-assertion-strong)
+- [Running what you wrote](#running-what-you-wrote)
+
+## Where a test goes
+
+Group tests by the wic module they exercise, one `test_<area>.py` per
+module. Start a new file when you begin covering a module that has no
+file yet; otherwise extend the existing one.
+
+## Shape of a test
+
+```python
+import pytest
+
+from wic.ksparser import sizetype
+
+
+class TestSizetype:
+    # sizetype(default, size_in_bytes=False) returns a parser f(arg);
+    # with default "M", a bare number is read as mebibytes-in-KiB.
+    def test_plain_value_uses_default_unit(self):
+        parse = sizetype("M")
+        assert parse("100") == 100 * 1024
+
+    @pytest.mark.parametrize("arg,expected", [
+        ("1K", 1),
+        ("1G", 1 * 1024 * 1024),
+        ("0", 0),
+    ])
+    def test_suffixes(self, arg, expected):
+        assert sizetype("M")(arg) == expected
+```
+
+Prefer `@pytest.mark.parametrize` to express each input/output pair as
+its own case rather than looping inside one test, so a single bad
+value shows up as one named failure.
+
+Use `pytest`'s `tmp_path` fixture for any scratch files; a passing
+test's scratch directory is removed automatically (see the retention
+policy in `pyproject.toml`), and a failing one is kept for inspection.
+
+## Assert the correct behaviour
+
+Every assertion states the behaviour wic is *expected* to provide,
+never the behaviour it currently happens to have. A test that locks in
+a wrong result is worse than no test: it gives false confidence and it
+turns red exactly when someone repairs the code.
+
+When a test exposes a defect, the fix to the wic source lands in the
+same change as the test, so the test asserts the correct behaviour and
+passes. Do not commit a test that asserts a known-wrong result.
+
+Wrong -- bakes in the bug:
+
+```python
+# value lost its comma; asserting the broken output
+assert result["file"] == "s3://bucket/a"
+```
+
+Right -- asserts the correct output (and the fix lands with it):
+
+```python
+assert result["file"] == "s3://bucket/a,b.img"
+```
+
+## Probe the boundaries
+
+For a parameter, probe the boundary and just past it rather than only
+the comfortable middle of its range. The classes worth covering:
+
+- numbers: empty, zero, negative, the maximum and one beyond it,
+  non-power-of-two and non-multiple-of-1024/1000 values, primes,
+  fractional values where an integer is assumed, `0` versus `0.0`
+  versus `"0"`, off-by-one at a block or sector boundary, and very
+  large magnitudes near `2**31` and `2**63`
+- strings: the empty string, whitespace only, embedded spaces, tabs,
+  newlines and CRLF endings, non-ASCII characters, shell
+  metacharacters in a value that becomes part of a command line,
+  leading-dash values that look like options, over-delimited or
+  malformed forms (stray commas, doubled or missing separators), and
+  very long values
+- paths: traversal (`../`), doubled slashes, a trailing slash or its
+  absence, `.` and `..` components, broken or looping symlinks, and a
+  non-existent path
+- types and shape: each wrong type the parameter might receive (a
+  string where an integer is expected, a list where a string is
+  expected, `None`, and so on), the wrong argument count, and
+  duplicate keys or entries
+- state: a second call that must not see stale cached data, a
+  `cache=False` path that must evict, calling an operation twice, and
+  finalising a half-constructed object
+
+Each case asserts a specific outcome -- an exact value, or a specific
+exception -- rather than merely "it did not crash." A test that only
+checks for the absence of an exception will pass against badly wrong
+output.
+
+## Keep the assertion strong
+
+If a test is hard to make pass, the answer is in the code or in
+understanding the correct behaviour, never in weakening the assertion
+to something vague enough to pass. Loosening an assertion to get green
+is how a suite quietly stops catching regressions.
+
+## Running what you wrote
+
+```bash
+tests/run-tests.sh tests/unit/test_<area>.py -v
+tests/run-tests.sh --coverage tests/unit/test_<area>.py
+```
+
+A new test should leave the suite green, with zero failures.