From patchwork Wed Jul 1 07:40:21 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D61FC43602 for ; Wed, 1 Jul 2026 07:40:41 +0000 (UTC) Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.39828.1782891637108077429 for ; Wed, 01 Jul 2026 00:40:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=YXjYQc/x; spf=pass (domain: gmail.com, ip: 209.85.160.181, mailfrom: twoerner@gmail.com) Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-51a8f5ef23bso2069171cf.2 for ; Wed, 01 Jul 2026 00:40:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891636; x=1783496436; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=wgA1KZBX54HVNPOyagtXsuIOHW3aatFGsduASI2pxTw=; b=YXjYQc/xJJSL2G2Y7//vH6T6V0sKRbalV8VU7ZX7ad1tmuxDNl6ZObtNlNQx8i8sWY 7U7OhhU10qsR5UOM92pb0KnO0eITq+W7GCZm2A8JG7exLu7sPmDOo8W5d4dN9Tunkm2Y JKIhmNyXbZW59kB4pz88N9lnghBBkJj+p0S6FP+gDwPFIlzkF7wqoKFy+c/sfZ8PXAVf w8IWEvWccngZxWHMIiHx9YMhXGKT+x8L3hIeA58dYa8qoorog8mTq2JW6Y35TjWmFFRF NfcLQmE+15YByAb/bNERGgfiHirF+FniEHj04zx0uHUOSqdXfrjDYpRJoO4HM1cBR3/x HERg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891636; x=1783496436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=wgA1KZBX54HVNPOyagtXsuIOHW3aatFGsduASI2pxTw=; b=Q/5Df43jRvnFFB7pCoA6HeJBXtlJ6gdelspIG006hx9aRv8NwP2bBX2lTir2o7EjJW Q02XvEcnTeo9/6nOZpz/77HLF6i4GSfoWwQ0oAknNX1myMowpPZ2vgQ2AWdvls5s+lQk RVnN+zORZ+gtHx4MFLEuXmEFWaXc6ysmTYcbJlaom1qUpMc93zGoBnf4mOAFcm8kRDiV M9GXtsYruYphJ5ND5SKYAS92563eEBMV2gwbwslKy49Ud4iuEdsHgLGybm6AOcbpKPGS PyjkrkfTZbUu69wB8T3vG0pgcMQknC+7QMMVECkEUVMuRuI5vp1HScn6FCAyG+1q4O6Z M78g== X-Gm-Message-State: AOJu0Yx7a1fBEGy5ezM/AHAFc13kM7pB2/K8CZbJ64STv4bxI/yNd9iD T8oZbBgLFtB7Y86HqRFMorUbZ7GFNbyvJHryzgGGeri9AOWu8SpMj+6ECtYo4Q== X-Gm-Gg: AfdE7cm3uzC6FSD5XgBQwbgV0Fz8lyrcVGicxZaFIxVyXLbFGndgYPvhFchMyRA0a1f Pjt4hIa+F/Oxz+F9yBHYMiymBZFYzRKzLBETgpl6DiQvBwga32nPVyZ9XB5YdMTMK5zkaYpKwkG CwZFUDoFQxHZPubYCgKYXPZk6FwZ3poQuaeGZm1QptpDMvQUcL2AqYzDewMAnDoBy0sPb3FlZV+ tzSsLCjOLnrc1O9sXPWaHQ8Oftr9CZjsjESKeN65i0U1W/CxdiHE8Olhwjt38hKvE65S+yekWDL ZzM93XatUNujl9i0Qwh/BtVSuY66PUxV+7/RTi18QnIi793xUMndbU4P2LZ/Zo+T5NEOUHjO3i4 K7C7yDLshKaC+2Su6AaW+UbMqSj09wx+NMV/9pJF2Uc4tyHvhGpOaMLvhN0YbwZbrtCLM7+JLa+ s+xlVirWUA3niCzxuN0q3lWhGupJcIvFEU8DRqBRMl6Vg+YtHN4UWyWaw= X-Received: by 2002:ac8:5a95:0:b0:51c:ebd:fb42 with SMTP id d75a77b69052e-51c26b20e48mr6086181cf.59.1782891635522; Wed, 01 Jul 2026 00:40:35 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:34 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 01/10] tests: add the standalone test-suite skeleton Date: Wed, 1 Jul 2026 03:40:21 -0400 Message-ID: <20260701074030.1090807-2-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:40:41 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4333 wic currently has no test mechanism of its own; it relies on the oe-selftest from oe-core for all its testing, which means a full bitbake build is needed to exercise even pure-Python logic. This commit lays the groundwork for a small standalone suite that runs from a plain checkout with nothing but pytest, so that logic can be pinned down and kept stable as the code evolves. This commit adds the skeleton only; it does not add any tests. What this adds: - pyproject.toml: a "tests" optional-dependency group (pytest only) so that "pip install -e .[tests]" pulls in what the suite needs, plus a [tool.pytest.ini_options] section. The pytest options keep a test's scratch directory only when that test fails, so repeated runs do not accumulate leftover files. - tests/unit/ and tests/docs/: the directories that hold the in-process unit suites and the suite documentation. Git cannot track an empty directory, so a placeholder .gitkeep is committed in each to create it. The .gitkeep files have no content and are removed once real files land in those directories. - .gitignore: ignore the .pytest_cache/ directory that pytest writes at the top of the checkout. - README.md: a Testing section pointing at the suite and at tests/docs/, and a tests/* entry in the project layout list. AI-Generated: codex/claude-opus 4.7 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - no change in this revision. changes in v2: - v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it. --- .gitignore | 3 +++ README.md | 14 ++++++++++++++ pyproject.toml | 12 ++++++++++++ tests/docs/.gitkeep | 0 tests/unit/.gitkeep | 0 5 files changed, 29 insertions(+) create mode 100644 tests/docs/.gitkeep create mode 100644 tests/unit/.gitkeep diff --git a/.gitignore b/.gitignore index eeb8a6ec4087..07992096c0fb 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,4 @@ **/__pycache__ + +# pytest cache +/.pytest_cache/ diff --git a/README.md b/README.md index 75229421763c..497ebb1de0f0 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,20 @@ environment file or folder (generated via `bitbake -c rootfs_wicenv - `src/wic/*`: core engine, plugins, and helpers. - `src/bb/*`: various bitbake helpers that were brought along and used in other parts of wic. - `src/oe/*`: various oe-core helpers that were brought along and used in other parts of wic. +- `tests/*`: the standalone test suite and its documentation (see Testing below). + +## Testing + +wic ships a standalone test suite under `tests/` that runs from a plain +checkout, with no bitbake and no OpenEmbedded build required. The test +extras pull in everything the suite needs: + +``` +pip install -e ".[tests]" +``` + +See [tests/docs/](tests/docs/) for how to run the suite and the +conventions it follows. ## Contributing diff --git a/pyproject.toml b/pyproject.toml index fdc1ce0f5ece..d660e39007c4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,6 +21,11 @@ classifiers = [ Homepage = "https://git.yoctoproject.org/wic" Repository = "https://git.yoctoproject.org/wic" +[project.optional-dependencies] +tests = [ + "pytest >= 7.0", +] + [project.scripts] wic = "wic.cli:main" @@ -36,3 +41,10 @@ build-backend = "hatchling.build" [tool.hatch.version] path = "src/wic/cli.py" + +[tool.pytest.ini_options] +# Keep a test's scratch directory only when it fails; a passing test's +# tmp_path is removed automatically so repeated runs do not accumulate +# leftover files under the pytest base temp directory. +tmp_path_retention_policy = "failed" +tmp_path_retention_count = 1 diff --git a/tests/docs/.gitkeep b/tests/docs/.gitkeep new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/unit/.gitkeep b/tests/unit/.gitkeep new file mode 100644 index 000000000000..e69de29bb2d1 From patchwork Wed Jul 1 07:40:22 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 824D6C44500 for ; Wed, 1 Jul 2026 07:40:41 +0000 (UTC) Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.39829.1782891638676099415 for ; Wed, 01 Jul 2026 00:40:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=s2BEWoHz; spf=pass (domain: gmail.com, ip: 209.85.222.173, mailfrom: twoerner@gmail.com) Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-915ab38ac14so33678685a.0 for ; Wed, 01 Jul 2026 00:40:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891637; x=1783496437; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=g+JbDIq7r4PS1tgWPnuiK0Rqjmn0yHxyhY8c27yUaxo=; b=s2BEWoHz0ebwEeLObYk5G+Jx1yX7+qsk+W7/ofLBs8F03fOK8jSE7amXRC5EML6I0x JsOakvmJgUnUhVVqQtMd/RTtQBALgVZ/9QDSxXpr8W9EcndQ2+4/mDzmVbxwsuppflki h4OtSw3SajvxAf12j1Q16Nz6eTbnzEDJslzmI1jVyQnK9zIUpDR+p+tkHiXR9twDfDJS WRzL3cSVCi/+v/F3hE0PzHh2fQEPv0jK4je7fOrRM3M7HUGCYn2Y/p17bdOFGedeG3hi N6Icvdejf0AYzXFYaHb89sF0+6LLGt+7bIHXaWKb1o4CiaN8MUAJkkC1Q4XrqUMO3b/N iFKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891637; x=1783496437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=g+JbDIq7r4PS1tgWPnuiK0Rqjmn0yHxyhY8c27yUaxo=; b=W3/foCccb+8GsP2oDchS1Wayhga+oYLxFtVsa7ScyjqJ3KZabMwBXgdBn3eZlyIUFd bUph3l/1n51kbPddkB3fbPgnLib9Q8463Ue2u8x4mTsqoAuxdSonZzoWQNaios5+4i1y uWv/0heZEZl5IGXNBw2d74qwX6raQjW1k0HVvUuq672HC7FUSjPp4kLbA2LVFSWzUBtY /dntPG1td8iRw5ibNWR6ZyPguTkKjFAHW647IxB+8za6V8ZQaNkMrPsviz/IS/Yu8qWR mdOkFJwd4xTaQHhIhn+RQ2G9Ym1aVESkCD7pPCjtCknNU67QG0Rfctioygz2qpUef3RL e8BQ== X-Gm-Message-State: AOJu0Yyqz5Q7UKtV2E6qr6kAxYzXbVW0bVTZd9JzBQShCEk3m9cP9vQw 4gm8CBkd+kdDSjPZNCKwPuxTy5DoTmPcA4Yat0rdPjCy1Ks5ou/7gsvIZVkv7w== X-Gm-Gg: AfdE7cnpqOQnccS2qd4BqCn1bJBWlXXSo2q9MS+b9baMU72LZKNxIVekpWgL9WbJ2s5 6F2Eby92Ore4sn8roRMIGTZTIBVJLIU4EOqDRt2Eu7rF0qGtjXIrYeTviN9SpilQn4Z3Z6ZcaKL YPalmKuZ4ixL3Uor+gYaIuFmudSYUbftvZRdjsgC5597X4Tx/ocu9pY5IH/W41YjNRbX6AGHLFT QXt9YUdk9+4Dn1EJZpCmpDfTcKOmEgoEGtqU0N6HU4oM1tztAHUWs0YsJf2/sH4ZJOrUhmxFSO2 2nF2jf6CS3zdjqzAzzhaCQWZganIz+CKjSPyJzWy9XiXNqCIRS0OCPj9CWWZOYugfQWEiRY7Gx0 eHcil55IK18VN9nom3IBy6azhpJ/R2KBNJsB1cb5XrWSqZW9jnnssovHcdkaM54z72bA0vSb3h4 M53V4dP8+Lwn494LAYruleTQfjMwgd4hjaXcfG6QPQeftk4cgcB8qcPNo= X-Received: by 2002:a05:620a:408c:b0:92e:6bbf:9672 with SMTP id af79cd13be357-92e784d12f8mr73882985a.46.1782891637225; Wed, 01 Jul 2026 00:40:37 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:35 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 02/10] tests: add a session banner via conftest.py Date: Wed, 1 Jul 2026 03:40:22 -0400 Message-ID: <20260701074030.1090807-3-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:40:41 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4334 When a test run starts it is useful to see, at a glance, exactly what is being tested: which host the run is on, which Python and pytest are in use, and -- most importantly -- which wic the suite imported and what version it reports. Without that, a passing or failing run is hard to attribute, especially when more than one wic checkout or virtualenv is in play. pytest looks for a file named conftest.py and, among other things, calls its pytest_report_header() hook to print extra lines in the session header before collection begins. This commit adds that file with a single hook that prints a short banner: - the host node name, operating system, and machine type; - the running Python version and the pytest version; - the version wic reports and the filesystem path the wic module was imported from. The wic lookup is defensive: if wic cannot be imported (for example the test extras were not installed, or the suite is being run outside the checkout) the banner says so rather than aborting the run, so the failure is visible in the header instead of as an opaque collection error. The file name conftest.py is mandatory; pytest discovers it by name, so it cannot be called anything else. AI-Generated: codex/claude-opus 4.7 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - no change in this revision. changes in v2: - v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it. --- tests/conftest.py | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 tests/conftest.py diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 000000000000..1c368ebf8595 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,38 @@ +# Session banner for the wic test suite. +# +# Before collection, print the environment the tests exercise: +# - the host +# - the Python and pytest versions +# - the wic under test (its version and the module path of its import) + +import platform + +import pytest + + +def _wic_under_test(): + """Return (version, module_path) for the wic being tested.""" + try: + import wic.cli as wic_cli + except Exception as exc: # pragma: no cover - reported in the banner + return ("(import failed: %s)" % exc, "(unimported)") + version = getattr(wic_cli, "__version__", "(unknown)") + module_path = getattr(wic_cli, "__file__", "(unknown)") + return (version, module_path) + + +def _format_banner(): + wic_version, wic_path = _wic_under_test() + lines = [ + "wic test suite", + " host: %s %s %s" % ( + platform.node(), platform.system(), platform.machine()), + " python: %s pytest: %s" % ( + platform.python_version(), pytest.__version__), + " wic: %s (%s)" % (wic_version, wic_path), + ] + return "\n".join(lines) + + +def pytest_report_header(config): + return _format_banner() From patchwork Wed Jul 1 07:40:23 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91464 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93976C43327 for ; Wed, 1 Jul 2026 07:40:41 +0000 (UTC) Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.39523.1782891639846996442 for ; Wed, 01 Jul 2026 00:40:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=lrqdqqLq; spf=pass (domain: gmail.com, ip: 209.85.219.50, mailfrom: twoerner@gmail.com) Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-8f032b47e3cso2540346d6.0 for ; Wed, 01 Jul 2026 00:40:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891639; x=1783496439; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=l8AwUyqD3YHDjcfObBhFvOsCWa4Eh1L7njz67SGuOEQ=; b=lrqdqqLq+tlM/yJbLzL6OK1bdoMTfbyUaArMInll3z3yA1MHfgZETvwFxKFXLZJ5qJ dMFOd/C2raFd/4sPPBttGbc31s7Q63uUoCaBX+xqSB+7Vv15/1lkpe5gxEfLg+hkDGpq 8xGjr6LdeUTHVxVPk42aqtNMBVtMhnQwV13Eb3nrnFQCln5x6VfZALd6HTn8pkHnQKrr JyTXErtFymenmsz02EWEH39Gx/jNGd+izHSSNDAquyziAuwF4mDAippkZadBcJpUR9DX hHInQMplFScn50hQ7Kj3PWHD09fc4uq2TGZmch6BrCXBD3oWNIyZR1fYehigRCb9UM+I SCPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891639; x=1783496439; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=l8AwUyqD3YHDjcfObBhFvOsCWa4Eh1L7njz67SGuOEQ=; b=dDUP1DB1TYUV2+6Eq5waF7Cq/a5CBNWaVuQUPg+oa4ncswzhNB6bhrxaCwvP5GBWsd OtLlo/TmayyMxZpN758MatfEEJ/X32Nw7bovvqNBzA0w3N3vpj6Vt3ZRnoxb5jioHlfq OD6cOz/e8QIC+6Emr3xUYr9NjWzYLP5sS3kU3k1sqy94W5OKx87Ifae0iGMYC5rjj8fd p/bD2S3pBINRh/etDjizqeEFlxtVjHN412Inab/BeR+yUyP7LD3C+xUL0EQhtiMDP/mO GU1kL7aHnbRQSlk8sP9mMCsB7HT99lz4eEN5THjxHCzj5fjD1EPmFK7YK9RchL9c1qXg 3DZA== X-Gm-Message-State: AOJu0Yz16WzqqUmW/32Whvfp6+7ayGHizBN12QDmUYdjmCLoCuHtoqyA gZFAFXQVJSisNX9t7impbcz+Prm1M34EeCYc5L8HIr+jlVbwo6MaYu109ITgMg== X-Gm-Gg: AfdE7cnFF24JMnDGmToIdsAxD4CwEwtzM08b8pBc/eVQQcLC7OYl7uFPWExQYurS33x 27osGiD+d47cxBABNLNItLyQL4qcd7WN1RxJxZsvn4KaXKKWqzoS2Ey5wwgZcqCHYrbsfygrCUp +KelRuLn3OYzsvYg5PYGif7koWn55pGWxWlUfjagC5EAThmFdHfpiGbW/is4GYL33an4Te1+D21 u+PZNeHXhJjEWOJpes069rGYqBxZ0fCkDXyL8WdQyIclqA/+NKKLdtdNc2EfklCCnRtmOPuDk0A XX/Vi6/HkYY6eL2zhlgpakV1MsJOQpGBvXH6HXhrmhSMjcrxlGSNdXLgiuX6Op2xklxg62vFIF/ LXN+cgIU4f6XYNbYg2NQzEep2RtgxeGI91huZwuFMDaktq8GSWx8kyEIC4sjdHt+5l1zM6IHbaM dpnE8H4NQlTwKvUCjUmJo5aTpm88R4Jfl1VGdJcfE+VUOoqt/pNS53xgw= X-Received: by 2002:a05:6214:2604:b0:8f1:c359:43cf with SMTP id 6a1803df08f44-8f3c8ff7be5mr4987706d6.39.1782891638684; Wed, 01 Jul 2026 00:40:38 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:37 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 03/10] tests: add the run-tests.sh wrapper Date: Wed, 1 Jul 2026 03:40:23 -0400 Message-ID: <20260701074030.1090807-4-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:40:41 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4335 The suite is run with pytest, but invoking pytest directly has two sharp edges: it must be run from the right directory for the pyproject configuration to apply, and it happily runs against whatever Python happens to be first on PATH, even one that cannot import wic. A run against the wrong interpreter produces a misleading banner and, worse, can silently exercise a different wic than the one in the checkout. This commit adds tests/run-tests.sh, a thin wrapper that removes both hazards: - it locates the repository root from its own path and changes there before running, so it works regardless of the caller's directory; - it checks that the interpreter can import wic before handing off to pytest, and fails loudly with the install command if it cannot; - with no arguments it runs the whole suite under tests/; any argument it does not recognise (a path, -k EXPR, -v, ...) is passed straight through to pytest, and an explicit -- forwards everything after it. The interpreter can be overridden with the PYTHON environment variable (default python3). -h/--help prints usage and exits. The README Testing section gains the run-tests.sh invocation alongside the install step. AI-Generated: codex/claude-opus 4.7 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - no change in this revision. changes in v2: - v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it. --- README.md | 1 + tests/run-tests.sh | 75 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100755 tests/run-tests.sh diff --git a/README.md b/README.md index 497ebb1de0f0..e557eb435316 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,7 @@ extras pull in everything the suite needs: ``` pip install -e ".[tests]" +tests/run-tests.sh ``` See [tests/docs/](tests/docs/) for how to run the suite and the diff --git a/tests/run-tests.sh b/tests/run-tests.sh new file mode 100755 index 000000000000..9db6d50338d6 --- /dev/null +++ b/tests/run-tests.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# +# Run the wic test suite. +# +# Thin wrapper around pytest that works from anywhere in the checkout: +# it locates the repository root, makes sure the interpreter can import +# wic, and then hands off to pytest. +# +# Needs the test extras: pip install -e ".[tests]" + +set -euo pipefail + +usage() { + cat <<'USAGE' +Usage: + tests/run-tests.sh [pytest args] + +Options: + -h, --help show this help and exit + +Anything else is passed straight through to pytest (for example a +path, -k EXPR, or -v). With no such argument the whole suite under +tests/ is run. + +Examples: + tests/run-tests.sh # whole suite + tests/run-tests.sh -k filemap -v # pass args through to pytest + tests/run-tests.sh tests/unit # a single tier or file + +Needs the test extras: pip install -e ".[tests]" +USAGE +} + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +PY="${PYTHON:-python3}" + +pytest_args=() +while [ $# -gt 0 ]; do + case "$1" in + -h|--help) + usage + exit 0 + ;; + --) + shift + pytest_args+=("$@") + break + ;; + *) + pytest_args+=("$1") + ;; + esac + shift +done + +cd "$REPO_ROOT" + +# Make sure the interpreter that will run pytest can actually import wic. +# The suites pass even without an install (each adds src/ to sys.path), +# but the session banner and any install-dependent behaviour would be +# wrong, so fail loudly instead of running against the wrong interpreter. +if ! "$PY" -c "import wic" >/dev/null 2>&1; then + echo "error: '$PY' cannot import wic." >&2 + echo " install wic with its test extras:" >&2 + echo " $PY -m pip install -e \".[tests]\"" >&2 + exit 1 +fi + +# Default target: the whole suite. Overridden if the caller passed a +# path or -k/-m selector to pytest. +if [ ${#pytest_args[@]} -eq 0 ]; then + pytest_args=("tests") +fi + +exec "$PY" -m pytest "${pytest_args[@]}" From patchwork Wed Jul 1 07:40:24 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91466 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88D96C43458 for ; Wed, 1 Jul 2026 07:40:51 +0000 (UTC) Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.39524.1782891643629284491 for ; Wed, 01 Jul 2026 00:40:43 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=DhBr/udh; spf=pass (domain: gmail.com, ip: 209.85.222.175, mailfrom: twoerner@gmail.com) Received: by mail-qk1-f175.google.com with SMTP id af79cd13be357-92e622cc874so19018285a.0 for ; Wed, 01 Jul 2026 00:40:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891642; x=1783496442; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=/mQcRXUgEI8W4D4U44K3nCuFCRFgIDOVPPX/UhC27ik=; b=DhBr/udhOqi4LhHSlzPfA4PF5E5OATq65k+8gPBK7EI2OYePPb4YVgkoiEIyQ4erfi dgMMXbd7FoEhoxpNDrkTc9+ypKy8vN9admFkQnc8c6x9HUb/Y1KbP0wt6pPvRUemicIw 8bjqmLpo86q6iCo4Fgpw+nvcvYgvplhTup3l53SLan9tGYXeaDvY5GX17QwYX0RIu4oE su48V2bD+7e8raNBxGFJwGUO1beWPLUp2TnX+fm8Shq53tKP+IZVIpdwwnx4cEVVBXNC 8MfQwkBS7vpJEtFqQAtZyS0mgxQ+WkR7imTWnCXYwXAIFoaQxSiWItZW52wqRXRa6kqn 68+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891642; x=1783496442; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=/mQcRXUgEI8W4D4U44K3nCuFCRFgIDOVPPX/UhC27ik=; b=dtQ9WPP7gwwwAa59tsLjO6I2cFNwpClt9QR0qG+lPnGIkq3rLEIZBjuJc403SzAZIY V6JzQmlhaIQ/KumrzmWtBZPVOgVNDKPNBYplLw+t+bmmhfy0RBI2i36gkRBbZxuLjYwz dM6zOiS20M9YFot3aXw2NKEPqI9n/7/UzPMlSlJ0SLWMPi2gVjESnhMUBOqwCNpCyUFs 7zvraDx1eRukv5N7gglZedoe7Qp31eZRC3sKMW5p2QVqlgEedX9LxkxUCm+TAr2+ULYq 0uTNZuU4mtUtggnYJX/pYga2Oom9XrUNPY/euiZBTAv+/jsRJdAcQJGpA2g8sp/caV9g h2Iw== X-Gm-Message-State: AOJu0YwjGdM75dtIvOuC3wOpjVNnMUzGO4U/UqPRIfKuhjTlsMrHbHXA QIeR6UynJkBMaMF7otR4mVj2Mhpvrv0rvFG9NWGsQG+E9vERgGe1iHcWkhxdDg== X-Gm-Gg: AfdE7cm+yd8dz1D3Nq/WI9jQzJOTWsdgYGsA5Vi/pnuyVEoGT8tJxUibnagDKkttE1q PItGWy8tyH3JNEHpw085QgEI77pTxk401exgGsd9lw+SItH+YjehFIGmJzzmX0m+qDgH4VEHfVi PBh0HZCDeNIu8HJKmGXMmPHE29SUkkjt8x7JPxEUNkYDCmMU79m/9B60fEpkOSSuzixPaWwtpLR y2C8/yKmdo0MwOwJHtu1RmPM8WxVJPlnn6/4QO1hUKA/me7AsliMd6Yu09e5FgEariRHaNpWvoU r11xyzh92tXXws32Bhe51yCBsefWV+Z8tdqdKcxHRFF5bmtKhG7dIa+JAAaXo2bsG539W3mOz5L Di3Q//XF1Iuq6tMy9sTJbGLmY7YHr0EvgXIXvxmHaJAe/tySuqKTbHGtQcddy/4GWb94c05ny9X VuP6Xxryvg8c/F06MoQjKmEL00773gaFZUqX3cz4QI/w0eAIoqSO2brC0= X-Received: by 2002:a05:620a:4451:b0:92d:d721:3fd1 with SMTP id af79cd13be357-92e78534bdamr68029885a.74.1782891642326; Wed, 01 Jul 2026 00:40:42 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:39 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 04/10] tests: add optional coverage reporting to run-tests.sh Date: Wed, 1 Jul 2026 03:40:24 -0400 Message-ID: <20260701074030.1090807-5-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:40:51 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4336 Knowing which lines a test run actually exercised is the difference between "the suite is green" and "the suite is green and we know what it touched". This commit wires branch coverage of the wic source into the runner, kept entirely opt-in so a plain run stays fast and quiet. pyproject.toml gains coverage and pytest-cov in the tests extra, so "pip install -e .[tests]" pulls in what the new flags need. run-tests.sh gains two options: - --coverage measures branch coverage of src/wic during the run and prints a terminal report listing the lines that were missed; - --html [DIR] additionally writes a browsable HTML report (default htmlcov/, or DIR if given) and implies --coverage. If --coverage is requested but pytest-cov is not installed the runner fails loudly with the install command rather than running without the measurement it was asked for. The coverage data file and the HTML report directory are build artifacts, so .gitignore learns to ignore .coverage and htmlcov/. AI-Generated: codex/claude-opus 4.7 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - no change in this revision. changes in v2: - v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it. --- .gitignore | 4 ++++ pyproject.toml | 2 ++ tests/run-tests.sh | 39 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 44 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 07992096c0fb..534c49538091 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,7 @@ # pytest cache /.pytest_cache/ + +# coverage data and reports +/.coverage +/htmlcov/ diff --git a/pyproject.toml b/pyproject.toml index d660e39007c4..ece2757bb686 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -24,6 +24,8 @@ Repository = "https://git.yoctoproject.org/wic" [project.optional-dependencies] tests = [ "pytest >= 7.0", + "coverage >= 7.0", + "pytest-cov >= 4.0", ] [project.scripts] diff --git a/tests/run-tests.sh b/tests/run-tests.sh index 9db6d50338d6..a483da6a63a6 100755 --- a/tests/run-tests.sh +++ b/tests/run-tests.sh @@ -13,9 +13,13 @@ set -euo pipefail usage() { cat <<'USAGE' Usage: - tests/run-tests.sh [pytest args] + tests/run-tests.sh [--coverage] [--html [DIR]] [pytest args] Options: + --coverage also measure branch coverage of src/wic and print a + terminal report listing the lines that were missed + --html [DIR] also write an HTML coverage report (default dir: + htmlcov/); implies --coverage -h, --help show this help and exit Anything else is passed straight through to pytest (for example a @@ -24,6 +28,9 @@ tests/ is run. Examples: tests/run-tests.sh # whole suite + tests/run-tests.sh --coverage # + terminal coverage report + tests/run-tests.sh --html # + HTML report in htmlcov/ + tests/run-tests.sh --html /tmp/cov # + HTML report in /tmp/cov tests/run-tests.sh -k filemap -v # pass args through to pytest tests/run-tests.sh tests/unit # a single tier or file @@ -34,9 +41,25 @@ USAGE REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" PY="${PYTHON:-python3}" +coverage=0 +html=0 +html_dir="htmlcov" pytest_args=() while [ $# -gt 0 ]; do case "$1" in + --coverage) + coverage=1 + ;; + --html) + coverage=1 + html=1 + # Optional directory argument: consume the next token only if + # it is not another option or the pytest pass-through marker. + case "${2:-}" in + ""|-*|--) ;; + *) html_dir="$2"; shift ;; + esac + ;; -h|--help) usage exit 0 @@ -72,4 +95,18 @@ if [ ${#pytest_args[@]} -eq 0 ]; then pytest_args=("tests") fi +if [ "$coverage" -eq 1 ]; then + if ! "$PY" -c "import pytest_cov" >/dev/null 2>&1; then + echo "error: coverage requested but pytest-cov is not installed." >&2 + echo " run: $PY -m pip install -e \".[tests]\"" >&2 + exit 1 + fi + cov_args=(--cov=wic --cov-branch --cov-report=term-missing) + if [ "$html" -eq 1 ]; then + cov_args+=(--cov-report="html:${html_dir}") + echo "HTML coverage report: ${html_dir}/index.html" >&2 + fi + exec "$PY" -m pytest "${pytest_args[@]}" "${cov_args[@]}" +fi + exec "$PY" -m pytest "${pytest_args[@]}" From patchwork Wed Jul 1 07:40:25 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9963AC43602 for ; Wed, 1 Jul 2026 07:40:51 +0000 (UTC) Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.39525.1782891645088313295 for ; Wed, 01 Jul 2026 00:40:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=qe4zevRs; spf=pass (domain: gmail.com, ip: 209.85.219.53, mailfrom: twoerner@gmail.com) Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-8eefd4a8057so4494746d6.0 for ; Wed, 01 Jul 2026 00:40:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891644; x=1783496444; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=FNUNHVfuGVDIE7QbmCHde7NVjnWrHF198jUhaGuOORo=; b=qe4zevRsp3Oph99HdqsVA2k5pbGgMYIFOJ2DGy1g1Kn0qjv6NKoFs+9bp37cPGPpY4 UEEvLOFI6NEMP+2/RZ1ftFiq0X1LjRIW5jxoPNFwZQfzeuYRZYr++qZdLYyzcv/dSj0I IXANtAKLEmoRYMYhRvXihJnWXgQKt/0rzo4xvG63Oi4HKYn0QnBWVNIvY/JdwJqofWrE GGdpBYxIRWfJ/H93iT16pw43OOUZCg8JNZOt2rTJFdESyHEVKQE25y55dEjlRJM8csWb C+0RxrxFirhNEC3dz4EpN5zS8y9sEoVhn40iDjOa3DXZ2k8kMXPpZIcMFBi+wdNru71Z OVMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891644; x=1783496444; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=FNUNHVfuGVDIE7QbmCHde7NVjnWrHF198jUhaGuOORo=; b=sOaa1SPMDIOJWdOwPCLXU2ibByHlsOUHp2np7zU4hw3vp3oKnEf7cLXBWw3zfvcEsz 7V2aIEQUUlX4W4IM7UKiI3SvsTlfiYC+gLbyZYXHqNyh1I+YtyCz99DGgCEexXvUHnx5 upSwrGZHcnI+aIvi2/Jw/WchvTVZi+snfOIJLePR4KUYV2S2VIo9RG4JxmVtcxAuw6FZ 8Mfol/8xPbxqFLfpHzKfP7hvuNMXYwN90emX8tCnEDX8Wgm/9wsyBAjdPtNQbh7tDoIk 887CVfs5E/ebStfdUd6qWVStOklY/LFlYin/vcGmaOeONAvys8Iwz3+/dRy5cns4728G vmIA== X-Gm-Message-State: AOJu0YzdviUPvjRJNjOrwXZQASOmlR5uHi8DMqrBfFIUi416E8Aer9mF z0Vx2iVYNYO+RDi4XTgjA9MjLVidmbgJEqOze/RI3uxzGsuZBJojIY7KRJnwSw== X-Gm-Gg: AfdE7cnN5NjjMf9JAjKxRR2KTfpw21A3dBfEAfiO7ArTWSVZM1uANwMMZJVvLhCkALT AOs3NCu1vblBcDn0DQNAS/BGf+ZGrBxQP2sYrtAovzQEIuad16pk/yJn2NHigrQijpoU1KCcGjP 8+KFg4zfaht/LU+fwiC7SWtoL/bK9L4U/c6LMZ+4p/9NcqfBUIEEbQTnf7xkdtYtIxsLopzz6WJ QElW8fyruzGanzodtjNczGPOJAsaWLwZOBetIEA4dkOMneVsichFHWH9PwrMpb87I7h6xuIhmk7 BYwSHP7Nt5u377tkUMAMwFUk0MZZuj/PYbz+fuz5UMz6f6Ka7b2YVH725kWW9SZW7ZYwqecXb3Z ZU5sRTNqfelbIo88yq79wUshcJZTNvPudWoUerdOL2eVw/YQUyJ+9oFN4CAWsJdWq3/MdjwFqtB NYTSRJZaJ0ef5CnWuxC5QjbwbqTOjS8d03kkeNXezR8z/YqsZqiFpm7WQ= X-Received: by 2002:a05:6214:2dca:b0:8e4:d2ba:d677 with SMTP id 6a1803df08f44-8f3c7e708demr5879746d6.29.1782891643663; Wed, 01 Jul 2026 00:40:43 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:42 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 05/10] tests: add ruff linting to run-tests.sh Date: Wed, 1 Jul 2026 03:40:25 -0400 Message-ID: <20260701074030.1090807-6-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:40:51 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4337 A test suite is only trustworthy if its own code is clean, so this commit brings ruff into the runner and holds the test tree to a clean bar. It also makes it easy to preview what ruff thinks of the wic source, without yet enforcing it. pyproject.toml gains ruff in the tests extra and a [tool.ruff] section. The configuration is deliberately minimal for now: the test suite is the only tree under an enforced clean bar; the wic source under src/ is not yet ruff-clean and is reported, not gated. run-tests.sh gains two lint modes, each used on its own: - --lint-tests runs ruff over tests/ and exits. The test suite must report nothing; a finding here is a bug in our own test code and is expected to be fixed. - --lint-src runs ruff over src/ and exits. The source is not yet ruff-clean, so this is a preview: the runner prints ruff's findings and exits with its status, but nothing in the suite asserts on them. Keeping the two trees on separate flags means cleaning up the source later does not disturb the test-tree gate. A lint mode cannot be combined with coverage, with the other lint mode, or with pytest arguments; the runner rejects such combinations loudly rather than silently dropping the extras. If ruff is not installed it fails with the install command. tests/docs/linting.md documents the two modes and why src/ is held back for now. That file replaces the tests/docs/.gitkeep placeholder, which is no longer needed now that the directory has real content. .gitignore learns to ignore ruff's .ruff_cache/ directory. AI-Generated: codex/claude-opus 4.7 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - no change in this revision. changes in v2: - v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it. --- .gitignore | 3 +++ pyproject.toml | 8 ++++++++ tests/docs/.gitkeep | 0 tests/docs/linting.md | 39 +++++++++++++++++++++++++++++++++++ tests/run-tests.sh | 47 +++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 97 insertions(+) delete mode 100644 tests/docs/.gitkeep create mode 100644 tests/docs/linting.md diff --git a/.gitignore b/.gitignore index 534c49538091..3c3cfb328fb0 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,6 @@ # coverage data and reports /.coverage /htmlcov/ + +# ruff cache +/.ruff_cache/ diff --git a/pyproject.toml b/pyproject.toml index ece2757bb686..656adcd4930a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -26,6 +26,7 @@ tests = [ "pytest >= 7.0", "coverage >= 7.0", "pytest-cov >= 4.0", + "ruff >= 0.5", ] [project.scripts] @@ -50,3 +51,10 @@ path = "src/wic/cli.py" # leftover files under the pytest base temp directory. tmp_path_retention_policy = "failed" tmp_path_retention_count = 1 + +[tool.ruff] +# For now only the test suite is actually linted (run-tests.sh +# --lint-tests passes the tests/ path); the wic source under src/ is +# not yet ruff-clean and is left out until its findings are fixed (see +# tests/docs/linting.md). --lint-src can still be run to preview the +# source findings, but it is reported, not enforced. diff --git a/tests/docs/.gitkeep b/tests/docs/.gitkeep deleted file mode 100644 index e69de29bb2d1..000000000000 diff --git a/tests/docs/linting.md b/tests/docs/linting.md new file mode 100644 index 000000000000..71b4de21c100 --- /dev/null +++ b/tests/docs/linting.md @@ -0,0 +1,39 @@ +# Linting + +## Contents + +- [Running the linter](#running-the-linter) +- [tests/ must be clean](#tests-must-be-clean) +- [src/ is not linted yet](#src-is-not-linted-yet) + +The test suite is linted with [ruff](https://docs.astral.sh/ruff/). It +is configured in `pyproject.toml` (`[tool.ruff]`). + +## Running the linter + +The runner exposes ruff through two separate modes, each used on its +own: + +```bash +tests/run-tests.sh --lint-tests # ruff over tests/ +tests/run-tests.sh --lint-src # ruff over src/ (preview only) +``` + +A lint mode cannot be combined with coverage, with the other lint +mode, or with pytest arguments; the runner rejects such combinations. + +## tests/ must be clean + +Our own test code is held to a clean bar: `tests/run-tests.sh +--lint-tests` reports nothing. If you add a test that trips a rule, fix +the test before the change lands. + +## src/ is not linted yet + +`--lint-src` runs ruff over the wic source, but the source is **not** +yet ruff-clean, so its findings are a preview report rather than a +gate: the runner prints them and exits with ruff's status, but nothing +in the suite asserts on them. Treating `src/` findings as a hard +failure now would block every run on fixes that have not landed. Once +the source is cleaned up, `src/` can be promoted to the same clean bar +as `tests/`. diff --git a/tests/run-tests.sh b/tests/run-tests.sh index a483da6a63a6..085dcc93f91d 100755 --- a/tests/run-tests.sh +++ b/tests/run-tests.sh @@ -14,23 +14,34 @@ usage() { cat <<'USAGE' Usage: tests/run-tests.sh [--coverage] [--html [DIR]] [pytest args] + tests/run-tests.sh --lint-tests + tests/run-tests.sh --lint-src Options: --coverage also measure branch coverage of src/wic and print a terminal report listing the lines that were missed --html [DIR] also write an HTML coverage report (default dir: htmlcov/); implies --coverage + --lint-tests run ruff over tests/ and exit; the test suite is held + to a clean bar, so this must report nothing + --lint-src run ruff over src/ and exit; src/ is not yet ruff-clean, + so this is a preview report and is not enforced -h, --help show this help and exit Anything else is passed straight through to pytest (for example a path, -k EXPR, or -v). With no such argument the whole suite under tests/ is run. +The two lint modes each run on their own; they cannot be combined with +coverage, with each other, or with pytest arguments. + Examples: tests/run-tests.sh # whole suite tests/run-tests.sh --coverage # + terminal coverage report tests/run-tests.sh --html # + HTML report in htmlcov/ tests/run-tests.sh --html /tmp/cov # + HTML report in /tmp/cov + tests/run-tests.sh --lint-tests # ruff over tests/ + tests/run-tests.sh --lint-src # ruff over src/ (preview) tests/run-tests.sh -k filemap -v # pass args through to pytest tests/run-tests.sh tests/unit # a single tier or file @@ -44,6 +55,8 @@ PY="${PYTHON:-python3}" coverage=0 html=0 html_dir="htmlcov" +lint_tests=0 +lint_src=0 pytest_args=() while [ $# -gt 0 ]; do case "$1" in @@ -60,6 +73,12 @@ while [ $# -gt 0 ]; do *) html_dir="$2"; shift ;; esac ;; + --lint-tests) + lint_tests=1 + ;; + --lint-src) + lint_src=1 + ;; -h|--help) usage exit 0 @@ -78,6 +97,34 @@ done cd "$REPO_ROOT" +# The lint modes run ruff and exit, so each must be used on its own. +# Reject combining them with coverage, with each other, or with pytest +# arguments loudly instead of silently ignoring the extras. +if [ $((lint_tests + lint_src)) -gt 0 ]; then + if [ "$lint_tests" -eq 1 ] && [ "$lint_src" -eq 1 ]; then + echo "error: --lint-tests and --lint-src cannot be combined." >&2 + echo " run one lint mode at a time." >&2 + exit 2 + fi + if [ "$coverage" -eq 1 ] || [ "$html" -eq 1 ] || [ ${#pytest_args[@]} -gt 0 ]; then + echo "error: a lint mode must be used on its own." >&2 + echo " lint, or drop the lint flag to run the suite." >&2 + exit 2 + fi + if ! "$PY" -c "import ruff" >/dev/null 2>&1 && ! command -v ruff >/dev/null 2>&1; then + echo "error: lint requested but ruff is not installed." >&2 + echo " run: $PY -m pip install -e \".[tests]\"" >&2 + exit 1 + fi + if [ "$lint_tests" -eq 1 ]; then + # The test suite is held to a clean bar; a finding here is a bug + # in our own test code and must be fixed. + exec "$PY" -m ruff check tests + fi + # src/ is not yet ruff-clean; this is a preview report, not a gate. + exec "$PY" -m ruff check src +fi + # Make sure the interpreter that will run pytest can actually import wic. # The suites pass even without an install (each adds src/ to sys.path), # but the session banner and any install-dependent behaviour would be From patchwork Wed Jul 1 07:40:26 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6A65C43327 for ; Wed, 1 Jul 2026 07:40:51 +0000 (UTC) Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.39836.1782891646551409558 for ; Wed, 01 Jul 2026 00:40:46 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=UAAIVHwW; spf=pass (domain: gmail.com, ip: 209.85.160.175, mailfrom: twoerner@gmail.com) Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-51bfbe05683so1975581cf.2 for ; Wed, 01 Jul 2026 00:40:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891645; x=1783496445; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=sYpHsxnDd4qnOjxUN2Bdvy/1Rr9Jb8fbNiQPOPWe8ok=; b=UAAIVHwWZt9l/tClFqDig7y1HLIi3dJGaUlwsMBVgGnBJKhavDM8TJ10jxeHFz7qyq oKmTePI42mCCUg5Ljzws3McBlHT/qpZH3P7touDixUXb4tPEqpMmYtxLqksItR7+Lpz6 0B+gg6cKiaAQNdDekkmysmGXtSN/GmekH/21VCVWCytw8s9CWdiE/ntMh5+VJ1b1jTfW 5ybFgVA6jm04jIv4lazRgGGSx9uKrmHtzsiprpZt/g6fm3G2T/UGMPdLsKO9EDXAMrb+ PxBGElg4gDkecbXmVb7O8jiM7ieUmF9+6R9OlLDWYPM67+Y2HL/NRVotOEX1kdz0d5TV RonQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891645; x=1783496445; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=sYpHsxnDd4qnOjxUN2Bdvy/1Rr9Jb8fbNiQPOPWe8ok=; b=icUFkm0B4tU6QHbGFGQ0AUGT3jupCJuc8xi2jCFbQy4ujRVWT8zV4zXfUi1gpbO1cx +xJOkra6hGVOG9tWtHx19gxQPIzDhUOvhBEUGNpEsu3DkbrzlXpO2cATlMHWNBJAxfAv GOpBY8Xy7sSps8kGsgt98Y+EjoDI60E1TaMtZhN+gEjMjk8qvU5R8s193cLdGNNC7e9s ONQACnMTC87QbNiBRKd1qau1JQl9dg9AIM9GmTxrNPr4e/ewgDSxaFMq6m8XhgxeGuQz a6Av5s6RkcBBlCbcA9RjJ9Rn8BRGxc4M1POfpty+GE18RJyzI3raYMKtCxIxDm+rhYfW l9LQ== X-Gm-Message-State: AOJu0Yw05eHm2ouWT+iWhVm9cwYcuuyoG2PmWP+XLXNGSmFV5wfLqsnh XQ2DgbC2P7trARhZC98c42JiofonQVGozC9FQMxu9UUE/Zrvyz/xZOpgXBLjSA== X-Gm-Gg: AfdE7cn6S2ozC7H4MDTV+57nHQcECmVvyr85VbvCB6SEmAk8kd3QV79aSlsrojEjHIG W8oLsD1Ad48MuFJ9hI4Eo8FMOfvyT4FlSs6Wu+2eafFkR9rO0jrLlHGSTdlcKrllSanKZQdEMsi hclxXu2/PkFg7x+/PcgGixP86zn80LB9zHubVy0JH4FySnVE5LisBtvOguIFv5hx38pApQ3fu+/ +rBYh1UKf4vJu9ZfbunDpyZWUkbIxu6wa7Hqfm/DLPLY9AbTxHuqswt5UmOTYgdGBELHDxzexGY dye3C7uZFqfwOTFKoqV5UO/BpSZDG3Uk9e4yGUDbtPyCRqYfsFHk8YxcjBvuGNsl5yc9GcmhMUO VJbB0uwTPgv+6j0e/RD++AU3o4YitLN7MgKurzO/ZdkTmo/gVB0n8Nx0N43VWZ191faEmrwd9zy rsImF8voSZ/8bGsxiCVg/F0A3+oC1D6+j8Q4ZW1N1icywpzmJuM7JyDTvYiBEMCJ39yg== X-Received: by 2002:a05:622a:13c7:b0:51c:20:aea7 with SMTP id d75a77b69052e-51c26a54d4cmr6902621cf.9.1782891645293; Wed, 01 Jul 2026 00:40:45 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:44 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 06/10] tests/docs: add the suite overview README Date: Wed, 1 Jul 2026 03:40:26 -0400 Message-ID: <20260701074030.1090807-7-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:40:51 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4338 The test suite needs a front door: a short document that says what the suite is, how it is laid out, and how to run it. This commit adds tests/docs/README.md to be that overview. The README covers: - the layout of tests/, file by file; - that the suite is unit-only and needs no host tools or build environment, so it runs from a plain checkout; - how to install the test extras and invoke run-tests.sh, pointing at run-tests.sh --help as the authoritative, always-current list of options rather than duplicating them here where they would drift; - that anything run-tests.sh does not recognise is passed through to pytest; - a pointer to linting.md for how ruff is applied; - a small table of the other documents under tests/docs/. It deliberately does not enumerate every run-tests.sh option, since the runner is expected to grow more of them; the table and the --help output stay the source of truth. AI-Generated: codex/claude-opus 4.7 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - document the standard pytest scratch-directory mechanisms (TMPDIR and --basetemp) in the suite README, so there is no need for a custom temporary-directory variable. changes in v2: - v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it. --- tests/docs/README.md | 82 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 tests/docs/README.md diff --git a/tests/docs/README.md b/tests/docs/README.md new file mode 100644 index 000000000000..ba7bd5e192a3 --- /dev/null +++ b/tests/docs/README.md @@ -0,0 +1,82 @@ +# wic test suite + +A standalone test suite for the wic source tree. It runs from a plain +checkout with nothing but `pytest` -- no bitbake, no OpenEmbedded +build, and no target image -- so wic's logic can be exercised and kept +stable as the code changes. + +## Contents + +- [Layout](#layout) +- [Running](#running) +- [Linting](#linting) +- [Documentation](#documentation) + +## Layout + +``` +tests/ + conftest.py session banner describing the run + run-tests.sh wrapper for running the suite + unit/ unit tests that import wic modules directly + docs/ this documentation +``` + +The suite is unit-only: every test under `tests/unit/` imports a wic +module in-process and asserts on its behaviour, so none of it needs +host tools or a build environment. + +## Running + +Install wic with its test extras, then run the suite: + +```bash +pip install -e ".[tests]" +tests/run-tests.sh +``` + +`run-tests.sh` works from anywhere in the checkout. It can also report +branch coverage of the wic source; run `tests/run-tests.sh --help` for +the current list of options. + +Anything `run-tests.sh` does not recognise is handed straight to +pytest, so the whole pytest command line is available: + +```bash +tests/run-tests.sh -k filemap -v # one area, verbose +tests/run-tests.sh tests/unit # a single tier or file +``` + +## Scratch files + +Tests that need scratch space use pytest's `tmp_path` fixture, so there +is no wic-specific temporary-directory setting. Scratch directories are +created under pytest's base temporary directory and cleaned up according +to the retention policy in `pyproject.toml` (a passing test's directory +is removed, a failing one is kept). + +By default pytest roots that base directory at the system temporary +directory (usually `/tmp`). If that fills up, or you want the scratch +files somewhere else, use the standard pytest mechanisms rather than a +custom variable. Both are passed straight through by `run-tests.sh`: + +```bash +TMPDIR=/path/to/scratch tests/run-tests.sh # honoured by pytest +tests/run-tests.sh --basetemp=/path/to/scratch +``` + +`--basetemp` puts everything under the given directory and clears it at +the start of each run; `TMPDIR` keeps the last few sessions there. + +## Linting + +The test suite is linted with ruff and is held to a clean bar. See +[linting.md](linting.md) for the lint modes and how the source tree is +treated. + +## Documentation + +| File | Content | +|------|---------| +| [authoring.md](authoring.md) | How to add a unit test to the suite | +| [linting.md](linting.md) | How ruff is used on the suite | From patchwork Wed Jul 1 07:40:27 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEC09C44500 for ; Wed, 1 Jul 2026 07:40:51 +0000 (UTC) Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.39526.1782891647777707345 for ; Wed, 01 Jul 2026 00:40:47 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=Maaw2NDd; spf=pass (domain: gmail.com, ip: 209.85.219.46, mailfrom: twoerner@gmail.com) Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-8f29ec73064so2278686d6.1 for ; Wed, 01 Jul 2026 00:40:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891647; x=1783496447; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=DYdZzk0y49p9zkaYJLkHiICuaXzdqidNJRp2xn+NhJg=; b=Maaw2NDdDxSJ0dpXFxmWyTI/HmMK0IJb8YoCmBBCp5X6iWV8GLXiN06NYD373XTp/+ ep8lfy+AGQcgOvdkKJy6cwJ5CLCLImAQhCqzGmyR7/AgNJmL2ZYS9vg5aucWMHrKVCDL 3lCmiWz5p1p1Sbpv/F0PTlz2YckoQez4hH+Fq3lbc0Jc8LK3OozR5a5FMfeBVFrVTaKg 7sP3vesMBtFhwqiZr8C7m+CLX8G0tI3G/B4+BZSejhqWgNazjmSXWlDzLQb5VUy2hQT4 l9g0W6Qw89mA4K06QRfS5tcQTRgzLYD4HRD+JFvBguRZY1ApRoDScR21Y0ygh9ABAF1t nWxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891647; x=1783496447; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=DYdZzk0y49p9zkaYJLkHiICuaXzdqidNJRp2xn+NhJg=; b=aNv+cAdYQp6K3e1JO+TCXFNrMZBrELe3sIEwH7R2Wt9lN6X5+MWZ5EJXXOgYqGcjP6 gsYx5qwSsp7Bcj3xUMhhbWZo+DDUmDMjt9aepDLer90XVwUR/Du1w6lC59QYtTl3kmrV WWj8dO8O562cWaK9N9jUESeXOkMS7ujI/YpjMkRYBYogkG25Qu6wxoW+zwCZHRmvA8S7 xf0Nh9fqpAV/XLNXTuYj173hjp1xaDMqLLNiaEXeK+Fs9zRqQN3uI49LXJhfx6+xSM0s 3Ix/NZflA4pJNqBecSGpTwIHok+dV+XYUXKcZrYLMOorBMZptw5vX2DoNjx31Xq3jgjn fyOA== X-Gm-Message-State: AOJu0YxfFRXjSKLkcF2fi0PTXKjwxUf2sypV7tpYIHaaOz0HiuDNGTww 0UMSiFluBij0D5Q0xwFjiq2cHsDxiMLqJ26it55hzS3dvrT1xu9KQXCR21/Jkw== X-Gm-Gg: AfdE7cmSUYGS8VLP7EansAJDHoHzrbJkh9lIKhWrfEdHGL0Zm7dsyLrnwU9sNrmGCG7 ycQjgLFXpY/hT929w/d9dHzRUoD3h7fNj/5aj2aVmEW3Aas0sXumrXAOGFwmPwqT3nZyWNX6vPc DhMdT+EilFAwsAykl7UNcaoiI2tVkEe+/iXN9lklQ/6msdOJloG8RELVusHPCrRai0NcvqwpUaH CttmoBLoOl6rDw+tpbxdCKKGYQY+3+LudEfKOSwY07MaIq3fxZ55ghXdNe6C/QhiHNj6/YyPzo1 S3jUCXxRhM4PyCAyyCYtEK8CrPC9tMM1HQPo4VBLv/2jfwcY0L/HNWofQOMg5JgTPt3yClWwI5k etSYeUYeiXgiNGwTEhZ6Tix38lrabk33nW+6z6+Gwh9Y6U3xRwCJW8EqPi0/iQJfkwj1D56mr5E d4RbZGYSgZGXjRYLM7JSLr/vKlJcSkfIiEZoUI0LvIn5nIl+Hb1ejhnws= X-Received: by 2002:ad4:5fc9:0:b0:8ef:cfa:2a36 with SMTP id 6a1803df08f44-8f3c63db19emr6074936d6.9.1782891646570; Wed, 01 Jul 2026 00:40:46 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:45 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 07/10] tests/docs: add the test-authoring guide Date: Wed, 1 Jul 2026 03:40:27 -0400 Message-ID: <20260701074030.1090807-8-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:40:51 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4339 A suite is only as good as the tests added to it over time, so it needs a guide that says where a new test goes, what one looks like, and how to choose its inputs and assertions. This commit adds tests/docs/authoring.md. The guide covers: - where a test goes: one test_.py per wic module under tests/unit/, extending an existing file or starting a new one; - the shape of a test: a small worked example, a preference for parametrize over in-test loops so each input/output pair is its own named case, and use of the tmp_path fixture for scratch files; - asserting the correct behaviour rather than whatever wic currently does, so a test never bakes in a bug; when a test exposes a defect, the source fix lands in the same change so the test passes; - probing the boundaries: a checklist of input classes worth covering for numbers, strings, paths, types/shape, and state, each asserting a specific value or exception rather than merely "it did not crash"; - keeping the assertion strong: never weaken an assertion to get green; - how to run just the test you wrote, with and without coverage. AI-Generated: codex/claude-opus 4.7 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - no change in this revision. changes in v2: - v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it. --- tests/docs/authoring.md | 124 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 tests/docs/authoring.md diff --git a/tests/docs/authoring.md b/tests/docs/authoring.md new file mode 100644 index 000000000000..c60cceb2d32d --- /dev/null +++ b/tests/docs/authoring.md @@ -0,0 +1,124 @@ +# Authoring a unit test + +Unit tests live under `tests/unit/`, one module per area of wic, named +`test_.py`. They import the wic module under test directly and +run in-process, so they need no host tools and no build environment. + +## Contents + +- [Where a test goes](#where-a-test-goes) +- [Shape of a test](#shape-of-a-test) +- [Assert the correct behaviour](#assert-the-correct-behaviour) +- [Probe the boundaries](#probe-the-boundaries) +- [Keep the assertion strong](#keep-the-assertion-strong) +- [Running what you wrote](#running-what-you-wrote) + +## Where a test goes + +Group tests by the wic module they exercise, one `test_.py` per +module. Start a new file when you begin covering a module that has no +file yet; otherwise extend the existing one. + +## Shape of a test + +```python +import pytest + +from wic.ksparser import sizetype + + +class TestSizetype: + # sizetype(default, size_in_bytes=False) returns a parser f(arg); + # with default "M", a bare number is read as mebibytes-in-KiB. + def test_plain_value_uses_default_unit(self): + parse = sizetype("M") + assert parse("100") == 100 * 1024 + + @pytest.mark.parametrize("arg,expected", [ + ("1K", 1), + ("1G", 1 * 1024 * 1024), + ("0", 0), + ]) + def test_suffixes(self, arg, expected): + assert sizetype("M")(arg) == expected +``` + +Prefer `@pytest.mark.parametrize` to express each input/output pair as +its own case rather than looping inside one test, so a single bad +value shows up as one named failure. + +Use `pytest`'s `tmp_path` fixture for any scratch files; a passing +test's scratch directory is removed automatically (see the retention +policy in `pyproject.toml`), and a failing one is kept for inspection. + +## Assert the correct behaviour + +Every assertion states the behaviour wic is *expected* to provide, +never the behaviour it currently happens to have. A test that locks in +a wrong result is worse than no test: it gives false confidence and it +turns red exactly when someone repairs the code. + +When a test exposes a defect, the fix to the wic source lands in the +same change as the test, so the test asserts the correct behaviour and +passes. Do not commit a test that asserts a known-wrong result. + +Wrong -- bakes in the bug: + +```python +# value lost its comma; asserting the broken output +assert result["file"] == "s3://bucket/a" +``` + +Right -- asserts the correct output (and the fix lands with it): + +```python +assert result["file"] == "s3://bucket/a,b.img" +``` + +## Probe the boundaries + +For a parameter, probe the boundary and just past it rather than only +the comfortable middle of its range. The classes worth covering: + +- numbers: empty, zero, negative, the maximum and one beyond it, + non-power-of-two and non-multiple-of-1024/1000 values, primes, + fractional values where an integer is assumed, `0` versus `0.0` + versus `"0"`, off-by-one at a block or sector boundary, and very + large magnitudes near `2**31` and `2**63` +- strings: the empty string, whitespace only, embedded spaces, tabs, + newlines and CRLF endings, non-ASCII characters, shell + metacharacters in a value that becomes part of a command line, + leading-dash values that look like options, over-delimited or + malformed forms (stray commas, doubled or missing separators), and + very long values +- paths: traversal (`../`), doubled slashes, a trailing slash or its + absence, `.` and `..` components, broken or looping symlinks, and a + non-existent path +- types and shape: each wrong type the parameter might receive (a + string where an integer is expected, a list where a string is + expected, `None`, and so on), the wrong argument count, and + duplicate keys or entries +- state: a second call that must not see stale cached data, a + `cache=False` path that must evict, calling an operation twice, and + finalising a half-constructed object + +Each case asserts a specific outcome -- an exact value, or a specific +exception -- rather than merely "it did not crash." A test that only +checks for the absence of an exception will pass against badly wrong +output. + +## Keep the assertion strong + +If a test is hard to make pass, the answer is in the code or in +understanding the correct behaviour, never in weakening the assertion +to something vague enough to pass. Loosening an assertion to get green +is how a suite quietly stops catching regressions. + +## Running what you wrote + +```bash +tests/run-tests.sh tests/unit/test_.py -v +tests/run-tests.sh --coverage tests/unit/test_.py +``` + +A new test should leave the suite green, with zero failures. From patchwork Wed Jul 1 07:40:28 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E44FAC44502 for ; Wed, 1 Jul 2026 07:40:51 +0000 (UTC) Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.39837.1782891649718384700 for ; Wed, 01 Jul 2026 00:40:49 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=gfCrmOYn; spf=pass (domain: gmail.com, ip: 209.85.160.176, mailfrom: twoerner@gmail.com) Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-516d0db9372so2659121cf.2 for ; Wed, 01 Jul 2026 00:40:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891648; x=1783496448; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=q92M89GGIt1K9FOezygJ/AHJtuX7nVA7F7g4W6P8R/g=; b=gfCrmOYnGk6IxYG0JMy5Ai7rjglN/hah6TSWcsaeRaSc9ShPBo049KKsWLngj5J8Fu bwUq0OWS5I18OANzOO3kjE/q1vD66FSj/voulwhyh+xyvQGkEmlLB4bz4Hz/BFtM5QJ/ 2u1IqBeB2hM/0vvzDxsreoSlpm+gDOzZ2TRN2f60w1yxlhdqkTHOraoWLnUmc19lswCv ZQ/pVTFraeXpYUC0kDdXgcLEU44MR9KljY5wb+0vsJSAxqpm/KY0pQsbLgeF9gZUMTaK TUIDl3KHGgq+ccfEtF6Jspb1gNbz6kGILLQ1oEIDGL6YYFstZHDdYj/Zs/nO6Rirn9kI HsjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891648; x=1783496448; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=q92M89GGIt1K9FOezygJ/AHJtuX7nVA7F7g4W6P8R/g=; b=AkJm9mnsa+6UWv+sz6jncAxAHNOHZxrc6900/KUz0w30cslkJOo0dEA8EFmqzFrx7N xIfKAvVVQs3t7NQM+qqCkes5pyUfyMwGPRmVFQYLGcN6maueKkKAM7fkO2LEFMCbsW3P XfAzx8UIDQg2mS7FDCZLxuqvsRfvBXP7PPtvEhFUttLs0CtAIrEJi7rIagZIr7GKewQE GJtyNC3TZyteBDnafJv0KGr842jNJKolDEQbDttTMc1iBR04xvnms/I7mpoc6bVAuJkQ ug82FDkiTufuh8G0o8xZx4HkLON7O9PPIhdCXg9qAtaCQiLaQ7z/AmpQKN0qwqBdY5Rc Kx2w== X-Gm-Message-State: AOJu0Yx7uxqtQTDeGb5oAevO+vLqHM7+zchy5/kAfphyFU5GyrJnYyM7 WrZfnd5w1smzwMic43HE81s06KrWL+71fIcAX/PhTMnikP9tSizwFkkS4li0xw== X-Gm-Gg: AfdE7clzdGugzE9usmyfsI4On6tiFIibsCCv+QELLS5yiJe7IQ/XZG0B9LwAMKwvmLJ J7+tHMw2SnTJytXB7y4mJOjwQi6gK+0NtNhACbn+P26oJnv9hjR7aqGrFZSb4iB5PPtNjFDkGpo PCkmRQSvomWh5hkZCau5UT99Mk8GiSAn3B9rBNwsbueE/+69iehgTCG2yYauaZCBFAxqSU9zqDA Y7JLef+a4aTD9Trk1ukKEsiii4zxu/W4Ri0v09/g8TPub3ORcE+YQWcJUuuxbbDqQsYjxORIrmO HgOSYeKLUEDjQ2ZoED+UfZf7WM6r5I5/B3DfGP6QZu5FAfGN2kwnbN1i6VQAFh2Rzd9v5voVvt2 r9DDj0U03R3XRLOo1GEWho9zASajenWMTNwsdQLOLmra2HZpSzDKNb2a1+wdCFldOHta8Rdz9XN UPeI0E1Thgt1Uz6KYDcEjq1KbBaK/0nTfoEjFUOjQCkBjJLfilhfS7ILQ= X-Received: by 2002:ac8:7f81:0:b0:51c:ee2:fe75 with SMTP id d75a77b69052e-51c26b1d1dbmr5648211cf.47.1782891648549; Wed, 01 Jul 2026 00:40:48 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:46 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 08/10] tests: ignore E402 in the test tree for the sys.path bootstrap Date: Wed, 1 Jul 2026 03:40:28 -0400 Message-ID: <20260701074030.1090807-9-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:40:51 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4340 The unit tests that follow run against a plain checkout that has not been installed. To do that, each test module prepends the in-tree src/ directory to sys.path and only then imports wic: _SRC = Path(__file__).resolve().parent.parent.parent / "src" if str(_SRC) not in sys.path: sys.path.insert(0, str(_SRC)) from wic.bb.utils import mkdirhier That bootstrap necessarily runs before the wic imports, which trips ruff's E402 (module-level import not at top of file). The ordering is required, not accidental: the import would fail without the sys.path adjustment ahead of it. Rather than scatter per-line noqa comments through every test module, this commit relaxes E402 for the test tree once, via a [tool.ruff.lint.per-file-ignores] entry scoped to tests/**. It is the only rule relaxed for tests/; the suite is otherwise held to a clean bar by --lint-tests. tests/docs/linting.md gains a section describing this single exception and why it exists. AI-Generated: codex/claude-opus 4.7 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - no change in this revision. changes in v2: - v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it. --- pyproject.toml | 8 ++++++++ tests/docs/linting.md | 20 ++++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/pyproject.toml b/pyproject.toml index 656adcd4930a..fb42fe6dd135 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -58,3 +58,11 @@ tmp_path_retention_count = 1 # not yet ruff-clean and is left out until its findings are fixed (see # tests/docs/linting.md). --lint-src can still be run to preview the # source findings, but it is reported, not enforced. + +[tool.ruff.lint.per-file-ignores] +# Test modules prepend the in-tree src/ directory to sys.path before +# importing wic, so the suite runs against a checkout that has not been +# installed. That bootstrap necessarily runs before the wic imports, +# which trips E402 (module-level import not at top of file). The +# ordering is deliberate; ignore E402 for the test tree only. +"tests/**" = ["E402"] diff --git a/tests/docs/linting.md b/tests/docs/linting.md index 71b4de21c100..1c1bfcc06f82 100644 --- a/tests/docs/linting.md +++ b/tests/docs/linting.md @@ -4,6 +4,7 @@ - [Running the linter](#running-the-linter) - [tests/ must be clean](#tests-must-be-clean) +- [The one intentional exception in tests/](#the-one-intentional-exception-in-tests) - [src/ is not linted yet](#src-is-not-linted-yet) The test suite is linted with [ruff](https://docs.astral.sh/ruff/). It @@ -28,6 +29,25 @@ Our own test code is held to a clean bar: `tests/run-tests.sh --lint-tests` reports nothing. If you add a test that trips a rule, fix the test before the change lands. +## The one intentional exception in tests/ + +Test modules prepend the in-tree `src/` directory to `sys.path` before +importing `wic`, so the suite runs against a plain checkout that has not +been installed: + +```python +_SRC = Path(__file__).resolve().parent.parent.parent / "src" +if str(_SRC) not in sys.path: + sys.path.insert(0, str(_SRC)) + +from wic.bb.utils import mkdirhier +``` + +That bootstrap necessarily runs before the `wic` imports, which trips +`E402` (module-level import not at top of file). The ordering is +required, so `E402` is ignored for the test tree via `per-file-ignores` +in `pyproject.toml`. This is the only rule relaxed for `tests/`. + ## src/ is not linted yet `--lint-src` runs ruff over the wic source, but the source is **not** From patchwork Wed Jul 1 07:40:29 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00AD5C44501 for ; Wed, 1 Jul 2026 07:40:52 +0000 (UTC) Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.39530.1782891651510482484 for ; Wed, 01 Jul 2026 00:40:51 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=Fa6eFU/4; spf=pass (domain: gmail.com, ip: 209.85.160.179, mailfrom: twoerner@gmail.com) Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-51c05dcdf49so3277401cf.0 for ; Wed, 01 Jul 2026 00:40:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891650; x=1783496450; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=axQ2ssvVXKU6yhRY9smbJOUcyfNFkOt7WLLJlfd4Vdo=; b=Fa6eFU/4gYfC9bLOPnAYhq2FbNC9rSX0aEeozRUCDEN4icc+HShprAHHpryMj6XzwJ 7+I67dnM1TUHbNndMrNXuSeF5r/u12DTM9fr4RKYhbju7x/qMDb/L+10elHP4ziIDARo hAwQ1PkurVnR+/d8MR/YNKOdg1xt7ceCqtu2c4JZwa0keOguQJ6tyZz4JGo5MxJW/YZO Wx9k6hylQshe3KitesgnZb6wz4QRjZvZAfBxNkxvuQbixJSLIFIQ5K20kzfN8fECxVcu YqiT1ttratupmIbqbe1u0mLSzihvNmc57dhV/YEi7XbOnOQWH0pcU/HmlFDZ+vVB0NpD +aAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891650; x=1783496450; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=axQ2ssvVXKU6yhRY9smbJOUcyfNFkOt7WLLJlfd4Vdo=; b=Kfy7CUApmkGXkvi3bsHCBFp2bAJU+qPAUdjiA0diWuosuYzQ6X6VWWMuowt7dN3w6H 3fQ1o+b2fGrGY+VHyjydfF9F2H7IV3kSQekWx0O36ZYwvpX9qi6TCjmMDPgxgkx5IWj5 OL8UrrYdmPDopjHaU+G9qeVoU4wOxFSiF3/osRZ8lgnmhEYqb3zjE6KlsQ50lkVbOQLe ubAZJFt1vvTW04CphRrh6sWmo7c0AGAyASUfH5DM21P+TUX2a/gnDnhFlCyM+63EHiWQ 87fiJkdFu8LD9zqXpSkmjqrPRlF2ep8q13tb7h+WIQPJ5ptH2BpbgyZF3glSL37B+hL1 VIyQ== X-Gm-Message-State: AOJu0Yx0woPaU3hN1v6Ht1b82ogrKZay+cD8DnrL4TzvkmDcUozpGM5u W3qzvRiA4BcaS2abtXaZaEIocR/Q4QgL1eWRpO2odSkmfRZ8nO2xmKYtUb3L3A== X-Gm-Gg: AfdE7cklA8sXe+6TkzqpD8dwAo5qfOv3NOZheXdwsEEmps9FaWrKkN1ZtM/zmCVg5LI uiMl9FpGaYUYmqubGRYo8P7NVtHx0dehS31+PMJinbUt3BwvsMxXgTnKE1IgAYMdxhYFaYvWREt /kjED3FNaCIidwhD9Vdy6M6uo92Tp2pVK0tjvp6Z9FPbMCgO43NdnKHmQqUam5FcFImDJ0uEm+9 GAFGQmBvKxtV3UmjgdubiB2aTLSXdCWjoPXhY6tMWPxCwcK6LrKyi9uZLyxhJMR+6hekRveSlUL n46peHwcJlS/9tA5sS0xnprniIBqVHhTuaNpisAtbPSARFrMnID6k3ua/XYlijVCldmj4F4AI/i DIU0jY9ohY+zAKPQVXBoF5HU2yzbUWwdXZy3nQSq00iKtNEbaX0hzAd3mYjrE0z/xl1sHzucZ9c LjsQ20e6G5ciYgNHNO7LwUJExJ5CGhHpRiO+38cXmH7UK7UBjfSxHiIiI= X-Received: by 2002:a05:622a:4d8d:b0:51a:8bc9:2915 with SMTP id d75a77b69052e-51c26a3b27fmr6773711cf.8.1782891650286; Wed, 01 Jul 2026 00:40:50 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:48 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 09/10] tests/unit/test_bb_utils: test mkdirhier() and fix its missing errno import Date: Wed, 1 Jul 2026 03:40:29 -0400 Message-ID: <20260701074030.1090807-10-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:40:51 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4341 mkdirhier() wraps os.makedirs() and, on OSError, re-checks the error to decide whether to swallow a benign "already exists" race or re-raise: except OSError as e: if e.errno != errno.EEXIST or not os.path.isdir(directory): raise e but the module never imports errno. The reference to errno.EEXIST is only evaluated when os.makedirs() actually raises, so the defect is invisible on the happy path and on the unexpanded-${} guard. The moment a real OSError occurs -- a parent component that is a file, a permission error, anything -- the handler itself raises NameError: name 'errno' is not defined, masking the original error with a misleading one. The fix is a one-line import. With errno available the handler behaves as intended: an EEXIST on an existing directory is swallowed, while any other OSError (and an EEXIST whose path is not a directory) propagates unchanged. This commit adds tests/unit/test_bb_utils.py, covering mkdirhier() end to end: - the happy path (nested, single-level, idempotent, existing dir); - the unexpanded-${} guard (rejected, nothing created, and that a bare brace without a dollar is allowed); - the OSError handler: a real mkdir under a file component surfaces an OSError, an arbitrary OSError propagates with its errno intact, an EEXIST on an existing directory is swallowed, and an EEXIST on a regular file propagates. The error-path tests fail with NameError without the import and pass with it, so the test and the fix belong together in this one change. AI-Generated: codex/claude-opus 4.8 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - switch the tests from tempfile.mkdtemp() to pytest's tmp_path fixture so each test's scratch directory is cleaned up instead of leaking under /tmp; no change to what is tested. changes in v2: - v1 recorded this bug with an xfail marker in one large commit; v2 drops the xfail, asserts the correct behaviour directly, and lands the one-line errno-import fix in this same commit so the test passes green. --- src/wic/bb/utils.py | 1 + tests/unit/test_bb_utils.py | 116 ++++++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 tests/unit/test_bb_utils.py diff --git a/src/wic/bb/utils.py b/src/wic/bb/utils.py index 3750056ba563..0b40dd0c1d59 100644 --- a/src/wic/bb/utils.py +++ b/src/wic/bb/utils.py @@ -1,6 +1,7 @@ """ Minimal subset of BitBake's bb.utils used by standalone wic. """ +import errno import os # from bitbake/lib/bb/utils.py diff --git a/tests/unit/test_bb_utils.py b/tests/unit/test_bb_utils.py new file mode 100644 index 000000000000..aeca054109ff --- /dev/null +++ b/tests/unit/test_bb_utils.py @@ -0,0 +1,116 @@ +r""" +Unit tests for wic.bb.utils -- a small standalone subset of BitBake's +bb.utils. mkdirhier() is a mkdir -p wrapper with two guards: an +unexpanded-${} check and an OSError handler that re-checks EEXIST. +""" +import errno +import os +import sys +import unittest.mock as mock +from pathlib import Path + +import pytest + +_SRC = Path(__file__).resolve().parent.parent.parent / "src" +if str(_SRC) not in sys.path: + sys.path.insert(0, str(_SRC)) + +from wic.bb.utils import mkdirhier + + +# --------------------------------------------------------------------------- +# Happy path +# --------------------------------------------------------------------------- + +class TestMkdirhierHappyPath: + def test_creates_nested_directory(self, tmp_path): + target = os.path.join(str(tmp_path), "a", "b", "c") + mkdirhier(target) + assert os.path.isdir(target) + + def test_idempotent_on_existing_directory(self, tmp_path): + target = os.path.join(str(tmp_path), "x") + mkdirhier(target) + # Second call must not raise (exist_ok=True). + mkdirhier(target) + assert os.path.isdir(target) + + def test_single_level(self, tmp_path): + target = os.path.join(str(tmp_path), "single") + mkdirhier(target) + assert os.path.isdir(target) + + def test_existing_root_directory(self, tmp_path): + # An already-existing directory (the tmpdir itself) is a no-op. + mkdirhier(str(tmp_path)) + assert os.path.isdir(str(tmp_path)) + + +# --------------------------------------------------------------------------- +# Unexpanded bitbake-variable guard +# --------------------------------------------------------------------------- + +class TestMkdirhierUnexpandedVar: + def test_unexpanded_var_rejected(self): + with pytest.raises(Exception) as ei: + mkdirhier("/tmp/${WORKDIR}/x") + assert "unexpanded bitbake variable" in str(ei.value) + + def test_unexpanded_var_not_created(self, tmp_path): + bad = os.path.join(str(tmp_path), "${VAR}", "sub") + with pytest.raises(Exception): + mkdirhier(bad) + # Nothing should have been created. + assert not os.path.exists(os.path.join(str(tmp_path), "${VAR}")) + + def test_brace_without_dollar_is_allowed(self, tmp_path): + # Only the literal '${' marker triggers the guard; a bare '{' is a + # legal (if unusual) directory name. + target = os.path.join(str(tmp_path), "plain{brace") + mkdirhier(target) + assert os.path.isdir(target) + + +# --------------------------------------------------------------------------- +# OSError handler +# --------------------------------------------------------------------------- + +class TestMkdirhierErrorPath: + def test_oserror_under_a_file_component(self, tmp_path): + # Create a regular file, then try to mkdir a subdir *under* it. On + # Linux os.makedirs raises NotADirectoryError (an OSError subclass); + # the handler must let it surface rather than swallow it. + f = os.path.join(str(tmp_path), "afile") + Path(f).write_text("x") + target = os.path.join(f, "sub") + with pytest.raises(OSError): + mkdirhier(target) + + def test_generic_oserror_propagates(self): + # An arbitrary OSError (EACCES != EEXIST) must propagate unchanged. + err = OSError() + err.errno = errno.EACCES + with mock.patch("wic.bb.utils.os.makedirs", side_effect=err): + with pytest.raises(OSError) as ei: + mkdirhier("/whatever/path") + assert ei.value.errno == errno.EACCES + + def test_eexist_on_existing_dir_is_swallowed(self, tmp_path): + # An EEXIST OSError on a path that is already a directory is the + # benign race the handler exists to absorb; it must not propagate. + err = OSError() + err.errno = errno.EEXIST + with mock.patch("wic.bb.utils.os.makedirs", side_effect=err): + mkdirhier(str(tmp_path)) # exists and is a dir -> swallowed, no raise + + def test_eexist_on_existing_file_propagates(self, tmp_path): + # EEXIST but the path is a regular file, not a directory: the + # handler's isdir() re-check fails, so the error must propagate. + f = os.path.join(str(tmp_path), "afile") + Path(f).write_text("x") + err = OSError() + err.errno = errno.EEXIST + with mock.patch("wic.bb.utils.os.makedirs", side_effect=err): + with pytest.raises(OSError) as ei: + mkdirhier(f) + assert ei.value.errno == errno.EEXIST From patchwork Wed Jul 1 07:40:30 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trevor Woerner X-Patchwork-Id: 91472 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5083C43458 for ; Wed, 1 Jul 2026 07:41:01 +0000 (UTC) Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.39838.1782891653478339622 for ; Wed, 01 Jul 2026 00:40:53 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=bS15sLf3; spf=pass (domain: gmail.com, ip: 209.85.219.43, mailfrom: twoerner@gmail.com) Received: by mail-qv1-f43.google.com with SMTP id 6a1803df08f44-8e9c9d63815so2175226d6.2 for ; Wed, 01 Jul 2026 00:40:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782891652; x=1783496452; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=V3BkQMZhZStIsv6P9kXJlKtdsa8KzwetRzr3UKOFwuQ=; b=bS15sLf3bVC8TmNtQD8vQ5kdyKDvkciroF+d2lTSlnqnhskbrc+emZHMr2CNefoImH b9n3mcHHSRTEMUvYeVq6oAgn1WuUqnEZPFh4Ppzsv2Ea3zubJu45C7LOZbfTd/2fxiTg aFKKsXMJ4fKSFlGtnAJJMWl1KscRIMoEoZHuv/ZxsbaHhAPO8mqWv2bX7x4alVqrGLyb /Rq2zdKaMkTFIOI2GxiVlBvv84vyPd/x95Bg+3vNtEudLqqd8O0G15gQkt4slZbeeaKh blo5HRXHy9uRgWEF3hDdjCiwgQ0oc51SrB+lYFm8edeIdROlQqx72kNm5Y0Sa+NfqDHn 2Ysw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782891652; x=1783496452; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=V3BkQMZhZStIsv6P9kXJlKtdsa8KzwetRzr3UKOFwuQ=; b=NmNalezAbQ5Awd8lL+zpWxboIzbYBOdR3xRFbKixX4SvuZ4wzRfkdwCNzMhhknZjce cjL/JpswAE8RUPY6v3kHFecfhMSnjiI38bWrzAZniYGtPUYGfLAcS09DU/blSJX9Djq7 /099JIR37jvgj7EH5C1VOIjZwSV+Amrt2rUzEhPAPb0yFovrhoQ3nf2gPMDT7TA2+4CS a8gvfmtvY7D9YSYpXZh4fM3Vw0LCaPDElYiDLN27B2mwCTPnicR4L9DkAcipyNoRn8Hl 2t6P1b9FKqOblJTIvdafWnWZFpfxRodyqm/epnZa9cUxjqBME79gacCGeI3pa7TeuNO/ Udvg== X-Gm-Message-State: AOJu0Yxe3mEUvn6ADRlnFbGxj9PeCKV0PBeINcVkkrshk7S4A06Jx5To TZe369scTuOshv6FfL4+mHU8Y2kb48zV1BixCncZT+uOUZxXklDCn5Od6lmtag== X-Gm-Gg: AfdE7clUQagpd4gVnKXjlw5bIoN50yD93BNwFeA0rkTABqZb8Jo6Rvs8NZtutkzWR5u cP8ECg6KCad7lqg0L5deKNdbqrddyOOpVHnwDAQ5ms6lTCXM/vyjrjSvWZIglTcAqr0rDyLtJRu HNnyu5BvKzFvHH6bCUCz1vmOkG/B+DzluonHLCry7qtF2qBN9VMl1LWTSukCEfz6gOpUnWKoHNs Jg19PSYbpUeFFfEbaUG1NCasFvM8YlCN5gskb1T5MmYeS3btZojnbL1l18kNLL9cVe+k3kqWRn1 OPKJg4NUobEQnLkTfU+cWMfq0N4bE9SGO4K/KpEveIqlxSjDZPBAEY2yyB2UECZrup+MN88Hf5y lSji9trhumdTHBnkqWF+4sLlm32qmkuN3EEbRPsaetVJuOmuOfegztzNHMy85IkVGix+jeHSB3G BVsUwmZVvTu8r57jjxTGWAvwRVAOdW3Ksux3y6M3eGPRCiy8fnxSxOHpI= X-Received: by 2002:a05:620a:1a26:b0:915:cb40:f76a with SMTP id af79cd13be357-92e782c44c1mr74431285a.39.1782891652131; Wed, 01 Jul 2026 00:40:52 -0700 (PDT) Received: from localhost.localdomain (pppoe-209-91-167-254.vianet.ca. [209.91.167.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8f35e790229sm15822316d6.2.2026.07.01.00.40.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2026 00:40:50 -0700 (PDT) From: Trevor Woerner To: yocto-patches@lists.yoctoproject.org Subject: [wic][PATCH v3 10/10] tests/docs: add the review rubric Date: Wed, 1 Jul 2026 03:40:30 -0400 Message-ID: <20260701074030.1090807-11-twoerner@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com> References: <20260701074030.1090807-1-twoerner@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Jul 2026 07:41:01 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/4342 The suite already documents how to write a test (authoring.md) and how it is linted (linting.md), but not how a test change is judged. This adds tests/docs/reviewing.md, the review-time companion to authoring.md, so the standard a change is held to is written down where both reviewers and contributors can see it. The rubric records the conventions the suite is built on: - one function per commit, with the subject keyed to the test file; - a test asserts the correct behaviour, and when it exposes a defect the source fix lands in the same commit rather than as an xfail or a test that bakes in the wrong result; - a fix must be proven to matter by backing it out and watching the test go red; - the suite is green and lint-clean at every commit, not only at the tip of a series; - assertions are specific, boundaries are probed, and an assertion is never weakened to force a pass; - coverage is read as a guide to untested branches, not a score; - each commit message stands alone and references no other commit. It closes with a short reviewer checklist that collects those points, and is linked from the docs table in the suite README. AI-Generated: codex/claude-opus 4.8 (xhigh) Signed-off-by: Trevor Woerner --- changes in v3: - new in v3: adds tests/docs/reviewing.md, the review rubric for a test change, and links it from the suite README. --- tests/docs/README.md | 1 + tests/docs/reviewing.md | 164 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 165 insertions(+) create mode 100644 tests/docs/reviewing.md diff --git a/tests/docs/README.md b/tests/docs/README.md index ba7bd5e192a3..d44212a287ab 100644 --- a/tests/docs/README.md +++ b/tests/docs/README.md @@ -79,4 +79,5 @@ treated. | File | Content | |------|---------| | [authoring.md](authoring.md) | How to add a unit test to the suite | +| [reviewing.md](reviewing.md) | How a test change is reviewed | | [linting.md](linting.md) | How ruff is used on the suite | diff --git a/tests/docs/reviewing.md b/tests/docs/reviewing.md new file mode 100644 index 000000000000..a9b002a6dabd --- /dev/null +++ b/tests/docs/reviewing.md @@ -0,0 +1,164 @@ +# Reviewing a test change + +This is the rubric a reviewer applies to a change that adds or edits +tests, and the bar a contributor should meet before sending one. It is +the review-time companion to [authoring.md](authoring.md): authoring.md +says how to write a good test, this says how such a change is judged. + +## Contents + +- [One function per commit](#one-function-per-commit) +- [Test and fix land together](#test-and-fix-land-together) +- [Prove the fix matters](#prove-the-fix-matters) +- [Green at every commit](#green-at-every-commit) +- [Assertions and boundaries](#assertions-and-boundaries) +- [Scratch files](#scratch-files) +- [Coverage](#coverage) +- [Commit message](#commit-message) +- [Reviewer checklist](#reviewer-checklist) + +## One function per commit + +A commit covers one function under test. A module with a single +function is one commit; a module with five functions is five commits, +one per function. All the tests for that function (happy path, guards, +boundaries, error paths) belong in the same commit, because they share +the function and its setup; do not split a commit per assertion. + +The subject line names the test file and the function: + +```text +tests/unit/test_: test () +tests/unit/test_: test () and fix +``` + +The second form is for when a fix to the wic source rides along (see +below). Keep the subject short: it says only "and fix", and the full +explanation of what was fixed goes in the commit body. There is no +leading `wic:`; the subject is keyed to the test file. + +## Test and fix land together + +A test asserts the behaviour wic is *expected* to provide, never the +behaviour it currently happens to have. When a test exposes a defect, +the fix to the wic source lands in the same commit, so the test asserts +the correct behaviour and passes. A change that adds a test asserting a +known-wrong result, or that marks a failing case `xfail` to defer the +fix, does not meet the bar: it either bakes in the bug or leaves a red +test behind. Reject both. + +## Prove the fix matters + +When a commit carries a fix, the test must actually exercise the bug. +The cheap proof is to back the source change out and watch the test go +red, then restore it and watch it pass: + +```bash +git stash push -- src/wic/ +tests/run-tests.sh tests/unit/test_.py # the new tests fail +git stash pop +tests/run-tests.sh tests/unit/test_.py # the new tests pass +``` + +A test that stays green with the fix removed is not testing the fix. +Either the assertion is too weak or the wrong code path is exercised; +in review, treat that as a finding. + +## Green at every commit + +The suite is green at every commit boundary, not just at the tip of the +series: zero failures and zero `xfail` markers, with +`tests/run-tests.sh --lint-tests` passing 100%, reporting nothing. The +test tree is held to a clean bar; a single lint finding fails the +commit. A series that only goes green at the end cannot be bisected and +is not acceptable; check intermediate commits, not just `HEAD`. + +`--lint-src` is a preview report over the wic source, not a gate (see +[linting.md](linting.md)); findings there do not block a test change. + +## Assertions and boundaries + +Every case asserts a specific outcome, an exact value or a specific +exception, never merely that nothing was raised. A test that only +checks for the absence of a crash passes against badly wrong output and +gives false confidence. + +A good test probes the boundary of each parameter and just past it, +rather than only the comfortable middle of its range. The classes worth +covering (empty, zero, negative, maximum and one beyond, wrong types, +path traversal, stale state, and so on) are enumerated in +authoring.md under "Probe the boundaries"; in review, look for the ones +that matter for the function at hand and note the ones that are +missing. + +If an assertion was weakened to make a test pass, the change is going +the wrong way. The answer to a hard-to-pass test is in the code or in +understanding the correct behaviour, never in loosening the assertion. + +## Scratch files + +A test that needs scratch space uses pytest's `tmp_path` fixture, which +gives each test its own directory and removes it automatically when the +test passes (a failing test's directory is kept for inspection). A +change that reaches for `tempfile.mkdtemp()`, `/tmp` directly, or any +other hand-rolled scratch path is going the wrong way: those leak +directories across runs and are worth a finding. + +The scratch tree lands under the system temporary directory by default. +To send it elsewhere, when `/tmp` is small or a run produces a lot of +scratch, use the standard pytest controls rather than a custom setting; +both are passed straight through by `run-tests.sh` and are described in +the suite [README](README.md#scratch-files): + +```bash +TMPDIR=/path/to/scratch tests/run-tests.sh +tests/run-tests.sh --basetemp=/path/to/scratch +``` + +## Coverage + +Coverage is a guide, not a score to maximise. Run it for the module +under review and read which lines and branches the new tests reach: + +```bash +tests/run-tests.sh --coverage tests/unit/test_.py +``` + +Use it to confirm the boundary rule above was actually met: the +function's own guards and error paths should be reached, not just its +happy path. A reachable branch of the function under review that no +test exercises usually means a boundary case is missing, so the fix is +to add that case, not to chase a coverage number. A branch that cannot +sensibly be triggered is not a gap. + +## Commit message + +Each commit stands alone. The body explains the function, the tests, +and, when a fix rides along, what the bug was, why it was invisible +before, and why the test and fix must land together; a short failing +snippet helps. Do not reference other commits by number or hash, since +the series may be reordered or rebased and such a reference would go +stale. When a fix rides along, the body is where the fix is explained +in full, since the subject only says "and fix". Do not weaken the +explanation to save space: a reviewer reading the commit in isolation +should understand the whole change. + +## Reviewer checklist + +- The commit covers exactly one function, with the subject in the form + above. +- All of that function's cases (happy path, guards, boundaries, error + paths) are present and each asserts a specific value or exception. +- If a fix rides along, it is in the same commit and the test fails + without it (verified by backing the fix out). +- No `xfail`, no test asserting a known-wrong result, no weakened + assertion. +- Scratch space uses the `tmp_path` fixture, not `mkdtemp()` or a raw + `/tmp` path. +- The suite is green and `--lint-tests` is clean at this commit, not + only at the tip. +- Coverage confirms the function's own guards and error paths are + reached; any reachable but untested branch is treated as a missing + boundary case. +- The commit message stands alone, references no other commit, and + fully explains any fix that the short "and fix" subject only names.