| Message ID | 20260227200241.66128-1-tim.orling@konsulko.com |
|---|---|
| State | Under Review |
| Headers | show |
| Series | ref-manual: document PTEST_RUNNER_TIMEOUT variable | expand |
Hi, On Fri Feb 27, 2026 at 9:02 PM CET, Tim Orling via lists.yoctoproject.org wrote: [...] > diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst > index 208b5e43d..587f67cc6 100644 > --- a/documentation/ref-manual/variables.rst > +++ b/documentation/ref-manual/variables.rst > @@ -7793,6 +7793,20 @@ system and gives an overview of their function and contents. > For more information, see > the :ref:`ref-classes-ptest-python-pytest` class documentation. > > + :term:`PTEST_RUNNER_TIMEOUT` > + Specifies the timeout in seconds for each ptest suite executed by > + ``ptest-runner`` during :ref:`ref-classes-testimage` runtime testing. > + The default value is ``"450"`` (7.5 minutes). > + > + This variable is set in the :ref:`ref-classes-testimage` class and can > + be overridden in image recipes to adjust the timeout for specific > + images or multiconfig variants. For example, in ``core-image-ptest``:: > + > + PTEST_RUNNER_TIMEOUT:virtclass-mcextend-python3-cffi = "600" Could you explain why you're using the :virtclass-mcextend-python3-cffi override here? Seems like a complicated example for this variable's description, but if it is needed, we might want to explain it? Antonin
On Tue, Mar 3, 2026 at 2:59 AM Antonin Godard <antonin.godard@bootlin.com> wrote: > Hi, > > On Fri Feb 27, 2026 at 9:02 PM CET, Tim Orling via lists.yoctoproject.org > wrote: > [...] > > diff --git a/documentation/ref-manual/variables.rst > b/documentation/ref-manual/variables.rst > > index 208b5e43d..587f67cc6 100644 > > --- a/documentation/ref-manual/variables.rst > > +++ b/documentation/ref-manual/variables.rst > > @@ -7793,6 +7793,20 @@ system and gives an overview of their function > and contents. > > For more information, see > > the :ref:`ref-classes-ptest-python-pytest` class documentation. > > > > + :term:`PTEST_RUNNER_TIMEOUT` > > + Specifies the timeout in seconds for each ptest suite executed by > > + ``ptest-runner`` during :ref:`ref-classes-testimage` runtime > testing. > > + The default value is ``"450"`` (7.5 minutes). > > + > > + This variable is set in the :ref:`ref-classes-testimage` class > and can > > + be overridden in image recipes to adjust the timeout for specific > > + images or multiconfig variants. For example, in > ``core-image-ptest``:: > > + > > + PTEST_RUNNER_TIMEOUT:virtclass-mcextend-python3-cffi = "600" > > Could you explain why you're using the :virtclass-mcextend-python3-cffi > override > here? Seems like a complicated example for this variable's description, > but if > it is needed, we might want to explain it? > The reason is simple, all the other variables in the "core-image-ptest.bb" recipe were set with "virtclass-mcextend-". [1] A prior attempt at setting with "pn-python3-cffi" in "ptest-packelists.inc" had not worked, although that was before I caught the need for the "testimage.bbclass" change. After further experimentation: 1. PTEST_RUNNER_TIMEOUT:pn-python3-cffi = "600" does NOT work [2][3] 2. PTEST_RUNNER_TIMEOUT:pn-core-image-ptest-python3-cffi = "600" DOES work [4][5] We also know from Trevor Gamblin experiments that PTEST_RUNNER_TIMEOUT:riscv-64 = "600" in local.conf works (and similar experiments with "conf" changes in "yocto-autobuilder-helper" worked). The goal of the "core-image-ptest.bb" changes were intended to avoid REQUIRING a change in "conf" only, and instead enable the change to happen in recipe scope. So the conclusion appears to be that "image recipe" and "conf" scope work, but an individual ptest enabled recipe doesnot work. The "image" scope probably makes sense since PTEST_RUNNER_TIMEOUT is actually enabled by the "testimage.bbclass"? Would it make more sense to use the "more obvious" usage in #2 above (even though it differs from other variable overrides in "core-image-ptest.bb" recipe)? [1] https://git.openembedded.org/openembedded-core/plain/meta/recipes-core/images/core-image-ptest.bb?id=729d81250922f324abf4375b8126382d1c757096 [2] https://autobuilder.yoctoproject.org/valkyrie/#/builders/61/builds/3163 [3] https://valkyrie.yocto.io/pub/non-release/20260303-115/testresults/qemuarm64-ptest/core-image-ptest-python3-cffi/log.do_testimage.473040.20260303192127 [4] https://autobuilder.yoctoproject.org/valkyrie/#/builders/61/builds/3165 [5] https://valkyrie.yocto.io/pub/non-release/20260303-117/testresults/qemuarm64-ptest/core-image-ptest-python3-cffi/log.do_testimage.519844.20260303194109 --Tim > Antonin >
Hi, On Tue Mar 3, 2026 at 9:17 PM CET, Tim Orling wrote: > On Tue, Mar 3, 2026 at 2:59 AM Antonin Godard <antonin.godard@bootlin.com> > wrote: > >> Hi, >> >> On Fri Feb 27, 2026 at 9:02 PM CET, Tim Orling via lists.yoctoproject.org >> wrote: >> [...] >> > diff --git a/documentation/ref-manual/variables.rst >> b/documentation/ref-manual/variables.rst >> > index 208b5e43d..587f67cc6 100644 >> > --- a/documentation/ref-manual/variables.rst >> > +++ b/documentation/ref-manual/variables.rst >> > @@ -7793,6 +7793,20 @@ system and gives an overview of their function >> and contents. >> > For more information, see >> > the :ref:`ref-classes-ptest-python-pytest` class documentation. >> > >> > + :term:`PTEST_RUNNER_TIMEOUT` >> > + Specifies the timeout in seconds for each ptest suite executed by >> > + ``ptest-runner`` during :ref:`ref-classes-testimage` runtime >> testing. >> > + The default value is ``"450"`` (7.5 minutes). >> > + >> > + This variable is set in the :ref:`ref-classes-testimage` class >> and can >> > + be overridden in image recipes to adjust the timeout for specific >> > + images or multiconfig variants. For example, in >> ``core-image-ptest``:: >> > + >> > + PTEST_RUNNER_TIMEOUT:virtclass-mcextend-python3-cffi = "600" >> >> Could you explain why you're using the :virtclass-mcextend-python3-cffi >> override >> here? Seems like a complicated example for this variable's description, >> but if >> it is needed, we might want to explain it? >> > > The reason is simple, all the other variables in the "core-image-ptest.bb" > recipe were set with "virtclass-mcextend-". [1] A prior attempt at setting > with "pn-python3-cffi" in "ptest-packelists.inc" had not worked, although > that was before I caught the need for the "testimage.bbclass" change. > > After further experimentation: > > 1. PTEST_RUNNER_TIMEOUT:pn-python3-cffi = "600" does NOT work [2][3] > 2. PTEST_RUNNER_TIMEOUT:pn-core-image-ptest-python3-cffi = "600" DOES work > [4][5] > > We also know from Trevor Gamblin experiments that > PTEST_RUNNER_TIMEOUT:riscv-64 = "600" in local.conf works (and similar > experiments with "conf" changes in "yocto-autobuilder-helper" worked). The > goal of the "core-image-ptest.bb" changes were intended to avoid REQUIRING > a change in "conf" only, and instead enable the change to happen in recipe > scope. > > So the conclusion appears to be that "image recipe" and "conf" scope work, > but an individual ptest enabled recipe doesnot work. The "image" scope > probably makes sense since PTEST_RUNNER_TIMEOUT is actually enabled by the > "testimage.bbclass"? > > Would it make more sense to use the "more obvious" usage in #2 above (even > though it differs from other variable overrides in "core-image-ptest.bb" > recipe)? I think we need to write this variable's documentation as a point of view of someone using the testimage class, not the core-image-ptest recipe. The core-image-ptest recipe appears to use some niche use-cases of overrides with mcextend (which also isn't documented). So, if I were a user writing an image that inherits the testimage class, and my image recipe does not contain any mcextend shenanigans, would I really be able to use syntax #1 or #2? I can obviously override the value of PTEST_RUNNER_TIMEOUT for all my underlying ptests, with something like: my-image.bb: IMAGE_CLASSES += "testimage" PTEST_RUNNER_TIMEOUT = "1000" But that's it, I guess? Antonin
diff --git a/documentation/ref-manual/classes.rst b/documentation/ref-manual/classes.rst index eb76c8916..7a045b6b7 100644 --- a/documentation/ref-manual/classes.rst +++ b/documentation/ref-manual/classes.rst @@ -3110,6 +3110,11 @@ after it is built, you can set :term:`TESTIMAGE_AUTO`:: TESTIMAGE_AUTO = "1" +The class defines :term:`PTEST_RUNNER_TIMEOUT`, which controls +the per-suite timeout for ``ptest-runner`` during runtime testing. +Image recipes can override this variable to adjust the timeout for +specific images or multiconfig variants. + For information on how to enable, run, and create new tests, see the ":ref:`test-manual/runtime-testing:performing automated runtime testing`" section in the Yocto Project Test Environment Manual. diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index 208b5e43d..587f67cc6 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -7793,6 +7793,20 @@ system and gives an overview of their function and contents. For more information, see the :ref:`ref-classes-ptest-python-pytest` class documentation. + :term:`PTEST_RUNNER_TIMEOUT` + Specifies the timeout in seconds for each ptest suite executed by + ``ptest-runner`` during :ref:`ref-classes-testimage` runtime testing. + The default value is ``"450"`` (7.5 minutes). + + This variable is set in the :ref:`ref-classes-testimage` class and can + be overridden in image recipes to adjust the timeout for specific + images or multiconfig variants. For example, in ``core-image-ptest``:: + + PTEST_RUNNER_TIMEOUT:virtclass-mcextend-python3-cffi = "600" + + The value is passed to the ``ptest-runner -t`` option on the target + at test execution time. + :term:`PV` The version of the recipe. The version is normally extracted from the recipe filename. For example, if the recipe is named
Add PTEST_RUNNER_TIMEOUT to the variable glossary and mention it in the testimage class documentation. The variable was added to testimage.bbclass to control the per-suite timeout for ptest-runner during runtime testing. It defaults to 450 seconds and can be overridden in image recipes. Signed-off-by: Tim Orling <tim.orling@konsulko.com> --- documentation/ref-manual/classes.rst | 5 +++++ documentation/ref-manual/variables.rst | 14 ++++++++++++++ 2 files changed, 19 insertions(+)