| Message ID | 20250609222143.3275-1-anibal@limonsoftware.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | test-manual/ptest: Add info about fail state of ptest-runner/ptests | expand |
Hello, Le mar. 10 juin 2025 à 00:21, Anibal Limon <anibal@limonsoftware.com> a écrit : > On the ML was pointed about no specific information about > ptest-runner/ptests so improve docs around. > > https://lists.yoctoproject.org/g/yocto-patches/message/1622 > > Signed-off-by: Anibal Limon <anibal@limonsoftware.com> > Thank you for working on this! :) Can you reference the bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832? With "[YOCTO #15832]" somewhere in your commit message. > --- > documentation/test-manual/ptest.rst | 17 ++++++++++++----- > 1 file changed, 12 insertions(+), 5 deletions(-) > > diff --git a/documentation/test-manual/ptest.rst > b/documentation/test-manual/ptest.rst > index 2c021af51..3b50e354a 100644 > --- a/documentation/test-manual/ptest.rst > +++ b/documentation/test-manual/ptest.rst > @@ -46,13 +46,19 @@ Running ptest > ============= > > The ``ptest-runner`` package installs a shell script that loops through > -all installed ptest test suites and runs them in sequence. Consequently, > -you might want to add this package to your image. > +all installed ptest test suites and runs them in sequence. > + > +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` > +and use it as exit value, failing the execution if any of the > ``run-ptest`` > +scripts fails. > Counting failed tests might lead to weird overflows. For example, there is a special error code for "ptest-runner itself failed" (127 IIRC). If you have 127 failing tests, the calling code may reach a wrong conclusion. I'd recommend returning a static non-zero error code on error (e.g 1). > + > +Consequently, you might want to add this package to your image. > + > > Getting Your Package Ready > ========================== > > -In order to enable a recipe to run installed ptests on target hardware, > +In order to enable a recipe to run installed ``ptests`` on target > hardware, > you need to prepare the recipes that build the packages you want to > test. Here is what you have to do for each recipe: > > @@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: > > - *Create run-ptest:* This script starts your test. Locate the > script where you will refer to it using > - :term:`SRC_URI`. Here is an > - example that starts a test for ``dbus``:: > + :term:`SRC_URI`. Be sure your *run-ptest* exit with 0 to mark it > Use ``run-ptest`` to stay consistent with the rest of the doc? + as succesfully executed. > + Here is an example that starts a test for ``dbus``:: > > #!/bin/sh > cd test > -- > 2.39.5 >
On Mon, Jun 9, 2025 at 4:48 PM Yoann Congal <yoann.congal@smile.fr> wrote: > Hello, > > Le mar. 10 juin 2025 à 00:21, Anibal Limon <anibal@limonsoftware.com> a > écrit : > >> On the ML was pointed about no specific information about >> ptest-runner/ptests so improve docs around. >> >> https://lists.yoctoproject.org/g/yocto-patches/message/1622 >> >> Signed-off-by: Anibal Limon <anibal@limonsoftware.com> >> > > Thank you for working on this! :) > > Can you reference the bug > https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832? > With "[YOCTO #15832]" somewhere in your commit message. > > >> --- >> documentation/test-manual/ptest.rst | 17 ++++++++++++----- >> 1 file changed, 12 insertions(+), 5 deletions(-) >> >> diff --git a/documentation/test-manual/ptest.rst >> b/documentation/test-manual/ptest.rst >> index 2c021af51..3b50e354a 100644 >> --- a/documentation/test-manual/ptest.rst >> +++ b/documentation/test-manual/ptest.rst >> @@ -46,13 +46,19 @@ Running ptest >> ============= >> >> The ``ptest-runner`` package installs a shell script that loops through >> -all installed ptest test suites and runs them in sequence. Consequently, >> -you might want to add this package to your image. >> +all installed ptest test suites and runs them in sequence. >> + >> +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` >> +and use it as exit value, failing the execution if any of the >> ``run-ptest`` >> +scripts fails. >> > > Counting failed tests might lead to weird overflows. > For example, there is a special error code for "ptest-runner itself > failed" (127 IIRC). > If you have 127 failing tests, the calling code may reach a wrong > conclusion. > > I'd recommend returning a static non-zero error code on error (e.g 1). > I will look into the bug description, and see how we can fix the behaviour you are experiencing. https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832#c0 > > >> + >> +Consequently, you might want to add this package to your image. >> + >> >> Getting Your Package Ready >> ========================== >> >> -In order to enable a recipe to run installed ptests on target hardware, >> +In order to enable a recipe to run installed ``ptests`` on target >> hardware, >> you need to prepare the recipes that build the packages you want to >> test. Here is what you have to do for each recipe: >> >> @@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: >> >> - *Create run-ptest:* This script starts your test. Locate the >> script where you will refer to it using >> - :term:`SRC_URI`. Here is an >> - example that starts a test for ``dbus``:: >> + :term:`SRC_URI`. Be sure your *run-ptest* exit with 0 to mark it >> > > Use ``run-ptest`` to stay consistent with the rest of the doc? > I sent a v2. Cheers!, Anibal > > + as succesfully executed. >> + Here is an example that starts a test for ``dbus``:: >> >> #!/bin/sh >> cd test >> -- >> 2.39.5 >> > > > -- > Yoann Congal > Smile ECS >
diff --git a/documentation/test-manual/ptest.rst b/documentation/test-manual/ptest.rst index 2c021af51..3b50e354a 100644 --- a/documentation/test-manual/ptest.rst +++ b/documentation/test-manual/ptest.rst @@ -46,13 +46,19 @@ Running ptest ============= The ``ptest-runner`` package installs a shell script that loops through -all installed ptest test suites and runs them in sequence. Consequently, -you might want to add this package to your image. +all installed ptest test suites and runs them in sequence. + +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` +and use it as exit value, failing the execution if any of the ``run-ptest`` +scripts fails. + +Consequently, you might want to add this package to your image. + Getting Your Package Ready ========================== -In order to enable a recipe to run installed ptests on target hardware, +In order to enable a recipe to run installed ``ptests`` on target hardware, you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe: @@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: - *Create run-ptest:* This script starts your test. Locate the script where you will refer to it using - :term:`SRC_URI`. Here is an - example that starts a test for ``dbus``:: + :term:`SRC_URI`. Be sure your *run-ptest* exit with 0 to mark it + as succesfully executed. + Here is an example that starts a test for ``dbus``:: #!/bin/sh cd test
On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around. https://lists.yoctoproject.org/g/yocto-patches/message/1622 Signed-off-by: Anibal Limon <anibal@limonsoftware.com> --- documentation/test-manual/ptest.rst | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)