| Message ID | 20250611015401.1342422-1-anibal@limonsoftware.com |
|---|---|
| State | New |
| Headers | show |
| Series | [PATCHv3] test-manual/ptest: Add info about fail state of ptest-runner/ptests | expand |
Hello, Le mer. 11 juin 2025 à 03:54, Anibal Limon <anibal@limonsoftware.com> a écrit : > On the ML was pointed about no specific information about > ptest-runner/ptests so improve docs around. > > https://lists.yoctoproject.org/g/yocto-patches/message/1622 > > [YOCTO #15832] > > Signed-off-by: Anibal Limon <anibal@limonsoftware.com> > --- > documentation/Pipfile | 2 ++ > documentation/test-manual/ptest.rst | 17 ++++++++++++----- > 2 files changed, 14 insertions(+), 5 deletions(-) > > diff --git a/documentation/Pipfile b/documentation/Pipfile > index 7ee1d2290..c99d09c75 100644 > --- a/documentation/Pipfile > +++ b/documentation/Pipfile > @@ -9,6 +9,8 @@ verify_ssl = true > sphinx = "*" > sphinx-rtd-theme = "*" > pyyaml = "*" > +vale = "*" > +sphinx-lint = "*" > ^ this change is unrelated and should be split into a separate commit (or removed if that was an error) > [requires] > python_version = "3" > diff --git a/documentation/test-manual/ptest.rst > b/documentation/test-manual/ptest.rst > index 2c021af51..e064419d7 100644 > --- a/documentation/test-manual/ptest.rst > +++ b/documentation/test-manual/ptest.rst > @@ -46,13 +46,19 @@ Running ptest > ============= > > The ``ptest-runner`` package installs a shell script that loops through > -all installed ptest test suites and runs them in sequence. Consequently, > -you might want to add this package to your image. > +all installed ptest test suites and runs them in sequence. > + > +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` > +and use it as exit value, failing the execution if any of the > ``run-ptest`` > +scripts fails. > + > +Consequently, you might want to add this package to your image. > + > "keeps counter of failed ``ptests`` and use it as exit value" is not entirely true. I went to look in the code of ptest-runner2, and found this commit: https://git.yoctoproject.org/ptest-runner2/commit/main.c?id=c10e7475353f4cfb5afb30ab1012c32c2efde85a (It was committed by you?) => ptest-runner does keep a count of failed ptests but returns 1 if the count is > 0. Getting Your Package Ready > ========================== > > -In order to enable a recipe to run installed ptests on target hardware, > +In order to enable a recipe to run installed ``ptests`` on target > hardware, > you need to prepare the recipes that build the packages you want to > test. Here is what you have to do for each recipe: > > @@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: > > - *Create run-ptest:* This script starts your test. Locate the > script where you will refer to it using > - :term:`SRC_URI`. Here is an > - example that starts a test for ``dbus``:: > + :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it > + as sucessfully executed. > Can you clarify here that ptest-runner expects 0 for a successful test or !=0 for a test failure. Also (but that may be tackled in another patch), oe-selftest's testimage seems to expect at least a "PASS: ...", "FAIL: ..." line somewhere. See https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832#c1. > + Here is an example that starts a test for ``dbus``:: > > #!/bin/sh > cd test > -- > 2.39.5 > Regards,
On Wed, Jun 11, 2025 at 8:33 AM Yoann Congal <yoann.congal@smile.fr> wrote: > Hello, > > Le mer. 11 juin 2025 à 03:54, Anibal Limon <anibal@limonsoftware.com> a > écrit : > >> On the ML was pointed about no specific information about >> ptest-runner/ptests so improve docs around. >> >> https://lists.yoctoproject.org/g/yocto-patches/message/1622 >> >> [YOCTO #15832] >> >> Signed-off-by: Anibal Limon <anibal@limonsoftware.com> >> --- >> documentation/Pipfile | 2 ++ >> documentation/test-manual/ptest.rst | 17 ++++++++++++----- >> 2 files changed, 14 insertions(+), 5 deletions(-) >> >> diff --git a/documentation/Pipfile b/documentation/Pipfile >> index 7ee1d2290..c99d09c75 100644 >> --- a/documentation/Pipfile >> +++ b/documentation/Pipfile >> @@ -9,6 +9,8 @@ verify_ssl = true >> sphinx = "*" >> sphinx-rtd-theme = "*" >> pyyaml = "*" >> +vale = "*" >> +sphinx-lint = "*" >> > ^ this change is unrelated and should be split into a separate commit (or > removed if that was an error) > Nice catch, > > >> [requires] >> python_version = "3" >> diff --git a/documentation/test-manual/ptest.rst >> b/documentation/test-manual/ptest.rst >> index 2c021af51..e064419d7 100644 >> --- a/documentation/test-manual/ptest.rst >> +++ b/documentation/test-manual/ptest.rst >> @@ -46,13 +46,19 @@ Running ptest >> ============= >> >> The ``ptest-runner`` package installs a shell script that loops through >> -all installed ptest test suites and runs them in sequence. Consequently, >> -you might want to add this package to your image. >> +all installed ptest test suites and runs them in sequence. >> + >> +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` >> +and use it as exit value, failing the execution if any of the >> ``run-ptest`` >> +scripts fails. >> + >> +Consequently, you might want to add this package to your image. >> + >> > "keeps counter of failed ``ptests`` and use it as exit value" is not > entirely true. > I went to look in the code of ptest-runner2, and found this commit: > https://git.yoctoproject.org/ptest-runner2/commit/main.c?id=c10e7475353f4cfb5afb30ab1012c32c2efde85a > (It was committed by you?) => ptest-runner does keep a count of failed > ptests but returns 1 if the count is > 0. > Yeah, I remembered, fixing doc. > > Getting Your Package Ready >> ========================== >> >> -In order to enable a recipe to run installed ptests on target hardware, >> +In order to enable a recipe to run installed ``ptests`` on target >> hardware, >> you need to prepare the recipes that build the packages you want to >> test. Here is what you have to do for each recipe: >> >> @@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: >> >> - *Create run-ptest:* This script starts your test. Locate the >> script where you will refer to it using >> - :term:`SRC_URI`. Here is an >> - example that starts a test for ``dbus``:: >> + :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it >> + as sucessfully executed. >> > > Can you clarify here that ptest-runner expects 0 for a successful test or > !=0 for a test failure. > Let me rephrase. > > Also (but that may be tackled in another patch), oe-selftest's testimage > seems to expect at least a "PASS: ...", "FAIL: ..." line somewhere. > See https://bugzilla.yoctoproject.org/show_bug.cgi?id=15832#c1. > > >> + Here is an example that starts a test for ``dbus``:: >> >> #!/bin/sh >> cd test >> -- >> 2.39.5 >> > > Regards, > -- > Yoann Congal > Smile ECS >
diff --git a/documentation/Pipfile b/documentation/Pipfile index 7ee1d2290..c99d09c75 100644 --- a/documentation/Pipfile +++ b/documentation/Pipfile @@ -9,6 +9,8 @@ verify_ssl = true sphinx = "*" sphinx-rtd-theme = "*" pyyaml = "*" +vale = "*" +sphinx-lint = "*" [requires] python_version = "3" diff --git a/documentation/test-manual/ptest.rst b/documentation/test-manual/ptest.rst index 2c021af51..e064419d7 100644 --- a/documentation/test-manual/ptest.rst +++ b/documentation/test-manual/ptest.rst @@ -46,13 +46,19 @@ Running ptest ============= The ``ptest-runner`` package installs a shell script that loops through -all installed ptest test suites and runs them in sequence. Consequently, -you might want to add this package to your image. +all installed ptest test suites and runs them in sequence. + +During the execution ``ptest-runner`` keeps counter of failed ``ptests`` +and use it as exit value, failing the execution if any of the ``run-ptest`` +scripts fails. + +Consequently, you might want to add this package to your image. + Getting Your Package Ready ========================== -In order to enable a recipe to run installed ptests on target hardware, +In order to enable a recipe to run installed ``ptests`` on target hardware, you need to prepare the recipes that build the packages you want to test. Here is what you have to do for each recipe: @@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe: - *Create run-ptest:* This script starts your test. Locate the script where you will refer to it using - :term:`SRC_URI`. Here is an - example that starts a test for ``dbus``:: + :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it + as sucessfully executed. + Here is an example that starts a test for ``dbus``:: #!/bin/sh cd test
On the ML was pointed about no specific information about ptest-runner/ptests so improve docs around. https://lists.yoctoproject.org/g/yocto-patches/message/1622 [YOCTO #15832] Signed-off-by: Anibal Limon <anibal@limonsoftware.com> --- documentation/Pipfile | 2 ++ documentation/test-manual/ptest.rst | 17 ++++++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-)