| Message ID | 20240626094130.255905-1-peter.hoyes@arm.com |
|---|---|
| State | New |
| Headers | show |
| Series | testimage: Gate artifact collection with variable | expand |
On Wed, 2024-06-26 at 10:41 +0100, Peter Hoyes via lists.openembedded.org wrote: > It does not always make sense to collect artifacts and data from the > target on failure, e.g. if testing firmware or if the target is not > running an SSH server. > > Introduce the variable TESTIMAGE_RUN_FAILURE_POST_ACTIONS, which > defaults to "1". If this variable is not true, skip the failed test post > actions. > > Signed-off-by: Peter Hoyes <peter.hoyes@arm.com> > --- > meta/classes-recipe/testimage.bbclass | 10 ++++++---- > 1 file changed, 6 insertions(+), 4 deletions(-) Is there a reason you can't just set TESTIMAGE_FAILED_QA ARTIFACTS = "" ? Cheers, Richard
On 27 Jun 2024, at 13:02, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Wed, 2024-06-26 at 10:41 +0100, Peter Hoyes via lists.openembedded.org wrote: >> It does not always make sense to collect artifacts and data from the >> target on failure, e.g. if testing firmware or if the target is not >> running an SSH server. >> >> Introduce the variable TESTIMAGE_RUN_FAILURE_POST_ACTIONS, which >> defaults to "1". If this variable is not true, skip the failed test post >> actions. >> >> Signed-off-by: Peter Hoyes <peter.hoyes@arm.com> >> --- >> meta/classes-recipe/testimage.bbclass | 10 ++++++---- >> 1 file changed, 6 insertions(+), 4 deletions(-) > > Is there a reason you can't just set TESTIMAGE_FAILED_QA ARTIFACTS = "” > ? Answering for Peter because I asked the same question: the next step after list_and_fetch_failed_tests_artifacts() is get_target_disk_usage() which does a target.run() so causes a SSH call. Ross
On Thu, 2024-06-27 at 12:34 +0000, Ross Burton wrote: > On 27 Jun 2024, at 13:02, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Wed, 2024-06-26 at 10:41 +0100, Peter Hoyes via > > lists.openembedded.org wrote: > > > It does not always make sense to collect artifacts and data from > > > the > > > target on failure, e.g. if testing firmware or if the target is > > > not > > > running an SSH server. > > > > > > Introduce the variable TESTIMAGE_RUN_FAILURE_POST_ACTIONS, which > > > defaults to "1". If this variable is not true, skip the failed > > > test post > > > actions. > > > > > > Signed-off-by: Peter Hoyes <peter.hoyes@arm.com> > > > --- > > > meta/classes-recipe/testimage.bbclass | 10 ++++++---- > > > 1 file changed, 6 insertions(+), 4 deletions(-) > > > > Is there a reason you can't just set TESTIMAGE_FAILED_QA ARTIFACTS > > = "” > > ? > > Answering for Peter because I asked the same question: the next step > after list_and_fetch_failed_tests_artifacts() is > get_target_disk_usage() which does a target.run() so causes a SSH > call. Rather than call this RUN_FAILURE_POST_ACTIONS could we have something like TESTIMAGE_SSH_AVAILABLE and then test against that in the appropriate places? Cheers, Richard
On 6/27/24 14:31, Richard Purdie wrote: > On Thu, 2024-06-27 at 12:34 +0000, Ross Burton wrote: >> On 27 Jun 2024, at 13:02, Richard Purdie >> <richard.purdie@linuxfoundation.org> wrote: >>> On Wed, 2024-06-26 at 10:41 +0100, Peter Hoyes via >>> lists.openembedded.org wrote: >>>> It does not always make sense to collect artifacts and data from >>>> the >>>> target on failure, e.g. if testing firmware or if the target is >>>> not >>>> running an SSH server. >>>> >>>> Introduce the variable TESTIMAGE_RUN_FAILURE_POST_ACTIONS, which >>>> defaults to "1". If this variable is not true, skip the failed >>>> test post >>>> actions. >>>> >>>> Signed-off-by: Peter Hoyes <peter.hoyes@arm.com> >>>> --- >>>> meta/classes-recipe/testimage.bbclass | 10 ++++++---- >>>> 1 file changed, 6 insertions(+), 4 deletions(-) >>> Is there a reason you can't just set TESTIMAGE_FAILED_QA ARTIFACTS >>> = "” >>> ? >> Answering for Peter because I asked the same question: the next step >> after list_and_fetch_failed_tests_artifacts() is >> get_target_disk_usage() which does a target.run() so causes a SSH >> call. > Rather than call this RUN_FAILURE_POST_ACTIONS could we have something > like TESTIMAGE_SSH_AVAILABLE and then test against that in the > appropriate places? > > Cheers, > > Richard In our test suite at least, SSH is sometimes available and we use SSH where it makes sense. We just can't guarantee that SSH is available in all failure scenarios (e.g. if the firmware failed to boot), and many of our test cases run pre-Linux shell. Peter
diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass index ed0d87b7a7..599a5563a5 100644 --- a/meta/classes-recipe/testimage.bbclass +++ b/meta/classes-recipe/testimage.bbclass @@ -18,9 +18,11 @@ inherit image-artifact-names TESTIMAGE_AUTO ??= "0" -# When any test fails, TESTIMAGE_FAILED_QA ARTIFACTS will be parsed and for -# each entry in it, if artifact pointed by path description exists on target, -# it will be retrieved onto host +# When any test fails, if TESTIMAGE_RUN_FAILURE_POST_ACTIONS equals "1", +# TESTIMAGE_FAILED_QA ARTIFACTS will be parsed and for each entry in it, if artifact +# pointed by path description exists on target, it will be retrieved onto host + +TESTIMAGE_RUN_FAILURE_POST_ACTIONS ??= "1" TESTIMAGE_FAILED_QA_ARTIFACTS = "\ ${localstatedir}/log \ @@ -367,7 +369,7 @@ def testimage_main(d): pass results = tc.runTests() complete = True - if results.hasAnyFailingTest(): + if oe.types.boolean(d.getVar('TESTIMAGE_RUN_FAILURE_POST_ACTIONS')) and results.hasAnyFailingTest(): run_failed_tests_post_actions(d, tc) except (KeyboardInterrupt, BlockingIOError) as err: if isinstance(err, KeyboardInterrupt):
It does not always make sense to collect artifacts and data from the target on failure, e.g. if testing firmware or if the target is not running an SSH server. Introduce the variable TESTIMAGE_RUN_FAILURE_POST_ACTIONS, which defaults to "1". If this variable is not true, skip the failed test post actions. Signed-off-by: Peter Hoyes <peter.hoyes@arm.com> --- meta/classes-recipe/testimage.bbclass | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)