mbox series

[v3,0/4] add failed tests artifacts retriever

Message ID 20230609064802.11777-1-alexis.lothore@bootlin.com
Headers show
Series add failed tests artifacts retriever | expand

Message

Alexis Lothoré June 9, 2023, 6:47 a.m. UTC 
From: Alexis Lothoré <alexis.lothore@bootlin.com>

This series is a proposal to bring in an "artifact retriever" to ease
debugging when some runtime tests fails. This is a follow-up to
the v2 ([1]) and its corresponding RFC ([2]), which in turn is
a proposal to address general debugging issues like [3]

The main change is based on Alexander's suggestion to defer ptests
artifacts list in core-image-ptest.bb, which as a consequence will
automatically filter out succeeding ptests artifacts.
For this new revision which now relies on multiconfig, I stumbled upon
issues because of PTEST_EXPECT_FAILURE = "1" in core-image-ptest.bb. From
my understanding, this is used with unittest.expectFailure to prevent
tests suffering from intermittent failures from making general test session
fail systematically. To keep this behaviour, a new helper is needed to know
about those failing tests which are not accounted.

Changes since v2:
- add helper to know about any tolerated ptest failure
- allow to retrieve ptests artifacts only for failing ptests, by benefiting
  from the multiconfig feature in ptest images

Changes since v1:
- remove legacy scp option
- put back target stop in 'finally' clause
- retrieve artifacts only in nominal target run (ie no exception from
  ssh/qemu target run)
- list artifacts directly in variable instead of using intermediate file
- use standard variables in artifacts paths
- allow glob patterns in artifacts paths
- expand artifacts path on target before tryiong to retrieve them

[1] https://lore.kernel.org/openembedded-core/20230607083015.20760-1-alexis.lothore@bootlin.com/
[2] https://lore.kernel.org/openembedded-core/20230523161619a8c871d9@mail.local/T/#t
[3] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901

Alexis Lothoré (4):
  oeqa/core/runner: add helper to know about expected failures
  oeqa/target/ssh: update options for SCP
  testimage: implement test artifacts retriever for failing tests
  core-image-ptest: append ptest directory to artifacts list

 meta/classes-recipe/testimage.bbclass        | 48 ++++++++++++++++++++
 meta/lib/oeqa/core/runner.py                 |  4 ++
 meta/lib/oeqa/core/target/ssh.py             |  5 +-
 meta/recipes-core/images/core-image-ptest.bb |  1 +
 4 files changed, 57 insertions(+), 1 deletion(-)