mbox series

[v2,0/2] add failed tests artifacts retriever

Message ID 20230607083015.20760-1-alexis.lothore@bootlin.com
Headers show
Series add failed tests artifacts retriever | expand

Message

Alexis Lothoré June 7, 2023, 8:30 a.m. UTC
This series is a proposal to bring in an "artifact retriever" to ease
debugging when some runtime tests fails. This is a follow-up to
the initial version ([1]) and its corresponding RFC ([2]), which in turn is
a proposal to address general debugging issues like [3]

In the proposed form the retriever is pretty simple/dumb: it waits for all
tests listed for a testimage run to be done, and if any of those tests has
failed, it tries to read a list of "artifacts of interest to retrieve", and
pulls those files onto the host system (next to testresults.json) for
further analysis. This is true for ALL runtime tests. So for example, a
failing test in a very basic test session running ping, ssh and ptests will
trigger artifacts retrieval, the failing test being either a ping test, a
ssh test or a ptest.
The artifacts list is provided in a new bitbake
TESTIMAGE_FAILED_QA_ARTIFACTS.
There is one single artifacts list to be provided for a whole test run (i.e
a run done with bitbake -c testimage). Artifacts list can then be overriden
by affecting the variable mentioned above.
Retrieved files are pulled through scp to allow compatibility for both
Qemu and SSH targets, and are currently stored "as is"
(unarchived/uncompressed) in tmp/log/oeqa/<image>/artifacts.

The series has been tested with the following process:
- add dummy failing ptest in lttng-modules through a custom patch in
  meta/recipes-kernel/lttng
- build core-image-minimal image with:
  - DISTRO_FEATURES:append = " ptest"
  - CORE_IMAGE_EXTRA_INSTALL += "dropbear lttng-tools-ptest busybox-ptest"
  - TEST_SUITES = "ping ssh ptest"
  - IMAGE_CLASSES += "testimage"
  - TESTIMAGE_FAILED_QA_ARTIFACTS="${libdir}/*/ptest /etc/version /etc/os-release /var/log"
- run tests: bitbake core-image-minimal -c testimage
- ensure artifacts are properly retrieved and stored

Changes since v1:
- remove legacy scp option
- put back target stop in 'finally' clause
- retrieve artifacts only in nominal target run (ie no exception from
  ssh/qemu target run)
- list artifacts directly in variable instead of using intermediate file
- use standard variables in artifacts paths
- allow glob patterns in artifacts paths
- expand artifacts path on target before tryiong to retrieve them

[1] https://lore.kernel.org/openembedded-core/20230602095037.97981-1-alexis.lothore@bootlin.com/
[2] https://lore.kernel.org/openembedded-core/20230523161619a8c871d9@mail.local/T/#t
[3] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901

Alexis Lothoré (2):
  oeqa/target/ssh: update options for SCP
  testimage: implement test artifacts retriever for failing tests

 meta/classes-recipe/testimage.bbclass | 49 +++++++++++++++++++++++++++
 meta/lib/oeqa/core/target/ssh.py      |  5 ++-
 2 files changed, 53 insertions(+), 1 deletion(-)