mbox series

[00/14] oeqa runtime tests when qemu hangs

Message ID 20230120144450.2913935-1-mikko.rapeli@linaro.org
Headers show
Series oeqa runtime tests when qemu hangs | expand

Message

Mikko Rapeli Jan. 20, 2023, 2:44 p.m. UTC 
I get a qemu hang on kirkstone, swtpm and optee. One of the
optee-test/xtest hangs the qemu machine in some kind of deadlock.
While this needs to be debugged and tested, the oeqa runtime tests
also hanged and never returned. Thus this patch set. With these changes
qemu deadlock is detected and with do_testimage() task eventually exits
with all correct tests failing and the hangin qemu system killed.
There are a lot of debug prints added by this patch set but I don't of
any other way to debug complex python code. strace output from the hang
doesn't tell where the deadlock happened.

Tested on kirkstone and cherry-picked to master. If something blows up
I'll do more testing on master branch based setup.

Mikko Rapeli (14):
  oeqa ssh.py: move output prints to new line
  oeqa ssh.py: fix hangs in run() and add debug prints
  oeqa ssh.py: clarify timeout API and add more debug prints to run()
  oeqa ssh.py: add connection keep alive options to ssh client
  testimage.bbclass: add debug prints for major state changes
  oeqa dump.py: add error counter and stop after 5 failures
  oeqa qemurunner.py: use os.set_blocking() instead of fcntl()
  oeqa qemurunner: read more data at a time from serial and log all of
    it
  oeqa qemurunner.py: increase boot failure warning log from 25 to 100
    lines
  oeqa qemurunner.py: add debug prints to stop()
  oeqa qemurunner.py: add timeout to QMP calls
  oeqa qemurunner.py: kill qemu if it hangs
  oeqa qemurunner.py: add debug output to run_serial()
  oeqa qemurunner.py: improve logging in run_serial()

 meta/classes-recipe/testimage.bbclass |  8 ++++
 meta/lib/oeqa/core/target/ssh.py      | 63 +++++++++++++++++++++------
 meta/lib/oeqa/utils/dump.py           | 23 +++++++++-
 meta/lib/oeqa/utils/qemurunner.py     | 41 ++++++++++++++---
 4 files changed, 113 insertions(+), 22 deletions(-)