mbox series

[00/15] Bitbake server thread enabling

Message ID 20221229170728.880367-1-richard.purdie@linuxfoundation.org
Headers show
Series Bitbake server thread enabling | expand

Message

Richard Purdie Dec. 29, 2022, 5:07 p.m. UTC 
This series adds 'idle' thread support to bitbake. These changes have
struggled a bit in testing as we're trying to add threading support to
code which never had it originally. We've been aiming at this for
some time (over a decade) but have reached the point where we need to 
do it and deal with any issues arrising.

The 'idle' thread is badly named as it is where the "real work"
of bitbake gets done, e.g. runqueue (task execution) is run as an 
idle handler. We can worry about renaming things later, getting the
basic changes working is my concern right now.

The problem these changes solve:
  * allows the UI to tell if the server is alive and still there
    even when async commands are running
  * allows the UI to interrupt the server and tell it to shutdown
    (i.e. make Ctrl+C repsonsive from the UI)
  * allow us to make progress on the various "bitbake hanging" bugs
    we have where the UI can't connect

I believe the changes are at the point where they should be able to 
merge. The series is a bit messy as the changes build on each other
and there are various fixes for various pieces of code along the way.
I could try and separate them into a different order but they're 
intertwined and I think the current form shows the issues each change
addresses which would be lost if it were restructured.

There is much more work needed here and the code can be considerably
cleaned up to make it clearer and most accessible, those changes
can follow and build from this though.

Richard Purdie (15):
  command: Tweak finishAsyncCommand ordering to avoid races
  cooker: Ensure commands clean up any parser processes
  knotty: Ping the server/cooker periodically
  event: Add enable/disable heartbeat code
  server/process: Run idle commands in a separate idle thread
  server/process: Improve idle loop exit code
  process: Improve client disconnect/idle sync
  process: Improve async command execution with idle interaction
  knotty: Avoid looping with tracebacks
  event: Always use threadlock
  server/process: Improve exception logging
  cooker/cookerdata: Rework the way the datastores are reset
  cooker: Ensure we clean up active idle handlers
  cache: Drop reciever side counting for SiggenRecipeInfo
  server/process: Add debug to show which handlers are active

 lib/bb/cache.py               |  15 ++---
 lib/bb/command.py             |  27 ++++-----
 lib/bb/cooker.py              |  58 ++++++++++++--------
 lib/bb/cookerdata.py          |  31 +++++++----
 lib/bb/event.py               |  82 +++++++++++++++-------------
 lib/bb/server/process.py      | 100 ++++++++++++++++++++++++----------
 lib/bb/server/xmlrpcserver.py |   2 +-
 lib/bb/tests/event.py         |  17 +-----
 lib/bb/ui/knotty.py           |  16 ++++--
 9 files changed, 206 insertions(+), 142 deletions(-)