mbox series

[bitbake-devel,RFC,v2,00/12] Bitbake Hash Server WebSockets and Alternate Database Backend

Message ID 20231003145249.1166276-1-JPEWhacker@gmail.com
Headers show
Series Bitbake Hash Server WebSockets and Alternate Database Backend | expand

Message

Joshua Watt Oct. 3, 2023, 2:52 p.m. UTC 
This patch series reworks the bitbake asyncrpc API to add a WebSockets
implementation for both the client and server. The hash equivalence
server is updated to allow using this new API (the PR server can also be
updated in the future if desired).

In addition, the database backed for the hash equivalence server is
abstracted so that sqlalchemy can optionally be used instead of sqlite.
This allows using "big metal" databases as the backend, which allows the
hash equivalence server to scale to a large number of queries.

Note that both websockets and sqlalchemy require 3rd party python
modules to function. However, these modules are optional unless the user
desires to use the APIs.



Joshua Watt (12):
  asyncrpc: Abstract sockets
  hashserv: Add remove API
  bitbake-hashclient: Add remove subcommand
  hashserv: Add websocket connection implementation
  asyncrpc: Add context manager API
  hashserv: tests: Add external database tests
  asyncrpc: Prefix log messages with client info
  bitbake-hashserv: Allow arguments from environment
  hashserv: Abstract database
  hashserv: Add SQLalchemy backend
  contrib: Update hashserv Dockerfile
  contrib: hashserv: Add docker-compose

 bitbake/bin/bitbake-hashclient               |  49 +-
 bitbake/bin/bitbake-hashserv                 |  87 ++-
 bitbake/contrib/hashserv/Dockerfile          |  33 +-
 bitbake/contrib/hashserv/docker-compose      |  11 +
 bitbake/contrib/hashserv/docker-compose.yaml |  36 ++
 bitbake/contrib/hashserv/requirements.txt    |   5 +
 bitbake/lib/bb/asyncrpc/__init__.py          |  32 +-
 bitbake/lib/bb/asyncrpc/client.py            | 100 ++--
 bitbake/lib/bb/asyncrpc/connection.py        | 146 ++++++
 bitbake/lib/bb/asyncrpc/exceptions.py        |  17 +
 bitbake/lib/bb/asyncrpc/serv.py              | 349 +++++++-----
 bitbake/lib/hashserv/__init__.py             | 137 ++---
 bitbake/lib/hashserv/client.py               |  40 +-
 bitbake/lib/hashserv/server.py               | 525 +++++++------------
 bitbake/lib/hashserv/sqlalchemy.py           | 268 ++++++++++
 bitbake/lib/hashserv/sqlite.py               | 229 ++++++++
 bitbake/lib/hashserv/tests.py                | 123 ++++-
 bitbake/lib/prserv/client.py                 |   8 +-
 bitbake/lib/prserv/serv.py                   |  37 +-
 19 files changed, 1495 insertions(+), 737 deletions(-)
 create mode 100755 bitbake/contrib/hashserv/docker-compose
 create mode 100644 bitbake/contrib/hashserv/docker-compose.yaml
 create mode 100644 bitbake/contrib/hashserv/requirements.txt
 create mode 100644 bitbake/lib/bb/asyncrpc/connection.py
 create mode 100644 bitbake/lib/bb/asyncrpc/exceptions.py
 create mode 100644 bitbake/lib/hashserv/sqlalchemy.py
 create mode 100644 bitbake/lib/hashserv/sqlite.py