mbox series

[bitbake-devel,v5,00/22] Bitbake Hash Server WebSockets, Alternate Database Backend, and User Management

Message ID 20231101154216.2758185-1-JPEWhacker@gmail.com
Headers show
Series Bitbake Hash Server WebSockets, Alternate Database Backend, and User Management | expand

Message

Joshua Watt Nov. 1, 2023, 3:41 p.m. UTC 
This patch series reworks the bitbake asyncrpc API to add a WebSockets
implementation for both the client and server. The hash equivalence
server is updated to allow using this new API (the PR server can also be
updated in the future if desired).

In addition, the database backed for the hash equivalence server is
abstracted so that sqlalchemy can optionally be used instead of sqlite.
This allows using "big metal" databases as the backend, which allows the
hash equivalence server to scale to a large number of queries.

Note that both websockets and sqlalchemy require 3rd party python
modules to function. However, these modules are optional unless the user
desires to use the APIs.

Also, user management is added. This allows user accounts to be
registered with the server and users can be given permissions to do
certain operations on the server. Users are not (necessarily) required
to login to access the server, as permissions can granted to anonymous
users. The default permissions will give anonymous users the same
permissions that they would have before user accounts were added so as
to retain backward compatibility, but server admins will likely want to
change this.

V3: Remove RFC status; patches are ready for review
V4: Fixed protocol breakage with mixing older and newer clients/servers
V5: Fixed compatibility with Python 3.8

Joshua Watt (22):
  asyncrpc: Abstract sockets
  hashserv: Add websocket connection implementation
  asyncrpc: Add context manager API
  hashserv: tests: Add external database tests
  asyncrpc: Prefix log messages with client info
  bitbake-hashserv: Allow arguments from environment
  hashserv: Abstract database
  hashserv: Add SQLalchemy backend
  hashserv: Implement read-only version of "report" RPC
  asyncrpc: Add InvokeError
  asyncrpc: client: Prevent double closing of loop
  asyncrpc: client: Add disconnect API
  hashserv: Add user permissions
  hashserv: Add become-user API
  hashserv: Add db-usage API
  hashserv: Add database column query API
  hashserv: test: Add bitbake-hashclient tests
  bitbake-hashclient: Output stats in JSON format
  bitbake-hashserver: Allow anonymous permissions to be space separated
  hashserv: tests: Allow authentication for external server tests
  hashserv: Allow self-service deletion
  hashserv: server: Add owner if user is logged in

 bin/bitbake-hashclient        | 145 +++++-
 bin/bitbake-hashserv          | 132 ++++-
 lib/bb/asyncrpc/__init__.py   |  33 +-
 lib/bb/asyncrpc/client.py     | 120 ++---
 lib/bb/asyncrpc/connection.py | 146 ++++++
 lib/bb/asyncrpc/exceptions.py |  21 +
 lib/bb/asyncrpc/serv.py       | 359 ++++++++-----
 lib/hashserv/__init__.py      | 190 +++----
 lib/hashserv/client.py        | 147 +++++-
 lib/hashserv/server.py        | 951 +++++++++++++++++++++-------------
 lib/hashserv/sqlalchemy.py    | 427 +++++++++++++++
 lib/hashserv/sqlite.py        | 408 +++++++++++++++
 lib/hashserv/tests.py         | 736 +++++++++++++++++++++++++-
 lib/prserv/client.py          |   8 +-
 lib/prserv/serv.py            |  37 +-
 15 files changed, 3053 insertions(+), 807 deletions(-)
 create mode 100644 lib/bb/asyncrpc/connection.py
 create mode 100644 lib/bb/asyncrpc/exceptions.py
 create mode 100644 lib/hashserv/sqlalchemy.py
 create mode 100644 lib/hashserv/sqlite.py