| Message ID | 20231103142640.1936827-1-JPEWhacker@gmail.com |
|---|---|
| Headers | show |
| Series | Bitbake Hash Server WebSockets, Alternate Database Backend, and User Management | expand |
Hello, I missed it earlier but I got this: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/5673/steps/12/logs/stdio On 03/11/2023 08:26:18-0600, Joshua Watt wrote: > This patch series reworks the bitbake asyncrpc API to add a WebSockets > implementation for both the client and server. The hash equivalence > server is updated to allow using this new API (the PR server can also be > updated in the future if desired). > > In addition, the database backed for the hash equivalence server is > abstracted so that sqlalchemy can optionally be used instead of sqlite. > This allows using "big metal" databases as the backend, which allows the > hash equivalence server to scale to a large number of queries. > > Note that both websockets and sqlalchemy require 3rd party python > modules to function. However, these modules are optional unless the user > desires to use the APIs. > > Also, user management is added. This allows user accounts to be > registered with the server and users can be given permissions to do > certain operations on the server. Users are not (necessarily) required > to login to access the server, as permissions can granted to anonymous > users. The default permissions will give anonymous users the same > permissions that they would have before user accounts were added so as > to retain backward compatibility, but server admins will likely want to > change this. > > V3: Remove RFC status; patches are ready for review > V4: Fixed protocol breakage with mixing older and newer clients/servers > V5: Fixed compatibility with Python 3.8 > V6: Fixed protocol incompatibility when exiting stream state that broke > mixing older and new clients/servers > > Joshua Watt (22): > asyncrpc: Abstract sockets > hashserv: Add websocket connection implementation > asyncrpc: Add context manager API > hashserv: tests: Add external database tests > asyncrpc: Prefix log messages with client info > bitbake-hashserv: Allow arguments from environment > hashserv: Abstract database > hashserv: Add SQLalchemy backend > hashserv: Implement read-only version of "report" RPC > asyncrpc: Add InvokeError > asyncrpc: client: Prevent double closing of loop > asyncrpc: client: Add disconnect API > hashserv: Add user permissions > hashserv: Add become-user API > hashserv: Add db-usage API > hashserv: Add database column query API > hashserv: test: Add bitbake-hashclient tests > bitbake-hashclient: Output stats in JSON format > bitbake-hashserver: Allow anonymous permissions to be space separated > hashserv: tests: Allow authentication for external server tests > hashserv: Allow self-service deletion > hashserv: server: Add owner if user is logged in > > bin/bitbake-hashclient | 145 +++++- > bin/bitbake-hashserv | 132 ++++- > lib/bb/asyncrpc/__init__.py | 33 +- > lib/bb/asyncrpc/client.py | 120 ++--- > lib/bb/asyncrpc/connection.py | 146 ++++++ > lib/bb/asyncrpc/exceptions.py | 21 + > lib/bb/asyncrpc/serv.py | 365 ++++++++----- > lib/hashserv/__init__.py | 190 +++---- > lib/hashserv/client.py | 147 +++++- > lib/hashserv/server.py | 952 +++++++++++++++++++++------------- > lib/hashserv/sqlalchemy.py | 427 +++++++++++++++ > lib/hashserv/sqlite.py | 408 +++++++++++++++ > lib/hashserv/tests.py | 736 +++++++++++++++++++++++++- > lib/prserv/client.py | 8 +- > lib/prserv/serv.py | 37 +- > 15 files changed, 3060 insertions(+), 807 deletions(-) > create mode 100644 lib/bb/asyncrpc/connection.py > create mode 100644 lib/bb/asyncrpc/exceptions.py > create mode 100644 lib/hashserv/sqlalchemy.py > create mode 100644 lib/hashserv/sqlite.py > > -- > 2.34.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#15421): https://lists.openembedded.org/g/bitbake-devel/message/15421 > Mute This Topic: https://lists.openembedded.org/mt/102364903/3617179 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
This patch series reworks the bitbake asyncrpc API to add a WebSockets implementation for both the client and server. The hash equivalence server is updated to allow using this new API (the PR server can also be updated in the future if desired). In addition, the database backed for the hash equivalence server is abstracted so that sqlalchemy can optionally be used instead of sqlite. This allows using "big metal" databases as the backend, which allows the hash equivalence server to scale to a large number of queries. Note that both websockets and sqlalchemy require 3rd party python modules to function. However, these modules are optional unless the user desires to use the APIs. Also, user management is added. This allows user accounts to be registered with the server and users can be given permissions to do certain operations on the server. Users are not (necessarily) required to login to access the server, as permissions can granted to anonymous users. The default permissions will give anonymous users the same permissions that they would have before user accounts were added so as to retain backward compatibility, but server admins will likely want to change this. V3: Remove RFC status; patches are ready for review V4: Fixed protocol breakage with mixing older and newer clients/servers V5: Fixed compatibility with Python 3.8 V6: Fixed protocol incompatibility when exiting stream state that broke mixing older and new clients/servers Joshua Watt (22): asyncrpc: Abstract sockets hashserv: Add websocket connection implementation asyncrpc: Add context manager API hashserv: tests: Add external database tests asyncrpc: Prefix log messages with client info bitbake-hashserv: Allow arguments from environment hashserv: Abstract database hashserv: Add SQLalchemy backend hashserv: Implement read-only version of "report" RPC asyncrpc: Add InvokeError asyncrpc: client: Prevent double closing of loop asyncrpc: client: Add disconnect API hashserv: Add user permissions hashserv: Add become-user API hashserv: Add db-usage API hashserv: Add database column query API hashserv: test: Add bitbake-hashclient tests bitbake-hashclient: Output stats in JSON format bitbake-hashserver: Allow anonymous permissions to be space separated hashserv: tests: Allow authentication for external server tests hashserv: Allow self-service deletion hashserv: server: Add owner if user is logged in bin/bitbake-hashclient | 145 +++++- bin/bitbake-hashserv | 132 ++++- lib/bb/asyncrpc/__init__.py | 33 +- lib/bb/asyncrpc/client.py | 120 ++--- lib/bb/asyncrpc/connection.py | 146 ++++++ lib/bb/asyncrpc/exceptions.py | 21 + lib/bb/asyncrpc/serv.py | 365 ++++++++----- lib/hashserv/__init__.py | 190 +++---- lib/hashserv/client.py | 147 +++++- lib/hashserv/server.py | 952 +++++++++++++++++++++------------- lib/hashserv/sqlalchemy.py | 427 +++++++++++++++ lib/hashserv/sqlite.py | 408 +++++++++++++++ lib/hashserv/tests.py | 736 +++++++++++++++++++++++++- lib/prserv/client.py | 8 +- lib/prserv/serv.py | 37 +- 15 files changed, 3060 insertions(+), 807 deletions(-) create mode 100644 lib/bb/asyncrpc/connection.py create mode 100644 lib/bb/asyncrpc/exceptions.py create mode 100644 lib/hashserv/sqlalchemy.py create mode 100644 lib/hashserv/sqlite.py