mbox series

[layerindex-web,0/4] Speed up the RRS pages, and test them

Message ID 20261116222410.35784-1-tim.orling@konsulko.com
Headers show
Series Speed up the RRS pages, and test them | expand

Message

Tim Orling Nov. 16, 2026, 10:24 p.m. UTC
The RRS recipe list, report, maintainers and recipe detail pages ran
queries per recipe, maintainer or upgrade: about 5,000 for the OE-Core
recipe list, which took 4 s, and over 2,500 for linux-yocto's detail
page. The database itself was busy for under 0.5 s of that, and the
lookups already use indexes, so these batch the queries instead.

1: RRS views had no tests, because the test database is built without
   the app. Add tests/rrs/, run by docker/run-tests.sh in a second pass
   with ENABLE_RRS=1 and --nomigrations (the RRS data migrations need the
   core layer first), plus a check for model changes without a
   migration. RRS stays optional: without it tests/rrs/ isn't collected.
2-4: Batch the queries of the recipe list and report, the maintainers
   page and the recipe detail page. Each comes with tests of the page
   data and of the query count, which fail without the change.

On a copy of the production database on a test bench instance, the page
data before and after is identical for 12 recipe list milestones, 9
maintainers pages and 153 recipe detail pages:

  Page (OE-Core 6.2)             Before    After
  recipes/                       4.3 s     0.32 s
  recipesreport/                 3.7 s     0.12 s
  maintainers/                   1.17 s    0.17 s
  recipedetail/ (linux-yocto)    3.7 s     0.30 s

Building the meta-openembedded recipe list goes from 9 s to 0.3 s.

This applies on master-next: patch 1 changes the lines of
docker/run-tests.sh that "requirements-dev.txt: pin the development and
test tools" changed:
https://lore.kernel.org/all/20261114070852.41000-1-piotr@qbee.io/

Tim Orling (4):
  docker/run-tests.sh: run tests for the RRS app with RRS enabled
  rrs/views.py: fetch the recipe list's patch counts in a few queries
  rrs/views.py: fetch the maintainers page's statistics in a few queries
  rrs/views.py: fetch the recipe detail page's upgrades in a few queries

 README.devel.md                 |   9 ++
 docker/run-tests.sh             |   9 +-
 rrs/views.py                    | 171 +++++++++++++++++++++++++++-----
 tests/rrs/conftest.py           |  52 ++++++++++
 tests/rrs/test_maintainers.py   |  94 ++++++++++++++++++
 tests/rrs/test_recipe_detail.py |  84 ++++++++++++++++
 tests/rrs/test_recipe_list.py   |  71 +++++++++++++
 tests/rrs/test_views.py         |  52 ++++++++++
 8 files changed, 514 insertions(+), 28 deletions(-)
 create mode 100644 tests/rrs/conftest.py
 create mode 100644 tests/rrs/test_maintainers.py
 create mode 100644 tests/rrs/test_recipe_detail.py
 create mode 100644 tests/rrs/test_recipe_list.py
 create mode 100644 tests/rrs/test_views.py