From patchwork Mon Nov 16 22:24:07 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104556 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.5213.1794867866985231752 for ; Mon, 16 Nov 2026 14:24:27 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=pq/BbiLG; spf=pass (domain: gmail.com, ip: 209.85.214.182, mailfrom: ticotimo@gmail.com) Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2efde3200c7so87895ad.2 for ; Mon, 16 Nov 2026 14:24:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794867866; x=1795472666; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/hfxRNEGx67WuwEx33thR5cDN6e6frCYHTHXoet/Y4Q=; b=pq/BbiLGFlcNgLUp/aLEXAZ5E+UlpKm8YdMVrBQH80ZOf/OeJrMDChrFkPfcMy1hte DVoyje3oRhfclGAcn2CfWTVAhJG+a04Jpuwgnk8NxbPi8zAL7kTlJWbC2VloyWcL9nj3 vth6pMeP9XVMQSCm1IboOuw+e5Hh2/rcAruJ4y7C1VpS4sZ+wTWbGFzJvxCQw1dYdVZm 1A6CN6eNYXKPo0H9032zb/IXjJqgmKXLH2iucthchYAJQDVVSWY/ZaVa6LyPL26sz2rf sYC5NWAMkOFr5uWuhTQFtCSNjIhbifzr9uXZH0sjWDCcEyE+Z6MjO+g2ptuABkmr2Xw2 11TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794867866; x=1795472666; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=/hfxRNEGx67WuwEx33thR5cDN6e6frCYHTHXoet/Y4Q=; b=AQ2YRul0ftwsKhhrn2X2Rudy+dMP8ZMlMvA6uoeASjGB2tUCrlPYTzknl5xeUStK31 BbKeYFpCbi6keSTw5amXs0NoPoXlbtBCLOKBcp146HZAlbxqoAaa6dmzw35YDYyro+cZ 2ITViGPdpqw74lrXext0xuuTpvZ3dQkMyZ6dTfl6qRbtoMsa1V5NcaLcuhl3uY2INITx 7wtwK9pg7en6sK9RJNgmwDcuHy+uk8eUkstia5phiBZfNLppWEt6TU9UyxVs2mGJ9Qmk rMus886ZaHF+KykHTwzPs4Fys2lLJChEwm5ylz27tq0fL5FMR/Ffmp0AgK1o1Qnb47jW qKXA== X-Gm-Message-State: APuU4Z+yilbvXg21ujbIo20DojFSaeiTVNiBxBs1iKOM5f+4mc6lE/d9 zYefc0P7ZXEQ9i4xMIBDf/28MxezuL8ZIos6+iiJ5zsbRb86aEPVLu+SmTAldg== X-Gm-Gg: AYBFou0+GaQ2rL5PtOtdf1AredHKV4l+tFurk1Xgu2avbTpLzl+riHJFgoDrKGbxGdR Ve/Jkmv49mzmwEyhG+XTrppouALZWtXBIQb2NGC7OURxKMFeEN3DnsBkSdDA/Wlz3is3xoO+uuh KzXxnmVjKBLZvcywUwFsAYMIG/ywyKubmWXzCLnH7Z5Ws30edEqghdrDHYsbh81i42vFYVwHDrE 7Z025EWiuyN42HAvSiZslaaJrdoVt2gviM8OL2r8PRCthORftTISrdHRqd6ABtFiWnchVs3PTFL oBT6RA9HNXrQ+uqzVentCeN0nZj8eW0xc7JdZRCglzndHkSY6+MxlyHFqygajx2IrN9Lkppd5X3 l7AIW5uQCYcdD9Fy8u9gZgc1xSZ/VcOpt568e8SAAs7p8RtvjyOcPOAd78fALaSED7WCZtJlVup YPzKrU1HZbOASYfwiTGcdmXGkYxv60BnqPjvY4AXhBkCvaZkdAf1JfSsp6pFrQP+MWV0SeE0Ev3 StTIEvCOtUvsFdUigijB7ffG3mTLZos4J4JzZ0g741597gkF0TJZu0/x5yfNFViDGek3AEMLDtG zLhnl6LM02INzMekoYcsDgU2ZZIuWXZ5bo7eisfpkg3ys2hxRilg2VxmVkH7RA== X-Received: by 2002:a17:902:d491:b0:2e9:821a:5527 with SMTP id d9443c01a7336-2ee0da7f5b7mr126617525ad.12.1794867866152; Mon, 16 Nov 2026 14:24:26 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2efd7ab4abcsm4618945ad.70.2026.11.16.14.24.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 16 Nov 2026 14:24:25 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 1/4] docker/run-tests.sh: run tests for the RRS app with RRS enabled Date: Mon, 16 Nov 2026 14:24:07 -0800 Message-ID: <20261116222410.35784-2-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261116222410.35784-1-tim.orling@konsulko.com> References: <20261116222410.35784-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Mon, 16 Nov 2026 22:24:32 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5430 RRS is optional, so its views had no tests: the test database is built without the app. Add tests/rrs/, collected only when RRS is installed, with a fixture for a maintenance plan and smoke tests for the RRS pages, and run it in a second pass with ENABLE_RRS=1. That pass builds the tables with --nomigrations, because the RRS data migrations need the core layer before they run (README.rrs.md), and checks that the RRS models have no changes missing a migration. AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- README.devel.md | 9 +++++++ docker/run-tests.sh | 9 ++++++- tests/rrs/conftest.py | 52 +++++++++++++++++++++++++++++++++++++++++ tests/rrs/test_views.py | 52 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 tests/rrs/conftest.py create mode 100644 tests/rrs/test_views.py diff --git a/README.devel.md b/README.devel.md index 64106d0..ea510a9 100644 --- a/README.devel.md +++ b/README.devel.md @@ -198,6 +198,15 @@ at a MariaDB/MySQL server where your database user may create databases pytest --ignore=tests/test_update.py tests/ ``` +The tests in `tests/rrs/` need the RRS app and are skipped without it. +`docker/run-tests.sh` runs them in a second pass with `ENABLE_RRS=1`. To run +them directly, add `'rrs'` to `INSTALLED_APPS` and build the test database from +the models, since the RRS data migrations need the core layer to exist first: + +```sh +pytest --nomigrations tests/rrs/ +``` + `tests/test_dockersetup.py` tests the files `dockersetup.py` writes. It needs only pytest and PyYAML, and is skipped inside the Docker image: diff --git a/docker/run-tests.sh b/docker/run-tests.sh index 5da56b3..2a22476 100755 --- a/docker/run-tests.sh +++ b/docker/run-tests.sh @@ -27,6 +27,10 @@ done # Runs as root only to prepare the container: tests/test_update.py rewrites # settings.py and bitbake writes its lock file next to it, the update scripts # expect bitbake already fetched, and the tests commit to git and call tree. +# The tests in tests/rrs/ need RRS, so a second pass runs them with +# ENABLE_RRS=1. --nomigrations builds the tables from the models: the RRS +# data migrations need the core layer to exist first (see README.rrs.md), +# which an empty database lacks. docker run --rm --network $NET -u root \ -e SECRET_KEY=ci-only \ -e GIT_AUTHOR_NAME=ci -e GIT_AUTHOR_EMAIL=ci@example.com \ @@ -40,4 +44,7 @@ docker run --rm --network $NET -u root \ su layers -s /bin/sh -c "set -e git clone -q https://git.openembedded.org/bitbake /opt/workdir/https___git_openembedded_org_bitbake pip3 install -q --user --no-warn-script-location -r /src/requirements-dev.txt - python3 -m pytest -rfE -p no:cacheprovider tests/"' + python3 -m pytest -rfE -p no:cacheprovider tests/ + export ENABLE_RRS=1 + python3 manage.py makemigrations --check --dry-run rrs + python3 -m pytest -rfE -p no:cacheprovider --nomigrations tests/rrs/"' diff --git a/tests/rrs/conftest.py b/tests/rrs/conftest.py new file mode 100644 index 0000000..060f2ef --- /dev/null +++ b/tests/rrs/conftest.py @@ -0,0 +1,52 @@ +# layerindex-web - fixtures for the tests that need the RRS app +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT +# +# AI-generated, Claude Opus 5.5 + +# RRS is optional (ENABLE_RRS=1 in docker/settings.py). Without it these tests +# are not collected; docker/run-tests.sh runs them in a second pass with it. + +import datetime +import types + +import pytest +from django.conf import settings + +if 'rrs' not in settings.INSTALLED_APPS: + collect_ignore_glob = ['test_*.py'] + + +@pytest.fixture +def rrs_plan(db): + """A maintenance plan "OE-Core" on the core layer's master branch, with + release 6.2 and its milestone M1 (October to December 2026)""" + from layerindex.models import Branch, LayerBranch, LayerItem + from rrs.models import ( + Maintainer, + MaintenancePlan, + MaintenancePlanLayerBranch, + Milestone, + Release, + ) + + # Created by rrs/migrations/0001_initial.py, which --nomigrations skips + for name in ('No maintainer', 'All'): + Maintainer.objects.get_or_create(name=name) + + branch, _ = Branch.objects.get_or_create(name='master', defaults={'bitbake_branch': 'master'}) + layer = LayerItem.objects.create(name='openembedded-core', status='P', layer_type='A', + summary='Core metadata', description='Core metadata', + vcs_url='https://git.example.com/openembedded-core') + layerbranch = LayerBranch.objects.create(layer=layer, branch=branch) + plan = MaintenancePlan.objects.create(name='OE-Core') + MaintenancePlanLayerBranch.objects.create(plan=plan, layerbranch=layerbranch) + release = Release.objects.create(plan=plan, name='6.2', start_date=datetime.date(2026, 10, 1), + end_date=datetime.date(2027, 4, 30)) + milestone = Milestone.objects.create(release=release, name='M1', + start_date=datetime.date(2026, 10, 1), + end_date=datetime.date(2026, 12, 31)) + return types.SimpleNamespace(layerbranch=layerbranch, plan=plan, release=release, + milestone=milestone) diff --git a/tests/rrs/test_views.py b/tests/rrs/test_views.py new file mode 100644 index 0000000..c494861 --- /dev/null +++ b/tests/rrs/test_views.py @@ -0,0 +1,52 @@ +# layerindex-web - smoke tests for the RRS pages +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT +# +# AI-generated, Claude Opus 5.5 + +import datetime + +import pytest + +from rrs.models import RecipeSymbol, RecipeUpgrade + +PAGES = ['recipes', 'recipesreport', 'maintainers', 'stats'] + + +@pytest.fixture +def busybox(rrs_plan): + symbol = RecipeSymbol.objects.create(layerbranch=rrs_plan.layerbranch, pn='busybox', + summary='Tiny versions of many common UNIX utilities') + date = datetime.datetime(2026, 11, 1) # USE_TZ is False + RecipeUpgrade.objects.create(recipesymbol=symbol, version='1.37.0', author_date=date, + commit_date=date, upgrade_type='U') + return symbol + + +def test_front_page_redirects_to_the_recipe_list(client, rrs_plan): + response = client.get('/rrs/') + assert response.status_code == 302 + assert response['Location'].startswith('/rrs/recipes/OE-Core/') + + +@pytest.mark.parametrize('page', PAGES) +def test_milestone_pages(client, busybox, page): + response = client.get(f'/rrs/{page}/OE-Core/6.2/M1/') + assert response.status_code == 200 + + +def test_recipe_list_shows_the_recipe(client, busybox): + response = client.get('/rrs/recipes/OE-Core/6.2/M1/') + assert [item.name for item in response.context['recipe_list']] == ['busybox'] + + +def test_recipe_detail(client, busybox): + response = client.get(f'/rrs/recipedetail/OE-Core/{busybox.pk}/') + assert response.status_code == 200 + assert b'busybox' in response.content + + +def test_unknown_milestone_is_404(client, rrs_plan): + assert client.get('/rrs/recipes/OE-Core/6.2/M9/').status_code == 404 From patchwork Mon Nov 16 22:24:08 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104557 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.5214.1794867870694909690 for ; Mon, 16 Nov 2026 14:24:30 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=RrnYdfqe; spf=pass (domain: gmail.com, ip: 209.85.214.178, mailfrom: ticotimo@gmail.com) Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2eb18f4da15so17292675ad.1 for ; Mon, 16 Nov 2026 14:24:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794867870; x=1795472670; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=S/46wrWFS3+/RL98Qk2K/MpXBPO2/wX7WoTjtR+1X9U=; b=RrnYdfqePzeAev5hhjwtMzNKp/Ab29N+xSpCUyWjhtI/JIh2VvETP13vObX0/jS66B 2gvSdOj1I6fmdPyJx9j7KzAyw5XUqF/ICHQcENGCZXX+q4o4zW1L0CfRrU728qU0rWLP GvbTHBGN1gKINVAREbBrKMU3yfLzqyXfA+Tqh0vbINUTwn7KgziF+Ajv3YoEMa+OaMni afRy1ewDjZome6p/NypAv6f9kc3X1ixAkXJP0hGLN+ACZ4gyiNbzNlNeWPcDFfzy6ICA eoqoNyFKZiIiJEyTTBkPgp5RXQhsyVuRhRZub10kIW7/hKTgRIG9aVnTeOLRVM9AqjyR ff/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794867870; x=1795472670; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=S/46wrWFS3+/RL98Qk2K/MpXBPO2/wX7WoTjtR+1X9U=; b=BnlNEV7soCaybdRIVtZYW8Zh9+T/xhsQxeNmml7M/uKtGCzm2X8VNXzCKLmwXZ1kzN ZkWWL1xMqC2VKYF658Tt2vxzf6gPxupLzPLkIVgPYjFrhmzbqjBVvRZvStOZYT9j8L6t WDgdhd/W12tLZjYBnbLjnUOGKcxPAg30ghmnh0YGDYxzfOuR9Mae0QvmqfazfBupi5M3 4OESU4GNQSA1VrwxNqfxez5OwK+x7ayM71aaWA6LrHub3t1i+GLx0VsD93V84C0+WDJX 7MP8u2147bmjyoH1SoauYL7u2sJruzkX1Oo4Dd9+YZgxADhtFHjmX81Y4P6VELnVO1+b oFmQ== X-Gm-Message-State: APuU4Z+LNeLnTQ2/2uCT3N10ubx56w/U5h5iuyZjICXiRIkXlHW9PVi4 /K27/69Dnb9XGcKN0h2ZbrWHoDv7E/5Jz++cBxa5DnxHGqFFoRRIsnSik1k9YQ== X-Gm-Gg: AZFdTZfLEzW95gip/5txd0ANyCCAfeYGjGoNzfQ0ivZutO0Kx64pF3eqkPC0ZDHDzWH CTQWAK4exzs7V31dRvBDWOo21TSEK7+OzJBcnh8m1B/liKc70pJX9nl868u9E68xaJw/zbZvXzV soWDxEOhDKl4GXPSu6yzw8R4X6X14HxFo+ZOSsiIxBVskDHLPaWRfrUN5S2PcH81AlxU/dSXDNK 57ozFXIxmYVl6hhFFb9Ks//WeOUYuqO/z2S7kVy3ro8+IGYZUVL7SK82Wxvy/8STuXbdRAXDbeB 2teQRgj3ySv+lwlNV0BmBaDJ0UsIGATVv9fM4MOAt4tcIqHTUDnfHxefNcLH2CFT7+3ykyGusVY qIlIqUm7fi+k9piyJWLyMNtw09S9LUdt+22tZdt5FoXKiS6fLf2satJ/hfjnXLFfeHQiwIkW/r/ 2MhSNbb8OPP5MJo70DEWrQmAxZ5WarHbtYkEbAqMtdO6GDsrx24QefC9iVwv6Y1l6m+LGwky4IN GAf4NgapelXgWkGChvT8mMNBe+m7SLMoa2DBRgxdubtn8S4zbg0zNEgzL3Ln9Xil7nP/bNwwzYG RxV4yZRMUD1lXGOqPDfbb5JlAOhclc4PkXULekK5JXfmnyk+RLc7eRO7/Q+T8w== X-Received: by 2002:a17:903:458d:b0:2dd:ad73:5b6d with SMTP id d9443c01a7336-2ee0dcd040cmr105815845ad.29.1794867869983; Mon, 16 Nov 2026 14:24:29 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2efd7ab4abcsm4618945ad.70.2026.11.16.14.24.28 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 16 Nov 2026 14:24:29 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 2/4] rrs/views.py: fetch the recipe list's patch counts in a few queries Date: Mon, 16 Nov 2026 14:24:08 -0800 Message-ID: <20261116222410.35784-3-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261116222410.35784-1-tim.orling@konsulko.com> References: <20261116222410.35784-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Mon, 16 Nov 2026 22:24:32 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5431 _get_recipe_list() ran five queries per recipe, so the recipes and report pages took about 4 s for OE-Core and 9 s for meta-openembedded, with the database busy for under 0.5 s of that. Fetch the recipe symbols with in_bulk() and the patch counts with one annotated query per layer branch, counting the same recipe as before when several share a pn. On a copy of the production database the recipe lists of 12 milestones are identical before and after, and OE-Core 6.2 takes 9 queries and 0.12 s instead of 4880 queries and 3.7 s. AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- rrs/views.py | 30 ++++++++++----- tests/rrs/test_recipe_list.py | 71 +++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+), 9 deletions(-) create mode 100644 tests/rrs/test_recipe_list.py diff --git a/rrs/views.py b/rrs/views.py index b2520ce..b580818 100644 --- a/rrs/views.py +++ b/rrs/views.py @@ -17,6 +17,7 @@ from django.shortcuts import get_object_or_404 from django.views.generic import TemplateView, ListView, DetailView, RedirectView from django.urls import resolve, reverse, reverse_lazy from django.db import connection +from django.db.models import Count, Q from django.contrib import messages from layerindex.models import Recipe, StaticBuildDep, Patch @@ -432,12 +433,29 @@ def _get_recipe_list(milestone): for ma in maintainers_all: maintainers_dict_all[ma['recipesymbol_id']] = ma['name'] + # Fetch the recipe symbols, and the patch counts of their layer index + # recipes, in a few queries rather than several per recipe + # (AI-generated, Claude Opus 5.5) + recipesymbols = RecipeSymbol.objects.in_bulk([recipe['id'] for recipe in recipes]) + patch_counts = {} + if recipesymbols: + recipeobjs = Recipe.objects.filter( + layerbranch_id__in={rs.layerbranch_id for rs in recipesymbols.values()}, + pn__in={rs.pn for rs in recipesymbols.values()}) \ + .order_by('id') \ + .annotate(patches_total=Count('patch'), + patches_pending=Count('patch', filter=Q(patch__status='P'))) + for recipeobj in recipeobjs: + # Keep the first recipe by id for a pn, as .first() did + patch_counts.setdefault((recipeobj.layerbranch_id, recipeobj.pn), + (recipeobj.patches_total, recipeobj.patches_pending)) + for recipe in recipes: upstream_version = '' upstream_status = '' no_update_reason = '' outdated = '' - recipesymbol = RecipeSymbol.objects.filter(id=recipe['id']).first() + recipesymbol = recipesymbols.get(recipe['id']) if recipe_upstream_history: recipe_upstream = recipe_upstream_dict_all.get(recipe['id']) @@ -479,14 +497,8 @@ def _get_recipe_list(milestone): recipe_list_item.upstream_status = upstream_status recipe_list_item.upstream_version = upstream_version recipe_list_item.outdated = outdated - recipeobj = recipesymbol.layerbranch.recipe_set.filter(pn=recipesymbol.pn, layerbranch=recipesymbol.layerbranch).first() - if recipeobj: - patches = recipeobj.patch_set - recipe_list_item.patches_total = patches.count() - recipe_list_item.patches_pending = patches.filter(status='P').count() - else: - recipe_list_item.patches_total = 0 - recipe_list_item.patches_pending = 0 + recipe_list_item.patches_total, recipe_list_item.patches_pending = \ + patch_counts.get((recipesymbol.layerbranch_id, recipesymbol.pn), (0, 0)) recipe_list_item.maintainer_name = maintainer_name recipe_list_item.no_update_reason = no_update_reason recipe_list.append(recipe_list_item) diff --git a/tests/rrs/test_recipe_list.py b/tests/rrs/test_recipe_list.py new file mode 100644 index 0000000..95b2595 --- /dev/null +++ b/tests/rrs/test_recipe_list.py @@ -0,0 +1,71 @@ +# layerindex-web - tests for the RRS recipe list +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT +# +# AI-generated, Claude Opus 5.5 + +# The recipe list shows each recipe's patch counts, and must get them without +# running queries per recipe. + +import datetime + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from layerindex.models import Patch, Recipe +from rrs.models import RecipeSymbol, RecipeUpgrade + +URL = '/rrs/recipes/OE-Core/6.2/M1/' + + +def add_recipe(layerbranch, pn, patch_statuses=(), recipe=True): + """A recipe that RRS lists for the milestone, with patches of the given statuses""" + symbol = RecipeSymbol.objects.create(layerbranch=layerbranch, pn=pn, summary=pn) + date = datetime.datetime(2026, 11, 1) # USE_TZ is False + RecipeUpgrade.objects.create(recipesymbol=symbol, version='1.0', author_date=date, + commit_date=date, upgrade_type='U') + if recipe: + obj = Recipe.objects.create(layerbranch=layerbranch, filename=f'{pn}_1.0.bb', + filepath='recipes', pn=pn, pv='1.0') + for i, status in enumerate(patch_statuses): + Patch.objects.create(recipe=obj, path=f'{pn}-{i}.patch', + src_path=f'{pn}-{i}.patch', status=status) + return symbol + + +def recipe_list(client): + response = client.get(URL) + assert response.status_code == 200 + return {item.name: item for item in response.context['recipe_list']} + + +def test_patch_counts(client, rrs_plan): + layerbranch = rrs_plan.layerbranch + add_recipe(layerbranch, 'busybox', ['P', 'P', 'A']) + add_recipe(layerbranch, 'zlib', []) + add_recipe(layerbranch, 'norecipe', recipe=False) + # A second recipe with the same pn: the first one by id is counted + add_recipe(layerbranch, 'gcc', ['P']) + later = Recipe.objects.create(layerbranch=layerbranch, filename='gcc_2.0.bb', + filepath='recipes', pn='gcc', pv='2.0') + Patch.objects.create(recipe=later, path='x.patch', src_path='x.patch', status='P') + Patch.objects.create(recipe=later, path='y.patch', src_path='y.patch', status='P') + + counts = {name: (item.patches_total, item.patches_pending) + for name, item in recipe_list(client).items()} + + assert counts == {'busybox': (3, 2), 'zlib': (0, 0), 'norecipe': (0, 0), 'gcc': (1, 1)} + + +def test_queries_do_not_grow_with_recipes(client, rrs_plan): + add_recipe(rrs_plan.layerbranch, 'recipe-0', ['P']) + recipe_list(client) # fills the per-process caches first + with CaptureQueriesContext(connection) as one: + assert len(recipe_list(client)) == 1 + for i in range(1, 6): + add_recipe(rrs_plan.layerbranch, f'recipe-{i}', ['P']) + with CaptureQueriesContext(connection) as six: + assert len(recipe_list(client)) == 6 + assert len(six) == len(one) From patchwork Mon Nov 16 22:24:09 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104558 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.5326.1794867873830920147 for ; Mon, 16 Nov 2026 14:24:33 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=Xq/OpMA+; spf=pass (domain: gmail.com, ip: 209.85.214.180, mailfrom: ticotimo@gmail.com) Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2ed0890476fso31690545ad.0 for ; Mon, 16 Nov 2026 14:24:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794867873; x=1795472673; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=CWeZ812I+YU5VXnKnv0OHTOJ/JcosAl5H4QM/CTZEBI=; b=Xq/OpMA+UpifZ2gNPXNuhw+SUnLbDR+bWVYCKz5lH6D+kaWohSRGuPNNv4J+McvXzf b7szwBcdBSZGbu9xTvvAXjPcPzdbLaZWPdis4Aaq3ozsc/cdQ784LIvKAaL+G5kNAA5X 6hyqh8cXDaVWSzNSimAKF10O7jYGcgyqlNYKwC2Jo99t9i16vt5qdp91QPWJykd6fGzt tKM4Wrf/CgX1GV/uxoLvChj4NQlP6SqCMaFD6tk2mVgfK3bkzSk6jtvoxxvj0FrZGh58 Ml83J0ORpeKPGd4f9kIQjjM8TeKf6/N7aCTjzqvzJVu7JqllxYBTAqZ/admZ8okcmZtU wu3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794867873; x=1795472673; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=CWeZ812I+YU5VXnKnv0OHTOJ/JcosAl5H4QM/CTZEBI=; b=nInWSNJMi6ziSeUQnLdBDbzDEdsDXpttHmuedw2zgIA6YnkRcl8hGbw0hx/L8wwOal y6RKazSQVegk7G4ds3451TIabZNiRFB0hHOpOdy1K9nuG0voV9MQZnLklZCJqPJfEvn+ aJAnCUeEsiFmpa8BX/e9y/wnleRU8Jta+REIDxpa9+vCJmMNtjA4a6l6iYBzZpn/gvES fk+9jEbWdxHCnCF8p6rkWGZHiguVIyFdxcb8zqHYm515lACaLvdvP0mBbL0EXBTUMalq 2VIN7/xf638/6FES0pSy/OWzUJIHon4QlUho8BLuJE9WTvF9s7X576zDAluw6LDrYM1G W6vg== X-Gm-Message-State: APuU4Z/jePU/SqOUB/jXuXSvsBvQc+KK2TL610JDZ5j0lTc59kGB/l4+ NgZ8pKLMtrGHq/UNdRdnzCRYIlWU/nJCDvpVA9Up+slrLU2clLT7V97vI4STWQ== X-Gm-Gg: AZFdTZenmn/5A1/MVWR2G7e3Qex0nEs0/G6l/Q/Qk4yWygcNCP9SwUEvZMEiMQ/qr1q 2iaXOdN7TMQRnUPreZKvkATEhmsNGS/R9Aa9+W0Y6e+WxFJ0KG6rgBrfHDn99ARtdzq6hJb9QoR f+VarxYkFMUqAwRWXk1uLK+/xu12wyPcqDQItfWfdhFxj1W0USAwr7O/Qaa5ptr6VbzMtmpCqvm pxTr0RE8tmeQVBnQ8iTjnlGF1lzJ4ydjnU6YTCVHyRiPsC9WnXCR8yz6djHUzCxB8ke+e+r8Zyz XmWdsONDzkZIdsFdu/Mk/AUe5O54keLm45jeZcKqou5ZAz/6DbRr6WJpfYZsoFI6hiTggT6oi9G mSJ+iRKHomKngfY7BHoM6WJGdEU+bNv51hJcBHnsQb/uUm/7rGgYfdGznO2v/wy1TGBn2qQv/cS 1BPYVKKCBfKELCYG0XiBIRaLM0Tlz97Xaq4+EB+j7/H2T1pW6jjvpqcZ3avwJuDlHCMBlwALvES AiA6mb3N0uZ6aPtyGPZyB+VsPRfazal8w3Tqh3DU1OacraXkO9WONcFwMRpFC9LGJRSo4lJaoEB zH9bPNIGpgHvhMileiULll3CsnCle4+wLFLL2lrnKorKTFwIlBsx3CxOyjUkfRhffFqENLatd4p ebtJSOt8= X-Received: by 2002:a17:902:e744:b0:2e9:6333:fbb8 with SMTP id d9443c01a7336-2ee0dd2ed49mr111856255ad.12.1794867873111; Mon, 16 Nov 2026 14:24:33 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2efd7ab4abcsm4618945ad.70.2026.11.16.14.24.31 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 16 Nov 2026 14:24:32 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 3/4] rrs/views.py: fetch the maintainers page's statistics in a few queries Date: Mon, 16 Nov 2026 14:24:09 -0800 Message-ID: <20261116222410.35784-4-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261116222410.35784-1-tim.orling@konsulko.com> References: <20261116222410.35784-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Mon, 16 Nov 2026 22:24:42 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5432 The maintainers page ran about a dozen queries per maintainer: the statistics of each maintainer separately, then one count of upgrades per maintainer per interval. Compute every maintainer's statistics from one read of the maintainer and upstream records per layer branch, and count each interval's upgrades for all maintainers in one grouped query. On a copy of the production database the page data for 9 milestones is identical before and after, and OE-Core 6.2 takes 28 queries and 0.08 s instead of 952 queries and 0.75 s. AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- rrs/views.py | 78 +++++++++++++++++++++++++++-- tests/rrs/test_maintainers.py | 94 +++++++++++++++++++++++++++++++++++ 2 files changed, 168 insertions(+), 4 deletions(-) create mode 100644 tests/rrs/test_maintainers.py diff --git a/rrs/views.py b/rrs/views.py index b580818..81ef35e 100644 --- a/rrs/views.py +++ b/rrs/views.py @@ -373,6 +373,68 @@ def _get_milestone_statistics(milestone, maintainer_name=None): return milestone_statistics +def _get_maintainers_statistics(milestone, maintainer_names): + """ + The statistics of _get_milestone_statistics(milestone, name) for every + name in maintainer_names, with a few queries per layer branch rather + than several per maintainer (AI-generated, Claude Opus 5.5) + """ + statistics = {name: {'all': 0, 'up_to_date': 0, 'not_updated': 0, + 'cant_be_updated': 0, 'unknown': 0} + for name in maintainer_names} + if not statistics: + return statistics + + for maintplanlayer in milestone.release.plan.maintenanceplanlayerbranch_set.all(): + layerbranch = maintplanlayer.layerbranch + + recipe_upstream_history = RecipeUpstreamHistory.get_last_by_date_range( + layerbranch, + milestone.start_date, + milestone.end_date + ) + recipe_maintainer_history = Raw.get_remahi_by_end_date( + layerbranch.id, milestone.end_date) + if not recipe_maintainer_history: + continue + + maintainer_recipes = {} + for recipesymbol_id, name in RecipeMaintainer.objects.filter( + history_id=recipe_maintainer_history[0]).values_list( + 'recipesymbol_id', 'maintainer__name'): + maintainer_recipes.setdefault(name, set()).add(recipesymbol_id) + + recipe_upstreams = {} + if recipe_upstream_history and maintainer_recipes: + for ru in RecipeUpstream.objects.filter( + history=recipe_upstream_history).values( + 'recipesymbol_id', 'status', 'no_update_reason'): + recipe_upstreams.setdefault(ru['recipesymbol_id'], []).append(ru) + + for name, maintainer_statistics in statistics.items(): + recipes = maintainer_recipes.get(name, set()) + maintainer_statistics['all'] += len(recipes) + for recipesymbol_id in recipes: + for ru in recipe_upstreams.get(recipesymbol_id, []): + if ru['status'] == 'Y': + maintainer_statistics['up_to_date'] += 1 + elif ru['status'] == 'N': + if ru['no_update_reason'] == '': + maintainer_statistics['not_updated'] += 1 + else: + maintainer_statistics['cant_be_updated'] += 1 + else: + maintainer_statistics['unknown'] += 1 + + for maintainer_statistics in statistics.values(): + maintainer_statistics['percentage'] = '0' + if maintainer_statistics['all'] > 0: + maintainer_statistics['percentage'] = "%.0f" % \ + ((float(maintainer_statistics['up_to_date']) / + float(maintainer_statistics['all'])) * 100) + + return statistics + class RecipeList(): pk = None name = None @@ -879,8 +941,18 @@ class MaintainerListView(ListView): self.intervals = sorted(intervals.keys()) current_date = date.today() + maintainer_names = {ml.name for ml in maintainer_list} + all_statistics = _get_maintainers_statistics(milestone, maintainer_names) + # Upgrades per maintainer in each interval, one query per interval + interval_upgrades = [] + for i in (self.intervals if maintainer_names else []): + interval_upgrades.append(dict(RecipeUpgrade.objects.filter( + maintainer__name__in=maintainer_names, + commit_date__gte=intervals[i]['start_date'], + commit_date__lte=intervals[i]['end_date']).values_list( + 'maintainer__name').annotate(Count('id')).order_by())) for ml in maintainer_list: - milestone_statistics = _get_milestone_statistics(milestone, ml.name) + milestone_statistics = all_statistics[ml.name] ml.recipes_all = milestone_statistics['all'] ml.recipes_up_to_date = ('' if milestone_statistics['up_to_date'] == 0 else milestone_statistics['up_to_date']) @@ -901,9 +973,7 @@ class MaintainerListView(ListView): if current_date >= start_date and current_date <= end_date: self.current_interval = idx - number = RecipeUpgrade.objects.filter(maintainer__name = ml.name, - commit_date__gte = start_date, - commit_date__lte = end_date).count() + number = interval_upgrades[idx].get(ml.name, 0) ml.interval_statistics.append('' if number == 0 else number) # To add Wk prefix after get statics to avoid sorting problems diff --git a/tests/rrs/test_maintainers.py b/tests/rrs/test_maintainers.py new file mode 100644 index 0000000..ace1458 --- /dev/null +++ b/tests/rrs/test_maintainers.py @@ -0,0 +1,94 @@ +# layerindex-web - tests for the RRS maintainers page +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT +# +# AI-generated, Claude Opus 5.5 + +# The maintainers page shows each maintainer's recipe statistics and upgrades +# per week, and must get them without running queries per maintainer. + +import datetime + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from rrs.models import ( + Maintainer, + RecipeMaintainer, + RecipeMaintainerHistory, + RecipeSymbol, + RecipeUpgrade, + RecipeUpstream, + RecipeUpstreamHistory, +) + +URL = '/rrs/maintainers/OE-Core/6.2/M1/' +# USE_TZ is False +UPGRADE_DATE = datetime.datetime(2026, 11, 1) + + +def histories(layerbranch): + author, _ = Maintainer.objects.get_or_create(name='No maintainer') + maintainer_history = RecipeMaintainerHistory.objects.create( + title='Update maintainers', date=datetime.datetime(2026, 10, 15), author=author, + sha1='0' * 40, layerbranch=layerbranch) + upstream_history = RecipeUpstreamHistory.objects.create( + layerbranch=layerbranch, start_date=datetime.datetime(2026, 10, 20), + end_date=datetime.datetime(2026, 10, 20, 1)) + return maintainer_history, upstream_history + + +def add_recipe(layerbranch, history, pn, maintainer_name, status, no_update_reason=''): + """A recipe maintained by maintainer_name, with an upstream status and one upgrade""" + maintainer_history, upstream_history = history + maintainer, _ = Maintainer.objects.get_or_create(name=maintainer_name) + symbol = RecipeSymbol.objects.create(layerbranch=layerbranch, pn=pn, summary=pn) + RecipeMaintainer.objects.create(recipesymbol=symbol, maintainer=maintainer, + history=maintainer_history) + RecipeUpstream.objects.create(recipesymbol=symbol, history=upstream_history, version='2.0', + type='A', status=status, no_update_reason=no_update_reason, + date=upstream_history.end_date) + RecipeUpgrade.objects.create(recipesymbol=symbol, maintainer=maintainer, version='1.0', + author_date=UPGRADE_DATE, commit_date=UPGRADE_DATE, + upgrade_type='U') + + +def maintainer_list(client): + response = client.get(URL) + assert response.status_code == 200 + return {ml.name: ml for ml in response.context['maintainer_list']} + + +def test_statistics(client, rrs_plan): + history = histories(rrs_plan.layerbranch) + add_recipe(rrs_plan.layerbranch, history, 'busybox', 'Alice', 'Y') + add_recipe(rrs_plan.layerbranch, history, 'zlib', 'Alice', 'N') + add_recipe(rrs_plan.layerbranch, history, 'gcc', 'Bob', 'N', 'Needs a newer binutils') + + maintainers = maintainer_list(client) + + alice, bob = maintainers['Alice'], maintainers['Bob'] + assert (alice.recipes_all, alice.recipes_up_to_date, alice.recipes_not_updated, + alice.recipes_cant_be_updated, alice.percentage_done) == (2, 1, 1, '', '50%') + assert (bob.recipes_all, bob.recipes_up_to_date, bob.recipes_cant_be_updated, + bob.percentage_done) == (1, '', 1, '0%') + # One upgrade each, in the week of UPGRADE_DATE + assert [n for n in alice.interval_statistics if n] == [2] + assert [n for n in bob.interval_statistics if n] == [1] + assert alice.interval_statistics.index(2) == bob.interval_statistics.index(1) + + +def test_queries_do_not_grow_with_maintainers(client, rrs_plan): + history = histories(rrs_plan.layerbranch) + # All 'N', so the plan-wide statistics take the same path both times + add_recipe(rrs_plan.layerbranch, history, 'recipe-0', 'Maintainer 0', 'N') + maintainer_list(client) # fills the per-process caches first + with CaptureQueriesContext(connection) as one: + assert 'Maintainer 0' in maintainer_list(client) + for i in range(1, 6): + add_recipe(rrs_plan.layerbranch, history, f'recipe-{i}', f'Maintainer {i}', 'N') + with CaptureQueriesContext(connection) as six: + assert len(maintainer_list(client)) == 6 + assert len(six) == len(one) From patchwork Mon Nov 16 22:24:10 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104559 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.5330.1794867876944840247 for ; Mon, 16 Nov 2026 14:24:37 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=IBo45bz/; spf=pass (domain: gmail.com, ip: 209.85.215.173, mailfrom: ticotimo@gmail.com) Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-cd61cef87f1so1809197a12.0 for ; Mon, 16 Nov 2026 14:24:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794867876; x=1795472676; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=mD2Vg/w9hnc/PLl0n0TiVJQ86JmMisraooNAksUKMzk=; b=IBo45bz/YTXSqSsoYRTUdxHAnG10s9Uq0nHN/ABt8x0zacoyYqycbHLGqp+JugrdK9 VZeRMIUveIQnh/DQcIL5mAo6BN3Q0lEmUX3j58Nvq/tZDGXtnOBFB6Jiw6lgd/oZqqtL LIPatxPYtT/sLhE6+pNyRY3LkiH7Ue5mxj3WjoOBz+4Jd3SIG+Wddy8TiqwQgkRiiOwR KqlWcu4DbKUGEqP1ugPU/CnbYYvPWpS9OIC1nVTQc7KGuyNmgs/982zRflZzmK7lsfb7 Xdpa5A9xtvTn3hn9r9CG1g316xPzBDNS/AjVpgcq6OSrN/4aURnuWD1AHR0knxRWD3qW EuzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794867876; x=1795472676; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=mD2Vg/w9hnc/PLl0n0TiVJQ86JmMisraooNAksUKMzk=; b=2LMeaZ/wuLcSYFBAX2EcfG6df96FvwapRvbZSxTB3mTfVQqaQeugPBE9luMiZMlPEj /6N7pA0dIFXwkgMxM1SyHXGQNT+QxZ66seE/Bd5dO75t2e1YtbwqsCLoA2aoQn7EMlQE qgCETIIk2PJr7J9YIdFMBxslSVtkkRdogh957uh/luifMjfjq9QrkO++0atwXETyPUBN MMalvf0HHvus8dG/K1w/BkrVu2V7+x1NQrRmNDM3W9YcP11Et2MeDblh4luXrid32c7v sFUjcSLtRzuXlT3PmBvdFSnvQhAjb5GApJP0r/ugIcgVM0EjOZOsQNEqjXq48INmKG14 /Glg== X-Gm-Message-State: APuU4Z8U4u9RYWUGF2Y7N2hRy5g7vxTA3AOoMgKzDHqcJSmrI6O5PTUV UnPBhJr2fmpoBC/nMAF8YtvT3KyW4FFXydc563oBNsM1kCIY/4j+MnONRfdbxw== X-Gm-Gg: AZFdTZdJCq9kQWHCV5fjZgG9i14aLodFn9Zf28Yqm9JbfvUYtP13OiK4SkPy9E+YdhK l1MdN/nERtJqGbYBPgSTBf14RXJtvh3oCk1Rn0lXtmw3nCh9Y+Hf0mCTDwhh7Axk34MMdl6WPSc DOQ0Miovf7TitJMDSbTqRzGiqR4BYLLHwaz6F8YB2oAzbOY4fJ1rA9MiikD2KpqmuoMhca8mTrs NUADKXWCujFAG8MEf+PaOP0EwXuTD2f6ozcIjrhy21iZNmhg3OeAt7tJf5vuGgbs5E0VgJNbwqc 8LetzkWDzKRR6ouhCJSkf8evsAtY9IBNEW6jDQpxx0adBEGx/WXxM8OHKfdAbOad5xIdQQgaWVZ yL6Yw+l5obtbkY5W0G23UJsq6CdsMUwC6nWYKPfcaOohhDAjBJNyZzMzDa2d3yVrX42yPP6ULo+ 9Kn6IdtxSA3tlsOkJiorUmj/jpswupf1Xx4s6+JPO9C3wB69lLPzruR1gjHd51Ry44a52tsJPMX O+nzG7CJJJxf7BcXtB9PsqxROOi2nTt8AVcbxYAIdKDp9Et+uZYaMqIw6z9p9tvU62Qcp/bw3pL PxJ0tAOVzhHdsKCxt2+aotDIzhyp6esGzir75XwQ8L++cLqXIogc8ng1A7YXkBwyXmNN7PD1re7 4w1bxThk= X-Received: by 2002:a17:90b:4a84:b0:3b0:ad42:dcb3 with SMTP id 98e67ed59e1d1-3b1b09fa377mr10543250a91.14.1794867876091; Mon, 16 Nov 2026 14:24:36 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2efd7ab4abcsm4618945ad.70.2026.11.16.14.24.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 16 Nov 2026 14:24:35 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 4/4] rrs/views.py: fetch the recipe detail page's upgrades in a few queries Date: Mon, 16 Nov 2026 14:24:10 -0800 Message-ID: <20261116222410.35784-5-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261116222410.35784-1-tim.orling@konsulko.com> References: <20261116222410.35784-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Mon, 16 Nov 2026 22:24:42 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5433 The recipe detail page ran about nine queries per upgrade to find the release and milestone it falls in and whether its author maintained the recipe then, so linux-yocto, with 282 upgrades, took 3.7 s. Read the plan's releases and milestones once and pick from them by date, and look up each maintainer history and maintainer once per page. Releases with the same dates are picked in id order, as the queries returned them. Also fetch the other branches' recipes with their layer branch and branch. On a copy of the production database the page data for 153 OE-Core recipes is identical before and after, and linux-yocto takes 0.3 s instead of 3.6 s. AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- rrs/views.py | 63 +++++++++++++++++++------ tests/rrs/test_recipe_detail.py | 84 +++++++++++++++++++++++++++++++++ 2 files changed, 133 insertions(+), 14 deletions(-) create mode 100644 tests/rrs/test_recipe_detail.py diff --git a/rrs/views.py b/rrs/views.py index 81ef35e..529b203 100644 --- a/rrs/views.py +++ b/rrs/views.py @@ -754,31 +754,63 @@ class RecipeUpgradeDetail(): self.upgrade_type = upgrade_type self.group = group -def _get_recipe_upgrade_detail(maintplan, recipe_upgrade): +def _get_release_and_milestone(maintplan, commit_date, lookups): + """ + Release.get_by_date() and Milestone.get_by_release_and_date() for + commit_date, from the plan's releases and milestones read once into + lookups (AI-generated, Claude Opus 5.5) + """ + if 'releases' not in lookups: + # id breaks ties between releases with the same dates as the queries did + lookups['releases'] = list(Release.objects.filter(plan=maintplan).order_by('-end_date', 'id')) + lookups['milestones'] = {} + for milestone in Milestone.objects.filter(release__plan=maintplan) \ + .exclude(name='All').order_by('-end_date', 'id'): + lookups['milestones'].setdefault(milestone.release_id, []).append(milestone) + # The queries compare the DateFields with the commit's date + commit_day = commit_date.date() + release = next((r for r in lookups['releases'] + if r.start_date <= commit_day <= r.end_date), None) + milestone = None + if release: + milestone = next((m for m in lookups['milestones'].get(release.id, []) + if m.start_date <= commit_day <= m.end_date), None) + return release, milestone + +def _get_recipe_upgrade_detail(maintplan, recipe_upgrade, lookups=None): + # lookups keeps what several upgrades of a recipe share, so a caller + # that passes the same dict for each runs fewer queries + if lookups is None: + lookups = {} release_name = '' milestone_name = '' recipe_maintainer_history = None - release = Release.get_by_date(maintplan, recipe_upgrade.commit_date) + release, milestone = _get_release_and_milestone(maintplan, recipe_upgrade.commit_date, lookups) if release: release_name = release.name - milestone = Milestone.get_by_release_and_date(release, - recipe_upgrade.commit_date) if milestone: milestone_name = milestone.name - recipe_maintainer_history = RecipeMaintainerHistory.get_by_end_date( - recipe_upgrade.recipesymbol.layerbranch, - milestone.end_date) + key = ('maintainer_history', recipe_upgrade.recipesymbol.layerbranch_id, milestone.end_date) + if key not in lookups: + lookups[key] = RecipeMaintainerHistory.get_by_end_date( + recipe_upgrade.recipesymbol.layerbranch, + milestone.end_date) + recipe_maintainer_history = lookups[key] is_recipe_maintainer = False maintainer_name = '' if not recipe_upgrade.maintainer is None: maintainer_name = recipe_upgrade.maintainer.name - if not recipe_maintainer_history is None and \ - RecipeMaintainer.objects.filter(maintainer__name - = maintainer_name, history = recipe_maintainer_history) \ - .count() > 0: + key = ('is_maintainer', maintainer_name, + recipe_maintainer_history.id if recipe_maintainer_history else None) + if key not in lookups: + lookups[key] = not recipe_maintainer_history is None and \ + RecipeMaintainer.objects.filter(maintainer__name + = maintainer_name, history = recipe_maintainer_history) \ + .exists() + if lookups[key]: is_recipe_maintainer = True commit_date = recipe_upgrade.commit_date.date().isoformat() @@ -849,8 +881,11 @@ class RecipeDetailView(DetailView): details = [] multigroup = False lastgroup = '' # can't use None here - for ru in RecipeUpgrade.objects.filter(recipesymbol=recipesymbol).exclude(upgrade_type='M').order_by('group', '-commit_date', '-id'): - details.append(_get_recipe_upgrade_detail(maintplan, ru)) + lookups = {} + # Through the related manager, each upgrade shares recipesymbol + for ru in recipesymbol.recipeupgrade_set.select_related('maintainer', 'group') \ + .exclude(upgrade_type='M').order_by('group', '-commit_date', '-id'): + details.append(_get_recipe_upgrade_detail(maintplan, ru, lookups)) if not multigroup: if lastgroup == '': lastgroup = ru.group @@ -882,7 +917,7 @@ class RecipeDetailView(DetailView): context['recipe_depends'] = [] context['recipe_distros'] = [] - context['otherbranch_recipes'] = Recipe.objects.filter(layerbranch__layer=recipesymbol.layerbranch.layer, layerbranch__branch__comparison=False, pn=recipesymbol.pn).order_by('layerbranch__branch__sort_priority') + context['otherbranch_recipes'] = Recipe.objects.filter(layerbranch__layer=recipesymbol.layerbranch.layer, layerbranch__branch__comparison=False, pn=recipesymbol.pn).select_related('layerbranch__branch').order_by('layerbranch__branch__sort_priority') return context diff --git a/tests/rrs/test_recipe_detail.py b/tests/rrs/test_recipe_detail.py new file mode 100644 index 0000000..d29d136 --- /dev/null +++ b/tests/rrs/test_recipe_detail.py @@ -0,0 +1,84 @@ +# layerindex-web - tests for the RRS recipe detail page +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT +# +# AI-generated, Claude Opus 5.5 + +# The recipe detail page lists a recipe's upgrades with the release and +# milestone each falls in, and whether its author maintained the recipe then. +# It must get them without running queries per upgrade. + +import datetime + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from rrs.models import ( + Maintainer, + RecipeMaintainer, + RecipeMaintainerHistory, + RecipeSymbol, + RecipeUpgrade, +) + + +def setup_recipe(layerbranch): + """busybox, maintained by Alice since 2026-10-15""" + alice, _ = Maintainer.objects.get_or_create(name='Alice') + Maintainer.objects.get_or_create(name='Bob') + history = RecipeMaintainerHistory.objects.create( + title='Update maintainers', date=datetime.datetime(2026, 10, 15), author=alice, + sha1='0' * 40, layerbranch=layerbranch) + symbol = RecipeSymbol.objects.create(layerbranch=layerbranch, pn='busybox', summary='busybox') + RecipeMaintainer.objects.create(recipesymbol=symbol, maintainer=alice, history=history) + return symbol + + +def add_upgrade(symbol, version, when, maintainer_name): + # USE_TZ is False + RecipeUpgrade.objects.create(recipesymbol=symbol, version=version, author_date=when, + commit_date=when, upgrade_type='U', sha1='1' * 40, + maintainer=Maintainer.objects.get(name=maintainer_name)) + + +def details(client, symbol): + response = client.get(f'/rrs/recipedetail/OE-Core/{symbol.pk}/') + assert response.status_code == 200 + return response.context['recipe_upgrade_details'] + + +def test_release_milestone_and_maintainer(client, rrs_plan): + symbol = setup_recipe(rrs_plan.layerbranch) + add_upgrade(symbol, '1.37.0', datetime.datetime(2026, 11, 1, 10), 'Alice') + add_upgrade(symbol, '1.36.1', datetime.datetime(2026, 10, 2, 10), 'Bob') + # In release 6.2 but after its only milestone + add_upgrade(symbol, '1.38.0', datetime.datetime(2027, 2, 1, 10), 'Alice') + # Before any release + add_upgrade(symbol, '1.30.0', datetime.datetime(2025, 1, 1, 10), 'Alice') + + rows = {d.version: (d.release_name, d.milestone_name, d.maintainer_name, d.is_recipe_maintainer) + for d in details(client, symbol)} + + assert rows == { + '1.38.0': ('6.2', '', 'Alice', False), + '1.37.0': ('6.2', 'M1', 'Alice', True), + '1.36.1': ('6.2', 'M1', 'Bob', False), + '1.30.0': ('', '', 'Alice', False), + } + + +def test_queries_do_not_grow_with_upgrades(client, rrs_plan): + symbol = setup_recipe(rrs_plan.layerbranch) + add_upgrade(symbol, '1.0', datetime.datetime(2026, 10, 2), 'Alice') + add_upgrade(symbol, '1.1', datetime.datetime(2026, 10, 3), 'Bob') + details(client, symbol) # fills the per-process caches first + with CaptureQueriesContext(connection) as two: + assert len(details(client, symbol)) == 2 + for day in range(4, 10): + add_upgrade(symbol, f'1.{day - 2}', datetime.datetime(2026, 10, day), + 'Alice' if day % 2 else 'Bob') + with CaptureQueriesContext(connection) as eight: + assert len(details(client, symbol)) == 8 + assert len(eight) == len(two)