From patchwork Sun May 10 00:46:59 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2427CD3447 for ; Sun, 10 May 2026 00:47:31 +0000 (UTC) Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.22527.1778374042962490160 for ; Sat, 09 May 2026 17:47:23 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=n/sE4Uw/; spf=pass (domain: gmail.com, ip: 209.85.216.51, mailfrom: ticotimo@gmail.com) Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-366330b6751so1869766a91.1 for ; Sat, 09 May 2026 17:47:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1778374042; x=1778978842; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=F+MbkurJq2hCgqsBefyNRhs8rLsr1eUv9isVsciz45Q=; b=n/sE4Uw/e6FwU4G4BdMvMK/qKZ1jydfavNARnESXHuFwqXzGBUDpCtCTIpnvtr79FZ Af84ZTOWJq05Gbb3GGRVbfDhcQKgY26vnIDPLX5D3U/dB/Gwr0aQT4TQE0aEqK4v8jV8 Rp30ZYLKY80OyOFo8fMW5wOqQshAnM4Yrkk/MO8z5OfMhlEmuwosHnR0TG6W63n1Z8un UtQU4e8t3jNB7wenFog4q54swRDZnrdJHoNYwvgTdrLsSCOzw9G3C+twK9r7SpYEKVAI VYwMY9RiQMV/RD0ApzSiTlI2OF5lmQ0bu+OSYYnbxV26p9S35t4g8/xu3IR9w3F/Ijo9 W0Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778374042; x=1778978842; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=F+MbkurJq2hCgqsBefyNRhs8rLsr1eUv9isVsciz45Q=; b=JiDra9zfCoVLn8T2vcxgfQUoHLl5OqQgvHeg3hlIQV+7xZz1ijdNBjDnGTc50Dia5y ipbU2tWGku53StkUG3LNKyhs5ClnremaZkf6NOL+B5UyRhjFIHMLEJzgOD7aE/Ym0Hz1 724tLqIG4tZ6TCEZC1YPhvxJmht/kJu0R4lvHrRG8kqz33l10hc/wDD5GAzR4tD1pKB9 raf6aquSJ2tMSdXuIDgIEbo1tUnIunkgmAi+6S7CEAV3v1bvbxHgcLAErFIcQmhWEOgg rWNcGKIB5TPl55ol777dycpMg2HtbGf/rZnVojuLBLeV1ZXAlWo47gdGDhFTfoOBjDU3 zlMA== X-Gm-Message-State: AOJu0YwTRcRs/BJVLebbtUpg/Y4m2XlknFqxzTkFIkWZcdWOnpDKDZR0 YuDtSeL6pto9U9ihpXl8W0RMzk/fHLsmPz5n/yrfkV97dxW6bzQvQv//LF+9qkbJ X-Gm-Gg: Acq92OEUU2ATqMChFkGUq0QgwtUoXkDjxE95GbEnoLZJfOmf3OdpaYj/62LKuqRn+ls oMoInyI4CBivS97u87rjtNSGzO+bNvx3Bchfa3d5uyVdZapPbnW7AMhjq/ejRRx9Rfg/Wr/OUZR Q6Du8yH8QCfVV01LDUJH1W1uHle26VWZ4ojy6SeUdq7uxU9oTo1GzfB1SMoabkkIWaUtgT4Xh3f e8SDH+NkzPLAZKf5Y15oNk9h1/PoRdIaZTaCESfFi61YNF8x3rO5WtwZBSprYUXnKpw3fmoc20T GEgn9LJnbxCzY2672DXpQ7qpUoxdMZmHRz5LSH1lDqlEx37IQQh4XmXvof9XIwL55QEruFQ+QQA dVAsIYYO2d+Un5ugOT8ZTzFQG0uLkWtg6sub81O4hMpXjV6kzEv43W/9lqIQhXjX6IPGV1CsO5v flNYiSNYLq8FV53gFg0mxzEVI5KusZfH9HgprzpmuTkxj/1AFxoaepPdEQRGBMVg7ly0zdPjDlz inpz8zNrOyHL18Zgc6tz2Q= X-Received: by 2002:a17:90b:578d:b0:35f:b5df:450 with SMTP id 98e67ed59e1d1-365ac2807f0mr18645025a91.19.1778374041604; Sat, 09 May 2026 17:47:21 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-367d627bc2bsm3178257a91.7.2026.05.09.17.47.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 09 May 2026 17:47:20 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 1/5] layerindex/views.py: fix StatsView statistics page timeout Date: Sat, 9 May 2026 17:46:59 -0700 Message-ID: <20260510004706.81282-1-tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 10 May 2026 00:47:31 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3958 The per-branch statistics query used a single ORM queryset with five Count(..., distinct=True) annotations spanning deep reverse FK chains (layerbranch__recipe, layerbranch__bbclass, etc.). Django translates this into one SQL query with multiple LEFT OUTER JOINs across all those tables simultaneously. At production data volumes (~541 layers, ~25k recipes) the resulting query exhausts gunicorn worker memory/time, causing a 504 Gateway Time-out. Fix by iterating over each visible branch individually and issuing simple COUNT queries per table. This replaces one huge cross-join with N×5 small indexed WHERE-clause queries that are far cheaper and independently cacheable. The template is unchanged; Django's template engine resolves dict keys and object attributes identically with dot notation. [YOCTO #15391] AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- layerindex/views.py | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/layerindex/views.py b/layerindex/views.py index 3cf91d2..23bafd0 100644 --- a/layerindex/views.py +++ b/layerindex/views.py @@ -1522,12 +1522,23 @@ class StatsView(TemplateView): context['class_count_distinct'] = BBClass.objects.values('name').distinct().count() context['machine_count_distinct'] = Machine.objects.values('name').distinct().count() context['distro_count_distinct'] = Distro.objects.values('name').distinct().count() - context['perbranch'] = Branch.objects.filter(hidden=False).order_by('sort_priority').annotate( - layer_count=Count('layerbranch', distinct=True), - recipe_count=Count('layerbranch__recipe', distinct=True), - class_count=Count('layerbranch__bbclass', distinct=True), - machine_count=Count('layerbranch__machine', distinct=True), - distro_count=Count('layerbranch__distro', distinct=True)) + # Compute per-branch counts with individual queries rather than a single + # annotated queryset. The multi-Count(distinct=True) approach generates one + # large SQL query with many LEFT JOINs that causes a gunicorn worker timeout + # at production data volumes. Simple per-branch COUNT queries are far cheaper. + # See: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15391 + perbranch = [] + for branch in Branch.objects.filter(hidden=False).order_by('sort_priority'): + perbranch.append({ + 'name': branch.name, + 'updates_enabled': branch.updates_enabled, + 'layer_count': LayerBranch.objects.filter(branch=branch).count(), + 'recipe_count': Recipe.objects.filter(layerbranch__branch=branch).count(), + 'class_count': BBClass.objects.filter(layerbranch__branch=branch).count(), + 'machine_count': Machine.objects.filter(layerbranch__branch=branch).count(), + 'distro_count': Distro.objects.filter(layerbranch__branch=branch).count(), + }) + context['perbranch'] = perbranch return context From patchwork Sun May 10 00:47:00 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD857CD37AF for ; Sun, 10 May 2026 00:47:31 +0000 (UTC) Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.22355.1778374050177293899 for ; Sat, 09 May 2026 17:47:30 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=CTvef6wm; spf=pass (domain: gmail.com, ip: 209.85.216.42, mailfrom: ticotimo@gmail.com) Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-366330b6751so1869785a91.1 for ; Sat, 09 May 2026 17:47:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1778374049; x=1778978849; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=US3J4gVDpCzawGPgdoHUZTsWJlUDN8GoAGwu6RyYxsk=; b=CTvef6wmh/hxB+5DB4So0HASdwFHGBQ/1igYsfDlXwiSupzodHnASWCoaF898gXrIo aWszZd99RTN+kT0dU1D8u5JVtA41O7FAXJ+WErXLVQ5XAn/MLHRCV7cvQpTolMXE4R2U ybOUFZegsGF8i1ZYHC0y/ZofjrQ1sIU/LIrBNCVhQIVgFvOaZuN/wJdMcSWcLfHcNcMD /sq3au8U5Yg8SFEMOwQOJ39msLL3Ck8z+plRZLRoQojYLPft5Q2tfH6jzATqP+iT2aJA XzISZAlDjT7QttXfygNciim9HS9M/CpYOC6BgnNrr965OCKe0FxdrWAaorWXLs/UhHjF E66w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778374049; x=1778978849; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=US3J4gVDpCzawGPgdoHUZTsWJlUDN8GoAGwu6RyYxsk=; b=ivZo3nq6yZ1fd7O8TcsSnpDJsTcFKDmq05YZNLKFvFVvzGYaYbXgh9V8I02pcJkSXJ ck9Mnts0cRKGRx0XiAW8iQSu1GFimw+ndp62DYTr01UoEVitoRnSdRSTxs4z5UWyCfCI fK17BBf7T+eN+yFR6KfmBlG/vV8aa0i5kip4pOSNjTgSftWpbpF+mOJ0zHtU9y+0rdc0 P8ZQF3TbNDIXs1wa6Dv8gdAqQpOyRE7YESgzuMlU9BpMX5wZMTXqq+mvlBsk9DIoC57/ Oro9HBnVqgxmCDQ9ypnyU1u40R6jXNxSritE8lnRmDbmLQtXPBtsBkvAynNPv/t23Bum /a2g== X-Gm-Message-State: AOJu0YzPAc6tsi3HSQC8/8xgcsH1U4n9Tsloxy4CTyYlGLQT7NbH1oXL BihaEDxQhTUZ9Qa7ysRXPW+K6wnpqYokCE7sWQ6HvQYHvar1UqlAVJPylexOObb3 X-Gm-Gg: Acq92OGOYMCTaMcJBNqwSdEoPHXGvSr3dVH8bccDr5NrKq//huI/tcYr7ngd9gd42h5 ptIovcei+NYu+4esX8zrrbzQKll9J9kzqHXiKnaQ0D60XmIME6TvAEWbXZWzabg34VNdhINptoG /DFUAr2nn/oTvUX3Eeg8fq9mceK9+lKXvjRECAsYwe5vNUzqnGZH6M1PYteI7NhElNUIagVeEhS IpLDDdwvviUHMf68T1syjnFfCG7RpV6VANLtuZKrocQaC6T+lnz6m3sN7vL69JOQ0P62mXh36MB bINzEu00Vgq/xYBfrmYBdsHVUkUnPHSmR7Fd0Y6JKqoyTJdwCmwpfRXG82YkUrD1A3geFRpmr5p DCwOVn7+dBlsAJT1250bwgcSGeJKbTLH6kP1RucF0Qhs6Dr95q6nMAlEdPP3GcGR+LBf4NGuLHb ZEroioiKtKbDBw3CEdNT9yjAikb6KnMoGojh7t+2DK53K65+9I1BeUOycWPHhT5VopmYYHoG4Kd jT4dtiUEBhLX2x29ycuE0s= X-Received: by 2002:a17:90b:1c8b:b0:35b:e4f8:7b2b with SMTP id 98e67ed59e1d1-365abe8a3d0mr20860800a91.18.1778374048938; Sat, 09 May 2026 17:47:28 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-367d627bc2bsm3178257a91.7.2026.05.09.17.47.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 09 May 2026 17:47:28 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 2/5] tests: add StatsView tests Date: Sat, 9 May 2026 17:47:00 -0700 Message-ID: <20260510004706.81282-2-tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260510004706.81282-1-tim.orling@konsulko.com> References: <20260510004706.81282-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 10 May 2026 00:47:31 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3959 Add test_stats_view.py covering the StatsView statistics page: - HTTP 200 response - perbranch context is a list of dicts (not a queryset) - hidden branches are excluded - branches are sorted by sort_priority - each dict contains all keys the template requires - per-branch counts match test fixture data - updates_enabled is correctly reflected per branch - overall context keys (layercount, *_count_distinct) are present [YOCTO #15391] AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- tests/test_stats_view.py | 164 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 164 insertions(+) create mode 100644 tests/test_stats_view.py diff --git a/tests/test_stats_view.py b/tests/test_stats_view.py new file mode 100644 index 0000000..c30cc17 --- /dev/null +++ b/tests/test_stats_view.py @@ -0,0 +1,164 @@ +# layerindex-web - tests for StatsView +# +# Copyright (C) 2026 Konsulko Group +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT + +# Tests for bug #15391 - Statistics page timeout +# https://bugzilla.yoctoproject.org/show_bug.cgi?id=15391 +# +# The fix replaces a single annotated queryset (which generated one huge SQL +# query with multiple COUNT(DISTINCT ...) JOINs) with per-branch individual +# COUNT queries that are much cheaper at production data volumes. + +import pytest +from django.test import TestCase +from django.urls import reverse + +from layerindex.models import (Branch, LayerItem, LayerBranch, Recipe, + BBClass, Machine, Distro) + + +@pytest.mark.django_db +class TestStatsView(TestCase): + """Tests for StatsView (bug #15391 - statistics page timeout fix).""" + + def setUp(self): + # Create two visible branches with different sort priorities + self.branch1 = Branch.objects.create( + name='main', + bitbake_branch='master', + short_description='Main branch', + sort_priority=1, + hidden=False, + updates_enabled=True, + ) + self.branch2 = Branch.objects.create( + name='wrynose', + bitbake_branch='2.18', + short_description='Wrynose branch', + sort_priority=2, + hidden=False, + updates_enabled=False, + ) + # A hidden branch that should never appear in perbranch + self.branch_hidden = Branch.objects.create( + name='old-hidden', + bitbake_branch='1.0', + short_description='Old hidden branch', + sort_priority=99, + hidden=True, + updates_enabled=False, + ) + + # A layer + self.layer = LayerItem.objects.create( + name='meta-test', + status='P', + layer_type='A', + summary='Test layer', + description='A test layer', + vcs_url='git://example.com/meta-test.git', + ) + + # Wire up layerbranches + self.lb1 = LayerBranch.objects.create(layer=self.layer, branch=self.branch1) + self.lb2 = LayerBranch.objects.create(layer=self.layer, branch=self.branch2) + + # Add objects only to branch1 + Recipe.objects.create( + layerbranch=self.lb1, + filename='test_1.0.bb', + pn='test', + pv='1.0', + filepath='recipes-test', + ) + BBClass.objects.create( + layerbranch=self.lb1, + name='testclass', + ) + Machine.objects.create( + layerbranch=self.lb1, + name='qemux86', + description='QEMU x86 machine', + ) + Distro.objects.create( + layerbranch=self.lb1, + name='testdistro', + description='Test distro', + ) + + def test_stats_view_returns_200(self): + """StatsView should return HTTP 200.""" + response = self.client.get(reverse('stats')) + self.assertEqual(response.status_code, 200) + + def test_perbranch_is_list_of_dicts(self): + """perbranch context should be a list of dicts, not a queryset.""" + response = self.client.get(reverse('stats')) + perbranch = response.context['perbranch'] + self.assertIsInstance(perbranch, list) + for item in perbranch: + self.assertIsInstance(item, dict) + + def test_perbranch_excludes_hidden_branches(self): + """Hidden branches should not appear in perbranch.""" + response = self.client.get(reverse('stats')) + names = [b['name'] for b in response.context['perbranch']] + self.assertIn('main', names) + self.assertIn('wrynose', names) + self.assertNotIn('old-hidden', names) + + def test_perbranch_sorted_by_priority(self): + """Branches should be ordered by sort_priority.""" + response = self.client.get(reverse('stats')) + names = [b['name'] for b in response.context['perbranch']] + self.assertEqual(names, ['master', 'main', 'wrynose']) + + def test_perbranch_dict_has_required_keys(self): + """Each perbranch dict must contain all keys the template expects.""" + required_keys = {'name', 'updates_enabled', 'layer_count', + 'recipe_count', 'class_count', 'machine_count', 'distro_count'} + response = self.client.get(reverse('stats')) + for item in response.context['perbranch']: + self.assertTrue(required_keys.issubset(item.keys()), + f"Missing keys in perbranch item: {required_keys - item.keys()}") + + def test_perbranch_counts_branch1(self): + """Per-branch counts for branch1 should reflect the test data.""" + response = self.client.get(reverse('stats')) + branch1_data = next(b for b in response.context['perbranch'] if b['name'] == 'main') + self.assertEqual(branch1_data['layer_count'], 1) + self.assertEqual(branch1_data['recipe_count'], 1) + self.assertEqual(branch1_data['class_count'], 1) + self.assertEqual(branch1_data['machine_count'], 1) + self.assertEqual(branch1_data['distro_count'], 1) + + def test_perbranch_counts_branch2_empty(self): + """Branch2 has a layerbranch but no recipes/classes/machines/distros.""" + response = self.client.get(reverse('stats')) + branch2_data = next(b for b in response.context['perbranch'] if b['name'] == 'wrynose') + self.assertEqual(branch2_data['layer_count'], 1) + self.assertEqual(branch2_data['recipe_count'], 0) + self.assertEqual(branch2_data['class_count'], 0) + self.assertEqual(branch2_data['machine_count'], 0) + self.assertEqual(branch2_data['distro_count'], 0) + + def test_perbranch_updates_enabled_field(self): + """updates_enabled should be correctly reflected per branch.""" + response = self.client.get(reverse('stats')) + perbranch = response.context['perbranch'] + b1 = next(b for b in perbranch if b['name'] == 'main') + b2 = next(b for b in perbranch if b['name'] == 'wrynose') + self.assertTrue(b1['updates_enabled']) + self.assertFalse(b2['updates_enabled']) + + def test_overall_context_keys_present(self): + """Overall statistics context keys should all be present.""" + response = self.client.get(reverse('stats')) + for key in ('layercount', 'recipe_count_distinct', 'class_count_distinct', + 'machine_count_distinct', 'distro_count_distinct'): + self.assertIn(key, response.context, + f"Missing context key: {key}") From patchwork Sun May 10 00:47:01 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2CDFCD37AF for ; Sun, 10 May 2026 00:47:41 +0000 (UTC) Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.22356.1778374053294611309 for ; Sat, 09 May 2026 17:47:33 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=XWWfiwuA; spf=pass (domain: gmail.com, ip: 209.85.216.46, mailfrom: ticotimo@gmail.com) Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-3660daea6a5so1590542a91.1 for ; Sat, 09 May 2026 17:47:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1778374051; x=1778978851; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8OlDScb9F8nFAbC6DFC0VEwl/qfbCVhiqUFTueUzwV0=; b=XWWfiwuAQRvdMAtuvhSg7pjyFNvmEbjOwFvosJWd+JNS2aMsDMYGtf0lDaFgyCaxxI WAIMiAlo7522vkEIeFD18y34x6rcNG8uVeEmEEJEf2Pr8Ztyq6D5gP0Ej3gh79zZpD7Q 7WCB3oP6dop58trZ7lfFQDGyIuSwmCG840QZnti77Ox3bMolopPLUk2Onwb5M30pzF1j JmB1M8L8qvHLDP/tT4PNVOCAnZ2LxOggMhm+D7LjF1ignPEwqh93RO8wAQW1ZLECqIOv CS16c+xuhDAG8viEj827Z8zJhnrQRDdaGfqSV9E8o+iI7BNz8DOnlVF2eHrKpNshjloU d6nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778374051; x=1778978851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8OlDScb9F8nFAbC6DFC0VEwl/qfbCVhiqUFTueUzwV0=; b=r6FzJm9NI6i3bqHBbctXkg1X/ZwpACd5h1xlbXWeitwx4/DWc8fqaN5htmqdUUCZAM +vsJnKG114mtiuq5LPRecu6Vd8u211iTCFjeROG9Jk3/nQ++phctxqvEs/D/ijhgzJkH pqHbdw5PZYwPZasFAagXhv09USmmZqFkSAf49KE9H7M1ABh1r23cjbt4rBlwqioyxyKI MA8M0XAxKI1Ruk0+MUzpE3X8IjwDhS0aCLWLH+WA5rhOir3F4v2ziS3PdZQ9nxHnP+3+ zgqAPjLCGXcUGi2DmzeDye8kuTWeXGn364sUjadfdghx0Mo7k21Mym7E/JRrpv3Anyno omBQ== X-Gm-Message-State: AOJu0Yz/ptuqmbcQ1MUkCkPq4/VsbNFPpxm6y+xZf1UTRr+owTZ+l9YQ As2yIGnm9+WNoIP4S1uOh79Y4E1gQL6v6Z3+1RsxXwCQU6G8UOaLFRWDIvmuP165 X-Gm-Gg: Acq92OGIWPb+Wn84XJPGvyKdeZlVLi2DmYUw7i8oPiTxGHI3oc+WPXR4RsFD5cBO2C6 pZUNdMinTUI2T5XQyNAyooCKUGeerkOGqi1qOOKvHdQmG1SfzUViXU3oCwIKX7unV2bZd5wc0ue 8YEOPGrpeZoT03x0nqD+wXlqHIR7AKugIs/4SK86uS/mbM6VrpgsengVSid5AusJ5Q8nro9xJog 83xIn55AmfJVZtJgNs9ee9eB6/scJCQmg/8KzB3lNb1h9E+fLB6fvdou7BcYKr4lHB6h0Jq8y5K wEx2Gt6XS7HRf64gIHPkKNz354jNLIAqF8MX2aE2I9rho1Bzl0LhvbMnBsjri4U+9jPsLUluu9E HMQPYU0EsUJZ8abKW+VMFax30LNZBIssZydHJ/6TjnrecUzhL7XK3ogv7Th0cAFg1L9YbMSlF6n gmu06nE2L19NUzkjBqVLBfVY0Buo9TQVTX9MscrzLSfgVX0IXek64c9qNiVz4SrR+krzEOFnhtj N5bhH3HtDyihiTrz9l9WMd5otq5WRAItg== X-Received: by 2002:a17:90a:d003:b0:366:173c:9a94 with SMTP id 98e67ed59e1d1-366173c9ea1mr12599455a91.14.1778374051409; Sat, 09 May 2026 17:47:31 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-367d627bc2bsm3178257a91.7.2026.05.09.17.47.30 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 09 May 2026 17:47:30 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 3/5] layerindex/views.py: fix DuplicatesView timeout Date: Sat, 9 May 2026 17:47:01 -0700 Message-ID: <20260510004706.81282-3-tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260510004706.81282-1-tim.orling@konsulko.com> References: <20260510004706.81282-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 10 May 2026 00:47:41 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3960 Two performance problems caused the duplicates page to take over 115 seconds to respond at production data volumes, eventually causing clients to disconnect (SIGPIPE / broken pipe): 1. Python list comprehension IN clause: each get_recipes/get_classes/ get_incfiles method evaluated its 'dupes' queryset in Python with a list comprehension to build a filter: filter(pn__in=[item['pn'] for item in dupes]) This loaded all matching values into memory and emitted a potentially huge SQL IN (...) literal. Replace with a Django ORM subquery by passing the queryset directly: filter(pn__in=dupes) # dupes is a .values('pn') queryset Django translates this into a SQL subquery (WHERE pn IN (SELECT pn FROM ... GROUP BY pn HAVING COUNT(...) > 1)), which the database can optimise without a round-trip through Python. 2. N+1 queries: the template renders recipe.layerbranch.layer.name, class.layerbranch.layer.name and incfile.layerbranch.layer.name for every row. Without select_related these trigger one extra query per row. Add select_related('layerbranch__layer') to each queryset so the JOIN is done once up front. [YOCTO #16175] AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- layerindex/views.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/layerindex/views.py b/layerindex/views.py index 23bafd0..742e5d5 100644 --- a/layerindex/views.py +++ b/layerindex/views.py @@ -656,24 +656,30 @@ class DuplicatesView(TemplateView): init_qs = Recipe.objects.filter(layerbranch__branch__name=self.kwargs['branch']) if layer_ids: init_qs = init_qs.filter(layerbranch__layer__in=layer_ids) - dupes = init_qs.values('pn').annotate(Count('layerbranch', distinct=True)).filter(layerbranch__count__gt=1) - qs = init_qs.all().filter(pn__in=[item['pn'] for item in dupes]).order_by('pn', 'layerbranch__layer', '-pv') + # Use a subquery instead of a Python list comprehension so that Django + # emits a single SQL subquery rather than loading all duplicate pn values + # into memory and building a potentially huge IN (...) clause. + # See: https://bugzilla.yoctoproject.org/show_bug.cgi?id=16175 + dupes = init_qs.values('pn').annotate(Count('layerbranch', distinct=True)).filter(layerbranch__count__gt=1).values('pn') + qs = init_qs.filter(pn__in=dupes).select_related('layerbranch__layer').order_by('pn', 'layerbranch__layer', '-pv') return recipes_preferred_count(qs) def get_classes(self, layer_ids): init_qs = BBClass.objects.filter(layerbranch__branch__name=self.kwargs['branch']) if layer_ids: init_qs = init_qs.filter(layerbranch__layer__in=layer_ids) - dupes = init_qs.values('name').annotate(Count('layerbranch', distinct=True)).filter(layerbranch__count__gt=1) - qs = init_qs.all().filter(name__in=[item['name'] for item in dupes]).order_by('name', 'layerbranch__layer') + # Use a subquery instead of a Python list comprehension (see bug #16175) + dupes = init_qs.values('name').annotate(Count('layerbranch', distinct=True)).filter(layerbranch__count__gt=1).values('name') + qs = init_qs.filter(name__in=dupes).select_related('layerbranch__layer').order_by('name', 'layerbranch__layer') return qs def get_incfiles(self, layer_ids): init_qs = IncFile.objects.filter(layerbranch__branch__name=self.kwargs['branch']) if layer_ids: init_qs = init_qs.filter(layerbranch__layer__in=layer_ids) - dupes = init_qs.values('path').annotate(Count('layerbranch', distinct=True)).filter(layerbranch__count__gt=1) - qs = init_qs.all().filter(path__in=[item['path'] for item in dupes]).order_by('path', 'layerbranch__layer') + # Use a subquery instead of a Python list comprehension (see bug #16175) + dupes = init_qs.values('path').annotate(Count('layerbranch', distinct=True)).filter(layerbranch__count__gt=1).values('path') + qs = init_qs.filter(path__in=dupes).select_related('layerbranch__layer').order_by('path', 'layerbranch__layer') return qs def get_context_data(self, **kwargs): From patchwork Sun May 10 00:47:02 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF8D5CD3447 for ; Sun, 10 May 2026 00:47:41 +0000 (UTC) Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.22529.1778374055170712861 for ; Sat, 09 May 2026 17:47:35 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=CjSSVbZp; spf=pass (domain: gmail.com, ip: 209.85.216.50, mailfrom: ticotimo@gmail.com) Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-367dd53815fso292196a91.0 for ; Sat, 09 May 2026 17:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1778374054; x=1778978854; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vs8RuFCxoBQjFS91jjdCkhuZq7hlN5GJgHV65iNw72c=; b=CjSSVbZpMvv0SMGGBGwxGjBzpI6o8ka3Uc2LOMIeD7oJK+bu5vTm3BcJweXNui3kb3 W0IEU///o7g6eHAMXtOOMAhTPsC2/gVtBBciQN+Q1cLaWchNl//yaqF1+cKfeLkBDOR/ cU0ZRNce74piDIpnWdC7tU2x7o2T2FBeIj+yF3G3ZRgUVYfCSvqu0OfW7Mpr82rMzgwG uVs989sGmDdakcxWwCezAbAV8raQ4sl8nItwHwhI11x2017VtWRDP608VOQU7YVsMXiF TUQJseFeNXHsR91b31Ty0MpShNbotGGPPIaQXw2m89D8DxDuNnt6MoOeNUjyMv5iCzFT XyAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778374054; x=1778978854; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Vs8RuFCxoBQjFS91jjdCkhuZq7hlN5GJgHV65iNw72c=; b=a0FcpLX+DN8/mNbY85R2Ze4MyoZMQo3zen+aZhTGtr1KjYzIA8/Tpay8iD1EIjILy+ 4kujwoo1tifvM9VRNwkRmY/AgAS4x96LKEK9Qu83ciyhwqJ853fwiYwvAKVCNK7Id8qX 9r6gZwcddKrcSSFB2d84NK5PzpdwQGtxmI5m6GyCchDSK8mup1Uf8SzqrOfWIeg3KPT0 Z2XyyWg/i/EOs2xsJFKx1JIsLHDC9W293Z8dUSwaOJKHPWYgp/sN9yW2y3Pij4v0zSdP iJ9vEhbmAB3ukaKePno0e1rydSswHfG0GiPJydS4ofOxRvW5MFwkgpAwlm/U0gRIqjZy yQ6Q== X-Gm-Message-State: AOJu0Yx59f+lNYa5aV+C8moVEDHbqAFEaafQPdsHfaMgr0NMAosnplzx I1AbmF2WLaNrSpq6o27TKcaKa6fZhlEMrsplMJ4KUcdvGriVTnGoTlBoU/yyTQek X-Gm-Gg: Acq92OGYudnDGX2ZUJmibTOArxS609IjCGhFcG+2LzGyhupQqT7TYGg+ZWbSzu+b6HU n1lX2JEpuj0PMsyQ2Q7KFEcObAbUAMvnY31r6EXDb0D7vpqLKiR2L362qH77EcQWXb0YfTtVhJi KAEHtBjdEnsb4owBNZ61qgBkveNVpYml9qehAVqXCrFdUlIEWkevJHqNQWR5cQwnI7O1jcanyKW Q4PTYfoRsuiXf/OBhb4fzq7FdmDzkrgG/ulcgl6rvMNT1RZbh+vzusyYz6slRRTh7wnWFXMVXbY CTrc+g6Woa6G37jPGu7qNGruu7fQf0pY6A+11Z4HigUIRR75E/g+LxmRQqM7s0rJ1hWoN6K48P/ JkSIKUfdmCXx85H6mmIJV26O//p03IpRDIoLb2fPDZ+/hF3BybQIyz8Ea4az8c+aW5IWFFlXatT 1YOKn9d/TPsPyIHnQGYJ/hmNu9uZSzU8KHkKcg0M3qAvoYuultkBv/aq53Iz3YwJdEUI3dU25uQ ObHnLsEM2EUs2vEnnqfsBs/ef4s6wzSBA== X-Received: by 2002:a17:90b:2d8a:b0:367:e13b:f52 with SMTP id 98e67ed59e1d1-367e13b11cemr2399916a91.0.1778374053880; Sat, 09 May 2026 17:47:33 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-367d627bc2bsm3178257a91.7.2026.05.09.17.47.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 09 May 2026 17:47:33 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 4/5] tests: add DuplicatesView tests Date: Sat, 9 May 2026 17:47:02 -0700 Message-ID: <20260510004706.81282-4-tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260510004706.81282-1-tim.orling@konsulko.com> References: <20260510004706.81282-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 10 May 2026 00:47:41 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3961 Add test_duplicates_view.py covering the DuplicatesView duplicates page: - HTTP 200 response - Duplicate recipes/classes/include files are included in context - Unique (non-duplicate) objects are excluded from context - Each duplicate appears once per layer (two rows for two layers) - Layer filter (?l=id) restricts results to the selected layer, suppressing duplicates when only one layer is in scope - Branch with no data returns empty querysets - select_related check: accessing layerbranch.layer.name after list evaluation fires no additional queries [YOCTO #16175] AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- tests/test_duplicates_view.py | 218 ++++++++++++++++++++++++++++++++++ 1 file changed, 218 insertions(+) create mode 100644 tests/test_duplicates_view.py diff --git a/tests/test_duplicates_view.py b/tests/test_duplicates_view.py new file mode 100644 index 0000000..304a35c --- /dev/null +++ b/tests/test_duplicates_view.py @@ -0,0 +1,218 @@ +# layerindex-web - tests for DuplicatesView +# +# Copyright (C) 2026 Tim Orling +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT + +# Tests for bug #16175 - Duplicates page timeout +# https://bugzilla.yoctoproject.org/show_bug.cgi?id=16175 +# +# The fix replaces Python list comprehensions used to build IN (...) clauses +# with Django ORM subqueries, and adds select_related to avoid N+1 queries +# when the template resolves layerbranch__layer names. + +import pytest +from django.test import TestCase +from django.urls import reverse + +from layerindex.models import (Branch, LayerItem, LayerBranch, Recipe, + BBClass, IncFile) + + +@pytest.mark.django_db +class TestDuplicatesView(TestCase): + """Tests for DuplicatesView (bug #16175 - duplicates page timeout fix).""" + + def setUp(self): + self.branch = Branch.objects.create( + name='main', + bitbake_branch='master', + short_description='Main branch', + sort_priority=1, + hidden=False, + updates_enabled=True, + ) + + # Two layers — objects in both will appear as duplicates + self.layer_a = LayerItem.objects.create( + name='meta-alpha', + status='P', + layer_type='A', + summary='Alpha layer', + description='Alpha test layer', + vcs_url='git://example.com/meta-alpha.git', + ) + self.layer_b = LayerItem.objects.create( + name='meta-beta', + status='P', + layer_type='S', + summary='Beta layer', + description='Beta test layer', + vcs_url='git://example.com/meta-beta.git', + ) + # A third layer whose objects are unique (should NOT appear as duplicates) + self.layer_c = LayerItem.objects.create( + name='meta-gamma', + status='P', + layer_type='S', + summary='Gamma layer', + description='Gamma test layer', + vcs_url='git://example.com/meta-gamma.git', + ) + + self.lb_a = LayerBranch.objects.create(layer=self.layer_a, branch=self.branch) + self.lb_b = LayerBranch.objects.create(layer=self.layer_b, branch=self.branch) + self.lb_c = LayerBranch.objects.create(layer=self.layer_c, branch=self.branch) + + # Duplicate recipe (same pn in layer_a and layer_b) + Recipe.objects.create(layerbranch=self.lb_a, filename='shared_1.0.bb', + pn='shared', pv='1.0', filepath='recipes-test') + Recipe.objects.create(layerbranch=self.lb_b, filename='shared_1.0.bb', + pn='shared', pv='1.0', filepath='recipes-test') + # Unique recipe (only in layer_c — should NOT appear) + Recipe.objects.create(layerbranch=self.lb_c, filename='unique_1.0.bb', + pn='unique', pv='1.0', filepath='recipes-test') + + # Duplicate class (same name in layer_a and layer_b) + BBClass.objects.create(layerbranch=self.lb_a, name='sharedclass') + BBClass.objects.create(layerbranch=self.lb_b, name='sharedclass') + # Unique class (only in layer_c — should NOT appear) + BBClass.objects.create(layerbranch=self.lb_c, name='uniqueclass') + + # Duplicate include file (same path in layer_a and layer_b) + IncFile.objects.create(layerbranch=self.lb_a, path='conf/shared.inc') + IncFile.objects.create(layerbranch=self.lb_b, path='conf/shared.inc') + # Unique include file (only in layer_c — should NOT appear) + IncFile.objects.create(layerbranch=self.lb_c, path='conf/unique.inc') + + def _url(self, branch='main'): + return reverse('duplicates', kwargs={'branch': branch}) + + def test_duplicates_view_returns_200(self): + """DuplicatesView should return HTTP 200.""" + response = self.client.get(self._url()) + self.assertEqual(response.status_code, 200) + + # --- Recipes --- + + def test_duplicate_recipes_included(self): + """Recipes appearing in more than one layer should be in context.""" + response = self.client.get(self._url()) + pns = [r.pn for r in response.context['recipes']] + self.assertIn('shared', pns) + + def test_unique_recipes_excluded(self): + """Recipes appearing in only one layer should not appear.""" + response = self.client.get(self._url()) + pns = [r.pn for r in response.context['recipes']] + self.assertNotIn('unique', pns) + + def test_duplicate_recipes_have_two_rows(self): + """Each layer's entry for the duplicate recipe should be present.""" + response = self.client.get(self._url()) + shared = [r for r in response.context['recipes'] if r.pn == 'shared'] + self.assertEqual(len(shared), 2) + + # --- Classes --- + + def test_duplicate_classes_included(self): + """Classes appearing in more than one layer should be in context.""" + response = self.client.get(self._url()) + names = [c.name for c in response.context['classes']] + self.assertIn('sharedclass', names) + + def test_unique_classes_excluded(self): + """Classes appearing in only one layer should not appear.""" + response = self.client.get(self._url()) + names = [c.name for c in response.context['classes']] + self.assertNotIn('uniqueclass', names) + + def test_duplicate_classes_have_two_rows(self): + """Both layer entries for the duplicate class should be present.""" + response = self.client.get(self._url()) + shared = [c for c in response.context['classes'] if c.name == 'sharedclass'] + self.assertEqual(len(shared), 2) + + # --- Include files --- + + def test_duplicate_incfiles_included(self): + """Include files appearing in more than one layer should be in context.""" + response = self.client.get(self._url()) + paths = [f.path for f in response.context['incfiles']] + self.assertIn('conf/shared.inc', paths) + + def test_unique_incfiles_excluded(self): + """Include files appearing in only one layer should not appear.""" + response = self.client.get(self._url()) + paths = [f.path for f in response.context['incfiles']] + self.assertNotIn('conf/unique.inc', paths) + + def test_duplicate_incfiles_have_two_rows(self): + """Both layer entries for the duplicate include file should be present.""" + response = self.client.get(self._url()) + shared = [f for f in response.context['incfiles'] if f.path == 'conf/shared.inc'] + self.assertEqual(len(shared), 2) + + # --- Layer filter --- + + def test_layer_filter_restricts_recipes(self): + """Passing ?l= should restrict results to that layer.""" + response = self.client.get(self._url() + f'?l={self.layer_a.id}') + self.assertEqual(response.status_code, 200) + # With only one layer selected, nothing can be a duplicate + pns = [r.pn for r in response.context['recipes']] + self.assertNotIn('shared', pns) + + def test_wrong_branch_returns_empty(self): + """A branch with no data should return empty querysets.""" + other = Branch.objects.create( + name='other', + bitbake_branch='other', + sort_priority=50, + ) + response = self.client.get(reverse('duplicates', kwargs={'branch': 'other'})) + self.assertEqual(response.status_code, 200) + self.assertEqual(len(list(response.context['recipes'])), 0) + self.assertEqual(len(list(response.context['classes'])), 0) + self.assertEqual(len(list(response.context['incfiles'])), 0) + + def test_select_related_avoids_extra_queries(self): + """Ensure layerbranch and layer are fetched with select_related. + + Checks that accessing layerbranch.layer.name on query results does not + trigger additional database queries (i.e. select_related is working). + """ + from django.db import connection, reset_queries + from django.conf import settings + + settings.DEBUG = True + reset_queries() + + view = __import__('layerindex.views', fromlist=['DuplicatesView']).DuplicatesView + v = view() + v.kwargs = {'branch': 'main'} + v.request = None + + recipes = list(v.get_recipes([])) + classes = list(v.get_classes([])) + incfiles = list(v.get_incfiles([])) + + # Record query count after fetching + query_count_after_fetch = len(connection.queries) + + # Access layerbranch.layer.name on every result — should NOT fire new queries + for r in recipes: + _ = r.layerbranch.layer.name + for c in classes: + _ = c.layerbranch.layer.name + for f in incfiles: + _ = f.layerbranch.layer.name + + query_count_after_access = len(connection.queries) + settings.DEBUG = False + + self.assertEqual(query_count_after_fetch, query_count_after_access, + "Unexpected extra queries when accessing layerbranch.layer.name — " + "select_related may not be working") From patchwork Sun May 10 00:47:03 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 87808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB87BCD37B2 for ; Sun, 10 May 2026 00:47:41 +0000 (UTC) Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.22359.1778374060063043499 for ; Sat, 09 May 2026 17:47:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=q+RoSm2U; spf=pass (domain: gmail.com, ip: 209.85.216.42, mailfrom: ticotimo@gmail.com) Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-366be8040a9so704737a91.3 for ; Sat, 09 May 2026 17:47:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1778374059; x=1778978859; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rWUHzosjP56wsLW+5LHup3T6f9+ajPtvKbgLXdesrqU=; b=q+RoSm2UvjNDzdZCq8uiNqEivKte9PfRWOJlH7hl6Bk9kZoF6D6aVPIyc/uX9mhE2F 67cL3N/NyjfvMETzbJbt4lPGcO53kX7l5ZaSExIx9jrY7ER5up6kdumiVuMgXdo0P8fM fud4RAHjT2NBwoAgayXVfpdJTKmApCGlpmhCE05/wIBjvDROmbQ7AYUxWeQc9dU6kMy8 4E4fLRd9oEIh1BgiVogdHb08C0nzWHfMChh2ri+h2cQ3YtTim0xDwDyv/UPCx60E3zKl nsTTc1WmR/wuuxu4lzkeJRIBejpXvmXYAs0HRXalz0ds6NLWVgWPqpStWKpRx539+Bl+ VUkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778374059; x=1778978859; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=rWUHzosjP56wsLW+5LHup3T6f9+ajPtvKbgLXdesrqU=; b=lHseR+9lZ4VE/ABCtW2pukVt0vLxrD8AhO5HFsx6RXsGnoman/YdnH54UUSURtFW5n G8pjqEM704VmoNPfoeol0bGU9u9ap4qGHg1Eb0lfVhch6o+yEeOQk3KcBCIfRxhRo2ca FLfOzVZwzcKLInWUVsvuWZrW1fFErCzJRapmtlKnzoi76b4vkgCMQZbY9RRo2Y/PUAIV zp+zOf0s7fRraNL6EZgHK9wgE40TIrHMWysyPsejK2h0FCeplzIf/ttTJLkZyftil32k maITRWjBLxuiqjWNlI4f0MSNqf3ZQo1RUfIYH9CVPjxmQVSrhfaVTcCYLEPDB/2mvLLU nZlA== X-Gm-Message-State: AOJu0YzT+i8FW/oxDwD00SxinrT81j+ckVATdzuvfwRnUOJufA+fJUUy m0+/hmqulIR/hJ4cA9txGa0WzfbIaB61Rn9f1tkDlkvj+6+KiNhL/6ZsWIOss7qp X-Gm-Gg: Acq92OEsWsozvL+FJVT9XCHNBymfv+3qmXkplZEb2vw8o2oVD7KHbOfz+4bS10P2VJm +XYqNcM2M52SzakHxoRRSHW9+I4KlzCqVp/INC3sa8iWHS5yg0tOZ/2hS8e9VB29wQurKj25+kq rOddJjMw/wkzhBtSdUhkrvnufeB5spzjFN/Hra2X3hFOvEGNCZNy5HnwFeFqEw6g+hy7Hgqappc 3UpZ8MY9pLrrY1nokfneT69w3TbpB/GDv/tcygJtf835HaM0WZe9UnNXSnlt+fsvTzAI48qZBSd Gy82xjXZolDiCFEK+GCXj4y57gpkbSMY4yT6Dh0QQswgBDgL54D2WvNmTJc5LCE/FgRggNodhzW thCpAyZgjyor546kOTM+PYYbEC/AW6fXk4RGMYwQaac5Ina4i6nUksLSv1jA7cRmzNknOnba4Gf VDVAqNI62U/9tKCO4t8giZxvb4OZ3A8TpRkmv9Kw+KvPsXBtXXTE5hov/Id1LEySHbIcF/SWEGc 7FDt2xYH15G7+5WFztuaio= X-Received: by 2002:a17:90b:224f:b0:35f:b69d:7292 with SMTP id 98e67ed59e1d1-365ac080dc9mr18646828a91.15.1778374058637; Sat, 09 May 2026 17:47:38 -0700 (PDT) Received: from localhost.localdomain (c-98-232-159-17.hsd1.or.comcast.net. [98.232.159.17]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-367d627bc2bsm3178257a91.7.2026.05.09.17.47.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 09 May 2026 17:47:38 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 5/5] layerindex/views.py: fix DuplicatesView 504 from per-row correlated subquery Date: Sat, 9 May 2026 17:47:03 -0700 Message-ID: <20260510004706.81282-5-tim.orling@konsulko.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260510004706.81282-1-tim.orling@konsulko.com> References: <20260510004706.81282-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 10 May 2026 00:47:41 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/3962 The previous fix (subquery IN clause + select_related) resolved the Python memory overhead but the 504 Gateway Timeout persisted because recipes_preferred_count() uses .extra(select={...}) to embed a correlated multi-table subquery that the database evaluates once for every result row. With hundreds of duplicate recipes on the master branch this creates an O(N) compounding scan that dominates page load time. Replace the call to recipes_preferred_count() in DuplicatesView.get_recipes with a two-step batch approach: 1. Fetch all duplicate recipes into a list using the existing subquery + select_related (one SQL query). 2. Run a single aggregating query to find, for each pn, the maximum index_preference among S/A-type layers in this branch that carry that pn (one SQL query returning one row per distinct pn). 3. Annotate each recipe object in Python: preferred_count is set to 1 if another S/A layer has a strictly higher index_preference for the same pn, otherwise 0. The template only checks preferred_count > 0, so the boolean semantics are identical to the original COUNT. recipes_preferred_count() is intentionally left unchanged — it is still used by RecipeSearchView where result sets are paginated to 50 rows and the per-row cost is negligible. Also add Max to the django.db.models import line. [YOCTO #16175] AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling --- layerindex/views.py | 30 ++++++++- tests/test_duplicates_view.py | 113 ++++++++++++++++++++++------------ 2 files changed, 100 insertions(+), 43 deletions(-) diff --git a/layerindex/views.py b/layerindex/views.py index 742e5d5..84750f7 100644 --- a/layerindex/views.py +++ b/layerindex/views.py @@ -25,7 +25,7 @@ from django.contrib.sites.models import Site from django.core.exceptions import PermissionDenied from django.urls import resolve, reverse, reverse_lazy from django.db import transaction -from django.db.models import Count, Q +from django.db.models import Count, Max, Q from django.db.models.functions import Lower from django.db.models.query import QuerySet from django.db.models.signals import pre_save @@ -661,8 +661,32 @@ class DuplicatesView(TemplateView): # into memory and building a potentially huge IN (...) clause. # See: https://bugzilla.yoctoproject.org/show_bug.cgi?id=16175 dupes = init_qs.values('pn').annotate(Count('layerbranch', distinct=True)).filter(layerbranch__count__gt=1).values('pn') - qs = init_qs.filter(pn__in=dupes).select_related('layerbranch__layer').order_by('pn', 'layerbranch__layer', '-pv') - return recipes_preferred_count(qs) + recipes = list(init_qs.filter(pn__in=dupes).select_related('layerbranch__layer').order_by('pn', 'layerbranch__layer', '-pv')) + if not recipes: + return recipes + # recipes_preferred_count() attaches a correlated subquery via .extra() that + # the database evaluates once per result row. With hundreds of duplicate + # recipes that compounds into a slow multi-table correlated scan that causes + # a 504 Gateway Timeout. Instead, compute preferred_count with a single + # batch MAX query across all pns and annotate the results in Python. + # + # A recipe is "non-preferred" (preferred_count > 0) when another layer of + # type S or A carries the same pn and has a higher index_preference. + pns = {r.pn for r in recipes} + pn_max_pref = dict( + Recipe.objects.filter( + layerbranch__branch__name=self.kwargs['branch'], + pn__in=pns, + layerbranch__layer__layer_type__in=['S', 'A'], + ).values('pn').annotate( + max_pref=Max('layerbranch__layer__index_preference') + ).values_list('pn', 'max_pref') + ) + for recipe in recipes: + own_pref = recipe.layerbranch.layer.index_preference + max_pref = pn_max_pref.get(recipe.pn) + recipe.preferred_count = 1 if (max_pref is not None and max_pref > own_pref) else 0 + return recipes def get_classes(self, layer_ids): init_qs = BBClass.objects.filter(layerbranch__branch__name=self.kwargs['branch']) diff --git a/tests/test_duplicates_view.py b/tests/test_duplicates_view.py index 304a35c..78d1036 100644 --- a/tests/test_duplicates_view.py +++ b/tests/test_duplicates_view.py @@ -9,9 +9,17 @@ # Tests for bug #16175 - Duplicates page timeout # https://bugzilla.yoctoproject.org/show_bug.cgi?id=16175 # -# The fix replaces Python list comprehensions used to build IN (...) clauses -# with Django ORM subqueries, and adds select_related to avoid N+1 queries -# when the template resolves layerbranch__layer names. +# The original code had two performance problems: +# +# 1. Python list comprehension IN clause: each get_* method pulled all +# duplicate names into Python memory and built a huge IN (...) literal. +# Fixed by passing the 'dupes' queryset directly so Django emits a subquery. +# +# 2. Per-row correlated subquery: get_recipes() called recipes_preferred_count() +# which embeds a correlated subquery via .extra() evaluated once per result +# row by the database. With hundreds of duplicate recipes this caused a 504 +# Gateway Timeout. Fixed by computing preferred_count with a single batch +# MAX query over all pns, then annotating results in Python. import pytest from django.test import TestCase @@ -35,7 +43,7 @@ class TestDuplicatesView(TestCase): updates_enabled=True, ) - # Two layers — objects in both will appear as duplicates + # layer_a: base layer (type A), lower preference self.layer_a = LayerItem.objects.create( name='meta-alpha', status='P', @@ -43,7 +51,9 @@ class TestDuplicatesView(TestCase): summary='Alpha layer', description='Alpha test layer', vcs_url='git://example.com/meta-alpha.git', + index_preference=0, ) + # layer_b: software layer (type S), higher preference — recipes here win self.layer_b = LayerItem.objects.create( name='meta-beta', status='P', @@ -51,8 +61,9 @@ class TestDuplicatesView(TestCase): summary='Beta layer', description='Beta test layer', vcs_url='git://example.com/meta-beta.git', + index_preference=10, ) - # A third layer whose objects are unique (should NOT appear as duplicates) + # layer_c: unique objects only, should never appear as duplicates self.layer_c = LayerItem.objects.create( name='meta-gamma', status='P', @@ -60,36 +71,42 @@ class TestDuplicatesView(TestCase): summary='Gamma layer', description='Gamma test layer', vcs_url='git://example.com/meta-gamma.git', + index_preference=5, ) self.lb_a = LayerBranch.objects.create(layer=self.layer_a, branch=self.branch) self.lb_b = LayerBranch.objects.create(layer=self.layer_b, branch=self.branch) self.lb_c = LayerBranch.objects.create(layer=self.layer_c, branch=self.branch) - # Duplicate recipe (same pn in layer_a and layer_b) - Recipe.objects.create(layerbranch=self.lb_a, filename='shared_1.0.bb', - pn='shared', pv='1.0', filepath='recipes-test') - Recipe.objects.create(layerbranch=self.lb_b, filename='shared_1.0.bb', - pn='shared', pv='1.0', filepath='recipes-test') - # Unique recipe (only in layer_c — should NOT appear) - Recipe.objects.create(layerbranch=self.lb_c, filename='unique_1.0.bb', - pn='unique', pv='1.0', filepath='recipes-test') - - # Duplicate class (same name in layer_a and layer_b) + # Duplicate recipe in both layer_a (pref=0) and layer_b (pref=10) + self.recipe_a = Recipe.objects.create( + layerbranch=self.lb_a, filename='shared_1.0.bb', + pn='shared', pv='1.0', filepath='recipes-test') + self.recipe_b = Recipe.objects.create( + layerbranch=self.lb_b, filename='shared_1.0.bb', + pn='shared', pv='1.0', filepath='recipes-test') + # Unique recipe only in layer_c — must NOT appear + Recipe.objects.create( + layerbranch=self.lb_c, filename='unique_1.0.bb', + pn='unique', pv='1.0', filepath='recipes-test') + + # Duplicate class in layer_a and layer_b BBClass.objects.create(layerbranch=self.lb_a, name='sharedclass') BBClass.objects.create(layerbranch=self.lb_b, name='sharedclass') - # Unique class (only in layer_c — should NOT appear) + # Unique class only in layer_c — must NOT appear BBClass.objects.create(layerbranch=self.lb_c, name='uniqueclass') - # Duplicate include file (same path in layer_a and layer_b) + # Duplicate include file in layer_a and layer_b IncFile.objects.create(layerbranch=self.lb_a, path='conf/shared.inc') IncFile.objects.create(layerbranch=self.lb_b, path='conf/shared.inc') - # Unique include file (only in layer_c — should NOT appear) + # Unique include file only in layer_c — must NOT appear IncFile.objects.create(layerbranch=self.lb_c, path='conf/unique.inc') def _url(self, branch='main'): return reverse('duplicates', kwargs={'branch': branch}) + # --- Basic page health --- + def test_duplicates_view_returns_200(self): """DuplicatesView should return HTTP 200.""" response = self.client.get(self._url()) @@ -110,11 +127,38 @@ class TestDuplicatesView(TestCase): self.assertNotIn('unique', pns) def test_duplicate_recipes_have_two_rows(self): - """Each layer's entry for the duplicate recipe should be present.""" + """Each layer's copy of the duplicate recipe should be present.""" response = self.client.get(self._url()) shared = [r for r in response.context['recipes'] if r.pn == 'shared'] self.assertEqual(len(shared), 2) + # --- preferred_count (batch MAX computation, replaces per-row correlated subquery) --- + + def test_lower_preference_recipe_is_deemphasised(self): + """The layer_a recipe (pref=0) should have preferred_count > 0 because + layer_b (pref=10, type S) has a higher-preference copy of the same pn.""" + response = self.client.get(self._url()) + recipe_from_a = next( + r for r in response.context['recipes'] + if r.pn == 'shared' and r.layerbranch.layer.name == 'meta-alpha') + self.assertGreater(recipe_from_a.preferred_count, 0) + + def test_higher_preference_recipe_is_not_deemphasised(self): + """The layer_b recipe (pref=10) should have preferred_count == 0 because + no other S/A layer has a higher preference for the same pn.""" + response = self.client.get(self._url()) + recipe_from_b = next( + r for r in response.context['recipes'] + if r.pn == 'shared' and r.layerbranch.layer.name == 'meta-beta') + self.assertEqual(recipe_from_b.preferred_count, 0) + + def test_preferred_count_attribute_present_on_all_recipes(self): + """Every recipe in the context must have a preferred_count attribute.""" + response = self.client.get(self._url()) + for recipe in response.context['recipes']: + self.assertTrue(hasattr(recipe, 'preferred_count'), + f"preferred_count missing on {recipe.pn}") + # --- Classes --- def test_duplicate_classes_included(self): @@ -158,20 +202,15 @@ class TestDuplicatesView(TestCase): # --- Layer filter --- def test_layer_filter_restricts_recipes(self): - """Passing ?l= should restrict results to that layer.""" + """With only one layer selected, nothing can be a duplicate.""" response = self.client.get(self._url() + f'?l={self.layer_a.id}') self.assertEqual(response.status_code, 200) - # With only one layer selected, nothing can be a duplicate pns = [r.pn for r in response.context['recipes']] self.assertNotIn('shared', pns) def test_wrong_branch_returns_empty(self): - """A branch with no data should return empty querysets.""" - other = Branch.objects.create( - name='other', - bitbake_branch='other', - sort_priority=50, - ) + """A branch with no data should return empty result sets.""" + Branch.objects.create(name='other', bitbake_branch='other', sort_priority=50) response = self.client.get(reverse('duplicates', kwargs={'branch': 'other'})) self.assertEqual(response.status_code, 200) self.assertEqual(len(list(response.context['recipes'])), 0) @@ -179,30 +218,24 @@ class TestDuplicatesView(TestCase): self.assertEqual(len(list(response.context['incfiles'])), 0) def test_select_related_avoids_extra_queries(self): - """Ensure layerbranch and layer are fetched with select_related. - - Checks that accessing layerbranch.layer.name on query results does not - trigger additional database queries (i.e. select_related is working). - """ + """Accessing layerbranch.layer.name after list evaluation fires no new queries.""" from django.db import connection, reset_queries from django.conf import settings + import layerindex.views as views_module settings.DEBUG = True reset_queries() - view = __import__('layerindex.views', fromlist=['DuplicatesView']).DuplicatesView - v = view() + v = views_module.DuplicatesView() v.kwargs = {'branch': 'main'} - v.request = None + v.request = type('req', (), {'GET': {}})() - recipes = list(v.get_recipes([])) + recipes = v.get_recipes([]) classes = list(v.get_classes([])) incfiles = list(v.get_incfiles([])) - # Record query count after fetching query_count_after_fetch = len(connection.queries) - # Access layerbranch.layer.name on every result — should NOT fire new queries for r in recipes: _ = r.layerbranch.layer.name for c in classes: @@ -214,5 +247,5 @@ class TestDuplicatesView(TestCase): settings.DEBUG = False self.assertEqual(query_count_after_fetch, query_count_after_access, - "Unexpected extra queries when accessing layerbranch.layer.name — " - "select_related may not be working") + "Extra queries fired when accessing layerbranch.layer.name — " + "select_related may not be effective")