From patchwork Wed Aug 12 15:04:47 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paolo Wattebled X-Patchwork-Id: 95031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B341CC5B56A for ; Wed, 12 Aug 2026 15:05:07 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.10284.1786547101723764644 for ; Wed, 12 Aug 2026 08:05:02 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=WHO9dxMM; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: paolo.wattebled@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id C63783D81738 for ; Wed, 12 Aug 2026 11:05:00 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id 5n5JEomIhqnV; Wed, 12 Aug 2026 11:04:53 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id C5D8B3D81867; Wed, 12 Aug 2026 11:04:53 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com C5D8B3D81867 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1786547093; bh=sLv0TZlwj91YwK0A0quFiFtE7LvdxRrWLNfr79qKEfs=; h=From:To:Date:Message-ID:MIME-Version; b=WHO9dxMMZylFNXh59mHMuFmMSWl4QRv3tjrN0p5T8A+I0a6CuAz12FEWiA35IPAgC J1cL1SGKKCQBFE0h8hwoPtgHvJ/E5Fve9fIx6g0oUffpdb+9Bq/KgLPGjUKwTIFdyE FNhY0wkTBY/CbNJOtxBl1pDdEfYyAt0HTrpt5pgBdwkLSEJXgpY11DYD0rMJFEq3v2 mErkf3u00WtqaTv2+UHGwTPTDH4MUzEYSamX+GlG2Xqq1tgHT+q2Snf0g5hZJiYASb mmqj/RBoVrprfqLYFGCtU9GdhZLLLexpvfNqnCUBIFkcy7vBLU+HoZ+7G/GZ+YqD9p gwO4tGLKmUMzA== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id fE-wznzsCo-R; Wed, 12 Aug 2026 11:04:53 -0400 (EDT) Received: from pwattebled.mtl.sfl (unknown [192.168.51.254]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 9A9A03D8185E; Wed, 12 Aug 2026 11:04:53 -0400 (EDT) From: Paolo Wattebled To: bitbake-devel@lists.openembedded.org Cc: Paolo Wattebled Subject: [PATCH] toaster: Add recipe variables view Date: Wed, 12 Aug 2026 11:04:47 -0400 Message-ID: <20260812150448.2343308-1-paolo.wattebled@savoirfairelinux.com> X-Mailer: git-send-email 2.55.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 12 Aug 2026 15:05:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19915 Toaster records global build variables but does not expose the final values associated with individual recipes. Collect variables defined or modified by recipes and their bbappends, resolve active overrides and expansions, and store one compressed JSON snapshot in the database for each build and recipe. Keep the variable count separately to avoid decompressing snapshots in recipe-list queries. Expose the values through a searchable and paginated Variables tab and show their counts in the built-recipes table and recipe package view. Preserve compatibility with older dependency payloads and invalidate stale BitBake caches after extending the cached recipe information. Tests cover variable collection, cache transport, database persistence, backward compatibility, searching, pagination, escaping, and counts. AI-Generated: Uses GitHub Copilot and OpenCode with GPT-5.6 Sol Signed-off-by: Paolo Wattebled --- bin/bitbake-selftest | 1 + lib/bb/cache.py | 2 +- lib/bb/cache_extra.py | 66 +++++++- lib/bb/cooker.py | 6 +- lib/bb/tests/cache_extra.py | 141 ++++++++++++++++ lib/bb/ui/buildinfohelper.py | 21 ++- .../orm/migrations/0022_recipevariable.py | 27 +++ lib/toaster/orm/models.py | 10 ++ lib/toaster/tests/db/test_db.py | 101 ++++++++++++ lib/toaster/tests/views/test_views.py | 155 +++++++++++++++++- lib/toaster/toastergui/buildtables.py | 11 +- .../templates/detail_pagination_bottom.html | 4 +- .../templates/detail_search_header.html | 2 +- lib/toaster/toastergui/templates/recipe.html | 32 ++++ .../toastergui/templates/recipe_packages.html | 7 + lib/toaster/toastergui/views.py | 68 +++++++- 16 files changed, 642 insertions(+), 12 deletions(-) create mode 100644 lib/bb/tests/cache_extra.py create mode 100644 lib/toaster/orm/migrations/0022_recipevariable.py diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest index fb7c57dd8..1b09815cd 100755 --- a/bin/bitbake-selftest +++ b/bin/bitbake-selftest @@ -21,6 +21,7 @@ except RuntimeError as exc: sys.exit(str(exc)) tests = ["bb.tests.codeparser", + "bb.tests.cache_extra", "bb.tests.color", "bb.tests.cooker", "bb.tests.cow", diff --git a/lib/bb/cache.py b/lib/bb/cache.py index 2361c5684..33fa61936 100644 --- a/lib/bb/cache.py +++ b/lib/bb/cache.py @@ -28,7 +28,7 @@ import shutil logger = logging.getLogger("BitBake.Cache") -__cache_version__ = "156" +__cache_version__ = "157" def getCacheFile(path, filename, mc, data_hash): mcspec = '' diff --git a/lib/bb/cache_extra.py b/lib/bb/cache_extra.py index bf4226d16..95a13c402 100644 --- a/lib/bb/cache_extra.py +++ b/lib/bb/cache_extra.py @@ -13,7 +13,47 @@ # SPDX-License-Identifier: GPL-2.0-only # +import logging +import json +import zlib + from bb.cache import RecipeInfoCommon +logger = logging.getLogger("BitBake.CacheExtra") + + +def _winning_override(metadata, variable_name): + active = {} + metadata.need_overrides() + for override_variable, override in metadata.overridedata.get(variable_name, ()): + if (override in metadata.overridesset or + ':' in override and + set(override.split(':')).issubset(metadata.overridesset)): + active[override] = override_variable + + match = None + modified = True + while modified: + modified = False + for override in metadata.overrides: + for candidate in active.copy(): + if candidate.endswith(':' + override): + active[candidate.removesuffix(':' + override)] = active.pop(candidate) + modified = True + elif candidate == override: + match = active.pop(candidate) + return match + + +def _recipe_value_event(metadata, event): + if not event.get('file', '').endswith(('.bb', '.bbappend')): + return False + if event.get('flag') not in (None, '_defaultval'): + return False + operation = event.get('op', '') + if '[' not in operation: + return True + override = operation.rsplit('[', 1)[1].removesuffix(']') + return set(override.split(':')).issubset(metadata.overridesset) class HobRecipeInfo(RecipeInfoCommon): __slots__ = () @@ -27,7 +67,7 @@ class HobRecipeInfo(RecipeInfoCommon): # that this class will provide cachefields = ['summary', 'license', 'section', 'description', 'homepage', 'bugtracker', - 'prevision', 'files_info'] + 'prevision', 'files_info', 'recipe_variables'] def __init__(self, filename, metadata): @@ -39,6 +79,28 @@ class HobRecipeInfo(RecipeInfoCommon): self.bugtracker = self.getvar('BUGTRACKER', metadata) self.prevision = self.getvar('PR', metadata) self.files_info = self.getvar('FILES_INFO', metadata) + recipe_variables = {} + for variable_name in metadata: + try: + if ':' in variable_name: + continue + history = metadata.varhistory.variable(variable_name) + winning_override = _winning_override(metadata, variable_name) + if winning_override: + history += metadata.varhistory.variable(winning_override) + if not any(_recipe_value_event(metadata, event) for event in history): + continue + if (metadata.getVarFlag(variable_name, 'func', False) or + winning_override and + metadata.getVarFlag(winning_override, 'func', False)): + continue + value = metadata.getVar(variable_name, True) + recipe_variables[variable_name] = '' if value is None else str(value) + except Exception as exc: + logger.debug("Omitting recipe variable %s from %s after %s", + variable_name, filename, type(exc).__name__) + self.recipe_variables = zlib.compress( + json.dumps(recipe_variables, separators=(',', ':')).encode('utf-8')) @classmethod def init_cacheData(cls, cachedata): @@ -51,6 +113,7 @@ class HobRecipeInfo(RecipeInfoCommon): cachedata.bugtracker = {} cachedata.prevision = {} cachedata.files_info = {} + cachedata.recipe_variables = {} def add_cacheData(self, cachedata, fn): cachedata.summary[fn] = self.summary @@ -61,3 +124,4 @@ class HobRecipeInfo(RecipeInfoCommon): cachedata.bugtracker[fn] = self.bugtracker cachedata.prevision[fn] = self.prevision cachedata.files_info[fn] = self.files_info + cachedata.recipe_variables[fn] = self.recipe_variables diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 4b6ba3196..fe80af22d 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -270,7 +270,8 @@ class BBCooker: if hasattr(self, "data"): consolelog = self.data.getVar("BB_CONSOLELOG") - if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset: + if (CookerFeatures.BASEDATASTORE_TRACKING in self.featureset or + CookerFeatures.HOB_EXTRA_CACHES in self.featureset): self.enableDataTracking() caches_name_array = ['bb.cache:CoreRecipeInfo'] @@ -310,7 +311,8 @@ class BBCooker: self.data.setVar('BB_CMDLINE', self.ui_cmdline) - if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset: + if (CookerFeatures.BASEDATASTORE_TRACKING in self.featureset and + CookerFeatures.HOB_EXTRA_CACHES not in self.featureset): self.disableDataTracking() for mc in self.databuilder.mcdata.values(): diff --git a/lib/bb/tests/cache_extra.py b/lib/bb/tests/cache_extra.py new file mode 100644 index 000000000..12a82f56a --- /dev/null +++ b/lib/bb/tests/cache_extra.py @@ -0,0 +1,141 @@ +# +# BitBake Tests for extra cache data +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import unittest +import json +import zlib + +import bb.data +import bb.parse +from bb.cache_extra import HobRecipeInfo + + +class HobRecipeInfoTest(unittest.TestCase): + + @staticmethod + def metadata(): + metadata = bb.data.init() + metadata.enableTracking() + return metadata + + def test_recipe_variables(self): + metadata = self.metadata() + metadata.setVar('TEXT', '${VALUE}', file='test.bb', line=1) + metadata.setVar('VALUE', 'expanded', file='test.bb', line=2) + metadata.setVar('EMPTY', '', file='test.bb', line=3) + metadata.setVar('NUMBER', 7, file='test.bbappend', line=1) + metadata.setVar('SPECIAL', 'café\nline\x00end', + file='test.bb', line=4) + metadata.setVar('INHERITED', 'global', file='conf/bitbake.conf', line=1) + metadata.setVar('do_function', 'echo test', file='test.bb', line=5) + metadata.setVarFlag('do_function', 'func', True) + + info = HobRecipeInfo('test.bb', metadata) + variables = json.loads(zlib.decompress(info.recipe_variables)) + + self.assertEqual(variables['TEXT'], 'expanded') + self.assertEqual(variables['EMPTY'], '') + self.assertEqual(variables['NUMBER'], '7') + self.assertEqual(variables['SPECIAL'], 'café\nline\x00end') + self.assertNotIn('INHERITED', variables) + self.assertNotIn('do_function', variables) + + def test_recipe_variable_operations_are_included(self): + metadata = self.metadata() + metadata.setVar('TEXT', 'global', file='conf/bitbake.conf', line=1) + metadata.setVar('TEXT:append', ' recipe', file='test.bb', line=1) + metadata.setVar('TEXT:remove', 'global', file='test.bbappend', line=1) + + info = HobRecipeInfo('test.bb', metadata) + variables = json.loads(zlib.decompress(info.recipe_variables)) + + self.assertEqual(variables['TEXT'], ' recipe') + + def test_only_effective_overrides_are_included(self): + metadata = self.metadata() + metadata.setVar('OVERRIDES', 'machine', file='conf/bitbake.conf', line=1) + metadata.setVar('ACTIVE:machine', 'recipe', file='test.bb', line=1) + metadata.setVar('INACTIVE:other', 'recipe', file='test.bb', line=2) + metadata.setVar('GLOBAL', 'global', file='conf/bitbake.conf', line=2) + metadata.setVar('GLOBAL:append:other', ' recipe', file='test.bb', line=3) + + info = HobRecipeInfo('test.bb', metadata) + variables = json.loads(zlib.decompress(info.recipe_variables)) + + self.assertEqual(variables['ACTIVE'], 'recipe') + self.assertNotIn('ACTIVE:machine', variables) + self.assertNotIn('INACTIVE', variables) + self.assertNotIn('INACTIVE:other', variables) + self.assertNotIn('GLOBAL', variables) + + def test_weak_default_is_included(self): + metadata = self.metadata() + metadata.setVarFlag('PACKAGECONFIG', '_defaultval', 'feature', + file='test.bb', line=1) + + info = HobRecipeInfo('test.bb', metadata) + variables = json.loads(zlib.decompress(info.recipe_variables)) + + self.assertEqual(variables['PACKAGECONFIG'], 'feature') + + def test_combined_override_is_included_under_logical_name(self): + metadata = self.metadata() + metadata.setVar('OVERRIDES', 'foo:bar:local', + file='conf/bitbake.conf', line=1) + metadata.setVar('COMBINED:local:foo:bar', 'recipe', + file='test.bb', line=1) + + info = HobRecipeInfo('test.bb', metadata) + variables = json.loads(zlib.decompress(info.recipe_variables)) + + self.assertEqual(variables, {'COMBINED': 'recipe'}) + + def test_active_override_function_is_omitted(self): + metadata = self.metadata() + metadata.setVar('OVERRIDES', 'machine', file='conf/bitbake.conf', line=1) + metadata.setVar('do_work:machine', 'echo test', file='test.bb', line=1) + metadata.setVarFlag('do_work:machine', 'func', True, + file='test.bb', line=1) + + info = HobRecipeInfo('test.bb', metadata) + variables = json.loads(zlib.decompress(info.recipe_variables)) + + self.assertNotIn('do_work', variables) + + def test_recipe_variables_survive_extra_cache_mapping(self): + metadata = self.metadata() + metadata.setVar('EMPTY', '', file='test.bb', line=1) + info = HobRecipeInfo('test.bb', metadata) + cachedata = type('CacheData', (), {})() + + HobRecipeInfo.init_cacheData(cachedata) + info.add_cacheData(cachedata, 'test.bb') + + self.assertIn('recipe_variables', HobRecipeInfo.cachefields) + variables = json.loads(zlib.decompress( + cachedata.recipe_variables['test.bb'])) + self.assertEqual(variables['EMPTY'], '') + + def test_recipe_variables_are_compressed_before_caching(self): + metadata = self.metadata() + metadata.setVar('LARGE', 'repeated-value-' * 1000, + file='test.bb', line=1) + + info = HobRecipeInfo('test.bb', metadata) + + self.assertLess(len(info.recipe_variables), len(metadata.getVar('LARGE'))) + + def test_unexpandable_variable_is_omitted(self): + metadata = self.metadata() + metadata.setVar('BROKEN', '${BROKEN}', file='test.bb', line=1) + + info = HobRecipeInfo('test.bb', metadata) + + variables = json.loads(zlib.decompress(info.recipe_variables)) + self.assertNotIn('BROKEN', variables) + +if __name__ == '__main__': + unittest.main() diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py index 4ee45d67a..679f016bd 100644 --- a/lib/bb/ui/buildinfohelper.py +++ b/lib/bb/ui/buildinfohelper.py @@ -8,8 +8,10 @@ import sys import bb +import json import re import os +import zlib import django from django.utils import timezone @@ -31,7 +33,7 @@ from orm.models import Target_Image_File, TargetKernelFile, TargetSDKFile from orm.models import Variable, VariableHistory from orm.models import Package, Package_File, Target_Installed_Package, Target_File from orm.models import Task_Dependency, Package_Dependency -from orm.models import Recipe_Dependency, Provides +from orm.models import Recipe_Dependency, RecipeVariable, Provides from orm.models import Project, CustomImagePackage from orm.models import signal_runbuilds @@ -1459,6 +1461,23 @@ class BuildInfoHelper(object): t.save() self.internal_state['recipes'][pn] = recipe + if any('recipe_variables' in recipe_data + for recipe_data in event._depgraph['pn'].values()): + recipe_variables = [] + for pn, recipe_data in event._depgraph['pn'].items(): + variables = recipe_data.get('recipe_variables') + if variables is None: + continue + recipe_variables.append(RecipeVariable( + build=self.internal_state['build'], + recipe=self.internal_state['recipes'][pn], + variables=variables, + variable_count=len(json.loads(zlib.decompress(variables))))) + with transaction.atomic(): + RecipeVariable.objects.filter( + build=self.internal_state['build']).delete() + RecipeVariable.objects.bulk_create(recipe_variables) + # we'll not get recipes for key w/ values listed in ASSUME_PROVIDED assume_provided = self.server.runCommand(["getVariable", "ASSUME_PROVIDED"])[0].split() diff --git a/lib/toaster/orm/migrations/0022_recipevariable.py b/lib/toaster/orm/migrations/0022_recipevariable.py new file mode 100644 index 000000000..455a8ce23 --- /dev/null +++ b/lib/toaster/orm/migrations/0022_recipevariable.py @@ -0,0 +1,27 @@ +# Generated by Django 4.2.5 on 2026-08-11 00:00 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('orm', '0021_eventlogsimports'), + ] + + operations = [ + migrations.CreateModel( + name='RecipeVariable', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('variables', models.BinaryField()), + ('variable_count', models.PositiveIntegerField(default=0)), + ('build', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='orm.build')), + ('recipe', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='orm.recipe')), + ], + options={ + 'unique_together': {('build', 'recipe')}, + }, + ), + ] diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py index e2f488ed8..c240836cc 100644 --- a/lib/toaster/orm/models.py +++ b/lib/toaster/orm/models.py @@ -1346,6 +1346,16 @@ class Recipe(models.Model): unique_together = (("layer_version", "file_path", "pathflags"), ) +class RecipeVariable(models.Model): + build = models.ForeignKey(Build, on_delete=models.CASCADE) + recipe = models.ForeignKey(Recipe, on_delete=models.CASCADE) + variables = models.BinaryField() + variable_count = models.PositiveIntegerField(default=0) + + class Meta: + unique_together = (("build", "recipe"), ) + + class Recipe_DependencyManager(models.Manager): use_for_related_fields = True diff --git a/lib/toaster/tests/db/test_db.py b/lib/toaster/tests/db/test_db.py index 072ab9436..7c48b1de3 100644 --- a/lib/toaster/tests/db/test_db.py +++ b/lib/toaster/tests/db/test_db.py @@ -24,6 +24,10 @@ import sys import pytest +from types import SimpleNamespace +from unittest.mock import Mock, patch +import json +import zlib try: from StringIO import StringIO @@ -34,6 +38,11 @@ from contextlib import contextmanager from django.core import management from django.test import TestCase +from django.utils import timezone + +from bb.ui.buildinfohelper import BuildInfoHelper, ORMWrapper +from orm.models import Build, Layer, Layer_Version, Project, Recipe +from orm.models import RecipeVariable @contextmanager @@ -56,3 +65,95 @@ class MigrationTest(TestCase): with capture(makemigrations) as output: self.assertEqual(output, "No changes detected\n") + + +class RecipeVariableTest(TestCase): + + def setUp(self): + now = timezone.now() + project = Project.objects.get_or_create_default_project() + self.build = Build.objects.create( + project=project, machine='', distro='', distro_version='', + started_on=now, completed_on=now, cooker_log_path='', + bitbake_version='', progress_item='') + layer = Layer.objects.create(name='test', layer_index_url='') + self.layer_version = Layer_Version.objects.create( + build=self.build, layer=layer, branch='', commit='', + local_path='/layer') + + def _store(self, recipe_data): + if 'recipe_variables' in recipe_data: + recipe_data = dict(recipe_data) + recipe_data['recipe_variables'] = zlib.compress(json.dumps( + recipe_data['recipe_variables'], separators=(',', ':') + ).encode('utf-8')) + helper = BuildInfoHelper.__new__(BuildInfoHelper) + helper.internal_state = {'build': self.build, 'targets': []} + helper.orm_wrapper = ORMWrapper() + helper.server = Mock() + helper.server.runCommand.return_value = ['', None] + helper._get_layer_version_for_path = Mock(return_value=self.layer_version) + event = SimpleNamespace(_depgraph={ + 'layer-priorities': [], + 'pn': {'test': dict({'filename': '/layer/test.bb'}, **recipe_data)}, + 'depends': {'test': []}, + 'tdepends': {}, + }) + + helper.store_dependency_information(event) + + return helper.internal_state['recipes']['test'] + + def test_dependency_information_stores_recipe_variables(self): + recipe = self._store({'recipe_variables': {'EMPTY': '', 'FOO': 'bar'}}) + + snapshot = RecipeVariable.objects.get(build=self.build, recipe=recipe) + self.assertEqual( + json.loads(zlib.decompress(snapshot.variables)), + {'EMPTY': '', 'FOO': 'bar'}) + self.assertEqual(snapshot.variable_count, 2) + + def test_old_payload_does_not_delete_recipe_variables(self): + recipe = self._store({'recipe_variables': {'FOO': 'bar'}}) + self._store({}) + + snapshot = RecipeVariable.objects.get(build=self.build, recipe=recipe) + self.assertEqual( + json.loads(zlib.decompress(snapshot.variables)), {'FOO': 'bar'}) + + def test_dependency_information_replaces_recipe_variables(self): + recipe = self._store({'recipe_variables': {'FOO': 'old'}}) + self._store({'recipe_variables': {'BAR': 'new'}}) + + snapshot = RecipeVariable.objects.get(build=self.build, recipe=recipe) + self.assertEqual( + json.loads(zlib.decompress(snapshot.variables)), {'BAR': 'new'}) + self.assertEqual(snapshot.variable_count, 1) + + def test_dependency_information_removes_stale_recipe_snapshot(self): + stale_recipe = Recipe.objects.create( + name='stale', version='', layer_version=self.layer_version, + file_path='stale.bb') + RecipeVariable.objects.create( + build=self.build, recipe=stale_recipe, + variables=zlib.compress(b'{"STALE":"value"}')) + + current_recipe = self._store({'recipe_variables': {'FOO': 'bar'}}) + + self.assertFalse(RecipeVariable.objects.filter( + build=self.build, recipe=stale_recipe).exists()) + self.assertTrue(RecipeVariable.objects.filter( + build=self.build, recipe=current_recipe).exists()) + + def test_dependency_information_rolls_back_failed_bulk_create(self): + recipe = self._store({'recipe_variables': {'FOO': 'old'}}) + with patch.object(RecipeVariable.objects, 'bulk_create', + side_effect=RuntimeError('injected failure')): + with self.assertRaisesRegex(RuntimeError, 'injected failure'): + self._store({'recipe_variables': {'FOO': 'new'}}) + + snapshots = RecipeVariable.objects.filter(build=self.build) + self.assertEqual(snapshots.count(), 1) + snapshot = snapshots.get(recipe=recipe) + self.assertEqual( + json.loads(zlib.decompress(snapshot.variables)), {'FOO': 'old'}) diff --git a/lib/toaster/tests/views/test_views.py b/lib/toaster/tests/views/test_views.py index e1adfcf86..78695a151 100644 --- a/lib/toaster/tests/views/test_views.py +++ b/lib/toaster/tests/views/test_views.py @@ -10,14 +10,16 @@ """Test cases for Toaster GUI and ReST.""" import os +import zlib import pytest from django.test import TestCase from django.test.client import RequestFactory from django.urls import reverse from django.db.models import Q -from orm.models import Project, Package +from orm.models import Build, Project, Package from orm.models import Layer_Version, Recipe +from orm.models import RecipeVariable from orm.models import CustomImageRecipe from orm.models import CustomImagePackage @@ -61,6 +63,14 @@ class ViewTests(TestCase): if BuildEnvironment.objects.count() == 0: BuildEnvironment.objects.create(betype=BuildEnvironment.TYPE_LOCAL) + @staticmethod + def _recipe_variables(build, recipe, values): + return RecipeVariable( + build=build, recipe=recipe, + variable_count=len(values), + variables=zlib.compress(json.dumps( + values, separators=(',', ':')).encode('utf-8'))) + def test_get_base_call_returns_html(self): """Basic test for all-projects view""" @@ -90,6 +100,149 @@ class ViewTests(TestCase): self.assertTrue(name_found, "project name not found in projects table") + def test_recipe_variables_tab_scopes_searches_and_escapes(self): + build = Build.objects.get(pk=1) + other_build = Build.objects.get(pk=2) + other_recipe = Recipe.objects.exclude(pk=self.recipe1.pk).first() + RecipeVariable.objects.bulk_create([ + self._recipe_variables(build, self.recipe1, { + 'SRC_URI': 'git://example.invalid/src', + 'SPECIAL': '', + }), + self._recipe_variables(other_build, self.recipe1, { + 'OTHER_BUILD': 'hidden', + }), + self._recipe_variables(build, other_recipe, { + 'OTHER_RECIPE': 'hidden', + }), + ]) + url = reverse('recipe', args=(build.pk, self.recipe1.pk, '5')) + + response = self.client.get(url, { + 'count': 100, + 'page': 1, + 'orderby': 'variable_name:+', + 'search': 'SRC_URI', + }) + + self.assertEqual(response.status_code, 200) + self.assertTemplateUsed(response, 'recipe.html') + self.assertContains(response, 'SRC_URI') + self.assertContains(response, 'git://example.invalid/src') + self.assertNotContains(response, 'OTHER_BUILD') + self.assertNotContains(response, 'OTHER_RECIPE') + + response = self.client.get(url, { + 'count': 100, + 'page': 1, + 'orderby': 'variable_name:+', + }) + self.assertContains(response, 'Variables (2)') + self.assertContains(response, '<script>alert(1)</script>') + self.assertNotContains(response, '') + + def test_recipe_variables_tab_redirects_paginates_and_handles_empty(self): + build = Build.objects.get(pk=1) + url = reverse('recipe', args=(build.pk, self.recipe1.pk, '5')) + + response = self.client.get(url) + self.assertEqual(response.status_code, 302) + self.assertIn('count=100', response.url) + self.assertIn('orderby=variable_name%3A%2B', response.url) + + RecipeVariable.objects.bulk_create([ + self._recipe_variables(build, self.recipe1, { + 'VAR_%02d' % index: str(index) for index in range(12) + }) + ]) + response = self.client.get(url, { + 'count': 10, + 'page': 2, + 'orderby': 'variable_name:+', + }) + self.assertEqual(response.status_code, 200) + self.assertEqual(response.context['objects'].paginator.count, 12) + self.assertEqual(response.context['objects'].number, 2) + self.assertContains(response, 'VAR_10') + self.assertContains(response, 'VAR_11') + self.assertNotContains(response, 'VAR_00') + + RecipeVariable.objects.filter(build=build, recipe=self.recipe1).delete() + response = self.client.get(url, { + 'count': 100, + 'page': 1, + 'orderby': 'variable_name:+', + }) + self.assertContains(response, 'defines or modifies no variables') + + def test_recipe_variables_tab_bounds_query_parameters(self): + build = Build.objects.get(pk=1) + RecipeVariable.objects.bulk_create([ + self._recipe_variables(build, self.recipe1, {'SRC_URI': 'value'}) + ]) + url = reverse('recipe', args=(build.pk, self.recipe1.pk, '5')) + + response = self.client.get(url, { + 'count': 'alert(document.domain)', + 'page': 1, + 'orderby': 'variable_value:+', + 'filter': 'variable_value__regex:(a+)+$', + }) + + self.assertEqual(response.status_code, 200) + self.assertEqual(response.context['objects'].paginator.per_page, 100) + self.assertNotContains(response, 'alert(document.domain)') + self.assertEqual( + [variable['variable_name'] + for variable in response.context['objects'].object_list], + ['SRC_URI']) + + response = self.client.get(url, { + 'count': 100, + 'page': 1, + 'orderby': 'variable_name:+', + 'search': ('SRC_URI ' * 20) + ('x' * 300), + }) + self.assertLessEqual(len(response.context['search_term']), 256) + self.assertLessEqual(len(response.context['search_term'].split()), 16) + + def test_built_recipes_table_shows_variable_count(self): + build = Build.objects.get(pk=1) + self._recipe_variables(build, self.recipe1, { + 'FOO': 'one', + 'BAR': 'two', + }).save() + url = reverse('recipes', args=(build.pk,)) + + response = self.client.get(url, { + 'format': 'json', + 'limit': 25, + 'page': 1, + }) + data = json.loads(response.content) + row = next(row for row in data['rows'] + if self.recipe1.name in row['name']) + + self.assertIn('Variables', [column['title'] for column in data['columns']]) + self.assertIn('>2', row['variable_count']) + + def test_recipe_packages_tab_shows_variable_count(self): + build = Build.objects.get(pk=1) + self._recipe_variables(build, self.recipe1, { + 'FOO': 'one', + 'BAR': 'two', + }).save() + + response = self.client.get(reverse( + 'recipe_packages', args=(build.pk, self.recipe1.pk)), { + 'count': 10, + 'page': 1, + 'orderby': 'name:+', + }) + + self.assertEqual(response.status_code, 200) + self.assertContains(response, 'Variables (2)') + def test_typeaheads(self): """Test typeahead ReST API""" layers_url = reverse('xhr_layerstypeahead', args=(self.project.id,)) diff --git a/lib/toaster/toastergui/buildtables.py b/lib/toaster/toastergui/buildtables.py index 327059d00..52c3dd779 100644 --- a/lib/toaster/toastergui/buildtables.py +++ b/lib/toaster/toastergui/buildtables.py @@ -239,7 +239,10 @@ class BuiltRecipesTable(BuildTablesMixin): def setup_queryset(self, *args, **kwargs): build = Build.objects.get(pk=kwargs['build_id']) self.static_context_extra['build'] = build - self.queryset = build.get_recipes() + self.queryset = build.get_recipes().annotate( + variable_count=Sum( + 'recipevariable__variable_count', + filter=Q(recipevariable__build=build), default=0)) self.queryset = self.queryset.order_by(self.default_orderby) def setup_columns(self, *args, **kwargs): @@ -323,6 +326,12 @@ class BuiltRecipesTable(BuildTablesMixin): hideable=False, field_name="version") + self.add_column( + title="Variables", + field_name="variable_count", + static_data_name="variable_count", + static_data_template='{{data.variable_count|default:0}}') + self.add_column(title="Dependencies", static_data_name="dependencies", static_data_template=depends_on_tmpl) diff --git a/lib/toaster/toastergui/templates/detail_pagination_bottom.html b/lib/toaster/toastergui/templates/detail_pagination_bottom.html index 15adfbc91..ddcba133e 100644 --- a/lib/toaster/toastergui/templates/detail_pagination_bottom.html +++ b/lib/toaster/toastergui/templates/detail_pagination_bottom.html @@ -41,8 +41,8 @@