From patchwork Mon Oct 19 21:35:07 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 101326 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.5864.1792445722706522357 for ; Mon, 19 Oct 2026 14:35:22 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=eePFpwqJ; spf=pass (domain: gmail.com, ip: 209.85.210.180, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-89d6bd67045so49563b3a.0 for ; Mon, 19 Oct 2026 14:35:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1792445722; x=1793050522; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=g0ilYJ69NZHzfNWIgFCKxy5rvc04Si6r3HLP9kL8JRU=; b=eePFpwqJ6dGnoQagIeaKS6zy6j48mMeuHnePcsYZdK61eOKRlydOWz2gMGJz39EkF3 VIKqBBWtIWzu7EMOgMmzHMVzCFDGEhNoUtMhOEo5qN3ZT5H0Z/hmPvP6gCA2DcYmKd4A ZiRF1talR87DWGn1VvZ/nAQLcC0Ktgq7nqirSeH8USaYJCOQr77rWv6zIsGz1wi/wV0e DeO6CknFtEaCUk4KYbm5+Yf4IOMtL1/b8kH+b4mKtgKDDePSJ1bixMxpDXbur4ylLiSP jPDGPJHSQTLjTlXkh3/VlP//e5KaUhq9+Vvfpj1M/S84OYf9iUERIRQglszjxXCVo0nE C9Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792445722; x=1793050522; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=g0ilYJ69NZHzfNWIgFCKxy5rvc04Si6r3HLP9kL8JRU=; b=Nc7TN2pXlv+FVS7sEtknicQRa4u33bYymM8tBVpesQJuZ+ZnKu8OA6VLlFhqg46NpK P1mUZrUonM48O5IrRCveL//mhYHNFaED9btaPHpc+Y+nBva1p0nyraQpEwUov3LsMlbn NwQnPJweeKJ+PFr6ruwaCj94HLV9O/CGlzzHCw0A6geFxJP/tG4iJ8NAqATsOldqgHhk T/YJzE7ZDBBQis/TOOSi1SkyaItTS2G1eJRTY9d/j8ZgWvIdKRVhA0heE2kgaZxMB9xf uYdWrcw522RQ6m/32gWa1uoNeSaI+SMkKQWG1u+GX2lZLTvcBHbsnvBTvsZ47RKuynWe OHyg== X-Gm-Message-State: AFq9FYKHufybhhfGAWMi8SC/gS7C4KdMK4pF9pCMapcLjISOzJZz2WD7 uUcEYiAREEsPe6wuY6EV5kozCJ3nqTCVqqbxqHYnjFuvvJeogjZczdLWfRCDBKBY X-Gm-Gg: AYBFou0/YTOdtLeSPbB7ukpZsHyWElqB3qM0x4kv6qc90hmsmA+ouhHN7vp2wXxRqN+ oOzqXyjE6ecJOynhPsltEii9qwqda3Eoll1g5ggBXQYabroejVod7tYaiVPbwUkqHDYMsAZoRdm AvYTHUsAUJTEUPzjFBNuNxxNnY+Vas/UBKAKW3Kgb8jfZXVUKlyqA0Lxq9UCQGUBvpICw+p76Tg 4Au1rdfle1qjB3eBXmA3imhykVjofeiFCAF2tCUdbO9LnClcaDMdHVUFUj14CDpmK9IbXI4Cfmi OQ51DMn8ETM2w+Ranvifjk0ljH5GGDQT5w2m+5OrOfcre/H9VSNvwU0IKXs06BEANl71aJ7VgEI ziFNuEDo/jPoIz3sb2WJEAo9VA8wqHSa0coZo63P/wnOkzVHo/qzrXhRGUO/Ybp+LXOQY74mxDP nqCQX5Gfc3/jK9AZjPFS0SjAm7HPLMRcWzsvWE9aekVKdz12PmOZLcpKhzMOQwoN0FijIkmwAND wC14yM3BVcEwJuh1kfU3ulwHCC5dXtcvHKgWGTxnw== X-Received: by 2002:a05:6a00:3908:b0:8a0:6417:8f9b with SMTP id d2e1a72fcca58-8a07c01b24emr212045b3a.32.1792445721876; Mon, 19 Oct 2026 14:35:21 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-89f5820b344sm4307747b3a.53.2026.10.19.14.35.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 19 Oct 2026 14:35:19 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 1/4] update_layer.py: skip layers within a layer on incremental updates Date: Mon, 19 Oct 2026 14:35:07 -0700 Message-ID: <20261019213510.4384-2-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261019213510.4384-1-tim.orling@konsulko.com> References: <20261019213510.4384-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Mon, 19 Oct 2026 21:35:28 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5101 On incremental updates the check for layers within a layer compared repository-relative git diff paths with absolute directory paths, so it never matched, and files in nested layers (e.g. meta-voltumna's vendored copies of oe-core and meta-openembedded) were indexed as part of the outer layer. Move the check into utils.nested_layer_dirs() and utils.in_nested_layer(), use it for all change types, and treat renames into a nested layer as deletes. Records already indexed this way are removed by a full rescan (update.py -r). AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- layerindex/update_layer.py | 39 ++++++++------------------------- layerindex/utils.py | 22 +++++++++++++++++++ tests/test_utils.py | 45 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+), 30 deletions(-) diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py index 55b3e14..441efc2 100644 --- a/layerindex/update_layer.py +++ b/layerindex/update_layer.py @@ -494,11 +494,7 @@ def main(): layerrecipes_add = [] # Check if any paths should be ignored because there are layers within this layer - removedirs = [] - for root, dirs, files in os.walk(layerdir): - for diritem in dirs: - if os.path.exists(os.path.join(root, diritem, 'conf', 'layer.conf')): - removedirs.append(os.path.join(root, diritem) + os.sep) + removedirs = utils.nested_layer_dirs(layerdir) if diff: # Apply git changes to existing recipe list @@ -515,19 +511,17 @@ def main(): for diffitem in diff.iter_change_type('R'): oldpath = diffitem.a_blob.path newpath = diffitem.b_blob.path - skip = False - for removedir in removedirs: - # FIXME what about files moved into removedirs? - if oldpath.startswith(removedir): - skip = True - break - if skip: + if utils.in_nested_layer(oldpath, removedirs, repodir): continue if oldpath.startswith(subdir_start): if not newpath.startswith(subdir_start): logger.debug("Treating rename of %s to %s as a delete since new path is outside layer" % (oldpath, newpath)) other_deletes.append(diffitem) continue + if utils.in_nested_layer(newpath, removedirs, repodir): + logger.debug("Treating rename of %s to %s as a delete since new path is in a layer within this layer" % (oldpath, newpath)) + other_deletes.append(diffitem) + continue (oldtypename, oldfilepath, oldfilename) = recipeparse.detect_file_type(oldpath, subdir_start) (newtypename, newfilepath, newfilename) = recipeparse.detect_file_type(newpath, subdir_start) if oldtypename != newtypename: @@ -606,12 +600,7 @@ def main(): for diffitem in itertools.chain(diff.iter_change_type('D'), other_deletes): path = diffitem.a_blob.path if path.startswith(subdir_start): - skip = False - for removedir in removedirs: - if path.startswith(removedir): - skip = True - break - if skip: + if utils.in_nested_layer(path, removedirs, repodir): continue (typename, filepath, filename) = recipeparse.detect_file_type(path, subdir_start) if typename == 'recipe': @@ -636,12 +625,7 @@ def main(): for diffitem in itertools.chain(diff.iter_change_type('A'), other_adds): path = diffitem.b_blob.path if path.startswith(subdir_start): - skip = False - for removedir in removedirs: - if path.startswith(removedir): - skip = True - break - if skip: + if utils.in_nested_layer(path, removedirs, repodir): continue (typename, filepath, filename) = recipeparse.detect_file_type(path, subdir_start) if typename == 'recipe': @@ -685,12 +669,7 @@ def main(): for diffitem in diff.iter_change_type('M'): path = diffitem.b_blob.path if path.startswith(subdir_start): - skip = False - for removedir in removedirs: - if path.startswith(removedir): - skip = True - break - if skip: + if utils.in_nested_layer(path, removedirs, repodir): continue (typename, filepath, filename) = recipeparse.detect_file_type(path, subdir_start) if typename == 'recipe': diff --git a/layerindex/utils.py b/layerindex/utils.py index e5bad40..fd29de0 100644 --- a/layerindex/utils.py +++ b/layerindex/utils.py @@ -57,6 +57,28 @@ def get_layer_var(config_data, var, logger): value = config_data.getVar(var, True) return value or '' +def nested_layer_dirs(layerdir): + """ + Return the directories below layerdir that contain a conf/layer.conf of + their own, i.e. other layers within this layer, each with a trailing + separator. + """ + dirs = [] + for root, subdirs, _ in os.walk(layerdir): + for subdir in subdirs: + if os.path.exists(os.path.join(root, subdir, 'conf', 'layer.conf')): + dirs.append(os.path.join(root, subdir) + os.sep) + return dirs + +def in_nested_layer(path, nested_dirs, repodir): + """ + Check whether path is within one of nested_dirs (as returned by + nested_layer_dirs()). path may be relative to repodir, as git diff paths + are, or absolute. + """ + fullpath = os.path.join(repodir, path) + return any(fullpath.startswith(nested_dir) for nested_dir in nested_dirs) + def is_deps_satisfied(req_col, req_ver, collections): """ Check whether required collection and version are in collections""" for existed_col, existed_ver in collections: diff --git a/tests/test_utils.py b/tests/test_utils.py index 2b9ebcb..95b5e04 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -5,6 +5,7 @@ # SPDX-License-Identifier: MIT import operator +import os import sys import types @@ -40,3 +41,47 @@ def fake_bb(monkeypatch): ]) def test_is_deps_satisfied(fake_bb, req_ver, collections, expected): assert utils.is_deps_satisfied('core', req_ver, collections) is expected + + +def make_layer(path): + (path / 'conf').mkdir(parents=True) + (path / 'conf' / 'layer.conf').write_text('') + + +@pytest.fixture +def repo_with_nested_layers(tmp_path): + """A repository whose layer (in subdirectory 'meta-top') contains other + layers, as when layers are vendored with git subtree.""" + repodir = tmp_path / 'repo' + layerdir = repodir / 'meta-top' + make_layer(layerdir) + (layerdir / 'recipes-core' / 'foo').mkdir(parents=True) + make_layer(layerdir / 'meta-sub') + make_layer(layerdir / 'meta-openembedded' / 'meta-oe') + # A directory whose name starts like a nested layer's, but isn't one + (layerdir / 'meta-subsidiary' / 'recipes-x').mkdir(parents=True) + return str(repodir), str(layerdir) + + +def test_nested_layer_dirs(repo_with_nested_layers): + repodir, layerdir = repo_with_nested_layers + assert sorted(utils.nested_layer_dirs(layerdir)) == [ + os.path.join(layerdir, 'meta-openembedded', 'meta-oe') + os.sep, + os.path.join(layerdir, 'meta-sub') + os.sep, + ] + + +@pytest.mark.parametrize('path, expected', [ + # Paths relative to the repository, as git diff reports them + ('meta-top/meta-openembedded/meta-oe/recipes-a/a/a_1.0.bb', True), + ('meta-top/meta-sub/recipes-b/b/b_1.0.bb', True), + ('meta-top/recipes-core/foo/foo_1.0.bb', False), + ('meta-top/meta-subsidiary/recipes-x/x/x_1.0.bb', False), + ('meta-top/meta-openembedded/README', False), +]) +def test_in_nested_layer(repo_with_nested_layers, path, expected): + repodir, layerdir = repo_with_nested_layers + nested = utils.nested_layer_dirs(layerdir) + assert utils.in_nested_layer(path, nested, repodir) is expected + # Absolute paths give the same answer + assert utils.in_nested_layer(os.path.join(repodir, path), nested, repodir) is expected From patchwork Mon Oct 19 21:35:08 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 101327 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.5865.1792445725757654947 for ; Mon, 19 Oct 2026 14:35:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=gv24MTCY; spf=pass (domain: gmail.com, ip: 209.85.210.182, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-8a06c60065fso964874b3a.1 for ; Mon, 19 Oct 2026 14:35:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1792445725; x=1793050525; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=t50t8m8INy93F4pg5Wr++Bl+EN8ToRBd6SBMm9clHis=; b=gv24MTCYnnn7qCaPbtxwn19dNpHXhRhc/sbNtkIBTn2tPiNxCTB3U0jTQ9aXMRMjAA 6IFvZXt7649Wl1oifw5sS6W5Asx/MYxZFYkU/0ie0l/vw+BhVPv9zkT6r8iH6nwwqPiS 24wZtjvpuQPxZtRX3T8c6DRmhwNLYvBHlqYxIi4PqakKDoVmQvKlLvMpUG2usxufeRuA KZlk7jy9u9gvFfZUaYgdfhAh7zHYxXgqoudTBLm7MxQqNNlsr6NVgh833nm9y8Okw/vn DVXWCH3WdxvL7zxKPN1h3K+EL5HeEib8K/nilnVxlqwv0mLIgGL+JF8/Icz1HpC94sPa 7d3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792445725; x=1793050525; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=t50t8m8INy93F4pg5Wr++Bl+EN8ToRBd6SBMm9clHis=; b=1YikZdHH8klx3MMvHp20P2ZZmuFcf+PYlnj5PGwsLKVRx9B60IztLP2hGhcZubLMBY +Z4SrFwLcxR7OR2vobIbKDd2wnPrQ+676V4WLeb23EAew//SEJzTwi0Fr3+EvNMocYJc G8fsn293FgYCzgc+4jrSjCrOHBW+wQCQ0sR6cMumQQIhTTUjEvfsiM7KFKX+h2/+bOXE g5lbdx1mbxqNOdU07Jno2Z8ePzSJ6fgd9+L/2Tc52XPWtpI++A8B71LIGQScKBG9A7DY uOn0aa7pDVBL0gjrZg0Cx8FZyrqEiTXSjld+I9OgVKUO0pbI3/SgAhKtIzfI1tsYm76Y I5RQ== X-Gm-Message-State: AFq9FYLmMrmAqI91hwb7yDMHA4TJ0LpNqh/9viAWWSHkPxfRqTNq/Eyt Vq1jg4k2WMLn4KrURSPTkDl3A6cK1pzVcdwafj2FlnQIoLIMh9Tn2VUMsek+/7fH X-Gm-Gg: AYBFou2bgYCl+3vSG2mIuIAfh/IDjqa00fvimm6xPG6uUjFjkrP/kw15WQ7QGHGKrJi ys3lIDJkXAqqGL05Or+3xlRLAGWTimg+wECplktjZ5RIkEWXrpVbWeO2gXfAzqmRBLug2b10R8E o6wBnOSM3l05JKNGWCgd10xRms7anYOL+wqmNAlX8wlkfEViIh928kGrx0kHHsy8UgiM8TPEPKP /vJchlfuGgR8DZVm/IFmfjlUHQAHeVKITyi3DYR8LoIxSecyzJWUlH89L2AjE6na4O9p04WIBPu zwX67fSfhXGjpD78xY8HP5o3ddNvIeJ9Gh7RTBxBm0kBYYPEgp3FJkDWMhPWqIdmGqkLW9g9gO5 S/cbB2kbHPG9r4OWclnw87mQmZNIhTr5HYxBJq780WVEFGyJe4StRFLYI+IA5oocnVhUn00/heA 4oIr8LqRyPve4eJ5igODkz6otXmgFFLOw5ctOOjjzx9Jvp3YvTFgMvfW6voVboqQMXZnoPhfPzK apCQsXvwAnRPywbK+eFHP+M/br+Zbs= X-Received: by 2002:a05:6a00:4106:b0:89e:5133:d98b with SMTP id d2e1a72fcca58-8a04ba22ddamr6806314b3a.24.1792445724987; Mon, 19 Oct 2026 14:35:24 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-89f5820b344sm4307747b3a.53.2026.10.19.14.35.22 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 19 Oct 2026 14:35:23 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 2/4] update: copy a new layer branch's subdirectory from the newest branch Date: Mon, 19 Oct 2026 14:35:08 -0700 Message-ID: <20261019213510.4384-3-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261019213510.4384-1-tim.orling@konsulko.com> References: <20261019213510.4384-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Mon, 19 Oct 2026 21:35:28 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5102 When a layer is first found on a new branch, vcs_subdir was copied from get_layerbranch(None), an arbitrary and usually the oldest layer branch. meta-tq has since moved into a subdirectory, so its wrynose update failed with "conf/layer.conf not found". Copy it from the newest non-comparison branch instead. AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- layerindex/models.py | 9 ++++++ layerindex/update.py | 2 +- layerindex/update_layer.py | 2 +- tests/test_latest_layerbranch.py | 52 ++++++++++++++++++++++++++++++++ 4 files changed, 63 insertions(+), 2 deletions(-) create mode 100644 tests/test_latest_layerbranch.py diff --git a/layerindex/models.py b/layerindex/models.py index a72a72b..a9e9bdf 100644 --- a/layerindex/models.py +++ b/layerindex/models.py @@ -162,6 +162,15 @@ class LayerItem(models.Model): return res[0] return None + def get_latest_layerbranch(self): + """ + Return the layer branch for the newest non-comparison branch (by + branch sort priority), e.g. to copy settings such as vcs_subdir from + when the layer is first seen on a new branch. + """ + return self.layerbranch_set.filter(branch__comparison=False).order_by( + models.F('branch__sort_priority').asc(nulls_last=True), '-id').first() + def active_maintainers(self): matches = None for layerbranch in self.layerbranch_set.all(): diff --git a/layerindex/update.py b/layerindex/update.py index dd730ca..ca68e3e 100755 --- a/layerindex/update.py +++ b/layerindex/update.py @@ -389,7 +389,7 @@ def main(): layerbranch.branch = branchobj layerbranch_source = layer.get_layerbranch(branchobj) if not layerbranch_source: - layerbranch_source = layer.get_layerbranch(None) + layerbranch_source = layer.get_latest_layerbranch() if layerbranch_source: layerbranch.vcs_subdir = layerbranch_source.vcs_subdir diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py index 441efc2..0f49371 100644 --- a/layerindex/update_layer.py +++ b/layerindex/update_layer.py @@ -380,7 +380,7 @@ def main(): layerbranch.actual_branch = options.actual_branch layerbranch_source = layer.get_layerbranch(branch) if not layerbranch_source: - layerbranch_source = layer.get_layerbranch(None) + layerbranch_source = layer.get_latest_layerbranch() if layerbranch_source: layerbranch.vcs_subdir = layerbranch_source.vcs_subdir layerbranch.save() diff --git a/tests/test_latest_layerbranch.py b/tests/test_latest_layerbranch.py new file mode 100644 index 0000000..33e0844 --- /dev/null +++ b/tests/test_latest_layerbranch.py @@ -0,0 +1,52 @@ +# layerindex-web - tests for choosing the layer branch to copy settings from +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT + +import pytest +from django.test import TestCase + +from layerindex.models import Branch, LayerBranch, LayerItem + + +@pytest.mark.django_db +class TestLatestLayerBranch(TestCase): + """When a layer is first seen on a new branch, the update script copies + vcs_subdir from an existing layer branch. It must use the newest branch: + meta-tq moved into a meta-tq/ subdirectory, and copying the empty + subdirectory of an old branch made the new branch's update fail with + "conf/layer.conf not found".""" + + def setUp(self): + self.layer = LayerItem.objects.create(name='meta-tq', status='P', layer_type='B', + summary='test', description='test', + vcs_url='https://example.com/meta-tq.git') + + def branch(self, name, sort_priority, comparison=False): + return Branch.objects.create(name=name, bitbake_branch=name, sort_priority=sort_priority, + comparison=comparison) + + def layerbranch(self, branch, subdir): + return LayerBranch.objects.create(layer=self.layer, branch=branch, vcs_subdir=subdir) + + def test_newest_branch_wins_over_oldest_record(self): + # Oldest branch first, so it is the first record (as for meta-tq) + self.layerbranch(self.branch('krogoth', 20), '') + self.layerbranch(self.branch('kirkstone', 5), 'meta-tq') + newest = self.layerbranch(self.branch('scarthgap', 3), 'meta-tq') + self.assertEqual(self.layer.get_latest_layerbranch(), newest) + self.assertEqual(self.layer.get_latest_layerbranch().vcs_subdir, 'meta-tq') + + def test_comparison_and_unprioritised_branches_are_not_preferred(self): + self.layerbranch(self.branch('comparison', 1, comparison=True), 'other') + self.layerbranch(self.branch('unsorted', None), 'other') + newest = self.layerbranch(self.branch('scarthgap', 3), 'meta-tq') + self.assertEqual(self.layer.get_latest_layerbranch(), newest) + + def test_only_unprioritised_branches(self): + lb = self.layerbranch(self.branch('unsorted', None), 'meta-tq') + self.assertEqual(self.layer.get_latest_layerbranch(), lb) + + def test_no_layer_branches(self): + self.assertIsNone(self.layer.get_latest_layerbranch()) From patchwork Mon Oct 19 21:35:09 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 101328 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.5866.1792445728808583086 for ; Mon, 19 Oct 2026 14:35:28 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=VXqxebly; spf=pass (domain: gmail.com, ip: 209.85.210.169, mailfrom: ticotimo@gmail.com) Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-8a06e79e152so592581b3a.0 for ; Mon, 19 Oct 2026 14:35:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1792445728; x=1793050528; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WonUais3wbrsAOKZn2Ok1GFMVSxmjcFXngmV8veTCMU=; b=VXqxeblyvLv+5xX6UC0ZOgUDhSWpKJ5fXyz0CPZWjkmrgqjQaix+FPeZ30QTBy7aDM G2EpAgsu4eNTYxyd8J9FEVv+a5FuOSEEveAPyZSKW+/lkl37apwdJOSgc99MhWyWjNuW XzxbNheFahIP9ssq5wi7im+OfyifUEDFMTiZGUH2HpwG0Cug+Wr4caLF4/5DtYNJFy7x KZytNHJt7FgqTOGPGDi0l+XEuhtdRIcsQ63sLjTYukHWGwIeWawtHg38FbnxQLYoWMDy TDOnukkdfLN0OF5enkk1pM32/rTc+eobnMzl05COPyU2lH2ayHTlNejRfwdKEpa+JEED 5vxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792445728; x=1793050528; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=WonUais3wbrsAOKZn2Ok1GFMVSxmjcFXngmV8veTCMU=; b=U7wakVCGX3yfCBtQiiXTDmCTwqlOPBpLox1uQInD9EaWIwZ4C2U7DDCJqP4ZYX2Zm/ eGICB3UD4cy/nUkpUsqt3oRx/XUNhtCVUendUD1LFCM1M+Bf447F3+sfcOxsZ2aI+WJ6 g1ZA8YZaMBxXNkcQbDqQTevqWjryqoj4AQfm4kVQs87l+9ktTjC819rS3vPvdBwNab31 qSsWUOVjarmkdM8u50d1qGEY4+7K7M2sge1tY9XDoJBCV87RyXb7NRpP5+5rnYnHcuVM j4sTEcgPc+KpNJLfeIZnLiXxGSxbZzEl1K+iM4SHoFyR54Xv3CqvyLZlj6itK6itl3GL mGiw== X-Gm-Message-State: AFq9FYJJXqwFKG5nYUQPvBYsWMecksw2YZK95twIB8a8yYXHjA7WkFkr 7tGoW+04lqXvCXfYVueV4UgtjMGu4WH4QiUDGvIgdtbTa8p5xwlf+5NgbPtN+g1c X-Gm-Gg: AYBFou0LfvNKtkQzcVJrQfw9pGqo9V1nQZtXhiqLrLql+Tk5H9+EWLJbiQrrp4Gs++N 5Zrjtw/FfO18NnuZsmKVNjr7TLeMq9XSN6KxpMhiNHBYEW9kH1789FwMbyYa7jbkvOEyHFAHh9l iU+pVBByryhLNr8n9YpXgMa37X4UZLsHtw9azj87dZl6Ybw0JYjHax3v7AiFFdexobThFPIytuS XSjT/pa5MA1jCQTlljPZC0WbxBh7QFOxiZKnIhEdMLniJfXrWVT8FnXTfiCOTluU0cTyNF/bLJm mLN6iXbgl53c/+eCjNXAbhAT53eg8f2/KnZ0OOqfYP62qEYLIc0uUBYqRQX+OruGyvz1DALcnJW g006lGrCbdJN0nQm17tkNkqTgQXfwWQJRrTFnYLwTY9FzGhlvH1frNLGVoE2+Gv8yR7VIjAMUAy 7NtWH7dHIG1pObq6iFK83nLtAAiWQOXnu11htTJ2sn7J5PtCsZGP81Y1lsYrKaA87ShrI1ESAIg sRGmgfV8aondEVQePuMGE3QOBQ/k1d2 X-Received: by 2002:a05:6a00:22ca:b0:891:d071:1e55 with SMTP id d2e1a72fcca58-8a04ba1ebf1mr5650900b3a.21.1792445728148; Mon, 19 Oct 2026 14:35:28 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-89f5820b344sm4307747b3a.53.2026.10.19.14.35.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 19 Oct 2026 14:35:26 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 3/4] recipeparse: don't add the core layer to BBPATH twice Date: Mon, 19 Oct 2026 14:35:09 -0700 Message-ID: <20261019213510.4384-4-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261019213510.4384-1-tim.orling@konsulko.com> References: <20261019213510.4384-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Mon, 19 Oct 2026 21:35:38 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5103 The core layer is in BBLAYERS, so its layer.conf already appends it to BBPATH. Adding it to the initial BBPATH as well made include_all read maintainers.inc twice, giving "Duplicate inclusion" warnings. Leave it out of the initial BBPATH, and use core_layerdir in BBLAYERS instead of assuming "meta". AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- layerindex/recipeparse.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/layerindex/recipeparse.py b/layerindex/recipeparse.py index d593def..d8486e7 100644 --- a/layerindex/recipeparse.py +++ b/layerindex/recipeparse.py @@ -66,10 +66,11 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout utils.checkout_repo(core_repodir, "origin/%s" % core_branchname, logger=logger) if not os.path.exists(os.path.join(core_layerdir, 'conf/bitbake.conf')): raise RecipeParseError("conf/bitbake.conf not found in core layer %s - is subdirectory set correctly?" % core_layer.name) - # The directory above where this script exists should contain our conf/layer.conf, - # so add it to BBPATH along with the core layer directory + # The directory above where this script exists should contain our conf/local.conf, + # so add it to BBPATH. The core layer is added by its layer.conf via BBLAYERS + # below; listing it here as well would put it in BBPATH twice. confparentdir = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')) - os.environ['BBPATH'] = str("%s:%s" % (confparentdir, core_layerdir)) + os.environ['BBPATH'] = str(confparentdir) # Change into a temporary directory so we don't write the cache and other files to the current dir if not os.path.exists(settings.TEMP_BASE_DIR): @@ -84,7 +85,7 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout if not classic: # We need this to avoid problems with AVAILABLE_LICENSES f.write('COREBASE = "%s"\n' % core_repodir) - f.write('BBLAYERS = "%s/meta"\n' % core_repodir) + f.write('BBLAYERS = "%s"\n' % core_layerdir) pass if logger: From patchwork Mon Oct 19 21:35:10 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 101329 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.5914.1792445732011318932 for ; Mon, 19 Oct 2026 14:35:32 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20251104 header.b=RvU/1USi; spf=pass (domain: gmail.com, ip: 209.85.215.180, mailfrom: ticotimo@gmail.com) Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-cd69954c755so16100a12.3 for ; Mon, 19 Oct 2026 14:35:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1792445731; x=1793050531; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=YA1gg6BoUoVKTH8l6gbRE0hIecZchtZ78EYq2sP4l78=; b=RvU/1USiHD2Suu58zyzmY8zli43RGUyf0yYdoCeINADdFJ6VDKCjt1CZg5LQTaKffn qFQGzJj/M0kOgOpvE6eyUvnLFG4Yqjc1LuOfcUEO6n3ogr/JKG8MBSt9vVpQZD7ljy/6 69B8EDJ4nkOSSU7GhieJ3P/1MF18mJ93CYZQkqKALJR2lUKBqPQ+txzT79VwbmYZDkuS I/D6AOQSOm+XSFT+XTZsNAaXOxLrkcnI78kGDT8x/Cxk0gtiXDtXMfhnlPuDq4AMNRmz zdnl4uyeGLQIcT0g6S8ox+vDCPUTXuJNVvGi2s1rStWtVDQTe+YuPLMih/Ly9Tz3WWPy CwbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792445731; x=1793050531; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=YA1gg6BoUoVKTH8l6gbRE0hIecZchtZ78EYq2sP4l78=; b=FeSpoIE5zqUjSaPWmsEsZQ7xOeAHYfkAdNdyDQiFyKWX4lhYTETVDtX9sl9Rw+wa3a vbLS/BX54MSld1F7ZkW+rBob00bR0iNYQ5AI7XL/S+hlx5tVGSJRdCcj+8m66VycO+RN tKQORPm8P5jDlTYKaBqgZ4Y9xEj+uc7kiKML72Aef+2PRHRuBGRDqnlKnViEBmHYrcG9 lAntM2VpFQXBwHsj7zMoKICq40PxR524zwhpxlj/LFOq2p0i7WmjvWYuD7GZ1UPiv32q y/IpfhRAoVwQ4/rvxXErwH+SxVmLzORgw41i9LxHH6IlY1oDqUftz7prdi6TkSSIRJ5D mNeA== X-Gm-Message-State: AFq9FYIZ7QCJXmaEPoWa+d79ex+plaVTaXs+XcS07yCLa72YBkWsPzj4 5sg3FIORG92M4YK34oqq1pZ/4auOLvitbK+2jQzrwgaTZ20cXFZVlSqb1rpweKUj X-Gm-Gg: AYBFou0Ubnay4G1FYi3r/U3LEvZOAAPvW/RBBX48Gv+alGhtL2ED4agMJU7g6SNlq+R Mk+Dpzh3yVZ03Jvb0ghQLhCfkYGHAwSOw1N5JvWER4THYl/mSc24OW12imtuD7E3tuQ4uzkSC2G zB1gQTemuKT7BNrmbZRIAbaxa6CVaSd0bs+B/+2etJ9GefVdZepmRtnJIEYgzalnMMFIvm3yxRz LCaoxhKhS03vvfuUYZ+O2w/tzx2jsb0Vyi5FMrYjAtJXJdaWqVZ7+bhH4FZtn7RQaGQpxVEJgcO fpfS195HhIBB9uUvU6pZvYrHUtFV0HpEuQSA/XIPu0KcBL+K5e8qgmylgIer3zACVMxgbTkFbHK 7csDM9VB9v6fhPibbkj6l/SMDBvzTHK2SXYz08Q/dxCGEk4nojxS2lo4OjnIRcNRC/Em+7aFWnC T6Nt3sH3zI6JL22LiqItLCwYikRs4K02tIRxUSEQSAUbl8ytlfIWBpv0r6poq12C7rQevSvHC/q payt4DChuCO3xmSylBxFgNfdNaau2I= X-Received: by 2002:a05:6a20:918b:b0:3e2:6b5a:fea3 with SMTP id adf61e73a8af0-3e26b5b095fmr658991637.71.1792445731357; Mon, 19 Oct 2026 14:35:31 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-89f5820b344sm4307747b3a.53.2026.10.19.14.35.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 19 Oct 2026 14:35:29 -0700 (PDT) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 4/4] layerindex/models.py: store long recipe values without truncating Date: Mon, 19 Oct 2026 14:35:10 -0700 Message-ID: <20261019213510.4384-5-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261019213510.4384-1-tim.orling@konsulko.com> References: <20261019213510.4384-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Mon, 19 Oct 2026 21:35:38 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5104 Several Recipe fields hold free-form values from the metadata with no real length limit, and the update script kept truncating them (e.g. linux-firmware's LICENSE at 4688 characters). Raising CharField limits again only moves the problem and runs into MariaDB's per-row VARCHAR limit, so make Recipe.summary, license, provides, bbclassextend, inherits and configopts, and PackageConfig.with_option and without_option, TextFields. Recipe.pn is indexed, so raise it to 255 characters instead. AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- .../migrations/0055_recipe_text_fields.py | 58 +++++++++++++++++++ layerindex/models.py | 18 +++--- tests/test_recipe_fields.py | 53 +++++++++++++++++ 3 files changed, 120 insertions(+), 9 deletions(-) create mode 100644 layerindex/migrations/0055_recipe_text_fields.py create mode 100644 tests/test_recipe_fields.py diff --git a/layerindex/migrations/0055_recipe_text_fields.py b/layerindex/migrations/0055_recipe_text_fields.py new file mode 100644 index 0000000..0714623 --- /dev/null +++ b/layerindex/migrations/0055_recipe_text_fields.py @@ -0,0 +1,58 @@ +# Generated by Django 5.2.18 on 2026-10-19 03:19 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('layerindex', '0054_alter_sitenotice_expires'), + ] + + operations = [ + migrations.AlterField( + model_name='packageconfig', + name='with_option', + field=models.TextField(blank=True), + ), + migrations.AlterField( + model_name='packageconfig', + name='without_option', + field=models.TextField(blank=True), + ), + migrations.AlterField( + model_name='recipe', + name='bbclassextend', + field=models.TextField(blank=True), + ), + migrations.AlterField( + model_name='recipe', + name='configopts', + field=models.TextField(blank=True), + ), + migrations.AlterField( + model_name='recipe', + name='inherits', + field=models.TextField(blank=True), + ), + migrations.AlterField( + model_name='recipe', + name='license', + field=models.TextField(blank=True), + ), + migrations.AlterField( + model_name='recipe', + name='pn', + field=models.CharField(blank=True, max_length=255), + ), + migrations.AlterField( + model_name='recipe', + name='provides', + field=models.TextField(blank=True), + ), + migrations.AlterField( + model_name='recipe', + name='summary', + field=models.TextField(blank=True), + ), + ] diff --git a/layerindex/models.py b/layerindex/models.py index a9e9bdf..425af53 100644 --- a/layerindex/models.py +++ b/layerindex/models.py @@ -469,22 +469,22 @@ class Recipe(models.Model): layerbranch = models.ForeignKey(LayerBranch, on_delete=models.CASCADE) filename = models.CharField(max_length=255) filepath = models.CharField(max_length=255, blank=True) - pn = models.CharField(max_length=100, blank=True) + pn = models.CharField(max_length=255, blank=True) pv = models.CharField(max_length=100, blank=True) pr = models.CharField(max_length=100, blank=True) pe = models.CharField(max_length=100, blank=True) - summary = models.CharField(max_length=200, blank=True) + summary = models.TextField(blank=True) description = models.TextField(blank=True) section = models.CharField(max_length=100, blank=True) - license = models.CharField(max_length=3072, blank=True) + license = models.TextField(blank=True) homepage = models.URLField(blank=True) bugtracker = models.URLField(blank=True) - provides = models.CharField(max_length=2048, blank=True) - bbclassextend = models.CharField(max_length=100, blank=True) - inherits = models.CharField(max_length=383, blank=True) + provides = models.TextField(blank=True) + bbclassextend = models.TextField(blank=True) + inherits = models.TextField(blank=True) updated = models.DateTimeField(auto_now=True) blacklisted = models.CharField(max_length=255, blank=True) - configopts = models.CharField(max_length=4096, blank=True) + configopts = models.TextField(blank=True) srcrev = models.CharField(max_length=64, blank=True) class Meta: @@ -639,8 +639,8 @@ class Patch(models.Model): class PackageConfig(models.Model): recipe = models.ForeignKey(Recipe, on_delete=models.CASCADE) feature = models.CharField(max_length=255) - with_option = models.CharField(max_length=255, blank=True) - without_option = models.CharField(max_length=255, blank=True) + with_option = models.TextField(blank=True) + without_option = models.TextField(blank=True) build_deps = models.CharField(max_length=255, blank=True) def __str__(self): diff --git a/tests/test_recipe_fields.py b/tests/test_recipe_fields.py new file mode 100644 index 0000000..8b8dc42 --- /dev/null +++ b/tests/test_recipe_fields.py @@ -0,0 +1,53 @@ +# layerindex-web - tests for storing long recipe values +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT + +import pytest +from django.test import TestCase + +from layerindex.models import Branch, LayerBranch, LayerItem, PackageConfig, Recipe + + +@pytest.mark.django_db +class TestLongRecipeValues(TestCase): + """Values longer than the old field limits, as seen in real layers + (e.g. linux-firmware's LICENSE, nativesdk recipes' inherits), must be + stored in full rather than truncated.""" + + def setUp(self): + branch = Branch.objects.create(name='master', bitbake_branch='master') + layer = LayerItem.objects.create(name='meta-test', status='P', layer_type='A', + summary='test', description='test', + vcs_url='git://example.com/meta-test.git') + self.layerbranch = LayerBranch.objects.create(layer=layer, branch=branch) + + def test_long_values_are_not_truncated(self): + values = { + 'pn': 'p' * 112, + 'summary': 's' * 264, + 'license': ' & '.join(['LicenseRef-Firmware-%d' % i for i in range(250)]), + 'provides': 'virtual/x ' * 270, + # As generated by core-image-ptest.bb: one multiconfig variant per ptest + 'bbclassextend': 'native nativesdk ' + ' '.join(['mcextend:python3-ptest-%d' % i for i in range(150)]), + 'inherits': 'class ' * 110, + 'configopts': '--enable-thing ' * 290, + } + with self.assertNoLogs('LayerIndexModels', 'WARNING'): + recipe = Recipe.objects.create(layerbranch=self.layerbranch, filename='test_1.0.bb', **values) + pc = PackageConfig.objects.create(recipe=recipe, feature='tests', + with_option='--with-x ' * 45, without_option='--without-x ' * 30) + recipe.refresh_from_db() + pc.refresh_from_db() + for field, value in values.items(): + self.assertEqual(getattr(recipe, field), value, field) + self.assertEqual(pc.with_option, '--with-x ' * 45) + self.assertEqual(pc.without_option, '--without-x ' * 30) + + def test_short_fields_are_still_truncated(self): + # Fields that keep a length limit are still truncated, with a warning + with self.assertLogs('LayerIndexModels', 'WARNING'): + recipe = Recipe.objects.create(layerbranch=self.layerbranch, filename='test_1.0.bb', pv='1' * 150) + recipe.refresh_from_db() + self.assertEqual(len(recipe.pv), 100)