From patchwork Sun Nov 15 00:35:01 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104072 X-Patchwork-Delegate: ticotimo@gmail.com Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.7626.1794702923357278221 for ; Sat, 14 Nov 2026 16:35:23 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=UPAySoeZ; spf=pass (domain: gmail.com, ip: 209.85.216.42, mailfrom: ticotimo@gmail.com) Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-3ada4c06b7bso1183437a91.2 for ; Sat, 14 Nov 2026 16:35:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794702923; x=1795307723; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=5zQIlcf6igv8ZMfOTPYoi4S+xdDj9g/4a1seM7IGeaI=; b=UPAySoeZZgRUiHe4FE+3/9NSDuX3AHWyN5uaPC9PGKhPuM4BjqnmFyzeWkLS8aScRR AWHxbBgFmusWoJBv7NQ+8uKe3odq/u3unanWySxFTg6NO/5pEe3sbDo6fkOXKCDbhtFz DSQUuBfGsl6NzGJw3LtU2Y7XWPgOoNjX16Usaif3k91gbWb+8BmupxPZtK7xwGms0V5Z 4mhXIeOKmp48bCEjfTImKnpONmYhnxwsmaSDiAfNkkKlqXbOrGaMa2tW/LDAh4Nxe6tc mpocmK+luBtS6tV0LWlooS6J9UkxnjwAHOG4+S43PzQnJkOvv0UiXfC2jGDhOEqBPmHs pTaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794702923; x=1795307723; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=5zQIlcf6igv8ZMfOTPYoi4S+xdDj9g/4a1seM7IGeaI=; b=WWqUrhQ9vSOBsDTJPq2nxJkUKu2QZdIB9Y6XQnZkb2wcLNsZ2fmyqSCX0dfclLErVR iiSdN9SM4r8q/NJ0htVoocyq3Duwywk5qZMI+5D+eo3qnt2WTApHjbTJMqUndXpR7E4p 1xdzdht7DiFdMfmIzF3BGEzVf95k+B/aIZKmZ5WgSr4B9a+AAH9x47xnSlwNXwrpCzlN wtcjom9K2AIf5d8hEuM+O1CkFjpf1LuImDSW1V4LlweedzkCxc2E3rcR9XzMbyyolbiQ 2c/jnuMA8XJZfJMYWTJdV2/LzQkWXX8rJ1DpzCwngCb5hUhIhCQ3PtSJjc6ldYnzPIi/ WrQA== X-Gm-Message-State: APuU4Z8fC0FfK9uAOApeOhE1xIKPZSm+nHXLS+wAh8siKrJngsjQX/8x 0Exm0wUGM/JQdLZEY5WNyphXy8X7A/9N0xFbzm/OZaIg7XYq8WdUWDbkH76LZKCK X-Gm-Gg: AYBFou2HIU1HbDrsj8jDAdYGY9e6fVuvPSwemKgXiTVnWseYKO8tjESihcIAf3OijFd TaZpRHStN2iSsYbBxHaBzNLagZ5zjHZyLXE6sffhuIzo9HlImXYG96Sqqc2WTFfHrUCTy3P7oh8 Sn1lQC1d3SqKsfpOmdyeXl/MrmcHGhSMhCvNHPFt/5kJ6mq2enIwRXtav/StkvlMJLevVbKBHlg iNsMDKKFptvmCKjIOOgo8ISig9w1n4KkxR2SpepZXnSQAuoDbcgEn0wAS1pd2j379EN4jigRcA/ HtXtuL8GVeOYFA8dWvtvj0eHJf8I9k6bCypUiGgPyOQoPGjeeYmKDxpoYTTsfdxkTixCvqoDevI 7g5icdlf1tXd1STkBZivbT6ItUWwzj6M1vMIFI1o6JRF5navd/JHApeUhM5HxIxhO8INGfLK7Iz 4nnglmI4+a7SmD1Ztpf8OUin+BPArHexYTjl+P24owdK2TzPgyUPRmr9NCedtiPcgRGoe/NTsH/ /VjGRSsF5wtw0hSdkj08+2pnKJ29Iaq7PYkCqiOZ7LptfamdXguwwqGE3mfaVbnrExfWDKQb+ZL KZ2OKmfyFw03xlWDN5r5OgFVRhLD6qWPRFJgYa7nZWEB1s5FWsO9BlJ4jU+H9g== X-Received: by 2002:a17:90b:574b:b0:3ae:2969:903f with SMTP id 98e67ed59e1d1-3b1b0b8bcb7mr6057137a91.45.1794702922483; Sat, 14 Nov 2026 16:35:22 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3b1b39fb9a3sm5464089a91.3.2026.11.14.16.35.21 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 14 Nov 2026 16:35:21 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 1/6] rrs/tools/common.py: don't write a log file without TOOLS_LOG_DIR Date: Sat, 14 Nov 2026 16:35:01 -0800 Message-ID: <20261115003506.53654-2-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261115003506.53654-1-tim.orling@konsulko.com> References: <20261115003506.53654-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Sun, 15 Nov 2026 00:35:26 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5379 get_logger() always added a log file at TOOLS_LOG_DIR/, but TOOLS_LOG_DIR is empty by default, so the file was created in the current directory. In the Docker image that is /opt/layerindex, which the layers user can't write to, so every RRS update script failed as soon as it started. Only log to the console when TOOLS_LOG_DIR is not set. [YOCTO #16499] AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- docker/settings.py | 2 +- rrs/tools/common.py | 16 ++++++++++------ settings.py | 2 +- tests/test_rrs_logger.py | 38 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 50 insertions(+), 8 deletions(-) create mode 100644 tests/test_rrs_logger.py diff --git a/docker/settings.py b/docker/settings.py index a610052..9f18a55 100644 --- a/docker/settings.py +++ b/docker/settings.py @@ -309,7 +309,7 @@ AXES_COOLOFF_TIME = 1 # Full path to directory to store logs for dynamically executed tasks TASK_LOG_DIR = "/opt/layerindex-task-logs" -# Full path to directory where rrs tools stores logs +# Full path to a directory where the rrs tools also write their logs, if any TOOLS_LOG_DIR = "" USE_X_FORWARDED_HOST = True diff --git a/rrs/tools/common.py b/rrs/tools/common.py index 0cac1f7..0caf1c9 100644 --- a/rrs/tools/common.py +++ b/rrs/tools/common.py @@ -34,12 +34,16 @@ def get_logger(name, settings): handler.setFormatter(formatter) logger.addHandler(handler) - filename = os.path.join(settings.TOOLS_LOG_DIR, name) - maxBytes = 8388608 # 8MB - handler = RotatingFileHandler(filename, maxBytes=maxBytes) - - handler.setFormatter(formatter) - logger.addHandler(handler) + # Also log to a file if a directory is set for it; otherwise the console + # output is all there is (as collected by cron, systemd or docker) + log_dir = getattr(settings, 'TOOLS_LOG_DIR', '') + if log_dir: + filename = os.path.join(log_dir, name) + maxBytes = 8388608 # 8MB + handler = RotatingFileHandler(filename, maxBytes=maxBytes) + + handler.setFormatter(formatter) + logger.addHandler(handler) logger.setLevel(logging.INFO) diff --git a/settings.py b/settings.py index 324ad50..495af1e 100644 --- a/settings.py +++ b/settings.py @@ -298,5 +298,5 @@ AXES_COOLOFF_TIME = 1 # Full path to directory to store logs for dynamically executed tasks TASK_LOG_DIR = "/tmp/layerindex-task-logs" -# Full path to directory where rrs tools stores logs +# Full path to a directory where the rrs tools also write their logs, if any TOOLS_LOG_DIR = "" diff --git a/tests/test_rrs_logger.py b/tests/test_rrs_logger.py new file mode 100644 index 0000000..b03c928 --- /dev/null +++ b/tests/test_rrs_logger.py @@ -0,0 +1,38 @@ +# layerindex-web - tests for the RRS tools' logger +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT + +import logging +import os +import sys +import types + +import pytest + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'rrs', 'tools')) +from common import get_logger + + +@pytest.fixture +def readonly_cwd(tmp_path, monkeypatch): + os.chmod(tmp_path, 0o555) + monkeypatch.chdir(tmp_path) + yield tmp_path + os.chmod(tmp_path, 0o755) + + +def test_no_log_file_without_tools_log_dir(readonly_cwd): + # [YOCTO #16499] the scripts run in a directory they can't write to + logger = get_logger('RRSTestNoDir', types.SimpleNamespace(TOOLS_LOG_DIR='')) + assert not [h for h in logger.handlers if isinstance(h, logging.FileHandler)] + assert os.listdir(readonly_cwd) == [] + + +def test_log_file_in_tools_log_dir(tmp_path): + logger = get_logger('RRSTestDir', types.SimpleNamespace(TOOLS_LOG_DIR=str(tmp_path))) + logger.info('hello') + for h in logger.handlers: + h.flush() + assert 'hello' in (tmp_path / 'RRSTestDir').read_text() From patchwork Sun Nov 15 00:35:02 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104071 X-Patchwork-Delegate: ticotimo@gmail.com Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.7627.1794702926434182393 for ; Sat, 14 Nov 2026 16:35:26 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=b4D+UPDd; spf=pass (domain: gmail.com, ip: 209.85.216.46, mailfrom: ticotimo@gmail.com) Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-3b09c07b3d2so1367424a91.3 for ; Sat, 14 Nov 2026 16:35:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794702926; x=1795307726; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=665cj0vIDvZBoTuppu7mw0+WJXCrB4+Kd0zZU7Qz+D8=; b=b4D+UPDde87TUcCfnvAcGfXhB0iPrcPi5viTNzpIm7eXG6bHPJJ/MCffaWczMqJZ1+ XVmqZ4fL86Pe58MA+6h6xHvpubB2JtcmlIj7QAyt2pRnIZsQb24dS+PkiX2qlDSTyNOS bBnWfOXa5tyoO2s4cA4pRp/f56vvTt/mEYXh2xQTpGkE3bIVS/SmhJt6gjqZRdbvqobx mymk/C+ovadxf2ykjudx4QAhb4z26h7smEooOVxQ7kEIUhRYzAcbpaARMx5nC8or171Q kWg8sEWoES5ZRarjdZFIaK+Iq+DyuLjGXf+Fh4xCJDEFtfyvuO0jrsoFtF+62aWj3MTZ 1UfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794702926; x=1795307726; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=665cj0vIDvZBoTuppu7mw0+WJXCrB4+Kd0zZU7Qz+D8=; b=RQ96VKmfhj4S250uD6DgDSwP6KKme2O1DCfzjecIKRPBIKPQIinBQxPXTuYl7c453c BqIrCfkYBCaTNRE6s6bPLxe4frnIIzb3n20eISdVqrgKgfGBNP+uebAn7GlzkeBnFmfp n07/oLq677BbQMhq3XiRE1b01sx1UxnamPvyPi4EdudQom4TOj52YZNWiZJezWUmhgUi K4IfwZ6I8i35Y9QDueXhcn7zEnyUBejWETh0AnsOmLQeYiwqv1+yU0b2r9TGtHP8qM3I Ai00J40vThFMQqhDsVjAXwrQQemzYT4HXrIrtF7J9T9KN2LbMDq6Ea8IkfSxmcHcYdK+ tx4g== X-Gm-Message-State: APuU4Z86arSNvZnqfgG34GymmH3yg4N6jNSHzfqWCma6rzH4JWjLI51Z otfURdCDSsiSTzcFN8rTfpDdHP2v5wV+glT1psePMHtaOPmY4tzMb4m2nt3jR3LM X-Gm-Gg: AYBFou3m+FHmXW2BTrbwxX2Ew8YQrK3fvq4sE2TGiVdAukPejvt2r3B6f4Jy8KysVQH gsrrttVxjVRfR8bU7De1RNsx2nam33+J0Lq6fLLbgx/6AIXZguPIKLoT6m6h6yrOMJV/+thWPVf zIqURh+iLeZ9VNpKF80eOCGLEwu+avMAtLVzzK25CjgyBFGuwaFKpZddtSsgfPptWs8TkILvbrl G7roqERQ14tsrqIWusOvWvREaZlQSRb6gmePOwrN/R96MJo7lKSXaPJITcj0jYVBdIKmCkQY1N9 ff20AX1M7ogUkJsM3gghK+5/xJJyddARyi6L/coNlR9pdGixPLfkwN/u4rRqrvCoj94zZeq15Sc GcoMBWvoEwy183IpBvUHEN7TyNu+D7oQZwG3ZLS/utNwRJ3jIOJz+chEkOoIhnPmkf5WvfO8CVA BOJNaONRk7olz8Twqd/0nD49Mxeq3TPfnHtv22zRz/DWWuqy/b7EJTJ8NPhd9Fb8W9fh4v2HjFn yGQf0HcTQKv5gP0JuhvoyYr+NH/GrgfwQbsUDVEUz9KXEUJxcbkfkZ2Tk/K43xmkZRWxJGZSa9C va32Zy8mBziQWWkraSoqElkT+fRD8Vl8Z/geo1447YvnSR4c8PK8VfTEQccx6g== X-Received: by 2002:a17:90a:c2d0:b0:3aa:f2c4:12ea with SMTP id 98e67ed59e1d1-3b1b0a43396mr5927983a91.20.1794702925787; Sat, 14 Nov 2026 16:35:25 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3b1b39fb9a3sm5464089a91.3.2026.11.14.16.35.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 14 Nov 2026 16:35:24 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 2/6] rrs/tools/common.py: import BitBake after checking out its revision Date: Sat, 14 Nov 2026 16:35:02 -0800 Message-ID: <20261115003506.53654-3-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261115003506.53654-1-tim.orling@konsulko.com> References: <20261115003506.53654-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Sun, 15 Nov 2026 00:35:26 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5380 load_recipes() imported bb.fetch before recipeparse.init_parser() checks out the BitBake revision for the branch, so the bb package came from whichever revision was checked out before, e.g. an old one left by rrs_upgrade_history.py, while the rest of BitBake came from the new one. rrs_upstream_history.py then failed with "cannot import name 'multiprocessing' from 'bb'". Import it after init_parser(). [YOCTO #16501] AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- rrs/tools/common.py | 4 +++- tests/test_rrs_load_recipes.py | 35 ++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) create mode 100644 tests/test_rrs_load_recipes.py diff --git a/rrs/tools/common.py b/rrs/tools/common.py index 0caf1c9..ccdbe62 100644 --- a/rrs/tools/common.py +++ b/rrs/tools/common.py @@ -93,7 +93,6 @@ def get_recipe_files(layerdir): def load_recipes(layerbranch, bitbakepath, fetchdir, settings, logger, recipe_files=None, nocheckout=False): from layerindex import recipeparse - from bb.fetch import FetchError try: (tinfoil, tempdir) = recipeparse.init_parser(settings, @@ -102,6 +101,9 @@ def load_recipes(layerbranch, bitbakepath, fetchdir, settings, logger, except recipeparse.RecipeParseError as e: logger.error(str(e)) sys.exit(1) + # Only now that init_parser() has checked out the BitBake revision for the + # branch, or bb would be loaded from whichever one was checked out before + from bb.fetch import FetchError layer = layerbranch.layer urldir = str(layer.get_fetch_dir()) diff --git a/tests/test_rrs_load_recipes.py b/tests/test_rrs_load_recipes.py new file mode 100644 index 0000000..97f3d26 --- /dev/null +++ b/tests/test_rrs_load_recipes.py @@ -0,0 +1,35 @@ +# layerindex-web - tests for load_recipes() in rrs/tools/common.py +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT + +import os +import sys +import types + +import pytest + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'rrs', 'tools')) + + +def test_load_recipes_checks_out_bitbake_before_importing_it(monkeypatch): + # [YOCTO #16501] bb must come from the revision init_parser() checks out + import common + # As common_setup() does for the scripts: recipeparse imports utils + monkeypatch.syspath_prepend(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'layerindex')) + from layerindex import recipeparse + + class Stop(Exception): + pass + + def fake_init_parser(*args, **kwargs): + assert 'bb.fetch' not in sys.modules + raise Stop() + + for name in [m for m in sys.modules if m == 'bb' or m.startswith('bb.')]: + monkeypatch.delitem(sys.modules, name) + monkeypatch.setattr(recipeparse, 'init_parser', fake_init_parser) + with pytest.raises(Stop): + common.load_recipes(types.SimpleNamespace(branch=None), '/nonexistent', '/nonexistent', + types.SimpleNamespace(), None) From patchwork Sun Nov 15 00:35:03 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104073 X-Patchwork-Delegate: ticotimo@gmail.com Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.7558.1794702930682029185 for ; Sat, 14 Nov 2026 16:35:30 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=erVy9lRf; spf=pass (domain: gmail.com, ip: 209.85.215.180, mailfrom: ticotimo@gmail.com) Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-cdab938dc83so1089449a12.3 for ; Sat, 14 Nov 2026 16:35:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794702930; x=1795307730; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=EJnI+aNKMFlsVqR1hCwKw1ABy0I+ha6c8y5pPwE561A=; b=erVy9lRfoFQG1De/4ny3ROsRXwdQgFJJZt8dM7NqELkBZfzcC+178e2GRfCgDX95Wb uBNvADhpaOaLfsDbhDAZiS6GCoXv2Sa63WD/I7Ci8ITkHX/ztusqbnmfkcQQf19xU91t qV7jIJX38HO4PBFWzmHPbEpQwkhm/I8sWP8jyZVBYxka8Sn4hUJcLkijVmMKUShDri9s T5TrYGeZny95+B0W+jUZ9N6bJ2y75XO0K5D/DFqqBPDnAKm4slRi+v3xcGrTQSeAISvm 8vZLRIJf7jaVhivoqCjN+SN5o9+dDFOK871hY7/agPJMvYTME5f9eoQLFBNtKIAffceQ 1LHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794702930; x=1795307730; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=EJnI+aNKMFlsVqR1hCwKw1ABy0I+ha6c8y5pPwE561A=; b=tM2G6UuZncJ/66kZsF1CtbZouRRhnaEQtdvv5bX3pG+HUZrU7eGFhjQbBlh2q+U2tH tHfg6ERA7Iwb4sxoRY6+ddFVqRTDY9+eRvv6vpJon498YNghzFuP6n8cRH2mr3DP6GAo fPot1H4himqVN4TjiOsoU59XXzD29qqrxZSY96SSfUVlsoGh891uqy7zCnTF1HUMCRSf Vb9+etNr9OnxNvPYF4ReCrvJix0LEjr5uwdH0ZLkM8WmpzMSHv+Vvgt0696uGstV9b2N oAFbsrIjcHtRO1IzxvRnh3YznJDXZ//D0EjOtbWUSBSTqyIWYAuzIWZeAIhIJq/7xzej z1pA== X-Gm-Message-State: APuU4Z+fOAPaE/aeXmi1W+r1it2rnMzhz5/DXsHojQFJfufP5gK7+/jF lBaQzU+uix4oJNEmT+Z0bTb24+XEvkjZB8cZ7mO37XtX7mG/PTxQ1J66W4c2Eg== X-Gm-Gg: AYBFou2FElQaJsDKkdgegs8UX9F9MYfAzl/SE45Xf3uT/+rwQ7udzylWcMh0lzvmkaj pm9BvFFgdftfi1/q+fedaeCsVIhXJI4+WmDP3vjwYyPh3zXcgvd75seLhMW5FMWpJgwW8/x+sI0 YtJy4p3IL14VQS/D9vzECHWqHWfcxy34AeOqfXkf8yFm7RQzIdq17mJyET9FWhh0r9px9eTsMjg qvNzZP7IWsCIOA7zQTB6JYxk7RYG7e5vAtrG7FKINRsVNJrxfm3iy5KA6wVVh5KIMBsWzBsNvj8 xirxglP2TtTy2kqet0mqjY7Ej5Ton39RIMFEdHAAt+JD+bwumybX8I98JBdrfwLenaB/ax/gW9N rL+wm99/YT/75HH62r2w2lRE3JrY8idKfCXuYijd8BUdKtqje6CLXLy1uusEJE3wgWbzLtahBW5 BEZQNkSJioBh9EXwXUT2WqVixtdRwrsXXFVYXWdCkJKk7VPc9WSZKvWdZVKwG3O+3Rzhh6nZ7wf J+dZ5ijMWdo7fjirceTkKo1I5HUh1OntHIqAmHjFJ2aQvtEKfvKnpDYY8D3NWcw9OAryZy9SMe+ T3oMGp0AxZXzEEeguotiMJcqNfGWHF9jzX9e9V7JRpmspVZPlc+489hFJM8Gfw== X-Received: by 2002:a17:90a:c88b:b0:3ac:8741:e2e4 with SMTP id 98e67ed59e1d1-3b1b0a4e341mr6527182a91.52.1794702929885; Sat, 14 Nov 2026 16:35:29 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3b1b39fb9a3sm5464089a91.3.2026.11.14.16.35.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 14 Nov 2026 16:35:27 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 3/6] rrs_upgrade_history: use the right BitBake around the 2025 fragments change Date: Sat, 14 Nov 2026 16:35:03 -0800 Message-ID: <20261115003506.53654-4-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261115003506.53654-1-tim.orling@konsulko.com> References: <20261115003506.53654-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Sun, 15 Nov 2026 00:35:36 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5381 BitBake 3b9d7bea9 ("parse/ast: add support for 'built-in' fragments", 2025-06-18) changed the addfragments syntax, which OE-Core only followed in e959f916b4 (2025-06-20). The OE-Core commits in between got that BitBake, could not be parsed with it, and the BitBake server then waited for ever, so the upgrade history never got past them. Use the BitBake revision before the change for those commits. The table of BitBake ranges to replace only matched on the BitBake revision, so let an entry apply only to commits before a given date, since OE-Core commits from e959f916b4 on need the new revision. [YOCTO #16500] AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- rrs/tools/rrs_upgrade_history.py | 32 ++++++++++++++++++----- tests/test_rrs_upgrade_history.py | 43 +++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 7 deletions(-) create mode 100644 tests/test_rrs_upgrade_history.py diff --git a/rrs/tools/rrs_upgrade_history.py b/rrs/tools/rrs_upgrade_history.py index 6326914..6bf230b 100755 --- a/rrs/tools/rrs_upgrade_history.py +++ b/rrs/tools/rrs_upgrade_history.py @@ -16,7 +16,7 @@ import sys import os.path import optparse import logging -from datetime import date, datetime, timedelta +from datetime import date, datetime, timedelta, timezone sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__)))) from common import common_setup, get_logger @@ -42,6 +42,16 @@ bitbakepath = os.path.join(fetchdir, bitbakeitem.get_fetch_dir()) if getattr(settings, 'BITBAKE_PATH', ''): bitbakepath = os.path.join(bitbakepath, settings.BITBAKE_PATH) +def remapped_bitbake_rev(bitbake_rev, commitdate, bitbake_map, logger): + """The bitbake revision to use for a layer commit made at commitdate + (local time), given the latest bitbake revision before it""" + check_rev, until = bitbake_map.get(bitbake_rev, (None, None)) + # until is timezone-aware, so compare as timestamps + if check_rev and (until is None or commitdate.timestamp() < until.timestamp()): + logger.debug('Preferring bitbake revision %s over %s' % (check_rev, bitbake_rev)) + return check_rev + return bitbake_rev + def run_internal(maintplanlayerbranch, commit, commitdate, options, logger, bitbake_map, initial=False): from layerindex.models import PythonEnvironment from rrs.models import Release @@ -63,10 +73,7 @@ def run_internal(maintplanlayerbranch, commit, commitdate, options, logger, bitb bitbake_rev = utils.runcmd(['git', 'rev-list', '-1', '--before=%s' % str(commitdate), 'origin/master'], bitbakepath, logger=logger) - check_rev = bitbake_map.get(bitbake_rev, None) - if check_rev: - logger.debug('Preferring bitbake revision %s over %s' % (check_rev, bitbake_rev)) - bitbake_rev = check_rev + bitbake_rev = remapped_bitbake_rev(bitbake_rev, commitdate, bitbake_map, logger) cmd = '%s upgrade_history_internal.py %s %s' % (cmdprefix, maintplanlayerbranch.layerbranch.id, commit) if initial: @@ -174,7 +181,10 @@ def upgrade_history(options, logger): commit_list = commits.split('\n') bitbake_map = {} - def remap_range(start, end, replacewith=None): + def remap_range(start, end, replacewith=None, until=None): + """Use replacewith (by default end) instead of the + bitbake revisions from start up to end, for layer + commits before until (all if None)""" if replacewith is None: replacewith = end bitbake_commits = utils.runcmd(['git', 'rev-list', '%s^..%s^' % (start, end)], @@ -182,12 +192,20 @@ def upgrade_history(options, logger): logger=logger) bitbake_commit_list = bitbake_commits.splitlines() for commit in bitbake_commit_list: - bitbake_map[commit] = replacewith + bitbake_map[commit] = (replacewith, until) # Filter out some bad commits remap_range('fef18b445c0cb6b266cd939b9c78d7cbce38663f', '39780b1ccbd76579db0fc6fb9369c848a3bafa9d') remap_range('5796ed550d127853808f38257f8dcc8c1cf59342', '547128731e62b36d2271c4390b3fee2b16c535dc') remap_range('a06619951a43acb80b80d92e0caac560657ca249', '2117db3146ce38bb4a6e2df40b6cd2ab11b514d5') + # "parse/ast: add support for 'built-in' fragments" changed the + # addfragments syntax, which OE-Core only followed in + # e959f916b4 "bitbake.conf: enable built-in fragments for MACHINE + # and DISTRO" (2025-06-20 11:03:34 UTC); before that use the + # bitbake revision before the change + remap_range('3b9d7bea915dc7f10e845854f1dae325743f9456', 'f1175778108a559e1dd0d55cb68f42816c86f393', + replacewith='378db0fdd95f8704fccd852452555bfddcad384d', + until=datetime(2025, 6, 20, 11, 3, 34, tzinfo=timezone.utc)) if options.stop_commit and (options.stop_commit not in [x.split()[0] for x in commit_list]): logger.error('Stop commit %s is not in repository %s' % (options.stop_commit, repodir)) diff --git a/tests/test_rrs_upgrade_history.py b/tests/test_rrs_upgrade_history.py new file mode 100644 index 0000000..6bf3582 --- /dev/null +++ b/tests/test_rrs_upgrade_history.py @@ -0,0 +1,43 @@ +# layerindex-web - tests for the BitBake revision rrs_upgrade_history.py uses +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT + +import logging +import os +import sys +from datetime import datetime, timezone + +import pytest + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'rrs', 'tools')) + +NEW = '3b9d7bea915dc7f10e845854f1dae325743f9456' # changed the addfragments syntax +OLD = '378db0fdd95f8704fccd852452555bfddcad384d' # the revision before it +SWITCH = datetime(2025, 6, 20, 11, 3, 34, tzinfo=timezone.utc) + + +def local(dt): + # The script works with naive local times (datetime.fromtimestamp()) + return datetime.fromtimestamp(dt.timestamp()) + + +@pytest.mark.parametrize('commitdate, expected', [ + # [YOCTO #16500] OE-Core commits before it switched need the old revision + (local(datetime(2025, 6, 19, 10, 5, 42, tzinfo=timezone.utc)), OLD), + # ... and those from the switch on need the new one + (local(SWITCH), NEW), + (local(datetime(2025, 6, 21, 9, 0, 0, tzinfo=timezone.utc)), NEW), +]) +def test_bitbake_revision_for_commit(commitdate, expected): + from rrs_upgrade_history import remapped_bitbake_rev + bitbake_map = {NEW: (OLD, SWITCH)} + assert remapped_bitbake_rev(NEW, commitdate, bitbake_map, logging.getLogger('test')) == expected + + +def test_bitbake_revision_without_limit(): + from rrs_upgrade_history import remapped_bitbake_rev + log = logging.getLogger('test') + assert remapped_bitbake_rev('a', datetime(2030, 1, 1), {'a': ('b', None)}, log) == 'b' + assert remapped_bitbake_rev('c', datetime(2030, 1, 1), {'a': ('b', None)}, log) == 'c' From patchwork Sun Nov 15 00:35:04 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104074 X-Patchwork-Delegate: ticotimo@gmail.com Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.7560.1794702933215221601 for ; Sat, 14 Nov 2026 16:35:33 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=fR/YSCLg; spf=pass (domain: gmail.com, ip: 209.85.215.176, mailfrom: ticotimo@gmail.com) Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-cd60c5da24cso2059125a12.0 for ; Sat, 14 Nov 2026 16:35:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794702932; x=1795307732; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=H269tTLNyZkqNS3gkiZLESQ/LQl4VTTA4YBp8IkThLI=; b=fR/YSCLgHvJS6Enf5v9+ouvsCn1jqJCrVGkoIxKTh1zrS74dtUGaRPVkk5h6TNWNtk n0s/T+N4osKhBNy2ZETvt5e6nIQjWV1zmZd6Dk0t7yrwDwSrnvC4lCe5VNCCHLIYBwP/ VqN4I0fGQClMwIMJbNZUNhYwfG0XR6s7cPp/DUVrOnka0RLeEK5aHFSbkCtm3HnMPzjF b1rH/H/zavRVLqhV6VSVmBkN5GZajdXzyo0WXNzqMUh1IACfp9FJhb3KzomfsQ8dH9on tMJViV/R5SBIACKdCDxweuHOJnX9puPbbxWgnjCjkNizVd0uBP8SCvTg36DynGhNrwCw XXJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794702932; x=1795307732; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=H269tTLNyZkqNS3gkiZLESQ/LQl4VTTA4YBp8IkThLI=; b=getNAxtzF2zul3eYVVc93w43bhh9it/Pta/2AFc7j17h3eOMcUj2d5jV1WDQxYtuxp PJSwl41knY2JxuNcXEhAp7KH4J920X2GLL3brc6erlK2j7wcsHcgCusnw+kj8wKff/b/ 8MZ0FfIYQDDv4UW1EMceHVe+qegIbGFkq1xfR16ZRSDWuxgI/QQuxISoPtimHeLW+mCG 8UU85uf5QHhAT6J7qLhTM+XzcnoVKKhkWcda4aEtyZ32Xw/OOuLudhzX6izavpmZdIYt yfjV+r5ITZNAV5bkQAYwh+IrfBQAOESfyaL4ekSbOMXcYc047GOuMuQ5B80OaiNxGDsN x/jA== X-Gm-Message-State: APuU4Z9s34tuD6qt5/GuiifJUeNQkPWFBv2wzV0/zbdIvLZu8/KSt6dw ZulNgvW7WH0NRm6pZIQiNyhyd7554qskiqSpvS2TTgcSu0O4rHMHtwY+wrFwBA== X-Gm-Gg: AYBFou3n6GBDwJ1hosxwJLlKrMmp9Y4vNWvbYDlmrpnbWmnqbTlIqB4dEMShWXwI4EU p6NnDL9bbtxOUEKkkt/Hhny+rmQdIkrNlKEflLSdJ0CLMXyHf0Cir3cN8kGB3kNPr2z6BatgFXx HocbIRXCtgGTvr3JyQZmrguW9Mz5pfIqgRS3ZJ7yWOTZpiMpra/psUxaD8x9u2A1LCr8iPaLf5x uUKjxnJCQWMSYGzxuo5aIBwBY2LS3Jlf5bYH/0DiRL0hDQ7/t4p9K7DNgUFpWLDhzbFKwESDeQ9 0XC9b1ErKUiB3HmlRNAFta3Wz7A9RFeUEQ+3FESt++78Ka6FdDXnyR84+h/8TQcbOr+bjwvb2qD 32innQSDdmVyaiccQpWZ+5/+mtS6cUbd/lv2uPm7iod/YV5BZi/Z69sku9m1JktmlHjkcfZlPpT mFpnttm+46ElbSLsQ5b5lDyol5Cvu989GCKJdhmGaNIC3PKrVGEJoUh2usDzosM/1v1lkTDbiok uqC4GJxth6aLb0Bz7N5W6T1heILIgGyXTaknJdeu5dFGyeue0kN8ayvcGsfG5fZfpxRP7t+w9Fp T4XCq4dPAFABgKC3H19iWXmXj3KavSQm5Ei67oP8OEzD0iPvix5/NwmqeO15rV0= X-Received: by 2002:a17:90b:5910:b0:3b0:a378:1cf with SMTP id 98e67ed59e1d1-3b1b09e7b8dmr5793688a91.22.1794702932487; Sat, 14 Nov 2026 16:35:32 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3b1b39fb9a3sm5464089a91.3.2026.11.14.16.35.31 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 14 Nov 2026 16:35:31 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 4/6] rrs/tools/common.py: check out the layer branch before parsing it Date: Sat, 14 Nov 2026 16:35:04 -0800 Message-ID: <20261115003506.53654-5-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261115003506.53654-1-tim.orling@konsulko.com> References: <20261115003506.53654-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Sun, 15 Nov 2026 00:35:37 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5382 load_recipes() let init_parser() check out BitBake and the core layer, but not the layer itself or the layers it depends on, so rrs_upstream_history.py and rrs_distros.py parsed whichever branch the layer index updates had left them on, e.g. meta-openembedded at dunfell while working on master. Check out the layer branch and its dependencies' unless nocheckout is set. [YOCTO #16502] AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- rrs/tools/common.py | 13 +++++++++- tests/test_rrs_load_recipes.py | 45 ++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 1 deletion(-) diff --git a/rrs/tools/common.py b/rrs/tools/common.py index ccdbe62..f97d4e1 100644 --- a/rrs/tools/common.py +++ b/rrs/tools/common.py @@ -92,7 +92,7 @@ def get_recipe_files(layerdir): def load_recipes(layerbranch, bitbakepath, fetchdir, settings, logger, recipe_files=None, nocheckout=False): - from layerindex import recipeparse + from layerindex import recipeparse, utils try: (tinfoil, tempdir) = recipeparse.init_parser(settings, @@ -110,6 +110,17 @@ def load_recipes(layerbranch, bitbakepath, fetchdir, settings, logger, repodir = os.path.join(fetchdir, urldir) layerdir = os.path.join(repodir, str(layerbranch.vcs_subdir)) + if not nocheckout: + # The repositories are left on whichever branch was updated last; + # check out this branch of the layer and of the layers it depends on + # (init_parser() has done BitBake and the core layer) + utils.checkout_layer_branch(layerbranch, repodir, logger=logger) + for dep in layerbranch.dependencies_set.all(): + deplayerbranch = dep.dependency.get_layerbranch(layerbranch.branch.name) + if deplayerbranch: + utils.checkout_layer_branch(deplayerbranch, + os.path.join(fetchdir, dep.dependency.get_fetch_dir()), logger=logger) + d = recipeparse.setup_layer(tinfoil.config_data, fetchdir, layerdir, layer, layerbranch, logger) diff --git a/tests/test_rrs_load_recipes.py b/tests/test_rrs_load_recipes.py index 97f3d26..8322350 100644 --- a/tests/test_rrs_load_recipes.py +++ b/tests/test_rrs_load_recipes.py @@ -33,3 +33,48 @@ def test_load_recipes_checks_out_bitbake_before_importing_it(monkeypatch): with pytest.raises(Stop): common.load_recipes(types.SimpleNamespace(branch=None), '/nonexistent', '/nonexistent', types.SimpleNamespace(), None) + + +@pytest.mark.django_db +@pytest.mark.parametrize('nocheckout', [False, True]) +def test_load_recipes_checks_out_the_layer_branch(monkeypatch, nocheckout): + # [YOCTO #16502] parse the branch being worked on, not whichever branch + # the repositories were left on + import common + monkeypatch.syspath_prepend(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'layerindex')) + from layerindex import recipeparse, utils + from layerindex.models import Branch, LayerBranch, LayerDependency, LayerItem + + master, _ = Branch.objects.get_or_create(name='master', defaults={'bitbake_branch': 'master'}) + layers = {} + for name in ('meta-oe', 'meta-python'): + layer = LayerItem.objects.create(name=name, status='P', layer_type='A', summary=name, + description=name, vcs_url='git://example.com/%s.git' % name) + layers[name] = LayerBranch.objects.create(layer=layer, branch=master, actual_branch='main') + LayerDependency.objects.create(layerbranch=layers['meta-oe'], dependency=layers['meta-python'].layer) + + class Stop(Exception): + pass + + def fake_setup_layer(*args, **kwargs): + raise Stop() + + # load_recipes() imports bb.fetch once init_parser() has checked out BitBake + bb = types.ModuleType('bb') + bb.fetch = types.ModuleType('bb.fetch') + bb.fetch.FetchError = Exception + monkeypatch.setitem(sys.modules, 'bb', bb) + monkeypatch.setitem(sys.modules, 'bb.fetch', bb.fetch) + + checkouts = [] + monkeypatch.setattr(recipeparse, 'init_parser', lambda *args, **kwargs: (types.SimpleNamespace(config_data=None), '/tmp')) + monkeypatch.setattr(recipeparse, 'setup_layer', fake_setup_layer) + monkeypatch.setattr(utils, 'checkout_repo', lambda repodir, ref, logger=None: checkouts.append((os.path.basename(repodir), ref))) + with pytest.raises(Stop): + common.load_recipes(layers['meta-oe'], '/bitbake', '/fetch', types.SimpleNamespace(), None, + recipe_files=[], nocheckout=nocheckout) + if nocheckout: + assert checkouts == [] + else: + assert checkouts == [('git___example_com_meta-oe_git', 'origin/main'), + ('git___example_com_meta-python_git', 'origin/main')] From patchwork Sun Nov 15 00:35:05 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104075 X-Patchwork-Delegate: ticotimo@gmail.com Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.7561.1794702937974810716 for ; Sat, 14 Nov 2026 16:35:38 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=rnZ5SqAS; spf=pass (domain: gmail.com, ip: 209.85.216.43, mailfrom: ticotimo@gmail.com) Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-3b37951a397so390300a91.0 for ; Sat, 14 Nov 2026 16:35:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794702937; x=1795307737; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=YIqlGcwSf45jpSzyi+CQ610ZKHKNvwSKeiA19woEK+g=; b=rnZ5SqAS77WIRkhsxSWIb5nXy77SzznvULLfH0NjgvrKFsJBoryaeOlOGRNXa/fbkT BK9IgRbZuhOslI9vWW9cIJY9YoDMhZunFdu5LV+2gAL6yR4kfmqEOY1qf8JDfwQDlpHr S+SFQy623E44sg639Ay7C1HphtuVqTCU/n0pwuD/6mKKo3G56W/BugW0RQc8/Zdf8zOG YopW2OMzFQ2KJ/Xx0TioHOEcsOPaj+ulHamxE12a9F9IwIwu6DaufReGf6Gpj8efkwLC 1yGuP1tnFz6I8Or6hSeu02cB2sGRi5w2mHjvvZb2qpnzL5RPTWjzvEVFWb0t3Y2QbHfd 8+rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794702937; x=1795307737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=YIqlGcwSf45jpSzyi+CQ610ZKHKNvwSKeiA19woEK+g=; b=s6jHRNToIBGjmhpRHl3ncCFJrsgClolgHzHgzj8WRzIt45xLl+7HDaD+/a4BtTeuD+ Vggveu663zZW9EVo37Y2rY/83PQfdhHOOiWPcrJOAK5beu1ZWDo44epx9K1uQxRZqrQc ECpNiDdXG7m9xMgNC2SB94G1T+WMMTw6pjPnWy4M8DAUAVdEFJ5DPF29WqGhdEu0gagY kmXHUaeVSs2ZAf+utQ20LDfvyg9I9rPKZAILMEwGAKfLnJWGHZs3AsbpoF+e5LOam/sr lVpaDHOmt91FUVZ9UZtEqArR6Va+DsBJYwcBrKDPkgiOJ272C/x30AoNB1nUzJxLXLIF Ba0w== X-Gm-Message-State: APuU4Z//WIPF+iVK400LBpnxncLuBWFaYXKNIEglECfGGxIYZ6DkqCxb YQYR1DtlvtnTrIGZWItdu+NU6P0cY5qCZCqjgBsYyTLDzo5o33CNpLrAFIz9sA== X-Gm-Gg: AYBFou0/nF65fU3b/Gc9aYJNrhdnI14OX84P8aHPyyBnwPZA8WhdJ66qUvSbgDepnAz TRTfFU4E/Eq9av8HLN7rttAkv5w51VyQ8Qm/HFrQR+x2qBn1xQdUF6M4BFxv5A/QzJuddjKOtbq r/sfn4ZKwwsulxkR9aGxY95BrsT8wt/kxEHPcnfSpPmsNa2eHnhFtgIiK9kKEc0FEYRQCXZBYH/ W7UaaaxzxXIcRZ1PR591zubD2t0LsTl97fCMTZQiGBTvawtT0lAZftwIjLUy1zbORkmk+WauDb9 Qtp7fzYSJ7uj4GEt31QohSiwlzhWPJ3z/jBl+YF4e27P0Sh3LD0m7rG4KYIuKjrHxnkPOXUDuSo EA/DjsTn4UQ8hXtB1F5TIIPg6xr/lnlxLETRKxNx1J9MCs3R2ZyD8N62kWo4OtPsQiFJzEeRi5/ JTrPUdUMhVTnhGgRApACvqDhMWiSE7FR1swfulm/oZwLhURcaTFmrxfoxyfUb0EUXEWMvMrE/HS 2sy+n+jObhR/uE/ncu7KZvnC3OPUkfnwG5//ZZ5HERoKssobtiHONsnO8+6kPWQY4wnJVqofOyX prhJntuYYeib5HHri+ntFTfe2lsJFEtXRqM0UnlIr5m/7AIueZqWr0z3NihtViF0lXuWkDhrA65 WMm525MQ= X-Received: by 2002:a17:90b:4f86:b0:3b0:9b3c:4434 with SMTP id 98e67ed59e1d1-3b1b0ca50f0mr6490649a91.36.1794702937229; Sat, 14 Nov 2026 16:35:37 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3b1b39fb9a3sm5464089a91.3.2026.11.14.16.35.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 14 Nov 2026 16:35:35 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 5/6] conf/local.conf: use OEBasicHash for parsing Date: Sat, 14 Nov 2026 16:35:05 -0800 Message-ID: <20261115003506.53654-6-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261115003506.53654-1-tim.orling@konsulko.com> References: <20261115003506.53654-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Sun, 15 Nov 2026 00:35:46 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5383 OE-Core 5596ea156d made OEEquivHash the default but BB_HASHSERVE was only set in the following commit, so parsing 3c00ad5d3b fails with "OEEquivHash requires BB_HASHSERVE to be set" and the BitBake of the time hangs instead of exiting, stalling rrs_upgrade_history.py. Parsing doesn't need hash equivalence, so select OEBasicHash; this also avoids starting a hashserv for every parse on current master. [YOCTO #16504] AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- conf/local.conf | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/conf/local.conf b/conf/local.conf index 82cc577..5ce9ade 100644 --- a/conf/local.conf +++ b/conf/local.conf @@ -18,6 +18,10 @@ HOSTTOOLS_NONFATAL := "${@d.getVar('HOSTTOOLS', True) or ''} ${@d.getVar('HOSTTO # oe.utils.get_gcc_version() in a python expression HOSTTOOLS = "gcc" +# Parsing doesn't need hash equivalence; OEEquivHash would also need +# BB_HASHSERVE, which some OE-Core revisions don't set +BB_SIGNATURE_HANDLER = "OEBasicHash" + # Recipe reporting system configuration #require conf/distro/include/maintainers.inc #require conf/distro/include/distro_alias.inc From patchwork Sun Nov 15 00:35:06 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Orling X-Patchwork-Id: 104076 X-Patchwork-Delegate: ticotimo@gmail.com Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.7630.1794702940645758562 for ; Sat, 14 Nov 2026 16:35:40 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20260707 header.b=rSgla2fE; spf=pass (domain: gmail.com, ip: 209.85.216.53, mailfrom: ticotimo@gmail.com) Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-3adac5b29ebso2124674a91.2 for ; Sat, 14 Nov 2026 16:35:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794702940; x=1795307740; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0DNeSFap/rZHMML5YMEAPmRDosYBMRU9l3QKTNHWdRc=; b=rSgla2fE2a5O3tcV62/JLSdSbfA+30zrg8w6Qcm0hxOG3lr8H6y3HYsz5bNbfqr3Pk ePulK+GLfRJHL5vwBs5NqIJbGHGhlURlfXPnYu4x9O8WXkiY+bczuQ5/7Xu6xUMSh1KD mQzTZC41skCrls4YXHdTn7Hc09DqcomTOp2Sm0SlYjgQsxW0KGKvvH98s75deEInmBC4 CD9F2hh/OP6KOaKJbRwMZf6sbtiwBhmeXoymC2dNr1lk7keYSMmJKJJIR1DlxRi8nNNu 8QpmailAZA3sIENqdu/EsObOawL5fidKRi/cCYQFcC7tONxzA3cES0mkFDS8NtsbMjDq 9MLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794702940; x=1795307740; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=0DNeSFap/rZHMML5YMEAPmRDosYBMRU9l3QKTNHWdRc=; b=hwgfcbWlocVrKWrBOROaVUqdnIHk5Wqkzvp9BF6hz1MB0++TJVoNXGcmb3CNzqZzd3 CA4jI53QGJjD2CePiZqeV6B2Ugr5i20UUm17Gfw5HSNTotTI/0IkczLsIEX9CohMREN0 jE4nf+3Isj4VoYdKG2M2rGP3D/d0kkDVPaGlAAeiDPGi3tCR3TvmYNNk79EFsLilmq0K ybP85BVi/LIs7pc3bR3AlR+c6h2FB12AT4kVzLrZumLhw3xvwDDE1D03kWOQBuwoIaOA 7ZJjkuSumZHg9LB83f6646mQxKJEYK7YqOfZwG3PSEnxq5VcQM4iiTDGFsJZ4rD6AW28 NL3g== X-Gm-Message-State: APuU4Z/+jYIE4kZJKZMZ60KhQPRo5rnix93lNnfRc++khNeaY+KDBF6i fcoVufqghw7SW/MhwFXJZ/B3XTyb1HVKYzlfsVzJAPH5EbLa580QUFUKjOjmGw0q X-Gm-Gg: AYBFou0HGLLAOH5ThHH2krXgvHLqMw1V2a7QBdNX59ISZzi/4GEhkjQgq7+sBa+Reg9 F+dXPZtUD6xtZehg4xpbCuz04TxGcIcmrcNkvkqF16B80wf6PUgNOAjoEzd1D5Ekss9SelzaQ0g DaTtThh9GqoJ8oDSwLMw7dOrMHScKk9Hjqlqk0YasQLdcAaBx3fCny7ypyf1vObZkF6IppYgUlt yBWSNsneBfIi5rq7xdUGv7CgegABug0Zwu+OKllMIlYpum9dv3lEAfs8Q9b7obahYjTzlSstURi x6YO8dCEAxK2pLZPxv6ntgs0XOkrwpxIGleq0NtE7uCJYSvCGQpwhnnHDCfRGQKeNYAdL5mi944 LlnF9TkvfpTpWj2k0yIh0OrrYGI7kH4K3mrS+GFw5S9MXrE65L9iSdQ2nVob1ilJNyKYVBtAdAy 5mDHp/xt5GPX9eh0ILOh8Ct9o3DE6jxNDKg1R3d2f2GhVPcnliSBGIjFm3Xp+VA908JsKf53A+v PIDVJpSR3fuhx130/oxENDRl+0xeynqL8VSFKARtFuFNaGahopE+UFRkMBC3cfh4fBVu16iKSaJ KxNG2wwhUs7t31NbMe4EyJrA/Ozv5PorB0EHgL3AXQbTUvQYdMumipuB+ESIEQ== X-Received: by 2002:a17:90b:314c:b0:3ac:86a0:5766 with SMTP id 98e67ed59e1d1-3b1b0b55624mr6913554a91.63.1794702939879; Sat, 14 Nov 2026 16:35:39 -0800 (PST) Return-Path: Received: from localhost.localdomain ([69.9.132.159]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3b1b39fb9a3sm5464089a91.3.2026.11.14.16.35.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 14 Nov 2026 16:35:39 -0800 (PST) From: Tim Orling X-Google-Original-From: Tim Orling To: yocto-patches@lists.yoctoproject.org Cc: Tim Orling Subject: [layerindex-web][PATCH 6/6] rrs_distros: keep the environment when fetching distro package lists Date: Sat, 14 Nov 2026 16:35:06 -0800 Message-ID: <20261115003506.53654-7-tim.orling@konsulko.com> X-Mailer: git-send-email 2.56.1 In-Reply-To: <20261115003506.53654-1-tim.orling@konsulko.com> References: <20261115003506.53654-1-tim.orling@konsulko.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Sun, 15 Nov 2026 00:35:47 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto-patches/message/5384 OE-Core's distro_check calls bb.utils.export_proxies(), which also exports the datastore's PATH and HOME. OE-Core's PATH puts its scripts/git wrapper first with no python3 beside it, so once the package lists were fetched for the first layer branch, every git command for the next one failed with "/usr/bin/env: 'python3': No such file or directory". Restore the environment after fetching the lists. [YOCTO #16506] AI-Generated: Claude Opus 5.5 Signed-off-by: Tim Orling --- rrs/tools/common.py | 18 +++++++++++++ rrs/tools/rrs_distros.py | 5 ++-- tests/test_rrs_distros_environ.py | 45 +++++++++++++++++++++++++++++++ 3 files changed, 66 insertions(+), 2 deletions(-) create mode 100644 tests/test_rrs_distros_environ.py diff --git a/rrs/tools/common.py b/rrs/tools/common.py index f97d4e1..ef213af 100644 --- a/rrs/tools/common.py +++ b/rrs/tools/common.py @@ -24,6 +24,24 @@ def common_setup(): os.environ['GIT_ASKPASS'] = '' os.environ['GIT_TERMINAL_PROMPT'] = '0' +def create_distro_packages_list(distro_check, fetchdir, d): + """ + Download the distros' package lists with OE-Core's oe.distro_check, + keeping this process's environment as it was. + + distro_check's create_socket() calls bb.utils.export_proxies(), which + exports PATH and HOME from the datastore along with the proxy + variables. OE-Core's PATH puts its scripts/git wrapper first, with no + python3 beside it, so left in place it breaks every later git command + in this process. + """ + saved_environ = os.environ.copy() + try: + distro_check.create_distro_packages_list(fetchdir, d) + finally: + os.environ.clear() + os.environ.update(saved_environ) + def get_logger(name, settings): from logging.handlers import RotatingFileHandler diff --git a/rrs/tools/rrs_distros.py b/rrs/tools/rrs_distros.py index e6717c1..3dc745d 100755 --- a/rrs/tools/rrs_distros.py +++ b/rrs/tools/rrs_distros.py @@ -17,7 +17,8 @@ from datetime import datetime sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__)))) from common import common_setup, load_recipes, \ - get_pv_type, get_logger, DryRunRollbackException + get_pv_type, get_logger, DryRunRollbackException, \ + create_distro_packages_list common_setup() from layerindex import utils @@ -138,7 +139,7 @@ if __name__=="__main__": # Only need to do this once from oe import distro_check logger.debug("Downloading distro's package information ...") - distro_check.create_distro_packages_list(fetchdir, d) + create_distro_packages_list(distro_check, fetchdir, d) pkglst_dir = os.path.join(fetchdir, "package_lists") RecipeDistro.objects.filter(recipe__layerbranch = layerbranch).delete() diff --git a/tests/test_rrs_distros_environ.py b/tests/test_rrs_distros_environ.py new file mode 100644 index 0000000..e07fa74 --- /dev/null +++ b/tests/test_rrs_distros_environ.py @@ -0,0 +1,45 @@ +# layerindex-web - tests for create_distro_packages_list() in rrs/tools/common.py +# +# Licensed under the MIT license, see COPYING.MIT for details +# +# SPDX-License-Identifier: MIT + +import os +import sys +import types + +import pytest + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'rrs', 'tools')) + +import common + + +def fake_distro_check(fail=False): + # Like OE-Core's create_socket(): bb.utils.export_proxies() writes the + # datastore's PATH and HOME into os.environ + def create_distro_packages_list(fetchdir, d): + os.environ['PATH'] = '/oe-core/scripts:/build/tmp/hosttools' + os.environ['HOME'] = '/build' + os.environ['http_proxy'] = 'http://proxy.example.com:3128' + if fail: + raise OSError('HTTP Error 404: Not Found') + return types.SimpleNamespace(create_distro_packages_list=create_distro_packages_list) + + +@pytest.mark.parametrize('fail', [False, True]) +def test_create_distro_packages_list_keeps_the_environment(monkeypatch, fail): + # [YOCTO #16506] OE-Core's PATH left in the environment made the next + # layer branch's git commands run OE-Core's scripts/git without python3 + monkeypatch.setenv('PATH', '/usr/local/bin:/usr/bin:/bin') + monkeypatch.setenv('HOME', '/home/layers') + monkeypatch.delenv('http_proxy', raising=False) + before = dict(os.environ) + + if fail: + with pytest.raises(OSError): + common.create_distro_packages_list(fake_distro_check(fail=True), '/fetch', None) + else: + common.create_distro_packages_list(fake_distro_check(), '/fetch', None) + + assert dict(os.environ) == before