From patchwork Tue Oct 8 12:36:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 50033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 733F9CEF164 for ; Tue, 8 Oct 2024 12:36:31 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.web11.8046.1728390990708555304 for ; Tue, 08 Oct 2024 05:36:31 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=LE/sLVmK; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-42e748f78d6so50080295e9.0 for ; Tue, 08 Oct 2024 05:36:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1728390989; x=1728995789; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=xkkPCPfKEMmjFLM8D1jGEC3Z/sB1AWbZTK+36QpNdFs=; b=LE/sLVmKYohWGvJaAd3QSsYHdn0T62oZ839pN4Npi1aH22qG2ZTRfHpPqZ+C5ei6ZK 9+/chh4jaoQmdAt/deg4HbUNuNkCoXcIE0MaBs7rFbcXtrWo4GP5nFG3vTfNPLcCccSt cuLqzvEK+szyDN6EeI4/7GFmlW2HCSaSuRD5o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728390989; x=1728995789; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xkkPCPfKEMmjFLM8D1jGEC3Z/sB1AWbZTK+36QpNdFs=; b=rKRGOPFzcOl0APiIt9iYE3BP1KUBsBMe9Lb+dAxg6v3YcMf7evxynFUOSAYV807aZu pruiKfiiSg7g3g0Z7n/dOPooCQaDV2ARE2e+bxvRirtauR+xyhw4tK/Dvi82nhd5IAwa Kdknp5uYw5g2bfwkkUF+U2eFsfZnGnM+SlXXpmLxV09ocXolHhF138iZ7rB6hgvI1tl7 ljbOi3h2T5Pgs+w40NhkOUWALak7BPHjwjyQWlPm4kKQFvLStCv3OrO9iMj03BXAgzNN h5mbVgbQJM9b+SGIGH8kRx1Ro6qbXzk8q+/GrBxJrUsjd3N9xIwWB6EBbrOLl4gyuIww KPzQ== X-Gm-Message-State: AOJu0Yx+BP0ZzkpB1V1a/ifNUW1Oaxy7h6ixed9gpdYkEU8GCa3vLCxM aTLYSR8zI7HI9wrvmrxTkz6InvRSp05WGtsXBDiPNM/GMYupXgLdR+SJ3czurJ0JIPHQuuxIJNe q X-Google-Smtp-Source: AGHT+IGczexDuxpqAabtRiVaFp3Y6htlvk59RQLlxfAyvzIst9CAKvGwml7vCTjf7GGO/CCG2q8E/A== X-Received: by 2002:a5d:5582:0:b0:37c:cc01:f7e with SMTP id ffacd0b85a97d-37d0e8d98e0mr7814938f8f.58.1728390988754; Tue, 08 Oct 2024 05:36:28 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f9a:5ae5:8636:3d57]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d1691a47esm8019492f8f.28.2024.10.08.05.36.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Oct 2024 05:36:28 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/8] COW: Fix hardcoded magic numbers and work with python 3.13 Date: Tue, 8 Oct 2024 13:36:20 +0100 Message-ID: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 12:36:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16654 The COW tests started failing on python 3.13. Looks like it is time to fix the FIXME and drop the magic numbers! Signed-off-by: Richard Purdie --- lib/bb/COW.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/bb/COW.py b/lib/bb/COW.py index 76bc08a3ea..4af03c54ad 100644 --- a/lib/bb/COW.py +++ b/lib/bb/COW.py @@ -36,8 +36,9 @@ class COWDictMeta(COWMeta): __marker__ = tuple() def __str__(cls): - # FIXME: I have magic numbers! - return "" % (cls.__count__, len(cls.__dict__) - 3) + ignored_keys = set(["__count__", "__doc__", "__module__", "__firstlineno__", "__static_attributes__"]) + keys = set(cls.__dict__.keys()) - ignored_keys + return "" % (cls.__count__, len(keys)) __repr__ = __str__ @@ -161,8 +162,9 @@ class COWDictMeta(COWMeta): class COWSetMeta(COWDictMeta): def __str__(cls): - # FIXME: I have magic numbers! - return "" % (cls.__count__, len(cls.__dict__) - 3) + ignored_keys = set(["__count__", "__doc__", "__module__", "__firstlineno__", "__static_attributes__"]) + keys = set(cls.__dict__.keys()) - ignored_keys + return "" % (cls.__count__, len(keys)) __repr__ = __str__ From patchwork Tue Oct 8 12:36:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 50035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55A4DCEF161 for ; Tue, 8 Oct 2024 12:36:41 +0000 (UTC) Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by mx.groups.io with SMTP id smtpd.web10.8081.1728390991277083516 for ; Tue, 08 Oct 2024 05:36:31 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=PJKp8E42; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.47, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-37cd0b5515fso3281250f8f.2 for ; Tue, 08 Oct 2024 05:36:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1728390989; x=1728995789; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=SauaoGIEcZqUv6fFacnfQczCeqWjwyPxLfM/hzdzCf4=; b=PJKp8E429JdQdNgcGarTdpMJrVFkap6NSsBdIKMN6QDpwHI9pNblqI/N3m+CaYUo8A AzxDEoT+0dDZgLYO4xdNTk86wiXEnrV/R9AqUac+2ggsEa1i6ti+SwB4RzVwTrE0Zpkl cvJi+xqsVoPAnwRuGkfeoijxpdiaU3uHSI7LM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728390989; x=1728995789; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SauaoGIEcZqUv6fFacnfQczCeqWjwyPxLfM/hzdzCf4=; b=j+HQh0z+CPkDP84ICnLBAObKEeKzsFhR8WZpxBTgOQUid8EDZKR9yhc8KYqXAqX33W pKa2aEfI9KKq8JRhoHHGwafwVSsxP/7XKrDQ/BEWizbIIaTM+vuoruzzjRwOekkBm2ct X8PqnHjHhBG/alE3JxG2ec1LEfeBzLaIMuVDSJ4yzWnhjHXz+dsK61cFSh+IDKxZ1EGP 5Jpi3Mjwge9ijROrOwCpU1w2Yiyk1S/+2rse+qID3DigZ9juP2ppQKGzaapZIH+IrS/l aImVLxwBpwglY4opzpLbipKhO1iPiVYhmZwUONlp169l1n+YVvrGLkOckGxzuoMfqczV bh4Q== X-Gm-Message-State: AOJu0Yw8sgTb5WP6FawfFE59ezo1RYJ+FlHZLq8JG6nXY59Xd0jV4EtF dx9Fx87FBiWFHYCnd7nxo/LFBFf8ar3X3saVWVOlri52gOevYmef0peyGZ4otkU2NuXKi+/dSgP J X-Google-Smtp-Source: AGHT+IErwk1pPxqrXLkxj7uzXmJaV7oEWg3Mw5ZYTDEITKClZrPtIyZqC5Fi4yb7Qy+XZKbeEEgZRw== X-Received: by 2002:a5d:67cc:0:b0:37d:3760:1d9a with SMTP id ffacd0b85a97d-37d37601e33mr382979f8f.17.1728390989401; Tue, 08 Oct 2024 05:36:29 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f9a:5ae5:8636:3d57]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d1691a47esm8019492f8f.28.2024.10.08.05.36.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Oct 2024 05:36:29 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/8] fetch2: Avoid deprecation warning Date: Tue, 8 Oct 2024 13:36:21 +0100 Message-ID: <20241008123627.252307-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> References: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 12:36:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16655 From re on python 3.13 onwards: "Passing count and flags as positional arguments is deprecated. In future Python versions they will be keyword-only parameters." Avoid the warning. Signed-off-by: Richard Purdie --- lib/bb/fetch2/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 33a8086f27..b194a79be9 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -237,7 +237,7 @@ class URI(object): # to RFC compliant URL format. E.g.: # file://foo.diff -> file:foo.diff if urlp.scheme in self._netloc_forbidden: - uri = re.sub("(?<=:)//(?!/)", "", uri, 1) + uri = re.sub(r"(?<=:)//(?!/)", "", uri, count=1) reparse = 1 if reparse: From patchwork Tue Oct 8 12:36:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 50038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FE08CEF16B for ; Tue, 8 Oct 2024 12:36:41 +0000 (UTC) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mx.groups.io with SMTP id smtpd.web10.8082.1728390992228153475 for ; Tue, 08 Oct 2024 05:36:32 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=FeyXEkyu; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.42, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-42cd46f3a26so49035055e9.2 for ; Tue, 08 Oct 2024 05:36:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1728390990; x=1728995790; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=PAKcmVglXg2xDwPEWF78KIe3J10S7NGW5+tqJS5IPG0=; b=FeyXEkyunw1c9QzUZZHaA+OCVgxgW9lkoWy0uVJgDHPo57JBXWeKAtUFDxfX7gowdO hXJ8bheVwiJgG0/YfH6o2+XxmABVDKd9X4LR1hGnUJ0W6wCCDb/OtqGfJSRJDNEYlV2C QdLV46yBh1qH5ofHzBMnlePj1VB31Mro+lz9Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728390990; x=1728995790; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PAKcmVglXg2xDwPEWF78KIe3J10S7NGW5+tqJS5IPG0=; b=pyWzoLRSxsaer/fvTSi+syWyMcemkpEP14be9KE0mnZuABYOje7UovVFUxMH5u0pm2 GAHcH1kSGcE+ksduUsnjCL9gI+dyZtU8+9cSOciDQln1x68/64CNLHwjlguB61jGvdTl jXx5oB285qDP1jTtQkq5otsmd2sJM8z3xq7ghAKlHXIxge5pm97NRMYLAb2WuaixCQSp BSksMrLz5V5P0qZhzkKqAJLd6w+4PKQLgYseBNjow24uCH9ktUQ7O2DmLy20MHJfp1ET 8i5ibCby+A0cVjtu4g9t7HtnfYqyqNXsU5PoGS4yEthK4LAwmLDCqxZyys1xbadublnT PXuA== X-Gm-Message-State: AOJu0Yx0TZtvPzr59F0RbIZXKMuE14ymyrTeNN8Mwp/ZdmwCJqtOj/Ao UL4H+HbRlnKK/vm0OzyxxTbLSvtjT4drL8G522pUCeMe1IwC0kwd/59Lr4v82+anc8NWWZzzEeh d X-Google-Smtp-Source: AGHT+IF0sIqYS3Xq0GQ8w1QAhzYtICbWbSgnvVGsmCwg4RsXWxTOK+hmQy5vrI6+UICl/xy+dCFu7A== X-Received: by 2002:a05:600c:b8d:b0:42f:7639:d88d with SMTP id 5b1f17b1804b1-42f85af56f9mr88024905e9.35.1728390990227; Tue, 08 Oct 2024 05:36:30 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f9a:5ae5:8636:3d57]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d1691a47esm8019492f8f.28.2024.10.08.05.36.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Oct 2024 05:36:29 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/8] checksum/fetch2: Switch from persist_data to a standard cache file Date: Tue, 8 Oct 2024 13:36:22 +0100 Message-ID: <20241008123627.252307-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> References: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 12:36:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16656 The sqlite connection handling is causing problems with python 3.13. The connection can be closed at gc time which causing warnings and those can appear at 'random' points and break output, causing weird failures in different tinfoil tools and other tests. Using sqlite as an IPC was never a great idea so drop that usage entirely and just use the standard cache mechanism we already have for other situations. Signed-off-by: Richard Purdie --- lib/bb/cache.py | 10 ++++++++++ lib/bb/checksum.py | 25 +++++++++++++++++++++++++ lib/bb/cookerdata.py | 5 +++-- lib/bb/fetch2/__init__.py | 33 ++++++++++++++++++++------------- 4 files changed, 58 insertions(+), 15 deletions(-) diff --git a/lib/bb/cache.py b/lib/bb/cache.py index 958652e0e3..ec7b023fc7 100644 --- a/lib/bb/cache.py +++ b/lib/bb/cache.py @@ -847,6 +847,16 @@ class MultiProcessCache(object): data = [{}] return data + def clear_cache(self): + if not self.cachefile: + bb.fatal("Can't clear invalid cachefile") + + self.cachedata = self.create_cachedata() + self.cachedata_extras = self.create_cachedata() + with bb.utils.fileslocked([self.cachefile + ".lock"]): + bb.utils.remove(self.cachefile) + bb.utils.remove(self.cachefile + "-*") + def save_extras(self): if not self.cachefile: return diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py index 557793d366..3fb39a303e 100644 --- a/lib/bb/checksum.py +++ b/lib/bb/checksum.py @@ -142,3 +142,28 @@ class FileChecksumCache(MultiProcessCache): checksums.sort(key=operator.itemgetter(1)) return checksums + +class RevisionsCache(MultiProcessCache): + cache_file_name = "local_srcrevisions.dat" + CACHE_VERSION = 1 + + def __init__(self): + MultiProcessCache.__init__(self) + + def get_revs(self): + return self.cachedata[0] + + def get_rev(self, k): + if k in self.cachedata_extras[0]: + return self.cachedata_extras[0][k] + if k in self.cachedata[0]: + return self.cachedata[0][k] + return None + + def set_rev(self, k, v): + self.cachedata[0][k] = v + self.cachedata_extras[0][k] = v + + def merge_data(self, source, dest): + for h in source[0]: + dest[0][h] = source[0][h] diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py index 3ad5cf3dd0..1f447d30c2 100644 --- a/lib/bb/cookerdata.py +++ b/lib/bb/cookerdata.py @@ -1,3 +1,4 @@ + # # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell @@ -267,8 +268,8 @@ class CookerDataBuilder(object): try: self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles) - if self.data.getVar("BB_WORKERCONTEXT", False) is None and not worker: - bb.fetch.fetcher_init(self.data) + servercontext = self.data.getVar("BB_WORKERCONTEXT", False) is None and not worker + bb.fetch.fetcher_init(self.data, servercontext) bb.parse.init_parser(self.data) bb.event.fire(bb.event.ConfigParsed(), self.data) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index b194a79be9..add742bfad 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -23,13 +23,14 @@ import collections import subprocess import pickle import errno -import bb.persist_data, bb.utils +import bb.utils import bb.checksum import bb.process import bb.event __version__ = "2" _checksum_cache = bb.checksum.FileChecksumCache() +_revisions_cache = bb.checksum.RevisionsCache() logger = logging.getLogger("BitBake.Fetcher") @@ -493,18 +494,23 @@ methods = [] urldata_cache = {} saved_headrevs = {} -def fetcher_init(d): +def fetcher_init(d, servercontext=True): """ Called to initialize the fetchers once the configuration data is known. Calls before this must not hit the cache. """ - revs = bb.persist_data.persist('BB_URI_HEADREVS', d) + _checksum_cache.init_cache(d.getVar("BB_CACHEDIR")) + _revisions_cache.init_cache(d.getVar("BB_CACHEDIR")) + + if not servercontext: + return + try: # fetcher_init is called multiple times, so make sure we only save the # revs the first time it is called. if not bb.fetch2.saved_headrevs: - bb.fetch2.saved_headrevs = dict(revs) + bb.fetch2.saved_headrevs = _revisions_cache.get_revs() except: pass @@ -514,11 +520,10 @@ def fetcher_init(d): logger.debug("Keeping SRCREV cache due to cache policy of: %s", srcrev_policy) elif srcrev_policy == "clear": logger.debug("Clearing SRCREV cache due to cache policy of: %s", srcrev_policy) - revs.clear() + _revisions_cache.clear_cache() else: raise FetchError("Invalid SRCREV cache policy of: %s" % srcrev_policy) - _checksum_cache.init_cache(d.getVar("BB_CACHEDIR")) for m in methods: if hasattr(m, "init"): @@ -526,9 +531,11 @@ def fetcher_init(d): def fetcher_parse_save(): _checksum_cache.save_extras() + _revisions_cache.save_extras() def fetcher_parse_done(): _checksum_cache.save_merge() + _revisions_cache.save_merge() def fetcher_compare_revisions(d): """ @@ -536,7 +543,7 @@ def fetcher_compare_revisions(d): when bitbake was started and return true if they have changed. """ - headrevs = dict(bb.persist_data.persist('BB_URI_HEADREVS', d)) + headrevs = _revisions_cache.get_revs() return headrevs != bb.fetch2.saved_headrevs def mirror_from_string(data): @@ -1662,13 +1669,13 @@ class FetchMethod(object): if not hasattr(self, "_latest_revision"): raise ParameterError("The fetcher for this URL does not support _latest_revision", ud.url) - revs = bb.persist_data.persist('BB_URI_HEADREVS', d) key = self.generate_revision_key(ud, d, name) - try: - return revs[key] - except KeyError: - revs[key] = rev = self._latest_revision(ud, d, name) - return rev + + rev = _revisions_cache.get_rev(key) + if rev is None: + rev = self._latest_revision(ud, d, name) + _revisions_cache.set_rev(key, rev) + return rev def sortable_revision(self, ud, d, name): latest_rev = self._build_revision(ud, d, name) From patchwork Tue Oct 8 12:36:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 50040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7090BCEF170 for ; Tue, 8 Oct 2024 12:36:41 +0000 (UTC) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mx.groups.io with SMTP id smtpd.web10.8083.1728390993388096386 for ; Tue, 08 Oct 2024 05:36:33 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=UfsTrWKP; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.52, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-37d372c1942so151487f8f.0 for ; Tue, 08 Oct 2024 05:36:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1728390991; x=1728995791; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=TjO5f4fmaPp+N1T97Q4hzIlEU7bNUAlFufez8ShXk1I=; b=UfsTrWKPsghmM0WIVzVc+rNT5ceWeg+1s9yjLQQkg7LtVtLtYHUcWQ8+0KxKNbi95y 9C3I4Cxdm2ynnUTWj+6015GJ8CipSEqh2IXxi+ep3Raqh04dSlgeRbGWLTFcjdEqN9HU FVTwe1Yj854+sIObIvhnv3SJdRtc+HwboqvqI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728390991; x=1728995791; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TjO5f4fmaPp+N1T97Q4hzIlEU7bNUAlFufez8ShXk1I=; b=K+cnK9AH74pUMqMZ9uwmFRPS3GITiYNPf2Fyd3Hu9YyCs7igpYKCn+C0WscK2QcHRm YWLFGoLjLtgRYK1MwxwL8UOjp9t82/eAjtmbgyKGD3n/X8Hije6bQ7nfoOqyntnUFlxh Rc4wcHqkyhMvoICOdakH70qtA558Mlq36E1CILrrGGgRXhXBTkk7wRpgps+hmFx8yc+6 igTA+E0Icv747y9svBAnVS/zriTUudFcDDNZMWGhmA77V+UGpr0j9YX0Ru4JDQHY5qTt LcBVOtJ/DGExzhKxyj7dc+TIZd0KDzi7GFKnkyHAFKV9ZhUseXVHZ5KmAfwSXdNmfZDS +LXg== X-Gm-Message-State: AOJu0YzfEXKlj6fZ6w4itAh7PEvyAROpa/k4Hk3VEunSKlJKFCFQ8rtB O6cv5LrgYipdgkgBXVO9NXEi0gKWb7phwXFff2LB9pYLLn8KE05eyBUctDc089zfPkotLL0k8jf l X-Google-Smtp-Source: AGHT+IFi1rUkO1gJ93zqvkdzpuwQ8FsSlC0NRfV5Ukk8kHJbZU2ZT5mAn3XA/KUj8vzfOv+XDIT09A== X-Received: by 2002:a5d:69ca:0:b0:37c:d02a:7e41 with SMTP id ffacd0b85a97d-37d0e738969mr8562036f8f.15.1728390991243; Tue, 08 Oct 2024 05:36:31 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f9a:5ae5:8636:3d57]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d1691a47esm8019492f8f.28.2024.10.08.05.36.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Oct 2024 05:36:30 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 4/8] persist_data: Remove it Date: Tue, 8 Oct 2024 13:36:23 +0100 Message-ID: <20241008123627.252307-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> References: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 12:36:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16657 It was never a great solution to persisting data and there are much better ones now. The last user has been replaced so drop the code and tests. Signed-off-by: Richard Purdie --- bin/bitbake-selftest | 1 - lib/bb/persist_data.py | 271 ----------------------------------- lib/bb/tests/persist_data.py | 129 ----------------- 3 files changed, 401 deletions(-) delete mode 100644 lib/bb/persist_data.py delete mode 100644 lib/bb/tests/persist_data.py diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest index ce901232fe..1b7a783fdc 100755 --- a/bin/bitbake-selftest +++ b/bin/bitbake-selftest @@ -28,7 +28,6 @@ tests = ["bb.tests.codeparser", "bb.tests.event", "bb.tests.fetch", "bb.tests.parse", - "bb.tests.persist_data", "bb.tests.runqueue", "bb.tests.siggen", "bb.tests.utils", diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py deleted file mode 100644 index bcca791edf..0000000000 --- a/lib/bb/persist_data.py +++ /dev/null @@ -1,271 +0,0 @@ -"""BitBake Persistent Data Store - -Used to store data in a central location such that other threads/tasks can -access them at some future date. Acts as a convenience wrapper around sqlite, -currently, providing a key/value store accessed by 'domain'. -""" - -# Copyright (C) 2007 Richard Purdie -# Copyright (C) 2010 Chris Larson -# -# SPDX-License-Identifier: GPL-2.0-only -# - -import collections -import collections.abc -import contextlib -import functools -import logging -import os.path -import sqlite3 -import sys -from collections.abc import Mapping - -sqlversion = sqlite3.sqlite_version_info -if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): - raise Exception("sqlite3 version 3.3.0 or later is required.") - - -logger = logging.getLogger("BitBake.PersistData") - -@functools.total_ordering -class SQLTable(collections.abc.MutableMapping): - class _Decorators(object): - @staticmethod - def retry(*, reconnect=True): - """ - Decorator that restarts a function if a database locked sqlite - exception occurs. If reconnect is True, the database connection - will be closed and reopened each time a failure occurs - """ - def retry_wrapper(f): - def wrap_func(self, *args, **kwargs): - # Reconnect if necessary - if self.connection is None and reconnect: - self.reconnect() - - count = 0 - while True: - try: - return f(self, *args, **kwargs) - except sqlite3.OperationalError as exc: - if count < 500 and ('is locked' in str(exc) or 'locking protocol' in str(exc)): - count = count + 1 - if reconnect: - self.reconnect() - continue - raise - return wrap_func - return retry_wrapper - - @staticmethod - def transaction(f): - """ - Decorator that starts a database transaction and creates a database - cursor for performing queries. If no exception is thrown, the - database results are committed. If an exception occurs, the database - is rolled back. In all cases, the cursor is closed after the - function ends. - - Note that the cursor is passed as an extra argument to the function - after `self` and before any of the normal arguments - """ - def wrap_func(self, *args, **kwargs): - # Context manager will COMMIT the database on success, - # or ROLLBACK on an exception - with self.connection: - # Automatically close the cursor when done - with contextlib.closing(self.connection.cursor()) as cursor: - return f(self, cursor, *args, **kwargs) - return wrap_func - - """Object representing a table/domain in the database""" - def __init__(self, cachefile, table): - self.cachefile = cachefile - self.table = table - - self.connection = None - self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT PRIMARY KEY NOT NULL, value TEXT);" % table) - - @_Decorators.retry(reconnect=False) - @_Decorators.transaction - def _setup_database(self, cursor): - cursor.execute("pragma synchronous = off;") - # Enable WAL and keep the autocheckpoint length small (the default is - # usually 1000). Persistent caches are usually read-mostly, so keeping - # this short will keep readers running quickly - cursor.execute("pragma journal_mode = WAL;") - cursor.execute("pragma wal_autocheckpoint = 100;") - - def reconnect(self): - if self.connection is not None: - self.connection.close() - self.connection = sqlite3.connect(self.cachefile, timeout=5) - self.connection.text_factory = str - self._setup_database() - - @_Decorators.retry() - @_Decorators.transaction - def _execute_single(self, cursor, *query): - """ - Executes a single query and discards the results. This correctly closes - the database cursor when finished - """ - cursor.execute(*query) - - @_Decorators.retry() - def _row_iter(self, f, *query): - """ - Helper function that returns a row iterator. Each time __next__ is - called on the iterator, the provided function is evaluated to determine - the return value - """ - class CursorIter(object): - def __init__(self, cursor): - self.cursor = cursor - - def __iter__(self): - return self - - def __next__(self): - row = self.cursor.fetchone() - if row is None: - self.cursor.close() - raise StopIteration - return f(row) - - def __enter__(self): - return self - - def __exit__(self, typ, value, traceback): - self.cursor.close() - return False - - cursor = self.connection.cursor() - try: - cursor.execute(*query) - return CursorIter(cursor) - except: - cursor.close() - - def __enter__(self): - self.connection.__enter__() - return self - - def __exit__(self, *excinfo): - self.connection.__exit__(*excinfo) - - @_Decorators.retry() - @_Decorators.transaction - def __getitem__(self, cursor, key): - cursor.execute("SELECT * from %s where key=?;" % self.table, [key]) - row = cursor.fetchone() - if row is not None: - return row[1] - raise KeyError(key) - - @_Decorators.retry() - @_Decorators.transaction - def __delitem__(self, cursor, key): - if key not in self: - raise KeyError(key) - cursor.execute("DELETE from %s where key=?;" % self.table, [key]) - - @_Decorators.retry() - @_Decorators.transaction - def __setitem__(self, cursor, key, value): - if not isinstance(key, str): - raise TypeError('Only string keys are supported') - elif not isinstance(value, str): - raise TypeError('Only string values are supported') - - # Ensure the entire transaction (including SELECT) executes under write lock - cursor.execute("BEGIN EXCLUSIVE") - - cursor.execute("SELECT * from %s where key=?;" % self.table, [key]) - row = cursor.fetchone() - if row is not None: - cursor.execute("UPDATE %s SET value=? WHERE key=?;" % self.table, [value, key]) - else: - cursor.execute("INSERT into %s(key, value) values (?, ?);" % self.table, [key, value]) - - @_Decorators.retry() - @_Decorators.transaction - def __contains__(self, cursor, key): - cursor.execute('SELECT * from %s where key=?;' % self.table, [key]) - return cursor.fetchone() is not None - - @_Decorators.retry() - @_Decorators.transaction - def __len__(self, cursor): - cursor.execute("SELECT COUNT(key) FROM %s;" % self.table) - row = cursor.fetchone() - if row is not None: - return row[0] - - def __iter__(self): - return self._row_iter(lambda row: row[0], "SELECT key from %s;" % self.table) - - def __lt__(self, other): - if not isinstance(other, Mapping): - raise NotImplementedError() - - return len(self) < len(other) - - def get_by_pattern(self, pattern): - return self._row_iter(lambda row: row[1], "SELECT * FROM %s WHERE key LIKE ?;" % - self.table, [pattern]) - - def values(self): - return list(self.itervalues()) - - def itervalues(self): - return self._row_iter(lambda row: row[0], "SELECT value FROM %s;" % - self.table) - - def items(self): - return list(self.iteritems()) - - def iteritems(self): - return self._row_iter(lambda row: (row[0], row[1]), "SELECT * FROM %s;" % - self.table) - - @_Decorators.retry() - @_Decorators.transaction - def clear(self, cursor): - cursor.execute("DELETE FROM %s;" % self.table) - - def has_key(self, key): - return key in self - -def persist(domain, d): - """Convenience factory for SQLTable objects based upon metadata""" - import bb.utils - cachedir = (d.getVar("PERSISTENT_DIR") or - d.getVar("CACHE")) - if not cachedir: - logger.critical("Please set the 'PERSISTENT_DIR' or 'CACHE' variable") - sys.exit(1) - - bb.utils.mkdirhier(cachedir) - cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3") - - try: - return SQLTable(cachefile, domain) - except sqlite3.OperationalError: - # Sqlite fails to open database when its path is too long. - # After testing, 504 is the biggest path length that can be opened by - # sqlite. - # Note: This code is called before sanity.bbclass and its path length - # check - max_len = 504 - if len(cachefile) > max_len: - logger.critical("The path of the cache file is too long " - "({0} chars > {1}) to be opened by sqlite! " - "Your cache file is \"{2}\"".format( - len(cachefile), - max_len, - cachefile)) - sys.exit(1) - else: - raise diff --git a/lib/bb/tests/persist_data.py b/lib/bb/tests/persist_data.py deleted file mode 100644 index f641b5acbc..0000000000 --- a/lib/bb/tests/persist_data.py +++ /dev/null @@ -1,129 +0,0 @@ -# -# BitBake Test for lib/bb/persist_data/ -# -# Copyright (C) 2018 Garmin Ltd. -# -# SPDX-License-Identifier: GPL-2.0-only -# - -import unittest -import bb.data -import bb.persist_data -import tempfile -import threading - -class PersistDataTest(unittest.TestCase): - def _create_data(self): - return bb.persist_data.persist('TEST_PERSIST_DATA', self.d) - - def setUp(self): - self.d = bb.data.init() - self.tempdir = tempfile.TemporaryDirectory() - self.d['PERSISTENT_DIR'] = self.tempdir.name - self.data = self._create_data() - self.items = { - 'A1': '1', - 'B1': '2', - 'C2': '3' - } - self.stress_count = 10000 - self.thread_count = 5 - - for k,v in self.items.items(): - self.data[k] = v - - def tearDown(self): - self.tempdir.cleanup() - - def _iter_helper(self, seen, iterator): - with iter(iterator): - for v in iterator: - self.assertTrue(v in seen) - seen.remove(v) - self.assertEqual(len(seen), 0, '%s not seen' % seen) - - def test_get(self): - for k, v in self.items.items(): - self.assertEqual(self.data[k], v) - - self.assertIsNone(self.data.get('D')) - with self.assertRaises(KeyError): - self.data['D'] - - def test_set(self): - for k, v in self.items.items(): - self.data[k] += '-foo' - - for k, v in self.items.items(): - self.assertEqual(self.data[k], v + '-foo') - - def test_delete(self): - self.data['D'] = '4' - self.assertEqual(self.data['D'], '4') - del self.data['D'] - self.assertIsNone(self.data.get('D')) - with self.assertRaises(KeyError): - self.data['D'] - - def test_contains(self): - for k in self.items: - self.assertTrue(k in self.data) - self.assertTrue(self.data.has_key(k)) - self.assertFalse('NotFound' in self.data) - self.assertFalse(self.data.has_key('NotFound')) - - def test_len(self): - self.assertEqual(len(self.data), len(self.items)) - - def test_iter(self): - self._iter_helper(set(self.items.keys()), self.data) - - def test_itervalues(self): - self._iter_helper(set(self.items.values()), self.data.itervalues()) - - def test_iteritems(self): - self._iter_helper(set(self.items.items()), self.data.iteritems()) - - def test_get_by_pattern(self): - self._iter_helper({'1', '2'}, self.data.get_by_pattern('_1')) - - def _stress_read(self, data): - for i in range(self.stress_count): - for k in self.items: - data[k] - - def _stress_write(self, data): - for i in range(self.stress_count): - for k, v in self.items.items(): - data[k] = v + str(i) - - def _validate_stress(self): - for k, v in self.items.items(): - self.assertEqual(self.data[k], v + str(self.stress_count - 1)) - - def test_stress(self): - self._stress_read(self.data) - self._stress_write(self.data) - self._validate_stress() - - def test_stress_threads(self): - def read_thread(): - data = self._create_data() - self._stress_read(data) - - def write_thread(): - data = self._create_data() - self._stress_write(data) - - threads = [] - for i in range(self.thread_count): - threads.append(threading.Thread(target=read_thread)) - threads.append(threading.Thread(target=write_thread)) - - for t in threads: - t.start() - self._stress_read(self.data) - for t in threads: - t.join() - self._validate_stress() - From patchwork Tue Oct 8 12:36:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 50039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73863CEF171 for ; Tue, 8 Oct 2024 12:36:41 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web11.8048.1728390993800310447 for ; Tue, 08 Oct 2024 05:36:34 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=dOeOgQlo; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-37ccfba5df5so3622926f8f.0 for ; Tue, 08 Oct 2024 05:36:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1728390992; x=1728995792; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=IdsHgCCNMzMQBiS01W4ReITWN+XfIodgK31FIy1hvJA=; b=dOeOgQloKrUFhSIutXzxJ5KKsijJtHD3v7omeCQeoPs0S/CGpgcli1gZz6aCcj6/0x 2ZZ1sJ9IfuQyUgBWH4tDULCPVfuXxkeu3CNTTRzAO6PzzLFF1pSkRoxDfTEY19cSxbOz pch3zGGnY9SFw7dr8Odtx8vSuV3xtwgtucf9g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728390992; x=1728995792; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IdsHgCCNMzMQBiS01W4ReITWN+XfIodgK31FIy1hvJA=; b=JpWc8lwl40iGhymLJQ/PEQfw6jJcF8uFgJiIT2YAKsnU6qP9vQDe38h55cTxUsRxTZ fXvJwg4XOT9mmhiVSbQwNHge0d0K2Lygbt0TaC9EMTcBHtXAlgpwX5JzBcttGxnuHGFX 1FAGCqm4fC1OdxTjWRqyXqGPsuhEryQvuIU5Mgvx11PBtfoHUcab7XtqCRJNhqUHRRG9 abQjoMjb2BzZKo7Mjxa/i254wz5ImYwmAb/rGte5tjdtSTSWnhfj0UnDwPNx5CcafM4P M88+l48mbpWf0p6/5G3ZZBWl+NVx9KTDZSPMzUCSWC+vCOriNMacTy0qizzABRTvWOZi M5FA== X-Gm-Message-State: AOJu0YxOCLjijNs566aPkwcVqekG5Bn4hj0xPjIIaGChgEoqNJEwYIm3 WemU+URHztCXMCDON7B8IWVURuCSrsz/pn32BHXR6h35boZ60H+Hw7GsSddOhtD8mKiPuCW/8ZK p X-Google-Smtp-Source: AGHT+IHclc4pIT2JjKiJYuGdif1Z9DsRwkaNckN97OEn1qli86A3SLyY7tnOoKQiykfS0ZeNbhjf+g== X-Received: by 2002:a5d:5c87:0:b0:37d:2f6b:a08b with SMTP id ffacd0b85a97d-37d2f6ba46cmr1005474f8f.28.1728390991974; Tue, 08 Oct 2024 05:36:31 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f9a:5ae5:8636:3d57]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d1691a47esm8019492f8f.28.2024.10.08.05.36.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Oct 2024 05:36:31 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 5/8] bb/exceptions: Drop obsolete 2.6 code Date: Tue, 8 Oct 2024 13:36:24 +0100 Message-ID: <20241008123627.252307-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> References: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 12:36:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16658 In python 3 we should always have named tuples in this context. Drop the obsolete exception handling. Signed-off-by: Richard Purdie --- lib/bb/exceptions.py | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bb/exceptions.py b/lib/bb/exceptions.py index 801db9c82f..acf546de1d 100644 --- a/lib/bb/exceptions.py +++ b/lib/bb/exceptions.py @@ -38,14 +38,8 @@ def _get_frame_args(frame): """Get the formatted arguments and class (if available) for a frame""" arginfo = inspect.getargvalues(frame) - try: - if not arginfo.args: + if not arginfo.args: return '', None - # There have been reports from the field of python 2.6 which doesn't - # return a namedtuple here but simply a tuple so fallback gracefully if - # args isn't present. - except AttributeError: - return '', None firstarg = arginfo.args[0] if firstarg == 'self': From patchwork Tue Oct 8 12:36:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 50037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64C0DCEF169 for ; Tue, 8 Oct 2024 12:36:41 +0000 (UTC) Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by mx.groups.io with SMTP id smtpd.web11.8049.1728390994558553320 for ; Tue, 08 Oct 2024 05:36:34 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=KzEnVyl7; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-37d1eda8c7cso1479059f8f.3 for ; Tue, 08 Oct 2024 05:36:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1728390993; x=1728995793; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=b4tRqMJv/6wlwSREQXxvS+uKbWNYZ9T6lLDeSZX8qdI=; b=KzEnVyl71yviXRPmczKK0XrzHkH+nAoPqRcldnbOz8qrAV+w/EiCRKZ8adn04skBSz nHklKHwq7IWHSB78povRPa+x60+lrOETC54fBMJW7VJZU6T1CFm9FQ3GVSiU63Xt1SID CYRPbqQ1T2KkC3XXJKkefEdWureWxpWrs2VnE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728390993; x=1728995793; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b4tRqMJv/6wlwSREQXxvS+uKbWNYZ9T6lLDeSZX8qdI=; b=u82oZflZNVwsLXAZlvkXtwZz1p81IguluE8VmyV5IoFwumB/i6Am1XIhkiZ3JODV6B Mnz9WiBGwSWKOMPuVA7oCJOMzZ9XEgicp7+iM0BXvVO6fY38eKEEoQFAJEzfEIhcp/2P jymb1cuGdzmsmJvnUnm17K8+EaPX4ZFabDvoc5z8K3FEtuqv2uKADCJoZDh8w+sAFWAJ +/P9Fh76R12fqx0FpNXBOLtCjVl6LOiwSoK1sOIAYf4J7UY6z/5+ipmvhINN1cGHV9MR oULeh67i2RgZ6mwEA894r1gSgqDhap29T4u+t+GcGBZcNm3SmsZ5BA4/OkvWaPa+hj6T FVDg== X-Gm-Message-State: AOJu0Yz5sO/TxpmVQU0bjn6qG9vf21WNKCwwtJo5y258kdINqv1aHeyo AtAyDbB1+zweTNWcILQFcJ3bQs0DJjf+zq6iX/wJ8tMV8ZGOHdpK2/XvhkISRI4h+wIf/WCYYQj O X-Google-Smtp-Source: AGHT+IGwpu/hkMx3KBzH/VX+WQQZnB1dLglcVK94URn1STyuXxThXZHvuD31GBsC3RuxnrOzZCwbfg== X-Received: by 2002:a5d:6e91:0:b0:37c:d344:8b42 with SMTP id ffacd0b85a97d-37d0e6f0dfemr8705042f8f.15.1728390992730; Tue, 08 Oct 2024 05:36:32 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f9a:5ae5:8636:3d57]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d1691a47esm8019492f8f.28.2024.10.08.05.36.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Oct 2024 05:36:32 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 6/8] exceptions: Add hack to workaround python 3.13 failures Date: Tue, 8 Oct 2024 13:36:25 +0100 Message-ID: <20241008123627.252307-6-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> References: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 12:36:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16659 Signed-off-by: Richard Purdie --- lib/bb/exceptions.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/bb/exceptions.py b/lib/bb/exceptions.py index acf546de1d..60643bd642 100644 --- a/lib/bb/exceptions.py +++ b/lib/bb/exceptions.py @@ -47,7 +47,11 @@ def _get_frame_args(frame): cls = self.__class__.__name__ arginfo.args.pop(0) - del arginfo.locals['self'] + try: + del arginfo.locals['self'] + except TypeError: + # FIXME - python 3.13 FrameLocalsProxy can't be modified + pass else: cls = None From patchwork Tue Oct 8 12:36:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 50036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FE35CEF16C for ; Tue, 8 Oct 2024 12:36:41 +0000 (UTC) Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by mx.groups.io with SMTP id smtpd.web10.8084.1728390995323358906 for ; Tue, 08 Oct 2024 05:36:35 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Xr2aqghy; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-37cc810ce73so2944776f8f.1 for ; Tue, 08 Oct 2024 05:36:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1728390993; x=1728995793; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=IVHyIEg8w++hN0pohrf4BdPgbxgVUKfJ8HephhM1P8Y=; b=Xr2aqghySTQ8eAF7YG3GuS2SQoOxt/6UjCCBl+fE5LoW0BtJ/GE65pwyDZUhtrTquD S4Fjg0vIiwco5o5r6N7DhT+RfvQxnw6GoRM6Pd+/xCKr2buZWAyQuLt60kRbm6Bsul4M RWWa5t416D+MkSSlgYu999+CtpQgostO2L2cA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728390993; x=1728995793; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IVHyIEg8w++hN0pohrf4BdPgbxgVUKfJ8HephhM1P8Y=; b=SXvsII6+AUXX936lacSrB7FVmDLHU1a+IF31OCEYgyhMxoMJ4N2W41/KHnQOiNQgi6 QB8rZqb42CJRXIV08r2qjLEeIJ8VCQPjSaelQ4rSaf8j4Wbd11RkXmfH46ciVcGN/niH XM54x7tNeIJEvfCPpt/RV5aMlI/AdG58tRVG6UEbNWLPp/XJjr3+ixZ6B2rkGwaXrlRl 5GIoKcOv8Mr0wHa8fqAQh7QTLaS/vgBoFuB4tFi3crXp6Rmj40JVK/G20B2IvC2prkIE ql1fFZSuAO/Ni1zfa0S/o4Ny9pR0oW4+gRLvwECTWCqoIQll6CFbDr6h/hweupwriDFC wF5g== X-Gm-Message-State: AOJu0YxjCEv3iLQqltciXUrK3nU/juFCd3wpJUfbxJhjhMRPXbJYYbOw b0p+kHzfhf6YNsdHft76zeEZkhvmMm6S/EqKoyYa95xrxhsUPnN4a6F8rEqt3JTq2/aV80iEB/D i X-Google-Smtp-Source: AGHT+IGdaPiEJ7xWpOPsSl3TdBqUbl0eCtfkMFR/6dqcKw3W9dVXnUanITK9BydjOjg4vnmGc10wdQ== X-Received: by 2002:a5d:5751:0:b0:37c:c605:218b with SMTP id ffacd0b85a97d-37d0e792649mr7210456f8f.35.1728390993441; Tue, 08 Oct 2024 05:36:33 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f9a:5ae5:8636:3d57]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d1691a47esm8019492f8f.28.2024.10.08.05.36.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Oct 2024 05:36:33 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 7/8] toaster/tests: Fix kirkstone numbering Date: Tue, 8 Oct 2024 13:36:26 +0100 Message-ID: <20241008123627.252307-7-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> References: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 12:36:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16660 When the fixtures were updated for the styhead release, the test cases were not updated to match. Fix this. Signed-off-by: Richard Purdie --- lib/toaster/tests/functional/test_create_new_project.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/toaster/tests/functional/test_create_new_project.py b/lib/toaster/tests/functional/test_create_new_project.py index cdfdd9ab20..c3b5679514 100644 --- a/lib/toaster/tests/functional/test_create_new_project.py +++ b/lib/toaster/tests/functional/test_create_new_project.py @@ -110,7 +110,7 @@ class TestCreateNewProject(SeleniumFunctionalTestCase): - Release: Yocto Project 4.0 "Kirkstone" (option value: 4) - Merge Toaster settings: True """ - release = '4' + release = '5' release_title = 'Yocto Project 4.0 "Kirkstone"' project_name = 'projectkirkstone' self._create_test_new_project( From patchwork Tue Oct 8 12:36:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 50034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 549B8CEF166 for ; Tue, 8 Oct 2024 12:36:41 +0000 (UTC) Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by mx.groups.io with SMTP id smtpd.web11.8050.1728390996243608117 for ; Tue, 08 Oct 2024 05:36:36 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=P7k5OE90; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.47, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-42cb1e623d1so57405395e9.0 for ; Tue, 08 Oct 2024 05:36:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1728390994; x=1728995794; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=harKblW+l4zBr3bytm9a4M83D6H970PrS9fvj1rsaMM=; b=P7k5OE909/wEFlA79MlZENTGZjQEijZGhOaYo+hvRnAPdj+9ro2DPyba7y5ncjlPaO 3uh4rfEX/nCDJuJB4eZVXFJTuBnjYzgZpGYzYGziGYlJl/5MslTU2zT0FX0mU9Af7/d2 nPBKugXmkNBZoD/wy1YuFdTgF6WzmDvrxLvqU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728390994; x=1728995794; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=harKblW+l4zBr3bytm9a4M83D6H970PrS9fvj1rsaMM=; b=oMZgYfbfvIYyos91ODYu56Ve939ygMt6TT4Dom5GoKf1/CVA79JQYcDfkS+d0sJf+/ a3dVitltQkkN1RI5RhpG7O1HJujWlPdI+xr1qD7+pCzUETO8Cf2GO2acuZ6MbU1zlkun iKiWzt7PniPrcX2gYLZ2n2obQQKblc7pNxwA0d1pxQgcwAaZLZaMmPc1XgOu/EsOCePP hcOYuylupa/XETruiFISTlbniuJf2AtvfVjIs9xH7mQl/z6EyxYfYBKNcmoWY6RScElJ Zc7w4mn6C9Ihcm8RL7KkgoWw+2KrsLclZLA6idrIdkBrwZm9/g3FKlJFCweglM/oLSYU DjsQ== X-Gm-Message-State: AOJu0Yyuj/XCpPXU6/bPOtFgNzzZCWqz/AVzf8gBTgYf27tF8Mt23DeD uACGvOYMWgxK3xUy5hfJB7JsHPuEdyFFmmE7ryZqXCDmkm6pY7nzcIguC1PPQWWJpofZ8dAYFiu z X-Google-Smtp-Source: AGHT+IGK300hoX+1qJX/kU22ARxaeU5zysH2Z8bKv8SdYjlswvAVcVRRDk1GChdxj9TF0vZkcChJDA== X-Received: by 2002:a05:600c:a49:b0:42c:b52b:4335 with SMTP id 5b1f17b1804b1-42f85aad1c3mr128095965e9.10.1728390994271; Tue, 08 Oct 2024 05:36:34 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f9a:5ae5:8636:3d57]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d1691a47esm8019492f8f.28.2024.10.08.05.36.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Oct 2024 05:36:33 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 8/8] toaster/tests: Improve a test to give better debug output Date: Tue, 8 Oct 2024 13:36:27 +0100 Message-ID: <20241008123627.252307-8-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> References: <20241008123627.252307-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 12:36:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16661 "False is not True" is not a helpful error messgae. "XXX not in YYY" is much more helpful. Tweak the test to give more meaninful failure messages using assertIn rather than assertTrue. Signed-off-by: Richard Purdie --- lib/toaster/tests/functional/test_project_page.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/toaster/tests/functional/test_project_page.py b/lib/toaster/tests/functional/test_project_page.py index 0e36b44ffd..deb26812ea 100644 --- a/lib/toaster/tests/functional/test_project_page.py +++ b/lib/toaster/tests/functional/test_project_page.py @@ -719,16 +719,16 @@ class TestProjectPage(SeleniumFunctionalTestCase): remove_layer_btn.click() self.wait_until_visible('#change-notification', poll=2) change_notification = self.find('#change-notification') - self.assertTrue( - f'You have removed 1 layer from your project' in str(change_notification.text) + self.assertIn( + f'You have removed 1 layer from your project', str(change_notification.text) ) # check add layer button works, 18 is the random layer id add_layer_btn = self.find('#add-remove-layer-btn') add_layer_btn.click() self.wait_until_visible('#change-notification') change_notification = self.find('#change-notification') - self.assertTrue( - f'You have added 1 layer to your project' in str(change_notification.text) + self.assertIn( + f'You have added 1 layer to your project', str(change_notification.text) ) # check tabs(layers, recipes, machines) are displayed tabs = self.find_all('.nav-tabs li')