From patchwork Sun Sep 27 07:46:30 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yoann Congal X-Patchwork-Id: 99279 X-Patchwork-Delegate: yoann.congal@smile.fr Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93469C98324 for ; Sun, 27 Sep 2026 07:46:48 +0000 (UTC) Received: from mail-wr2-f35.google.com (mail-wr2-f35.google.com [74.125.225.99]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.33618.1790495200173231951 for ; Sun, 27 Sep 2026 00:46:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@smile.fr header.s=google header.b=FYlMBGIR; spf=pass (domain: smile.fr, ip: 74.125.225.99, mailfrom: yoann.congal@smile.fr) Received: by mail-wr2-f35.google.com with SMTP id ffacd0b85a97d-4885a1480a2so1118493f8f.3 for ; Sun, 27 Sep 2026 00:46:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smile.fr; s=google; t=1790495198; x=1791099998; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=WIFxBJLTzbvqMPkKIkzCjT2UyQq6nDbDujxiEhH/Fvc=; b=FYlMBGIRngvgG2A45r4MPUqCMuZ6ORhc+PA7Qa78iZovcHiBonD0lK6suGyAYEjViJ cnMS4clr5JL1y5BD6oZYQ0Kuwqqv2NvU0tPHpI+17Y2V4XmaauNLNAcHNsCGsnfnjwrU ANqOKprKe3JXDo2Eoe4bhvUNTwKLy9zci8vts= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790495198; x=1791099998; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=WIFxBJLTzbvqMPkKIkzCjT2UyQq6nDbDujxiEhH/Fvc=; b=wyuMSNiXSLU9rv9Ht1R68mNX56x3ecFusEI7UDQr44GdcVIBrSEulnrVJve0PnvfcO zTlnJJQSAb5NER352Qq6eN0RLwJv5G0C02Ht9S8Nb5EBHTpCqLTg0t8PbXssTFjQyfw0 Ch1tf56Cw2FKVfB84YbKnllGXYjw+tkgEMuq1wF9NqPV95pZ1Didc2CWvgFjG6oVjZ1o eT7xndEat7xwwRS6QpAXLaLCG53hFOZciDaAPOVkIUfzLYA0UkvYz3fLIIKHUS3ZRWcL jH7z8tYaXlpn4AiAyXxnyz//IakfPO47pBAfjPlXyevbHqySVKkp0O0oRIBHzpBfGyB5 uPLA== X-Gm-Message-State: AFq9FYL3jlna7BmV2ZAqlASQWaJhysZvE+lTB0j/OCpLFtHm5OCG1izQ ZhwBLAOITP+PoluL/p0ARunIHnnAr5vUIciOltME7W8pOhbvpIjH5tcU1tzMqQHV96Cy5r3sH6B VLgoVtxY= X-Gm-Gg: AYBFou3EE2Npa4lMPsQnJ4WIXp+VEikq4pHGlbFirzZ1Nw3gXnmPbH48Dk+F3cXGkxQ 9mCYlln6BS9c1t1RPIM2JtIkil5YCmDb9NhGxz332hu9xY9zdNSlOHs8Ppt1E+QiB2myFDk5LGd +gHVhtiKyRaA7LwKlr76wdWabu+5W87R6VwgBN6/4ghEsfYSyKHgSmuxK1f1c0xKs+GiQ3QFMPp /Kmaw7QW1DdxGsAmWPeNleusm8uqbA1FnHe+SeLRikwSR/4dMh4Prw0IowR+EA2BzXmPJS9xS2V sYOMaFoKAfNbPTg0Jn/iRQONn8dwAKu1mKMvvKNKO8ibgfwgo8/p7yWd2J22k3cWDnvN4yhLPHX p6TqqPxwAMSDKwsuT9vw9h0/tQ5P2qlPCkawkLYkwkASEYsU/ZnrbBe1XAFotz470FQcytcZdvb dglwHBBEaJqpncmuc8tpJd+p2qYru8z98NYMoUZk4t9H7RDl+/w1hGdEnJ3rxgICHJnLd+3yPOo tVsDVssMwtr0fExUiP2NKmTP5RH3EseSazsRTW+/tpBBfrJnW+7ulnhw3p4N2wnmnN5Xt9BamLt pjxEdtKoGQ== X-Received: by 2002:a05:6000:2685:b0:488:8ab9:f74c with SMTP id ffacd0b85a97d-4888ab9f860mr3584470f8f.3.1790495198425; Sun, 27 Sep 2026 00:46:38 -0700 (PDT) Received: from FRSMI25-LASER.home (2a01cb001331aa00a2e4fb7b0d887544.ipv6.abo.wanadoo.fr. [2a01:cb00:1331:aa00:a2e4:fb7b:d88:7544]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4887a30bcdbsm18485266f8f.2.2026.09.27.00.46.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Sep 2026 00:46:38 -0700 (PDT) From: Yoann Congal To: bitbake-devel@lists.openembedded.org Subject: [bitbake][wrynose][2.18][PATCH 1/6] runqueue: Fix pending_hash_index not including taskname Date: Sun, 27 Sep 2026 09:46:30 +0200 Message-ID: <54d387f575de2a1cfb77a1d8c92f0f268fa2e3ee.1790495161.git.yoann.congal@smile.fr> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 27 Sep 2026 07:46:48 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/20260 From: Chris Laplante The intention was probably not to use the literal string, "taskname" Signed-off-by: Chris Laplante Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie (cherry picked from commit c6a1288cb15171297c5607a8f6f6c01595ba8b8e) Signed-off-by: Yoann Congal --- lib/bb/runqueue.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index a4689e134..544081053 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -84,7 +84,7 @@ def pending_hash_index(tid, rqdata): (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) pn = rqdata.dataCaches[mc].pkg_fn[taskfn] h = rqdata.runtaskentries[tid].unihash - return pn + ":" + "taskname" + h + return pn + ":" + taskname + h class RunQueueStats: """ From patchwork Sun Sep 27 07:46:31 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yoann Congal X-Patchwork-Id: 99280 X-Patchwork-Delegate: yoann.congal@smile.fr Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59ED6C98338 for ; Sun, 27 Sep 2026 07:46:58 +0000 (UTC) Received: from mail-wm2-f13.google.com (mail-wm2-f13.google.com [74.125.225.141]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.32879.1790495200622547802 for ; Sun, 27 Sep 2026 00:46:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@smile.fr header.s=google header.b=YuxCbs+O; spf=pass (domain: smile.fr, ip: 74.125.225.141, mailfrom: yoann.congal@smile.fr) Received: by mail-wm2-f13.google.com with SMTP id 5b1f17b1804b1-49e7bcb94d3so15248835e9.2 for ; Sun, 27 Sep 2026 00:46:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smile.fr; s=google; t=1790495199; x=1791099999; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=iHmQFGMaEGMRESwF+ByREQaQNnpah9YBaROXn1EILho=; b=YuxCbs+O786bhRSRzG51mCGODu/jwQs5ZBBc0i46HlPyT82SfbH1lWZ6BAC12e9B3M Ho2xeEl9bjoM4JZCHutNhU73jwKLdSsp27qxGXhKZtObplkRH4QKiJ4joz1y0oGlDUCJ 7cfEtWI7IGxox+HdBZ+yUIfy3FlUwXxcEPjD0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790495199; x=1791099999; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=iHmQFGMaEGMRESwF+ByREQaQNnpah9YBaROXn1EILho=; b=RcpHvtR6l3LJ0hAeKqLhDgWMaAqwoCw6BoNh3LPbIrNA0waaaV4D1Er7lP66F6Bc6D Ho51X/aumA7DjniWK8rp0CaeINKF9LbqmYc3R8jnak1VQsA3juLZ9pV6baLoOrQlodDt x/KreRduh+54uPmh4nolilUdHtFDS9q59wZJjWypg4ATIWPymK8peR6y8wJ8fErLvkTC 85tzP4Yq0ja8rLHJctkZkawufUrVHQhHgQEuCYyGDOrdjbf2zmQCaVw8JLqcGvLGSDdX ufuUGn9gmuZow//ePdqRlxZX3n2tsPLIICDQNPsJkv9mnC0EQev45WYT9bnW0R2GBf+H Iy6A== X-Gm-Message-State: AFuF++kBIXY8LwVaTczprw9ijTGo1IVvHeznA5OiIcNzHYN4xknJ3Xgq Ac7uqoyDmWb5iFYMz2o+wENUgT+/RHfoj73qFMGBj2MWrw4jZ/gSUZCKiVnn8udnrTyaXuFax4c GZgD3rs8= X-Gm-Gg: AYBFou2xSFzoEy5sbetuit9Ej3OIsgTYArKLA3B9au4Oxd9evzIBHzvRK1dFx1Sonln czMRBKTBmyJKcJn68QCMVQ3rdWt1wqh1T31yRpgKOFJXvcSGHVRaVnyRdS4ZDM3OMlSDCqD9V+v qDqtqQJnS7TJ8KpZ7gu93XIMlKKBcjezv6bom/kO70NPQvlOT3Oig3117KldJsP5wp21w7D4LGn DZQRlYx6/G/T3M/4wNkqiavOKWpVZSz++ffAZRdHG7OsTHgu/SAVZGYs/r9VJHZMBCEvqlR8L5c 4TnpH0GlWE90elmVBURYPDJtN4ATdoVijyTBPIecXZZFUCEdRv2FJYCTUCt6bga5iUCP7xUxRN2 6Dz39OhgtH79tT8ghPSJHEeu56ViZrZrhiv76FlZZIvUodcymnGNX2G3lOo5Cna5dRC8IZNkMvv DHz0+xDJHs50CwdKYDL8GHxrOxB25GX0C2pC58k7wllCLsePxlGtsQVzud6T/Ezvb/TFC5wXMAl BemJAh4I6rvO7aoU/lROloh7w2QxEBFprY5fs39KHPPeqpq9I2IS83EI1yOPVLH9aRT76Zlng== X-Received: by 2002:a05:600c:34c6:b0:49e:7cff:f8ca with SMTP id 5b1f17b1804b1-49fe66f4dc9mr210656105e9.29.1790495198895; Sun, 27 Sep 2026 00:46:38 -0700 (PDT) Received: from FRSMI25-LASER.home (2a01cb001331aa00a2e4fb7b0d887544.ipv6.abo.wanadoo.fr. [2a01:cb00:1331:aa00:a2e4:fb7b:d88:7544]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4887a30bcdbsm18485266f8f.2.2026.09.27.00.46.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Sep 2026 00:46:38 -0700 (PDT) From: Yoann Congal To: bitbake-devel@lists.openembedded.org Subject: [bitbake][wrynose][2.18][PATCH 2/6] runqueue: fix bad check in process_possible_migrations Date: Sun, 27 Sep 2026 09:46:31 +0200 Message-ID: <021043086fdb0e69d9ef543807edb845006a5604.1790495161.git.yoann.congal@smile.fr> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 27 Sep 2026 07:46:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/20261 From: Chris Laplante This was probably a copy-paste error from above. Detected by PyCharm's code analysis (Local variable p might be referenced before assignment). Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie (cherry picked from commit 7f24c51c049bde59d28d28bc83dad2a769ec1eb5) Signed-off-by: Yoann Congal --- lib/bb/runqueue.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 544081053..ff8ecdf81 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2610,7 +2610,7 @@ class RunQueueExecute: next = set() ready = {} for tid in current: - if self.rqdata.runtaskentries[p].depends and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): + if self.rqdata.runtaskentries[tid].depends and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): continue # get_taskhash for a given tid *must* be called before get_unihash* below ready[tid] = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches) From patchwork Sun Sep 27 07:46:32 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yoann Congal X-Patchwork-Id: 99282 X-Patchwork-Delegate: yoann.congal@smile.fr Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9879CA5FA0 for ; Sun, 27 Sep 2026 07:46:58 +0000 (UTC) Received: from mail-wr2-f34.google.com (mail-wr2-f34.google.com [74.125.225.98]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.32880.1790495201049154414 for ; Sun, 27 Sep 2026 00:46:41 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@smile.fr header.s=google header.b=JPaY0zxK; spf=pass (domain: smile.fr, ip: 74.125.225.98, mailfrom: yoann.congal@smile.fr) Received: by mail-wr2-f34.google.com with SMTP id ffacd0b85a97d-48884b926f0so963856f8f.3 for ; Sun, 27 Sep 2026 00:46:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smile.fr; s=google; t=1790495199; x=1791099999; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=ao6gpCEgEayUZcwXqtktm4i185YPAb2C/VXgHnnZThg=; b=JPaY0zxKFI0lK1NQIs7nTCFJqMvaDXIFKzLrpkq+YByqpeB3mT9R4xmA25RuSBdTUn yi+a2bDyx6LikfDMvocczT3OrqCgJKNJlQm4vP272hIAVl4UYjU3b36B6GPv83jYtSI2 Ip86js1CQQZNrfOiKBkKQoFlXdZQDNqw9KO3g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790495199; x=1791099999; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=ao6gpCEgEayUZcwXqtktm4i185YPAb2C/VXgHnnZThg=; b=Y7gScp+xFxLLKDSZ72WFzAzA3EjHatmDvXDjtSoMU0xUFTJe75pFp0GB/evi5LDJ7h nKXdVIboi5VDQ3m1plXxNpdZrpMNuVDHU64HUUqdJo4zK7zFMvY0mbhIwQc5+AWX9rnZ K6vSNEA0Vhm2fuXvP+/JTjYYP7LP332k1vwEKxWfGz5owsIf9Q5gzFwFSx2sYIvz5+EI PCNiqc96vAwaQdrZ71rWOo9Nqs02Glr7elK+lO8A8DO4I503hYpSQ0tEFwvNQHKx8ONu bV1RdrEtHzgRk/s45v5CnpW5P4JiZZcCEPoD//6acdeWAuKMIsHF9ShdMlG2k3+XBprV flWA== X-Gm-Message-State: AFq9FYJqDzdVTF5brEM2/SDOl11HTqCGZjrVYr48zmFJXFZrJeHnij9O dnpIqUsIqdJiB171MFPBwZbZsKAJppIZ+NDh1rXkcZpBz8phcNUDLIvuh96viUaLk8u/sBJGPmV zTbvlrew= X-Gm-Gg: AYBFou20aEmoWUeKlZNhHKQGJD4AQcFFxCMzrpkrC1707wh9V+QgcnvGHE3Dx6HquHc NZichuaj99HEuxQW00R9tHeCzbLWLGWSAQ6fypY8RMmyjrGPpsHS8/umj2aKTZhKwvKojoPbcpz fFUQZ0awQTDVp+58rCZZdgm2OEJe+weq9uNg/nm3qqpjCF/5hzT9byLseq4BL1HlBvZa3Z09VBj S7pWJoQI0+BZ3QaO+w4vlJTwzFsRuqiUJ3rCsQijCzqGJQXMJ1adp61wL3b8vuxhgn5wuDXR32P b57QjDXE/MZaytPPDYKCNu27wenMW3uqs6M1Z9p9chfWdR9lrp8sdj9vi03c6RoT0QG72IDUueX 1ZQdRW73UYJdrj1t4Ww3PCIWXli+dawX2jqy6T38zkhctNwJRHvfcRe1bU0wDrj9REyT6j/3wsu QF5O50klmHXyjNOBTX6ZCwSc6PgYdx5bZ/EpI9U2wY0Jyu4Kp8TPDHEm4zqGH3spDvMFIlWLixn OL2wpvfl0hL4oUYXMFS3tlFvdBuRwccirTEbBggVv1fOp33DY71gikXHVuMyM0ZZLEqyKEAog== X-Received: by 2002:a5d:5888:0:b0:488:57ad:d5c8 with SMTP id ffacd0b85a97d-4887171423cmr19465515f8f.3.1790495199308; Sun, 27 Sep 2026 00:46:39 -0700 (PDT) Received: from FRSMI25-LASER.home (2a01cb001331aa00a2e4fb7b0d887544.ipv6.abo.wanadoo.fr. [2a01:cb00:1331:aa00:a2e4:fb7b:d88:7544]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4887a30bcdbsm18485266f8f.2.2026.09.27.00.46.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Sep 2026 00:46:39 -0700 (PDT) From: Yoann Congal To: bitbake-devel@lists.openembedded.org Subject: [bitbake][wrynose][2.18][PATCH 3/6] runqueue: wait for covered tasks before setscene Date: Sun, 27 Sep 2026 09:46:32 +0200 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 27 Sep 2026 07:46:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/20262 From: Chris Laplante The existing continue only affected the inner loop, allowing setscene execution while covered tasks were still running. Check the collection directly so candidate selection waits as intended. AI-Generated: GPT-5.6 Sol Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie (cherry picked from commit 714bcfa7a91327cb2e9934f5b428129936e61e99) Signed-off-by: Yoann Congal --- lib/bb/runqueue.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index ff8ecdf81..870633368 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2250,9 +2250,8 @@ class RunQueueExecute: return True continue # If covered tasks are running, need to wait for them to complete - for t in self.sqdata.sq_covered_tasks[nexttask]: - if t in self.runq_running and t not in self.runq_complete: - continue + if any(t in self.runq_running and t not in self.runq_complete for t in self.sqdata.sq_covered_tasks[nexttask]): + continue if nexttask in self.sq_deferred: # Deferred tasks that were still deferred were skipped above so we now need to process logger.debug("Task %s no longer deferred" % nexttask) From patchwork Sun Sep 27 07:46:33 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yoann Congal X-Patchwork-Id: 99283 X-Patchwork-Delegate: yoann.congal@smile.fr Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65340C9830E for ; Sun, 27 Sep 2026 07:46:58 +0000 (UTC) Received: from mail-wr2-f35.google.com (mail-wr2-f35.google.com [74.125.225.99]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.32881.1790495201560456896 for ; Sun, 27 Sep 2026 00:46:41 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@smile.fr header.s=google header.b=Yugda2iI; spf=pass (domain: smile.fr, ip: 74.125.225.99, mailfrom: yoann.congal@smile.fr) Received: by mail-wr2-f35.google.com with SMTP id ffacd0b85a97d-4887f690df6so1338900f8f.1 for ; Sun, 27 Sep 2026 00:46:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smile.fr; s=google; t=1790495200; x=1791100000; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=Dh8XAfyBiEqNjj/hoh7yhL2Z4aYhIihMsXEh3JFHXDY=; b=Yugda2iIX3pnXYO9qdsnhwoBe3hpSYdTpxoKxMUngHf60VZdB7v8ZYoPp0CrImDP8k 8DPi1+7LznWgc6HrCPkehCsP9dkvZMPXBGhJ5a0Cz/XPBzz9N+zqrEUUu0dGVNdqe5YZ wQHkS9ly+56shuOU1klfNYQS0y0u1bczdju44= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790495200; x=1791100000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=Dh8XAfyBiEqNjj/hoh7yhL2Z4aYhIihMsXEh3JFHXDY=; b=IsfW+evV7PneB8yFIRzTp3te+z558hbdTVcQJmtYEsFDIenGEIukhbLJK7bAzsPMxG Eg6uwPxfTkowDG8AlPp47DvS/+torfebCwfwiCY1ow7EvZHYbYqtrSZYO8w/mpI0hdl2 B5MIK/nFKRzoimVN1bOD5SXjXc1zp4YMwegkd/dyJC7SQKBGTtQrtVXKD5YWAh5YH7Z2 Hs8Tx3titMgweF9+bml0IBsav3q+XG2f7/dneMzoDhmw4sTVtwNmX78Rg6MFWknrm75f tKd/lLWhYKYU5uQEGR/dON30eudPdDRvH/WtbmqaQPgU3voaOVXOmRo5wjlLLGQEz0pu iZ3Q== X-Gm-Message-State: AFq9FYLjDWzlDOrp93JP8mx8Sw4Sm3u81GNeuP5yJJHkY8xfsSz9qUh4 r2BGO+eoMs5Ca5Zl9QTdQ2UKKr3PROpWzsWoWy2SdRpjSfwM+NoCWB+PdkJiWmz88Y36lbOQJm3 5TRuG+r0= X-Gm-Gg: AYBFou1NydUNmNNIJDfFyeu0nCDZJBV23L4WMicajFJ3Xquus69bh6kPQgGxc4Zax51 WeVoOuIwGQlm5sHpOu5Bbup2u1nmhcp9Nfo2A5ObotvJSFNEAGknn9F4FtgGwdBmVbZMTwMGyzf NdO3fSOr44u7fJ4jmPszsKIuH5FF6FZwy246Qkb8qC8Oyv7bkwuA9+i6lorh8lZ6R22bpDZzjvx 5XeMab+Jnuy+ylX+jZ0ja957eDYOkCljF/OoEIHNd6se9vZlz0fP5i6Z2djb5j7lcfJ2kVbLz6/ qk8dsnqQyLJT+iIXx0vipu6RUeCPP/1401/xyjhcVAek/sfNczzS03XO15oLZ1uvJ8ky5FcqpVy f1sdohDtXQjEitzn/AufVNSocdFNu4+iE90ZUKsfenCLCXC/DHwdzcmRmjfdLVWVz85bpgPk25/ XxRXdEHNkG3tt7IQ0s42aEqVMA42+m7TIL4WKlWSx49jThGGFnTW0LzMVuqfi96srgAoUViMPl6 50wa3O4llmSSrLbGdy3Po2e1jQBKsYMyHoiGqNHPfOIKuw448uxzVvRVLUW4ZAMwybpAp9shw== X-Received: by 2002:a05:6000:40e1:b0:487:27f6:a4cd with SMTP id ffacd0b85a97d-488716a5fb3mr18846694f8f.29.1790495199749; Sun, 27 Sep 2026 00:46:39 -0700 (PDT) Received: from FRSMI25-LASER.home (2a01cb001331aa00a2e4fb7b0d887544.ipv6.abo.wanadoo.fr. [2a01:cb00:1331:aa00:a2e4:fb7b:d88:7544]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4887a30bcdbsm18485266f8f.2.2026.09.27.00.46.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Sep 2026 00:46:39 -0700 (PDT) From: Yoann Congal To: bitbake-devel@lists.openembedded.org Subject: [bitbake][wrynose][2.18][PATCH 4/6] fetch/wget: Handle long filenames correctly Date: Sun, 27 Sep 2026 09:46:33 +0200 Message-ID: <65f17fdeeebfc0213cf2c809d08f7c4129fbe355.1790495161.git.yoann.congal@smile.fr> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 27 Sep 2026 07:46:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/20263 From: Richard Purdie Long filenames, which may happen through sstate, were being handled incorrectly as they could exceed max path lengths when '.tmp' was added. Handle those cases by using slightly shorter filenames if that happens for the intermediate tmp file. [YOCTO #16293] Signed-off-by: Richard Purdie Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie (cherry picked from commit da1fcdf15c647a2a718ae921e261a76e451b442a) Signed-off-by: Yoann Congal --- lib/bb/fetch2/wget.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index c292dfe7e..58f1a86e7 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -105,7 +105,10 @@ class Wget(FetchMethod): fetchcmd = self.basecmd dldir = os.path.realpath(d.getVar("DL_DIR")) - localpath = os.path.join(dldir, ud.localfile) + ".tmp" + # Where we ultimately want the file + finalpath = os.path.join(dldir, ud.localfile) + # A temp location while processing, keeping in mind max path lengths + localpath = finalpath[:250] + ".tmp" bb.utils.mkdirhier(os.path.dirname(localpath)) fetchcmd += " --output-document=%s" % shlex.quote(localpath) @@ -142,7 +145,7 @@ class Wget(FetchMethod): # Remove the ".tmp" and move the file into position atomically # Our lock prevents multiple writers but mirroring code may grab incomplete files - os.rename(localpath, localpath[:-4]) + os.rename(localpath, finalpath) return True From patchwork Sun Sep 27 07:46:34 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yoann Congal X-Patchwork-Id: 99284 X-Patchwork-Delegate: yoann.congal@smile.fr Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F1F0C9833F for ; Sun, 27 Sep 2026 07:46:58 +0000 (UTC) Received: from mail-wr2-f12.google.com (mail-wr2-f12.google.com [74.125.225.76]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.32882.1790495202344351560 for ; Sun, 27 Sep 2026 00:46:42 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@smile.fr header.s=google header.b=Xp9Ink0d; spf=pass (domain: smile.fr, ip: 74.125.225.76, mailfrom: yoann.congal@smile.fr) Received: by mail-wr2-f12.google.com with SMTP id ffacd0b85a97d-482f6351831so1149553f8f.1 for ; Sun, 27 Sep 2026 00:46:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smile.fr; s=google; t=1790495200; x=1791100000; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=gsk/KUe37wis/3Q8ZYT1cBkKFWsg+vh4xy4sd7h1+kI=; b=Xp9Ink0d/ClGSndJhceK6h3/l8WlYWlk899fU2rWaaZv4D+AhWV/FnqgXB1NrrHzND iAgOCmQFg+3jroUIGc53aIU0pk9NfKuScevuQThqo3bdZzVo2x5Q2cFcG/VFmC+s4I4M 0meW77B6cpx1hunIlY5znHremEPapd3MulmpQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790495200; x=1791100000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=gsk/KUe37wis/3Q8ZYT1cBkKFWsg+vh4xy4sd7h1+kI=; b=tPsAuPmGAUtmqsRP2BhAv4hwl2umT5MosF2JdhMjZ6On+uW2Z+M3kxhH+W/2TGhEya hF4DxXPuJ0wIhAlUT9eewGESLqP+OgZWmRHoOWfqE6H9Y+XL6YMQTIK6BDXfOveG2cV2 imvlS61ockaOT5chYzfYxd9Q+IML0nsJWt1Dyi5yj75v1cTzlOj/QDRi1T3uYYb+efLE DkgwAA9WLd/P73Nb6XR1dXez4+OXsXbJwlgwmF+1zwMcLuILTX5csh0GV9XGHA6KCeYI pKmhk5PImt8Sh4Ic8mAvACHu85HuwRiCo1uqCDYqgGrbDiuVdTTK4teU8RRGXc7ERrSp N4pw== X-Gm-Message-State: AFq9FYLHAWJohuLeWP46TkYfJCk/Rk6JocCyEiSpbjS8vFy8InXXFP26 uMDsYmQJb1L7Ghydc0kOzKhPklIgqk1Mng2VWLsnfZUaq71D7Hm7hVwtIsmZmygKv3bcdG3Uemg XjR1Dt84= X-Gm-Gg: AYBFou0KX/yybrKENadsf0FVQ7PNEEu1mBoLV0zHxi4dJn4JFdmy04a4tdUgwXv8Y/d qDJsDxXeJkyn7as3ZZNzAWnLzByqFDcZEtsWnmwjgT/NmnJcynwj9TyCZMRMNwTpjLFkGcVtfml JO5u72toaoLTG6lh3YFOvW97ticCcRINlCYUG97hHI+5gViadiQaRXzlBzc1AHf2vPjoo3uaFcd Rzx+3L3HnfhWqvnpRuIFZCilaKLu7X+GK60NB3DYFqMyzXj8qE2IlO7J3i8JjdBHcnQ7g+v3igt aYCKpYKNamsYPoo9oLe9VI0Uxo3WEbYl7UhLkFoaWfrJevxgRJk90+lxhjJnZMAn5zJFmX8pj/t xcqqj6qGyF3cJ7CaLWIKLXu7NDCPJi75a6Iz9tVog6v4Hceij3HWh7u9/Y9BEue8q0aPpWLQIe8 rKyQH0V/Q2/JMhKZsCP+khCjAcbl6F5A7pw3nxAt81YMuzwigEFNjAsi5XCNXu9Z2GY8WLlaMFo gHfFjWb5bmgNqNR8iqzctB7kZlPxMLR6lSyMSYpu1fmMi1GibnKy3jrDo4Gu/NzpffP0yUlNA== X-Received: by 2002:a05:6000:2888:b0:488:8948:2187 with SMTP id ffacd0b85a97d-48889482da9mr5279506f8f.16.1790495200574; Sun, 27 Sep 2026 00:46:40 -0700 (PDT) Received: from FRSMI25-LASER.home (2a01cb001331aa00a2e4fb7b0d887544.ipv6.abo.wanadoo.fr. [2a01:cb00:1331:aa00:a2e4:fb7b:d88:7544]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4887a30bcdbsm18485266f8f.2.2026.09.27.00.46.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Sep 2026 00:46:39 -0700 (PDT) From: Yoann Congal To: bitbake-devel@lists.openembedded.org Subject: [bitbake][wrynose][2.18][PATCH 5/6] data: Fix expandKeys recursion issue Date: Sun, 27 Sep 2026 09:46:34 +0200 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 27 Sep 2026 07:46:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/20264 From: Richard Purdie Using an variable expansion in override such as: RDEPENDS:${PN}:append:pn-packagegroup-cross-canadian-${MACHINE} = ' XXX' wasn't functioning correctly. Looking through the code, renameVar does work correctly but it doesn't work if called multiple times on the sub components of the variable. expandKeys will already recurse variables so we need to ensure renameVar doesn't also do that as things break if we do it multiple times. Change the code to avoid recursion in renameVar which should also have a small performance improvement. Test case from Nguyen Minh Tien [YOCTO #14867] Signed-off-by: Richard Purdie Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie (cherry picked from commit a3c6a020da9a352d0474e85bc0591b4a174cb0c0) Signed-off-by: Yoann Congal --- lib/bb/data.py | 2 +- lib/bb/data_smart.py | 5 +++-- lib/bb/tests/data.py | 25 +++++++++++++++++++++++++ 3 files changed, 29 insertions(+), 3 deletions(-) diff --git a/lib/bb/data.py b/lib/bb/data.py index 7c01e6bf8..f81f44eac 100644 --- a/lib/bb/data.py +++ b/lib/bb/data.py @@ -94,7 +94,7 @@ def expandKeys(alterdata, readdata = None): val = alterdata.getVar(key, False) if val is not None: bb.warn("Variable key %s (%s) replaces original key %s (%s)." % (key, val, ekey, newval)) - alterdata.renameVar(key, ekey) + alterdata.renameVar(key, ekey, recurse=False) def inheritFromOS(d, savedenv, permitted): """Inherit variables from the initial environment.""" diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index 110dfa111..e26b0659a 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -661,7 +661,7 @@ class DataSmart(MutableMapping): def getVar(self, var, expand=True, noweakdefault=False, parsing=False): return self.getVarFlag(var, "_content", expand, noweakdefault, parsing) - def renameVar(self, key, newkey, **loginfo): + def renameVar(self, key, newkey, recurse=True, **loginfo): """ Rename the variable key to newkey """ @@ -693,7 +693,8 @@ class DataSmart(MutableMapping): self.overridedata[newkey] = [] for (v, o) in self.overridedata[key]: self.overridedata[newkey].append([v.replace(key, newkey), o]) - self.renameVar(v, v.replace(key, newkey)) + if recurse: + self.renameVar(v, v.replace(key, newkey)) if ':' in newkey and val is None: self._setvar_update_overrides(newkey, **loginfo) diff --git a/lib/bb/tests/data.py b/lib/bb/tests/data.py index a83c71e2c..491774747 100644 --- a/lib/bb/tests/data.py +++ b/lib/bb/tests/data.py @@ -406,6 +406,31 @@ class TestOverrides(unittest.TestCase): bb.data.expandKeys(self.d) self.assertEqual(self.d.getVar("VERSION"), "2") + # Test an :append whose override name is only resolved by key expansion + def test_append_in_expanded_override(self): + self.d.setVar("MACHINE", "qemux86") + self.d.setVar("PN", "gizmo") + self.d.setVar("OVERRIDES", "gizmo:pn-gizmo-qemux86") + self.d.setVar("TEST:${PN}", "base") + self.d.setVar("TEST:${PN}:append:pn-gizmo-${MACHINE}", " appended") + bb.data.expandKeys(self.d) + self.assertEqual(self.d.getVar("TEST"), "base appended") + + # Test renaming to a key which is itself not expanded yet, as native.bbclass + # does. The dependent override keys must be left for expandKeys() to rename. + def test_rename_to_unexpanded_key_with_override(self): + self.d.setVar("BPN", "gizmo") + self.d.setVar("PN", "gizmo-native") + self.d.setVar("OVERRIDES", "class-target") + self.d.setVar("TEST:${PN}-lib", "base") + self.d.setVar("TEST:${PN}-lib:class-target", "target") + with LogRecord() as logs: + self.d.renameVar("TEST:${PN}-lib", "TEST:${BPN}-lib-native") + bb.data.expandKeys(self.d) + self.assertFalse(logContains("renameVar with equivalent keys", logs)) + self.assertFalse(logContains("replaces original key", logs)) + self.assertEqual(self.d.getVar("TEST:gizmo-lib-native"), "target") + def test_remove_with_override(self): self.d.setVar("TEST:bar", "testvalue2") self.d.setVar("TEST:some_val", "testvalue3 testvalue5") From patchwork Sun Sep 27 07:46:35 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yoann Congal X-Patchwork-Id: 99281 X-Patchwork-Delegate: yoann.congal@smile.fr Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59E86C98324 for ; Sun, 27 Sep 2026 07:46:58 +0000 (UTC) Received: from mail-wm2-f12.google.com (mail-wm2-f12.google.com [74.125.225.140]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.32883.1790495202929987093 for ; Sun, 27 Sep 2026 00:46:43 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@smile.fr header.s=google header.b=XTXpsOIJ; spf=pass (domain: smile.fr, ip: 74.125.225.140, mailfrom: yoann.congal@smile.fr) Received: by mail-wm2-f12.google.com with SMTP id 5b1f17b1804b1-49ff9621c5dso6238685e9.0 for ; Sun, 27 Sep 2026 00:46:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smile.fr; s=google; t=1790495201; x=1791100001; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=WdIFVnZKxNPRQYye1Kj6tfTMTEp+O6oS1NBwhahoxJY=; b=XTXpsOIJhOceEkZjUugXP5XLPiDZ7p6Q3WZ4ltIrXxfSKUxT80KuGe5gGqvIVvuACP zo97mcOBrBcXYw0Xuw8d0+EPy0pXzSk9UvDR5SxF+nDmMjbonzRecXB5APF/WYypfJ3V TzGWmfTxj2Te8aoxiEiZZ/2xCwlXbhac3oRR8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790495201; x=1791100001; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=WdIFVnZKxNPRQYye1Kj6tfTMTEp+O6oS1NBwhahoxJY=; b=Ahy66dCOGYYDP6ubI4l5zb1bAB2RzWghu6K5XOnrebK1ioifP4ow6jzYI5nL95sN/X r/CCZAe/BzoZ+Syw27bA6vRizATBbA4B8WwQoXUHQm5i9mA6v8BxQ0FKrZBfvSsm2rnU XvsloMhHe41ZuEyL7cgnO5hjukGzRUEt2eNJ326M7UWFaXC3wzaFJLFfP+mGmvClC8vi zhkL1E3A3ANVcwlMJO8vsveKD33HvTZfIj4+9rokHd+mx5Jmx6NFCgtasB97ZK5PS5DP AQkDFNv8qTDUtRDzNLH5nW89fuRxoIepaO/kgrRq1SdvzT9c5eYGw15F4ypN2jOCYJVt 3teQ== X-Gm-Message-State: AFuF++luTUNvX+gqPtfv7nC2MJQF/oiKTsdBUfYwxjLCF1JOpFfW1KxP tJSwFlZ+KlOGBroubvCdvSs+j803Q6cSgT5mwmjvgdq1ah4SjrLFfGhtH7VMAe+YXidBXntonLD 97mgy738= X-Gm-Gg: AYBFou2J9J7aCyTmuWyEm0GCPjoiCF/fwjPep51toZfGo2SBbaBBBb5CefLjxyCPcc9 5DQZ3OIO6UfXCRAvfKustWmJ8WStFAf2qJ3CYLeFka5zeBVGUc38t79xRkZbYoX9uBAs07GAt9J pa1fXcEk6Lp/gj0sjV4wJ3mRIH4E43FV0f+07YK5tNBvUEYVhlcXGsEFBJz0meN06RUPAJ5lJjh iqrrMO0Nn5fBBLA2EkwytZspa7OD5dwoEmCDHfsl3Li87vEKhkhMDZ/yyFDhbbZcoVMmLmzXxOA aXCrrDBDKJkhCi58uAH6Hwdy0o22ESwsyZ4qyYaCkKR2r8w/sZMdq3dJ7N32ARsGf5wtwt4jbSU XxmhVyjgRh9imvpMcdzlEhnD8wPyYu0hTXRnHTNrI2b0KyyU/hp3TaR1n1GsGonK9uqnFJpFxcW HTAqMga/tCsOmTD/kyl8Ut3adsdI7Mi+nclBld8GMzHFMFJBzWIgI3BaEDRUK66UDPKQXKwgfMB Pap/WrqIOyidWXZPrwNeklGrK0FaQhdj12zX/V1YZR8cSE+PBFfizlK1i0BnxQHViWlnEhsvw== X-Received: by 2002:a05:600d:4445:10b0:4a0:37b:5c25 with SMTP id 5b1f17b1804b1-4a0037b5ccbmr1046095e9.11.1790495201111; Sun, 27 Sep 2026 00:46:41 -0700 (PDT) Received: from FRSMI25-LASER.home (2a01cb001331aa00a2e4fb7b0d887544.ipv6.abo.wanadoo.fr. [2a01:cb00:1331:aa00:a2e4:fb7b:d88:7544]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4887a30bcdbsm18485266f8f.2.2026.09.27.00.46.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Sep 2026 00:46:40 -0700 (PDT) From: Yoann Congal To: bitbake-devel@lists.openembedded.org Subject: [bitbake][wrynose][2.18][PATCH 6/6] asyncrpc: Close the client event loop when the client is collected Date: Sun, 27 Sep 2026 09:46:35 +0200 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 27 Sep 2026 07:46:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/20265 From: Francisco Pedraza Client creates its own event loop and installs it with asyncio.set_event_loop(), so a client dropped without close() being called keeps the loop, and any transport that loop owns, alive. The loop only becomes collectable once a later client replaces the global reference, and asyncio then reports the transport and the loop as unclosed: ResourceWarning: unclosed transport ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False> This is why the warning shows up at an unrelated later point rather than where the client was leaked. SignatureGeneratorUniHashMixIn caches its client and only closes it from reset() and exit(), so any teardown path missing those leaks it. Register a weakref.finalize() to close the loop when the client is collected. close() detaches the finalizer first, so the explicit path does the same work as before. Shutdown errors are logged rather than raised, since an exception from the finalizer would otherwise be discarded by the interpreter and reported only as "Exception ignored in". A peer that has gone away or stopped responding is expected for a client that was never closed and is logged at debug; anything else, including a loop that cannot be run, is logged at warning. The exceptions raised by websockets fall into the latter case as they cannot be named here, websockets being imported lazily in connect_websocket(). The loop is closed in every case. Reproduced with two clients where the first is never closed, on python 3.10 with websockets 10.4 and on python 3.14 with websockets 17.0.1. Both report the warnings without this change and neither does with it. Each handled exception type was checked to not escape the finalizer, to still close the loop and to log at the intended level. bitbake-selftest hashserv.tests passes. Fixes [YOCTO #16236] Reviewed-by: Joshua Watt Signed-off-by: Francisco Pedraza Signed-off-by: Richard Purdie (cherry picked from commit 7a3cb0d55e3698e883a9ff8491febc9d155482ca) Signed-off-by: Yoann Congal --- lib/bb/asyncrpc/client.py | 50 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 3 deletions(-) diff --git a/lib/bb/asyncrpc/client.py b/lib/bb/asyncrpc/client.py index 17b72033b..dd28fa730 100644 --- a/lib/bb/asyncrpc/client.py +++ b/lib/bb/asyncrpc/client.py @@ -7,15 +7,19 @@ import abc import asyncio import json +import logging import os import socket import sys import re import contextlib +import weakref from threading import Thread from .connection import StreamConnection, WebsocketConnection, DEFAULT_MAX_CHUNK from .exceptions import ConnectionClosedError, InvokeError +logger = logging.getLogger("bb.asyncrpc.client") + UNIX_PREFIX = "unix://" WS_PREFIX = "ws://" WSS_PREFIX = "wss://" @@ -224,8 +228,49 @@ class Client(object): # required (but harmless) with it. asyncio.set_event_loop(self.loop) + # The loop and its transports must be closed even if the caller never + # calls close(). Since set_event_loop() above replaces the reference + # held by the previous client, an unclosed loop only becomes reachable + # for collection once another client is created, at which point + # asyncio reports "unclosed transport" and "unclosed event loop" + # ResourceWarnings. The finalizer is detached by close() so that the + # normal path is unaffected. + self._finalizer = weakref.finalize( + self, self._close_loop, self.loop, self.client + ) + self._add_methods("connect_tcp", "ping") + @staticmethod + def _close_loop(loop, client): + if loop.is_closed(): + return + try: + loop.run_until_complete(client.close()) + loop.run_until_complete(loop.shutdown_asyncgens()) + # This can be called from the finalizer, where an exception would be + # discarded by the interpreter and reported only as "Exception ignored + # in", so report the error here instead. The loop is closed below in + # every case. + except (OSError, ConnectionClosedError, asyncio.TimeoutError) as exc: + # The peer has gone away or is not responding. That is expected + # for a client that was never closed, so it is not worth warning + # about. asyncio.TimeoutError is only distinct from OSError on + # python older than 3.11. + logger.debug("Client connection already closed or unreachable: %s" % exc) + except RuntimeError as exc: + # The loop could not be run, for example because another loop is + # already running in this thread. Unlike the above, this means + # something is wrong with how the client is being used. + logger.warning("Could not shut down client event loop: %s" % exc) + except Exception as exc: + # Also covers the exceptions raised by websockets, which cannot be + # named here as it is imported lazily in connect_websocket(), and + # whatever may be raised while the interpreter is shutting down. + logger.warning("Error shutting down client connection: %s" % exc) + finally: + loop.close() + @abc.abstractmethod def _get_async_client(self): pass @@ -258,9 +303,8 @@ class Client(object): def close(self): if self.loop: - self.loop.run_until_complete(self.client.close()) - self.loop.run_until_complete(self.loop.shutdown_asyncgens()) - self.loop.close() + self._finalizer.detach() + self._close_loop(self.loop, self.client) self.loop = None def __enter__(self):