From patchwork Mon Feb 20 22:25:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2CA9C6379F for ; Mon, 20 Feb 2023 22:25:42 +0000 (UTC) Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by mx.groups.io with SMTP id smtpd.web10.27824.1676931937536060404 for ; Mon, 20 Feb 2023 14:25:37 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=4CJUN95D; spf=softfail (domain: sakoman.com, ip: 209.85.210.170, mailfrom: steve@sakoman.com) Received: by mail-pf1-f170.google.com with SMTP id n5so1497511pfv.11 for ; Mon, 20 Feb 2023 14:25:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=5fjzvVWeAMUnjKW+J6nRaSXjDYfvhjvJjztLM6Z2bgY=; b=4CJUN95Du1Wrm/MriZqxI08rgm7GUtAECxEBXGp532Hj4iCxg4M9iKTa8h59hWyF8G Vm5csYU+aBGjYksw8hHpdSShPr+zRyCWSG2KJkxBs1YukYJptpbQes3dtntL+zswoAvD jtfO3kR7pLHsIOHIOjhinOCn+/0n86J8+svLeJ6+z3YgfP97hSXBpPC3Ntv9vAocpYcy +Yt1MhZByOC9WEjZO1g5XbFrHd5IS93hCR5kEnYVHGDRt1WEZ1zpapyHabIUCsGuI0rW cL14gNPJDscUaMqykqK0OBw3wvPVZoXMgQHP4MTCuXtsFhSFZ3poWqoyS6lvm8R/4qgx AdFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5fjzvVWeAMUnjKW+J6nRaSXjDYfvhjvJjztLM6Z2bgY=; b=hdjv9FJTGoR+77NNs/2ivo9h9e7McQnARt36h+fm/hQgBJmolLxb4TbtXcXwLk9SsZ C3ga7gOfu77bcR14ClI44Eczkw4kZgwkj3SGrWasxQ8TxpD5b9SgjFnf4KyltZQkMep6 DCDNcfoltQ/Ib/8z8xzNRmycD/YIAZ6NNIxTiCcLrFZ4/RS1ccnEm8wfcmPvx41KWtEe IkVodBBJnP+lQS2V+8oKQOq1Q+Q2ac5fKfzbtgwI9fy78PKIVX90WCjXx7Gj8nHkKsQJ iqgYe5nCF6GS3kaQice7paNleoRrPdXtQlFatQvelIMgEW64L+mzDlrINbNn/dP2JMA1 ti2g== X-Gm-Message-State: AO0yUKVKRjFsMI5NXaaZ+lZQDWEv3cf/YY36o6V6tSpeHQZNRhJTPogF ZRR3G6gc7ZLWPlKQF14AniqXH5uAiqXojHdOa3s= X-Google-Smtp-Source: AK7set/tamQwT0zV/dZkYGFiKXfkCuZzKfTVGRShL7hS/2O3XZgw6uGQwX0x0L2EaTjUeZwjMg9IRA== X-Received: by 2002:a62:1ad1:0:b0:5a9:c772:85ab with SMTP id a200-20020a621ad1000000b005a9c77285abmr3685401pfa.22.1676931936582; Mon, 20 Feb 2023 14:25:36 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id q21-20020a62ae15000000b005a9131b6668sm8191447pff.2.2023.02.20.14.25.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 14:25:36 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 1/7] runqueue: Fix multiconfig deferred task sstate validity caching issue Date: Mon, 20 Feb 2023 12:25:17 -1000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 20 Feb 2023 22:25:42 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14468 From: Richard Purdie We were testing the validity of deferred tasks setscene status "up front" which is very unlikely to succeed and leads to cache invalidation issues. With the change to rebuild the deferred task list, this status becomes out of sync. The result was tasks being executed when they should not have been leading to extra work for the build unnecessarily. Instead, don't process validity status for deferred tasks and assume their data will become available. If it doesn't, this will now result in a build error as the setscene task will fail and the main task will run instead. In theory we could try and track the state changes in the deferred list and re-test validity then but I'm not sure it is worth the effort when the other code path and errors in setscene tasks will give a pretty good idea of what is happening anyway. Signed-off-by: Richard Purdie (cherry picked from commit edcafac13b3b241b6687419e59018d21811507a1) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman --- lib/bb/runqueue.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 6cdc72a8..f82bc413 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2084,8 +2084,6 @@ class RunQueueExecute: logger.debug(1, "%s didn't become valid, skipping setscene" % nexttask) self.sq_task_failoutright(nexttask) return True - else: - self.sqdata.outrightfail.remove(nexttask) if nexttask in self.sqdata.outrightfail: logger.debug(2, 'No package found, so skipping setscene task %s', nexttask) self.sq_task_failoutright(nexttask) @@ -2843,6 +2841,8 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s sqdata.stamppresent.remove(tid) if tid in sqdata.valid: sqdata.valid.remove(tid) + if tid in sqdata.outrightfail: + sqdata.outrightfail.remove(tid) (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) @@ -2887,10 +2887,10 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s if tid in sqrq.scenequeue_covered: continue - sqdata.outrightfail.add(tid) - h = pending_hash_index(tid, rqdata) if h not in sqdata.hashes: + if tid in tids: + sqdata.outrightfail.add(tid) sqdata.hashes[h] = tid else: sqrq.sq_deferred[tid] = sqdata.hashes[h] From patchwork Mon Feb 20 22:25:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9760C677F1 for ; Mon, 20 Feb 2023 22:25:42 +0000 (UTC) Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by mx.groups.io with SMTP id smtpd.web11.28208.1676931939379517781 for ; Mon, 20 Feb 2023 14:25:39 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=GBrXH6BL; spf=softfail (domain: sakoman.com, ip: 209.85.210.172, mailfrom: steve@sakoman.com) Received: by mail-pf1-f172.google.com with SMTP id a26so1375014pfo.9 for ; Mon, 20 Feb 2023 14:25:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=5GLRgrgwcku4N1eh9Zl9lZLRl/6qnueRenx8JfDME5U=; b=GBrXH6BLsk9efUEFtv+DFjtxdaBzwEBmZ/qUz2E2M+92mYvLa6S5GoiHI/j8hCeHmd WpwtuWrWdbm5T3Bu+A73DP272UrMqrxY53kxgERuV2Fg9bjchQL8Wsw5EfiHcv53Eeld TuYTp4FgkmV1G/Au1bGFcI7mW8uU+LUnctHjs0UJqsjDHZca2R/eR811bznEzzHSqh+4 upBb52Ukz20QVOlHwp0ZkcHZwgyjg7WECILqgIGmYAkDH6G8yMkmgZXTPyFw3ZB1uQdx JS4neQDRDvo/4UeKaCphUg6CD+YdZQg4PoROhRHdRai8D4OmMsxoTUQf5nyVLTNhTqmX hcMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5GLRgrgwcku4N1eh9Zl9lZLRl/6qnueRenx8JfDME5U=; b=7B2zg6P8V7W0RdztA10U2kPe4lSJR+6XA1OBodcGJGGmoWguWx86YiBH32H9PweMki ACMDWBv0Ox16SXY1TuaGesT2CIyLHnWHsNasomm26UX+xGywuhKdrkjB3fz67bnq6MMa LRHGWJpQSARDBMcOHFqAn8DD+HPSjSMe+SrxDqfLAQryFoV9ZSbyiyyOui8GERImOUb2 1w9Ass2EqBs43CRNbemIUGRuTAmH6m4ZYeEqDCoVMUjMQ7D9Z2Ip84ncBC4sy/DxyoKe bUWmi46gXZL9dDJ0B3TYgdt5gE3S8tWia58hSUcwz6avrU5QKjCe/DxRlixvmu1c6q5g sfyw== X-Gm-Message-State: AO0yUKXb2LZJsqlnz7yOxPimA40++UUlkQD9ZH++P/Lg/hHkpapc8CnR QTYLfLPrLVwU8qzIP05V212bBzMfULUc4Dh4ZPg= X-Google-Smtp-Source: AK7set/qnEPe34aYgQzOzUhUZBZR9K4zsMFkWCIZHuyumkkwlqYKe9VgpeVMwVKcEYm5Adk9Z1DuWg== X-Received: by 2002:a62:19c5:0:b0:592:705a:5a5e with SMTP id 188-20020a6219c5000000b00592705a5a5emr3076313pfz.9.1676931938435; Mon, 20 Feb 2023 14:25:38 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id q21-20020a62ae15000000b005a9131b6668sm8191447pff.2.2023.02.20.14.25.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 14:25:38 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 2/7] runqueue: Handle deferred task rehashing in multiconfig builds Date: Mon, 20 Feb 2023 12:25:18 -1000 Message-Id: <01936b4dd8e680f1f8035ff2d6231673f61efeab.1676931822.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 20 Feb 2023 22:25:42 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14469 From: Richard Purdie If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig build), we need to ensure that the hash change propagates through to all the tasks else the build will run multiple copies of the task, sometimes with oddly differing results as the outhashes of native tasks built in differing locations can confuse things. Signed-off-by: Richard Purdie (cherry picked from commit 2db571324f755edc4981deecbcfdf0aaa5a97627) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman --- lib/bb/runqueue.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index f82bc413..a437aeef 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2348,10 +2348,16 @@ class RunQueueExecute: self.updated_taskhash_queue.remove((tid, unihash)) if unihash != self.rqdata.runtaskentries[tid].unihash: - hashequiv_logger.verbose("Task %s unihash changed to %s" % (tid, unihash)) - self.rqdata.runtaskentries[tid].unihash = unihash - bb.parse.siggen.set_unihash(tid, unihash) - toprocess.add(tid) + # Make sure we rehash any other tasks with the same task hash that we're deferred against. + torehash = [tid] + for deftid in self.sq_deferred: + if self.sq_deferred[deftid] == tid: + torehash.append(deftid) + for hashtid in torehash: + hashequiv_logger.verbose("Task %s unihash changed to %s" % (hashtid, unihash)) + self.rqdata.runtaskentries[hashtid].unihash = unihash + bb.parse.siggen.set_unihash(hashtid, unihash) + toprocess.add(hashtid) # Work out all tasks which depend upon these total = set() From patchwork Mon Feb 20 22:25:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C18FDC61DA3 for ; Mon, 20 Feb 2023 22:25:42 +0000 (UTC) Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by mx.groups.io with SMTP id smtpd.web11.28210.1676931941392833706 for ; Mon, 20 Feb 2023 14:25:41 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=qfDOkUep; spf=softfail (domain: sakoman.com, ip: 209.85.210.178, mailfrom: steve@sakoman.com) Received: by mail-pf1-f178.google.com with SMTP id a7so1495889pfx.10 for ; Mon, 20 Feb 2023 14:25:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=HNokp74Z/PaUvtcKBm8xvcXLVcgZ5ABJeQ8g4YVb9xg=; b=qfDOkUepDG6KOGnuvucl+Pzy9B+PdGCBmPVrgtg6ri0CZhC2Wi0M9midf67p+WXHPD HTCivrM5Cb6XcHaXA/XXiObL/HjNJB/AS4xJsBmYzo3r02gAQQ9kS2MlUyF2pK5s0VEl qvWMhZOkbcvwof9zg8X5T3vTp3hzSWvRExoECHkY7zVGfS8xBSiabtAOTD7T+EXb3t+2 MtkJ1YorCj5+CBiY/QwMShOW1TIXkTtXUaa8o1trITWtUfAb5TPvNBJ/v9Uqdg4INGCi PPX4ZWATkXRPIC7LDx3qE9fJQei2wHripXRER1ixAN2k6wNhHr6sS6d1/dzfkdKdzMcV xKDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HNokp74Z/PaUvtcKBm8xvcXLVcgZ5ABJeQ8g4YVb9xg=; b=gTlrX9RQPjDcrjfsPbKKzElcv0WxWxFmlOQ48LFXbPlUIbRPaOrkApDghHuTaLoizv 0OxHLnVa1nFIInkZUPWkbDHemoQ0QwRQd6CJyXgwiX4/f5txvx0Aivnmso3CV1hlUDok sc4No22x65kwhh1B4z4R4e8P9c8yTeuJvTsJ+Msx02fMEgg1RSb1m8USqyyr6Fj4Oiyp XpGB7T5IIlWFEzF4V8zIugvyve27zDoT4cbijuXTTyhF7kiJIk31nljqr+T/sT4nyBZ7 8p51OZslZcu54uX8DbxuJdwsz7tR4p3epBEhZ3zO66aG6smfbEWtObAO/AWy07C0gyUj MiVg== X-Gm-Message-State: AO0yUKWZGk8xt5bkwH1gtCv/AgOusQrF9Hit4QQ8h09XpJZTciXWSmLl BdxaM9WUwFmS/8i+47HzQwKUfv7K0NQmYB9Y39E= X-Google-Smtp-Source: AK7set8yVtvFvVtMURBbvfQJXHulsc4Xs79WTCJIQxsAmBgUhcFB5aS3E3bB4gMkKQoIuRYJBPgQYA== X-Received: by 2002:a62:1dd0:0:b0:5aa:6125:dbf4 with SMTP id d199-20020a621dd0000000b005aa6125dbf4mr1634678pfd.11.1676931940298; Mon, 20 Feb 2023 14:25:40 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id q21-20020a62ae15000000b005a9131b6668sm8191447pff.2.2023.02.20.14.25.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 14:25:39 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 3/7] runqueue: Improve multiconfig deferred task issues Date: Mon, 20 Feb 2023 12:25:19 -1000 Message-Id: <4316e9f60ce5fd250a16586a1772dcc0adfeb932.1676931822.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 20 Feb 2023 22:25:42 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14470 From: Richard Purdie The previous patches have exposed new issues with this code path, the issues being around what should happen when the hash of a task changes and the task is or is not on the deferred task list. Rather than rebuilding the deferred task list during each rehash event, build it once at the start of a build. This avoids the problem of tasks being added back after they have run and also avoids problems of always ensuring the same task is deferred. It also allows the 'outrightfail' codepath to be handled separately as the conditions are subtly differnt. One significant win for the new approch is the build is not continually printing out lists of deferred tasks, that list remains fairly static from the start of the build. Logic is added in to ensure a rehashed task with a hash matching other deferred tasks is deferred along with them as a small optimization. An interesting test case for this code was reported by Mark Hatle with four multiconfigs, each the same apart from TMPDIR and running a build of: bitbake buildtools-tarball mc:{one,two,three,four}:core-image-minimal which is interesting in that the build of buildtools partially overlaps core-image-minimal and the build has a rehash event for qemuwrapper-cross even without any external hash equivalence server or preexisting data. Signed-off-by: Richard Purdie (cherry picked from commit bb424e0a6d274d398f434f7df63951da9ce305b3) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman --- lib/bb/runqueue.py | 58 +++++++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 26 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index a437aeef..9efd1c86 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2496,6 +2496,11 @@ class RunQueueExecute: if update_tasks: self.sqdone = False + for tid in [t[0] for t in update_tasks]: + h = pending_hash_index(tid, self.rqdata) + if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: + self.sq_deferred[tid] = self.sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) for (tid, harddepfail, origvalid) in update_tasks: @@ -2836,6 +2841,19 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq): sqdata.stamppresent = set() sqdata.valid = set() + sqdata.hashes = {} + sqrq.sq_deferred = {} + for mc in sorted(sqdata.multiconfigs): + for tid in sorted(sqdata.sq_revdeps): + if mc_from_tid(tid) != mc: + continue + h = pending_hash_index(tid, rqdata) + if h not in sqdata.hashes: + sqdata.hashes[h] = tid + else: + sqrq.sq_deferred[tid] = sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h])) + update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, stampcache, sqrq, summary=True) def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, summary=True): @@ -2876,32 +2894,20 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s sqdata.valid |= rq.validate_hashes(tocheck, cooker.data, len(sqdata.stamppresent), False, summary=summary) - sqdata.hashes = {} - sqrq.sq_deferred = {} - for mc in sorted(sqdata.multiconfigs): - for tid in sorted(sqdata.sq_revdeps): - if mc_from_tid(tid) != mc: - continue - if tid in sqdata.stamppresent: - continue - if tid in sqdata.valid: - continue - if tid in sqdata.noexec: - continue - if tid in sqrq.scenequeue_notcovered: - continue - if tid in sqrq.scenequeue_covered: - continue - - h = pending_hash_index(tid, rqdata) - if h not in sqdata.hashes: - if tid in tids: - sqdata.outrightfail.add(tid) - sqdata.hashes[h] = tid - else: - sqrq.sq_deferred[tid] = sqdata.hashes[h] - bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h])) - + for tid in tids: + if tid in sqdata.stamppresent: + continue + if tid in sqdata.valid: + continue + if tid in sqdata.noexec: + continue + if tid in sqrq.scenequeue_covered: + continue + if tid in sqrq.scenequeue_notcovered: + continue + if tid in sqrq.sq_deferred: + continue + sqdata.outrightfail.add(tid) class TaskFailure(Exception): """ From patchwork Mon Feb 20 22:25:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C00C6C61DA3 for ; Mon, 20 Feb 2023 22:25:52 +0000 (UTC) Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by mx.groups.io with SMTP id smtpd.web10.27829.1676931943167761918 for ; Mon, 20 Feb 2023 14:25:43 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=wUWEC30S; spf=softfail (domain: sakoman.com, ip: 209.85.214.182, mailfrom: steve@sakoman.com) Received: by mail-pl1-f182.google.com with SMTP id c1so3251719plg.4 for ; Mon, 20 Feb 2023 14:25:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Qrp62N4m/KJgUUl9vEYDys9t6gRDYAoGrkEo6jg7RO4=; b=wUWEC30S7jx9eO5ZCI8C8oLPkVa8BK8eS6CbteII/cp+ZLmr9R6luSLfwguW/5roWz lgN93wXvnXnU1k9edHl2qwLkGktFBlZr2NSp2qFQkY8mUi1xwW4XLsitp2Aw9vzPOFoC SK//0NQgBpBvv17atLV2lCRMYBiqiauDE3poz8KRtiuGpz8Six8edGMxGbmUrhQyQi5E STxFnraxTkHohaUKNForaMb5f17BOVpDoxYrFerSlf6mShL3Ky7UiPaD6lNGM7PLAPq7 ++nIHx0myGtwA/anfE1kSfkUUfOJ0DBL8MRMKkqb8KMN1M3uEG+u4Wfs0uoaA1bRUhjW 3fJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qrp62N4m/KJgUUl9vEYDys9t6gRDYAoGrkEo6jg7RO4=; b=4UP0LGMvg1WRpgMl3qEM9wbUecsDKT+pTS2lEOX7skfBd7fIKqAC2ypzB31db5EY1O 7dNjx3v7za6Yunx+4Eo88oYr4GdDYlpqr1eR/kOPtonisdOOvdQQstxFae7Oz0z1wj7Z tKOCOzpJxes2Lng3nilOOmiaStbYn+iHScCIitXdlNxK6L7Wki4UHUjlULZaI0DtoypP DHCvviLMvBZuftiMDcW3F5kN6lxGi7qINZ3y54deg5I4a3SXbgrsxYhqHwXgt8ZpqTsx kFoo3Fg2ez6fMOE45ziEpfta+r19UMPtZUz+8hve/qrXu3dZu3ruyIkknnYbXsSwb6/Q fjmg== X-Gm-Message-State: AO0yUKUqjf5XDqhcY8qpaTy6yWkQcQVIfdXnAQXyMUy86P23Jh1NVx+A ZZ0tFJkiy5LrXs62RRWZ4E7V89OvAD5/nX5ySqM= X-Google-Smtp-Source: AK7set/atCgO91EKUhmvMrGqAlbDzyVvkn40LjJCLDWwMw4oyj7UwkQ9m1Wzx1cVLnPodo9bxNtclA== X-Received: by 2002:a05:6a20:8e23:b0:c7:26d0:1138 with SMTP id y35-20020a056a208e2300b000c726d01138mr3401095pzj.39.1676931942124; Mon, 20 Feb 2023 14:25:42 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id q21-20020a62ae15000000b005a9131b6668sm8191447pff.2.2023.02.20.14.25.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 14:25:41 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 4/7] runqueue: Avoid deadlock avoidance task graph corruption Date: Mon, 20 Feb 2023 12:25:20 -1000 Message-Id: <2e1354525217505ce34fe775ee6ec8af46ff5324.1676931822.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 20 Feb 2023 22:25:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14471 From: Richard Purdie If the deferred task deadlock avoidance code triggers, it could mark an executed task as failed which leads to "covered and not covered" error messages. Improve the logic so if the deadlock code is triggered, it doesn't cause the errors. Signed-off-by: Richard Purdie (cherry picked from commit 51bdd6cb3bd9e2c02e261fb578bb945b86b82c75) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman --- lib/bb/runqueue.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 9efd1c86..c7f50fde 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2234,7 +2234,8 @@ class RunQueueExecute: if self.sq_deferred: tid = self.sq_deferred.pop(list(self.sq_deferred.keys())[0]) logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s" % tid) - self.sq_task_failoutright(tid) + if tid not in self.runq_complete: + self.sq_task_failoutright(tid) return True if len(self.failed_tids) != 0: From patchwork Mon Feb 20 22:25:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19886 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAB44C677F1 for ; Mon, 20 Feb 2023 22:25:52 +0000 (UTC) Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by mx.groups.io with SMTP id smtpd.web10.27832.1676931944997546095 for ; Mon, 20 Feb 2023 14:25:45 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=KrpIYPKs; spf=softfail (domain: sakoman.com, ip: 209.85.216.44, mailfrom: steve@sakoman.com) Received: by mail-pj1-f44.google.com with SMTP id cp9so3865640pjb.0 for ; Mon, 20 Feb 2023 14:25:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=BARB8zVTcsA1ibO49/oB9csZGNsfIQ0kMo7J9xwylbg=; b=KrpIYPKsN093n7ptIUBpnmI6vsgCtLeWNOiLc+W2AL1VeJgI0d7exWJ9np4fSLa39E 8xn+ErF0F8pdtMZDrIfGklTt3c52GuxBWsWoYXJePLbCETXoK+TUumgty4Pe+kQwF/Km iDQIAb37byu99Z57csrIqOZey2fLT2EG8hAZ2ixlqYTvbibYHAcZJocQzRke8tYQp2EX yUiL9Ju+DxAHp/SNM00QrdVo2qyzSNWA5gI0I82BQ6OQ4mWwzFH6vCtF5Ij+g29ucFl7 oIgUszCW7J20gKEJUGYnkY49HKx1kzfTpdWvSON//Eu3WMOsFhrTROu3FzjjZrMvvlKs YBLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BARB8zVTcsA1ibO49/oB9csZGNsfIQ0kMo7J9xwylbg=; b=WEdV9xfJXBoSR+o07e2hF6HoRDUad1r8y/AUhGnUSAQr9urIkyCB2MJDtc9eo8LwP8 m6uLeldtXdhyfbPKISQ05tB7KfwbH0l07tSjxEXB8d6N/Rv9I2iJntP1QZ8jKCYv2bi7 m3wwTcE5sBoLjJgZcMGLsmjVDpSAaFee8EiIIINxNf4AjiDnmXAUBJV1Tnxlka8lIuzh ZG/5rEBsMG4DpDwEjBQZ/C3Vbzypb4iVfotZQWgz5pDM7uT3RRZl+P4zjw8d4yEXZwi/ FT6u9lfpeEXlrhJFSH2/8xtWXu7+QnaVCOVZ8vHs14J7Q+0GFkh6E/jj3E5PrVndMZbx i0tQ== X-Gm-Message-State: AO0yUKVTLY5AbHxMITukgmjU2wC0LJBqpfFIzVWov2ZzndDxP98oaOZz 4YBXZrEXtV5XnaQsCvdbgaetRKpdEFIQcur3FwU= X-Google-Smtp-Source: AK7set9a6AOXpZQl+z11LYnLHLd8TwdWawOlANdrrok2ckLUOB/ul/SO6u8hjg5GTuEg4B6kqt6JxQ== X-Received: by 2002:a05:6a21:789c:b0:c7:7afa:5016 with SMTP id bf28-20020a056a21789c00b000c77afa5016mr16985109pzc.9.1676931943971; Mon, 20 Feb 2023 14:25:43 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id q21-20020a62ae15000000b005a9131b6668sm8191447pff.2.2023.02.20.14.25.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 14:25:43 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 5/7] runqueue: Fix issues with multiconfig deferred task deadlock messages Date: Mon, 20 Feb 2023 12:25:21 -1000 Message-Id: <03cf0d9fedfef1ae43b3c3cac07710487857af36.1676931822.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 20 Feb 2023 22:25:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14472 From: Richard Purdie In multiconfig builds with large numbers of identical tasks, builds were deadlocking after recent runqueue changes upon rebuilds where there was heavy sstate usage (i.e. on second builds after a first completed). The issue was that deferred tasks were being left indefinitely on the deferred list. The deadlock handler was then "breaking" things by failing tasks that had already succeeded, leading to the task being on both covered and not covered lists, giving a further error. The fix is to clean up the deferred task list when each setscene task completes. I'd previously been hoping to avoid iterating that list but it appears unavoidable. [YOCTO #14342] Signed-off-by: Richard Purdie (cherry picked from commit ae24a0f2d2d8b4b5ec10efabd0e9362e560832ea) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman --- lib/bb/runqueue.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index c7f50fde..f63a2191 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1975,6 +1975,12 @@ class RunQueueExecute: self.setbuildable(revdep) logger.debug(1, "Marking task %s as buildable", revdep) + for t in self.sq_deferred.copy(): + if self.sq_deferred[t] == task: + logger.debug(2, "Deferred task %s now buildable" % t) + del self.sq_deferred[t] + update_scenequeue_data([t], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) + def task_complete(self, task): self.stats.taskCompleted() bb.event.fire(runQueueTaskCompleted(task, self.stats, self.rq), self.cfgData) From patchwork Mon Feb 20 22:25:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C00F5C6379F for ; Mon, 20 Feb 2023 22:25:52 +0000 (UTC) Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by mx.groups.io with SMTP id smtpd.web10.27833.1676931946832947150 for ; Mon, 20 Feb 2023 14:25:46 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=uFbPYdm6; spf=softfail (domain: sakoman.com, ip: 209.85.210.182, mailfrom: steve@sakoman.com) Received: by mail-pf1-f182.google.com with SMTP id g12so1332658pfi.0 for ; Mon, 20 Feb 2023 14:25:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=748QVKVjh4beT3M0e9KWd8EDGteCuKv1wpKxSdgS41A=; b=uFbPYdm6ZvDWfFCGnpnda7XjxTv0JNDSHKVz5ykthtd7UdNdrS/RJ3B04lq012sEa9 OXfjPy8MBEwRFMIdRwJ+5y8ZDGcIWFrVQLY7z5sCFSbqjcHW67D0N8QMKSKRFkxxXasX DrCn0TYB0Zb697rUT/FBM9zVx43/tD5jd4pwmXCcoUj+sDyal42iBNEYSd8Uw6vhJ7BZ 4Ymw9kq6peO8Llwo6+JE07j52gDHG4p6CLLKK42xql0R5Jweg+sEQVnd4/qeNJLFp4Qm sz/Ct1fx4zdh0f0mEHDle5tAXO6nL5ypqoDQXbmB9fXfTlP/yEEBj2ooiDIzbPVA/pcX UGJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=748QVKVjh4beT3M0e9KWd8EDGteCuKv1wpKxSdgS41A=; b=XNi6w/KOHArIh9v7uvQSNg/PqwIu7pzjqXmQHgj6cuHSioh/f3TdDQPyoVHEsv1Djr 2n+4FEyyp5RsJRbFMFllluMDve3GpsjRyNLX6YivT71VVGw8+Zx2zninOOnPBhha8kfU tiyH32uFNX/z4+GBKxKgeg8lIGmVkNq4r+Un2Ex6fmTrVdyFaJPf//49kSQBUeNgh4Ku puDgjRhwWZZmx8U0Q8MgSM2cx3kKzv4iWkY+kKF+dCp2MvBp5ytO/ajySMB0Q1yL2ugJ tF3IcBfxSMWfM/on5g/ZzhDIQ4hSRpQOiM3rtbN1ZUuwDVRLSvGZDYnkgt+0Yuf36CZ+ UEog== X-Gm-Message-State: AO0yUKUn2QDdpGmfMvKj+mZyK4jt/0uXFhZHJbdrgVaq8Rg0n+IBvkAi uBBhQEx7/5Q8iOZcgTm09tsskkJgeMvL/mknOMY= X-Google-Smtp-Source: AK7set8hSSywINr9oElQvSaAghQpLaVivNEDsyf8gOs5cc8geJsqJQzNG82CoH4V2AzKOnQ/2TkSLw== X-Received: by 2002:aa7:9543:0:b0:5a9:c797:a1d6 with SMTP id w3-20020aa79543000000b005a9c797a1d6mr2853310pfq.2.1676931945890; Mon, 20 Feb 2023 14:25:45 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id q21-20020a62ae15000000b005a9131b6668sm8191447pff.2.2023.02.20.14.25.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 14:25:45 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 6/7] runqueue: Ensure deferred tasks are sorted by multiconfig Date: Mon, 20 Feb 2023 12:25:22 -1000 Message-Id: <657940c7c2a9dea4963a5063e4bf900d6b454903.1676931822.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 20 Feb 2023 22:25:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14473 From: Richard Purdie We have to prefer one multiconfig over another when deferring tasks, else we'll have cross-linked build trees and nothing will be able to build. In the original population code, we sort like this but we don't after rehashing. Ensure we have the same sorting after rehashing toa void deadlocks. Signed-off-by: Richard Purdie (cherry picked from commit 27228c7f026acb8ae9e1211d0486ffb7338123a2) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman --- lib/bb/runqueue.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index f63a2191..2a1299db 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2503,11 +2503,14 @@ class RunQueueExecute: if update_tasks: self.sqdone = False - for tid in [t[0] for t in update_tasks]: - h = pending_hash_index(tid, self.rqdata) - if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: - self.sq_deferred[tid] = self.sqdata.hashes[h] - bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) + for mc in sorted(self.sqdata.multiconfigs): + for tid in sorted([t[0] for t in update_tasks]): + if mc_from_tid(tid) != mc: + continue + h = pending_hash_index(tid, self.rqdata) + if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: + self.sq_deferred[tid] = self.sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) for (tid, harddepfail, origvalid) in update_tasks: From patchwork Mon Feb 20 22:25:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 19883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAAD2C64ED9 for ; Mon, 20 Feb 2023 22:25:52 +0000 (UTC) Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by mx.groups.io with SMTP id smtpd.web11.28213.1676931948757938964 for ; Mon, 20 Feb 2023 14:25:48 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=i4xPqkyE; spf=softfail (domain: sakoman.com, ip: 209.85.210.182, mailfrom: steve@sakoman.com) Received: by mail-pf1-f182.google.com with SMTP id e24so1461634pfn.3 for ; Mon, 20 Feb 2023 14:25:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=OhIh0KZ6HH76BJ2iq/xQLWythhPNDltmlzcL8CupPAA=; b=i4xPqkyEzk8iOrz6i9TNtgrYQWVUhM0JxhqN268w+3K9B/xDlPXbnvAWBixH0jE7NW vbDkjp0kBM6YegJydC8Tyo1FRMvlnA1YsLpEy5F5uxMTU976DNXWBOLfpCyhUFfr8XN6 au/NIk9sR6qg+vKoscwC7zl9qBpLLjjuyUhw7djOaujasSt4pwRt0N0S0mKpliOeyevT 5vBAcBAJub9/em2Hg3OTe44x9CX5slwJkNXHHMa/heca/N/s+b0pYNnUCg+ILu1q9+Yd N5YOuJuZYSFGqRMoQLP/T6KCxGQEfVYZi76ASo8Kan+RL7CCSjKcWnE+UyRW2pZ4nfof AZAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OhIh0KZ6HH76BJ2iq/xQLWythhPNDltmlzcL8CupPAA=; b=aupTSSnHuTB1nx1/9lYuEC+c4kClutWxksA6iahrFlWF3teqFpxNGarKVKOSURzm1Y /WPClX6ttBoZWIBcpE0h+QDHQrs+ovjVeHsWG67LfKtEA7MjKheg0A+YQrSF4XeSD9cF p+mFQRaLn01zi0S3zW7KoyYjbLOj3vZQ5uIMXw2hIOww6ElqKP/OVHDXgfWsajyMWoZY qmEoLRFf+rGJj3MsAn/FZdDecHOAXXNUz7nkID7IY7B7MUYwAKyCnYu87pg/FeZr5Mg5 KbYlB0Wy6UY8C/n7voiPhnDhkfkTojsCF5gAY4e2uxn9s6H4mxoeXOGG49fn/3iFmkx4 YH5g== X-Gm-Message-State: AO0yUKX8G8Xkt9qedY5iClf5o2vRhcrVFfl3uQMFwP+FyLQxUEFshS/1 IAHIk2U1LB6MWuD+5dRpKuHK+Zf0go/ay59EZCE= X-Google-Smtp-Source: AK7set/pU2gta8cOqDJ9Jhi0FOCuEOAsf+bVEMXX1Ki/p1H4Z/BN/TXwYzngQKmDLbvyYtrHuEFRRw== X-Received: by 2002:aa7:98c3:0:b0:5a8:4ba7:5840 with SMTP id e3-20020aa798c3000000b005a84ba75840mr1700082pfm.26.1676931947852; Mon, 20 Feb 2023 14:25:47 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-4-112.hawaiiantel.net. [72.253.4.112]) by smtp.gmail.com with ESMTPSA id q21-20020a62ae15000000b005a9131b6668sm8191447pff.2.2023.02.20.14.25.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Feb 2023 14:25:47 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 7/7] cooker: Drop sre_constants usage Date: Mon, 20 Feb 2023 12:25:23 -1000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 20 Feb 2023 22:25:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14474 From: Richard Purdie As reported by Martin Jansa : bitbake/lib/bb/cooker.py:16: DeprecationWarning: module 'sre_constants' is deprecated import sre_constants it's deprecated since 3.11 with: https://github.com/python/cpython/issues/91308 The correct replacement for our usage is re.error so use that instead. Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman --- lib/bb/cooker.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index ac54d437..6743bce5 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -13,7 +13,6 @@ import sys, os, glob, os.path, re, time import itertools import logging import multiprocessing -import sre_constants import threading from io import StringIO, UnsupportedOperation from contextlib import closing @@ -1795,7 +1794,7 @@ class CookerCollectFiles(object): try: re.compile(mask) bbmasks.append(mask) - except sre_constants.error: + except re.error: collectlog.critical("BBMASK contains an invalid regular expression, ignoring: %s" % mask) # Then validate the combined regular expressions. This should never @@ -1803,7 +1802,7 @@ class CookerCollectFiles(object): bbmask = "|".join(bbmasks) try: bbmask_compiled = re.compile(bbmask) - except sre_constants.error: + except re.error: collectlog.critical("BBMASK is not a valid regular expression, ignoring: %s" % bbmask) bbmask = None