From patchwork Fri Nov 29 22:10:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 53389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA8DDD6EC1C for ; Fri, 29 Nov 2024 22:10:17 +0000 (UTC) Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by mx.groups.io with SMTP id smtpd.web10.123949.1732918207889421383 for ; Fri, 29 Nov 2024 14:10:08 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=JzSxgkSH; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-385de9f789cso830267f8f.2 for ; Fri, 29 Nov 2024 14:10:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1732918205; x=1733523005; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=6+M7qhcVEMGbmkTaEfkGOlF0rjLHfKb+JOq3lsHyVAs=; b=JzSxgkSHXnKzFeone8MznLWQ18oPCfxI2/TH7zhYgUx8QzdNSDPWWM76+fmw0lxUaU ojurm1i3RC+txSJJyVaByxvLFOz6qzxXzoK1es7SebTo8sN/xGO6u1nCQXyFTpTZIYCg 3FZxWBC43IqEkk0YgR8RwkDyvfNUFMcyKMdqI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732918205; x=1733523005; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6+M7qhcVEMGbmkTaEfkGOlF0rjLHfKb+JOq3lsHyVAs=; b=lwS8UUdxNnM14YyGqJjaK6yqzGTSSw5O5np/mQUyijEdvAg7N3/sOP5AQxcyTaIKFU QP9mpuyWwtMxu9osfoGUInX2060BYkpISqU6AJCIGHvv4WIH/8jsp2q58l5gA/iHRPfN ibU62wRnzEYdy9KgNx7/07/6ADPJqH5lgUe1eizV7fbnJuTpr/1hYKs5KdayYuxXxG9v Ap8gljq/Bu391WRxSjHP202aXLNSR8dSo0mSYPlRA0DIqNLnjq7PCLVjA7bNnUCB9kmR TP/P9sJgmEy+t6b4KVhxAlfqWmJ3rBcrZ4h928Aulm9Zq8iYIQhYZ70LYw7ptPvBQD4n NY1w== X-Gm-Message-State: AOJu0YzN6pwfF61k1Z8BoOo1io0row5iyFwFryV+eUuBA3aN0h+xC9qc 2iy5vNTgVpkx+h4xXIYgwyrhN7En5wUuujP7aB0zEmRT0SWzUqYryZtxRi2gWRW5znZyWJpp0ad h X-Gm-Gg: ASbGncvjNk10fzX2nz6irog6FVRY9daMrrVYIFgHpEHq/Jn7nzpSVC2yxYazLjqCljm spG3+kcQAzO7vlcNYyzJQULSCcfz6rsrGhhSoMDNLdOOMyGzMzWoDPIDVJh13rXAoOeYIh7td5U ugeuS8AYHPK4rwVqEHQmR3f8WlomVWowe+4B0ncxKkXxEqsA5spScHj5InWKExpGFATmpUwyPO8 CDR/xu6LsTNhNq0r01O0bweRwelcYvpKpT4t6YlW8GIEnWTZ74UO2vA6EKDKKbZXqG8KSNrQ+H2 4l4= X-Google-Smtp-Source: AGHT+IFAhYa7lv4cNeJd65+S54bPSOzOtwXmX38bG0el3UcKlMmhr2rzA0w5Ybbi1HjDwYi2anfWaw== X-Received: by 2002:a5d:64c6:0:b0:382:5aae:87cf with SMTP id ffacd0b85a97d-385c6ebe438mr10141916f8f.31.1732918205109; Fri, 29 Nov 2024 14:10:05 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1083:22d7:abf5:ae6f]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-385e16d722fsm1228038f8f.41.2024.11.29.14.10.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Nov 2024 14:10:03 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/2] runqueue: Optimise setscene loop processing Date: Fri, 29 Nov 2024 22:10:01 +0000 Message-ID: <20241129221002.770402-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.45.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 29 Nov 2024 22:10:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16852 Rather than looping through things we looped through on the previous execution, start looping where we left off for setscene processing. This gives speed improvements depending on the kind of build being executed. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 2179ee1302..6457726d96 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -14,6 +14,7 @@ import os import sys import stat import errno +import itertools import logging import re import bb @@ -2204,11 +2205,16 @@ class RunQueueExecute: if not hasattr(self, "sorted_setscene_tids"): # Don't want to sort this set every execution self.sorted_setscene_tids = sorted(self.rqdata.runq_setscene_tids) + # Resume looping where we left off when we returned to feed the mainloop + self.setscene_tids_generator = itertools.cycle(self.rqdata.runq_setscene_tids) task = None if not self.sqdone and self.can_start_task(): - # Find the next setscene to run - for nexttask in self.sorted_setscene_tids: + loopcount = 0 + # Find the next setscene to run, exit the loop when we've processed all tids or found something to execute + while loopcount < len(self.rqdata.runq_setscene_tids): + loopcount += 1 + nexttask = next(self.setscene_tids_generator) if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values() and nexttask not in self.sq_harddep_deferred: if nexttask in self.sq_deferred and self.sq_deferred[nexttask] not in self.runq_complete: # Skip deferred tasks quickly before the 'expensive' tests below - this is key to performant multiconfig builds From patchwork Fri Nov 29 22:10:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 53388 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB88AD6EC1E for ; Fri, 29 Nov 2024 22:10:17 +0000 (UTC) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by mx.groups.io with SMTP id smtpd.web10.123950.1732918210235948764 for ; Fri, 29 Nov 2024 14:10:10 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=SXWh761O; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-385e2c52c21so162661f8f.2 for ; Fri, 29 Nov 2024 14:10:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1732918208; x=1733523008; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=lx9cYxNz8yhfJM7qDA0Tzjh+ABKzdKA+nZZjnHghE7Y=; b=SXWh761OPT6wdW005Lss0SzuyCL0Zci5wkTzVh6SheO6On6bful33T0Dkbrgw4tFsS ZV0OfizQwOrCbWRQC3aoGr4Du7hoGd6ETQVl5Oqw4nvph6CNvo/IdJI3LjUWgg8//kXu dE4oae4NGYTpjFRUmSjVqvlnpHLP01yX5YMcI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732918208; x=1733523008; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lx9cYxNz8yhfJM7qDA0Tzjh+ABKzdKA+nZZjnHghE7Y=; b=iwQjm9XFlhlU1x34pATHACAbRR+939ATMMIuN2KfZpbaS3UNXD3gyLoCowCSbgv4Ab IqsqonMPLbcs+h3qB4lJw3HXW7XlqVKGmX++0xSMCttXXd4bBwj1w384EhPtIuqgfyjt MhmgjFU+NEP6pF0augfIoNgkgTEb3Mt+e9Ggj2IhPZ3SHCHEYoJiA3810bWrwN7OvbcR 5LWAokbuIBBRN2m5X2rAVItzQV38MZOAcHeCE/lk3YOQHa95lYMCbbYY3620J57XfTQj ybWFEa57WdtWDdrtANBoPARgRiQSEnJB4Pvmz8ThYpCp0h+OT/4mEXVFYWvLsDg6ZST/ xc6Q== X-Gm-Message-State: AOJu0Yy62B1eAjhV8p/3k48CnFQ5qFxtXcgwqCfHxpBZchRRj7FM0jol yg2J7bg0YTc9CEUZQ5xROqhsXH5j0eJJ6LTGvbUhJCcuiloagV6w1iXgNPN9JZxdbDef8i+o4C7 w X-Gm-Gg: ASbGncuSnP92YiuSAoc3HIr8viTPJzrEz0msNXFEJGTtmZLjDXFYLcRNcMgkrRS4XSl Qd2Rf0WIyWyMBs3n5vb/D5uadJcNpc/V4j3TNADiGx+sp8O7DYJ6dk8rSVYtuhasH5qTICVaNCS ooA6lOF4QxZYg6CBtdwOH7J3Y5zAN340TRD/k41agCZT/m76gmzkoGkG5XMZTK4stHZwG6uBM/P /G8RPhsGmtqxIYlkGAWnX5EFxeG2pNMfhrqmq8TYLAuNg1tJ7RJf/KdyCmrF7/HkwmWrxIirhbO G9E= X-Google-Smtp-Source: AGHT+IHQaA9Fxuau3AqmB2Ep1SMqmKpaT8A29VdUf9gerY8kvjNvD9gFpfzuQQM4Et+pWiKeUE5+Fw== X-Received: by 2002:a5d:64ac:0:b0:382:383e:84d9 with SMTP id ffacd0b85a97d-385c6ee1decmr12141133f8f.48.1732918208045; Fri, 29 Nov 2024 14:10:08 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:1083:22d7:abf5:ae6f]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-385e16d722fsm1228038f8f.41.2024.11.29.14.10.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Nov 2024 14:10:06 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/2] runqueue: Fix scenetask processing performance issue Date: Fri, 29 Nov 2024 22:10:02 +0000 Message-ID: <20241129221002.770402-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241129221002.770402-1-richard.purdie@linuxfoundation.org> References: <20241129221002.770402-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 29 Nov 2024 22:10:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16853 Analysis shows that "bitbake core-image-ptest-all" spends a lot of time in scenequeue_updatecounters and much of it is rebuilding a set which doens't change. Reorder the code to avoid that performance glitch. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 6457726d96..ffb2d28494 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2771,8 +2771,12 @@ class RunQueueExecute: logger.debug2("%s was unavailable and is a hard dependency of %s so skipping" % (task, dep)) self.sq_task_failoutright(dep) continue + + # For performance, only compute allcovered once if needed + if self.sqdata.sq_deps[task]: + allcovered = self.scenequeue_covered | self.scenequeue_notcovered for dep in sorted(self.sqdata.sq_deps[task]): - if self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covered | self.scenequeue_notcovered): + if self.sqdata.sq_revdeps[dep].issubset(allcovered): if dep not in self.sq_buildable: self.sq_buildable.add(dep)