From patchwork Tue May 28 20:44:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 44332 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E250C41513 for ; Tue, 28 May 2024 20:44:34 +0000 (UTC) Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by mx.groups.io with SMTP id smtpd.web11.1928.1716929065426333887 for ; Tue, 28 May 2024 13:44:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=cM7ENL4c; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-4202ca70289so11139585e9.1 for ; Tue, 28 May 2024 13:44:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1716929063; x=1717533863; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=87X6tU6YjjWqVk4im2UM05esmJj8CzYGf5RQIk6U3mw=; b=cM7ENL4cYeYUTBbOmYAYbtmNb4v7pvHTD1nGgP+YLj0M2dzqmM61eW08WKgn2vrY8l a1iVnNmw7zOpiCCUGpQxgqFjUDyKj04mM9NjkU25ljVPU/OBo5d8Jo+/Gj0uW4mibmf4 3CDR6XAItFIDGicxM/O4Scc892JnyQ80Q1jHY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716929063; x=1717533863; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=87X6tU6YjjWqVk4im2UM05esmJj8CzYGf5RQIk6U3mw=; b=HllQVUwyMJdY1zx5HxgAiT0IcR9SJK2CRnuVDkwv2yT3K2lEeiko+U0dOeLXqjL/56 9FF0sd+3w0LZ6ODkJMKBmrg9D/Ve94ND3W5Wv3IItwgy2+l8u+W0KpKVGHtKbKT5R6iC aXRJDzcb+P2ug5Atw0YjXYLHT/09rscKgyYMqij6HeJkyjr5onXarShhHDkAN/qxYEaA DB3LcMj+n/LbbaiYhAbBfWRp29914w7uHdQc99dc2nVRuB7SYjTmFn1OxqP8aLr3Kles 0A7oe811hXR7dbP/1ZkScNWHKjX/97ZUUnSvHfbRJl8YhO9LQ7Cqcl8kdpTp8eS4jblu xiSw== X-Gm-Message-State: AOJu0YwHXgAldPJchPPM2iKaNhZ8vcIT3UdFJLXfRklSlerEmiwx37LR UeUYgEfT3n6TgxBpWEHefiP5MWUCBTv0C5dQ7CXeC063M7dtSO+pw8piNBsMuD8JcL4KOwsjW6e j X-Google-Smtp-Source: AGHT+IHOxyPYCufuIBRmMeZ2tqeqpq2H5uE2vtQ52CDv3dOzaBGYcgKKqnfmueR94blvQh4V5LFJBQ== X-Received: by 2002:a05:6000:400c:b0:356:4a73:af59 with SMTP id ffacd0b85a97d-3564a73b0a5mr10469023f8f.28.1716929062689; Tue, 28 May 2024 13:44:22 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:b316:c62f:d434:bcd]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-35579f9649esm12635228f8f.29.2024.05.28.13.44.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 13:44:22 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/4] runqueue: Add timing warnings around slow loops Date: Tue, 28 May 2024 21:44:17 +0100 Message-Id: <20240528204420.2845521-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 May 2024 20:44:34 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16272 With hashserve enabled, there are two slow paths/loops, one at initial runqueue generation and also during the rehash process when new outhashes are found. Add timing information at the hashserve log level for when these loops take longer than 30s or 60s overall. This will leave evidence in the logs when things are running particularly slowly. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 6b43f303d5..83e5273449 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1273,6 +1273,9 @@ class RunQueueData: bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids) + starttime = time.time() + lasttime = starttime + # Iterate over the task list and call into the siggen code dealtwith = set() todeal = set(self.runtaskentries) @@ -1284,6 +1287,14 @@ class RunQueueData: self.prepare_task_hash(tid) bb.event.check_for_interrupts(self.cooker.data) + if time.time() > (lasttime + 30): + lasttime = time.time() + hashequiv_logger.verbose("Initial setup loop progress: %s of %s in %s" % (len(todeal), len(self.runtaskentries), lasttime - starttime)) + + endtime = time.time() + if (endtime-starttime > 60): + hashequiv_logger.verbose("Initial setup loop took: %s" % (endtime-starttime)) + bb.parse.siggen.writeout_file_checksum_cache() #self.dump_data() @@ -2557,6 +2568,9 @@ class RunQueueExecute: elif self.rqdata.runtaskentries[p].depends.isdisjoint(total): next.add(p) + starttime = time.time() + lasttime = starttime + # When an item doesn't have dependencies in total, we can process it. Drop items from total when handled while next: current = next.copy() @@ -2589,6 +2603,14 @@ class RunQueueExecute: total.remove(tid) next.intersection_update(total) + if time.time() > (lasttime + 30): + lasttime = time.time() + hashequiv_logger.verbose("Rehash loop slow progress: %s in %s" % (len(total), lasttime - starttime)) + + endtime = time.time() + if (endtime-starttime > 60): + hashequiv_logger.verbose("Rehash loop took more than 60s: %s" % (endtime-starttime)) + if changed: for mc in self.rq.worker: RunQueue.send_pickled_data(self.rq.worker[mc].process, bb.parse.siggen.get_taskhashes(), "newtaskhashes") From patchwork Tue May 28 20:44:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 44334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFD99C25B7E for ; Tue, 28 May 2024 20:44:34 +0000 (UTC) Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) by mx.groups.io with SMTP id smtpd.web10.1866.1716929065678095797 for ; Tue, 28 May 2024 13:44:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=R4xIE7Nt; spf=pass (domain: linuxfoundation.org, ip: 209.85.208.181, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-lj1-f181.google.com with SMTP id 38308e7fff4ca-2e95a1eff78so17477201fa.0 for ; Tue, 28 May 2024 13:44:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1716929064; x=1717533864; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=0Fbv0YTCF0bBEPAQEd+q/Tjc2oKfIvsfQAHHK9orXwk=; b=R4xIE7NtIrRSptbmCDUG6aC4Mip4UGAJ7bbi0Lt2ftQ/5voSSSUsyMXBxCMpas1GlW CjfobiOmZBoC4joJBwBMZQf5bIrWeoYgOwIql1acEf5yJkr3OA2K92jFvqCo/lqbN7ho TxOXUSFCMKv29haKXhZj3bnmNGNIeT0wXS9g8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716929064; x=1717533864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0Fbv0YTCF0bBEPAQEd+q/Tjc2oKfIvsfQAHHK9orXwk=; b=ZE02+im3he7BFOMPMQ2ZAqf25j4yBAiOr8ixqyYZWayS6TcTr4owYEJhXhQkJtRjW9 PCKXPZR4haKsuIorsiqMpmJR9K6wtAGNyarfpmrfpk9VnzzHcRCLsEQz9MSHm8Yw9h5B sXlwW/Bx5opZelsx5xmw9zKu2u4yu4Ah5DTaRSSmARgoZ6wDciXOJ04MolxgXCp1m+Xp +kbH0Ckrv3wwtBsK8QNeErvJaV4uVzkW7xswmz1ve9AdHrVPQ1LGrQHi5zetp2TJ8ssf cf9Z7N7iba0KpqpxSeGbx4peY7xD/Q5V3XzigD2hVlAN8nCjQOQ6VBgrG6kOWh4mfi1E wPDg== X-Gm-Message-State: AOJu0YxeZd4MdVXiosXtskzwH6aGpGmtaWDeZctEpJzsTCWhM6sgu8x3 NQR5jXSv+WqsI5j9b+cdED1VZ9bNl0C57abiOwuy3AMduBWP0u1p8GI9+DU5zRpV0SYpJ5CccHT + X-Google-Smtp-Source: AGHT+IFWVuwLXG+emsKFUOQygLXku3E54tTupUZt6trjE3HeYdmbLRgeFtpysAaREl0CtzibdU2lcA== X-Received: by 2002:a2e:9b97:0:b0:2e7:174b:8c1c with SMTP id 38308e7fff4ca-2e95b256336mr123908501fa.38.1716929063310; Tue, 28 May 2024 13:44:23 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:b316:c62f:d434:bcd]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-35579f9649esm12635228f8f.29.2024.05.28.13.44.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 13:44:23 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/4] runqueue: Allow rehash loop to exit in case of interrupts Date: Tue, 28 May 2024 21:44:18 +0100 Message-Id: <20240528204420.2845521-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240528204420.2845521-1-richard.purdie@linuxfoundation.org> References: <20240528204420.2845521-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 May 2024 20:44:34 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16273 The initial hash serve loop exits in the case where interrupts are present but probably checks a bit too often. Tweak that and also allow the slow rehash loop to break on interrupt, improving bitbake Ctrl+C response. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 83e5273449..84a6f4172c 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1285,7 +1285,8 @@ class RunQueueData: dealtwith.add(tid) todeal.remove(tid) self.prepare_task_hash(tid) - bb.event.check_for_interrupts(self.cooker.data) + + bb.event.check_for_interrupts(self.cooker.data) if time.time() > (lasttime + 30): lasttime = time.time() @@ -2602,6 +2603,7 @@ class RunQueueExecute: next |= self.rqdata.runtaskentries[tid].revdeps total.remove(tid) next.intersection_update(total) + bb.event.check_for_interrupts(self.cooker.data) if time.time() > (lasttime + 30): lasttime = time.time() From patchwork Tue May 28 20:44:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 44333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B6D4C25B7C for ; Tue, 28 May 2024 20:44:34 +0000 (UTC) Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by mx.groups.io with SMTP id smtpd.web10.1867.1716929066027175602 for ; Tue, 28 May 2024 13:44:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=eqvtFnj1; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.45, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-421124a04d6so11354045e9.3 for ; Tue, 28 May 2024 13:44:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1716929064; x=1717533864; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=sFoLgyCqdBtE7WpK0nWArbjkeHZtb0yq21OOFSKvciQ=; b=eqvtFnj104aVSG9JkT0awaBKXUXlm4cJpHW+k8BEDebwZdVCF2vVZWGehm1IBcqyjI Ylei73G94tSGB8xjPAGI9TUFi10vnP1XIFoH3bXgdVHyXQ62DbNI9bDPDk6GJqwTTBoT 61TG2mJAH2DLE/YHAZ2iFDPjUTFat2SaxHcus= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716929064; x=1717533864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sFoLgyCqdBtE7WpK0nWArbjkeHZtb0yq21OOFSKvciQ=; b=GYhhtfXrrjh5CSD8d9PkZFPjpCbQiZhoql26S/YxQDRVfr/2WpoJ5rMmqD5vMG0/f+ oOOH7kkvi3ZzwNqK2vWvMSkB8YFKzufL/f6u1AapxDHfcF1WRolwCbz3B6yh7gi3JhW3 CA/k1DrRmmG9BuUA6Qcz2cGKlydEawia8Tx+pWB7IC477Zm5YLfwEwWoKGP9/Z4BYMc6 MCWEyHQXC1QVae+GpgwPeH2OT4fyOeIU96MbKXJJlpm8+RrY/jFsXyLnZqqRy6nPP6eN WQ4GkJUGpplwbZMX7TizVAvB170d+tDBVi2HElWa4eiOagfYl9jyDeDfiRGNySafDxvn qYAQ== X-Gm-Message-State: AOJu0YwFQJsXckcWwthgfSGdIz67iGcGUNsx1/Rw/4AM3a9mkzXf67Ww 3b9ctRHhsmGHpjK0ZxBzl+b7wXbj27AQYo0xHKrJIWpuMbuKeYkxx1mWqFFcBDor86TetactB0x y X-Google-Smtp-Source: AGHT+IGSUwCnk32rPoUtJP0F00QUJW4vmM6uv3VsjgE/9+FGLS2byUfsj+2SwvNPbycP0HTGwkx5Iw== X-Received: by 2002:a5d:6a8f:0:b0:354:f2b8:c75e with SMTP id ffacd0b85a97d-3552219d4demr8175540f8f.33.1716929064082; Tue, 28 May 2024 13:44:24 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:b316:c62f:d434:bcd]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-35579f9649esm12635228f8f.29.2024.05.28.13.44.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 13:44:23 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/4] runqueue: Process unihashes in parallel at init Date: Tue, 28 May 2024 21:44:19 +0100 Message-Id: <20240528204420.2845521-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240528204420.2845521-1-richard.purdie@linuxfoundation.org> References: <20240528204420.2845521-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 May 2024 20:44:34 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16274 Improve the runqueue init code to call unihash queries in parallel since this is faster and more efficient, particularly on slower links with longer round trip times. The call to the function from cooker is unneeded since that function calls prepare() and hence this functionality will already have run, so drop that obsolete call. Signed-off-by: Richard Purdie Reviewed-By: Joshua Watt --- lib/bb/cooker.py | 1 - lib/bb/runqueue.py | 18 ++++++++++-------- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 076ddaa58d..255b9f199a 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1459,7 +1459,6 @@ class BBCooker: if t in task or getAllTaskSignatures: try: - rq.rqdata.prepare_task_hash(tid) sig.append([pn, t, rq.rqdata.get_task_unihash(tid)]) except KeyError: sig.append(self.getTaskSignatures(target, [t])[0]) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 84a6f4172c..999868dd72 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1280,11 +1280,18 @@ class RunQueueData: dealtwith = set() todeal = set(self.runtaskentries) while todeal: + ready = set() for tid in todeal.copy(): if not (self.runtaskentries[tid].depends - dealtwith): - dealtwith.add(tid) - todeal.remove(tid) - self.prepare_task_hash(tid) + self.runtaskentries[tid].taskhash_deps = bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) + # get_taskhash for a given tid *must* be called before get_unihash* below + self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) + ready.add(tid) + unihashes = bb.parse.siggen.get_unihashes(ready) + for tid in ready: + dealtwith.add(tid) + todeal.remove(tid) + self.runtaskentries[tid].unihash = unihashes[tid] bb.event.check_for_interrupts(self.cooker.data) @@ -1301,11 +1308,6 @@ class RunQueueData: #self.dump_data() return len(self.runtaskentries) - def prepare_task_hash(self, tid): - self.runtaskentries[tid].taskhash_deps = bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) - self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) - self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid) - def dump_data(self): """ Dump some debug information on the internal data structures From patchwork Tue May 28 20:44:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 44335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C01B6C25B78 for ; Tue, 28 May 2024 20:44:34 +0000 (UTC) Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by mx.groups.io with SMTP id smtpd.web11.1929.1716929067084593646 for ; Tue, 28 May 2024 13:44:27 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=hKmFMROh; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.53, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-4202ca70289so11139955e9.1 for ; Tue, 28 May 2024 13:44:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1716929065; x=1717533865; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=D+zhKfDtI6DAXg6cnmGPeU07vIux6vj01VoOFctCbl4=; b=hKmFMROhmi0cstgmBvgiSWL7Bilf0HTNdDwXArqYAeKl9pbvcYurCOoXNhlrHZn0ql QKwwoUq7bsmyCIkoIP9392YwYn5Thav8PtlU6CMU8B5vGKYFRS7IXvfRwOn3gmjLf67A j2hVsXmdXyJX5CIchb5h3XoG2spqJMBz2h0Z4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716929065; x=1717533865; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D+zhKfDtI6DAXg6cnmGPeU07vIux6vj01VoOFctCbl4=; b=CFXncBatXY95RSZkHQTxw6u0afPArdiwigxoZHTCObMDUCq9mb+LcRTbEyRFGggzJn bxrS+m6mP1TLPUp2KwWXkkpLhYKArGjZgI6ftwEWv1R/soNKyYUJWShZMPySqguWJIrP yndaagtY47vLXuzR5Cu4kr3d81jvzLH/V0DJ4YRvg6Uso8uU6qDVDRFWXjBCWnjPb6LO XrdkWe+QfOozpJlDzDd3xQj+nGo4qvpYHTBkZNF3rIcYmC8AsnV6CmmH0mFwZk5O8+0t LLOKWSHfapMhWywEM3Bu61OoxgczpljOTcJZv5TfLcIMYctLAv8i5rmXrWuUhba/0Pa9 soxA== X-Gm-Message-State: AOJu0YxD+mfKClxCb8AyDuWD/EgiD4hyRnrkf5PIRRZVLT2sp3fquNZV s64WH7XJUmOiO912ALtXzBZJSfOo5a/UhngGW5Pi+9Jk/nFyN2YGbaRb96KZUVazloSCYHiQSE2 g X-Google-Smtp-Source: AGHT+IE2CN1CpZe+yqSVKQTtDfTjhV95/XaPJ+tTfYzh6+th26hAdDt2IieJOISpxxWYiuJzXMenCw== X-Received: by 2002:a5d:59aa:0:b0:358:380:f44c with SMTP id ffacd0b85a97d-3580380f521mr7546136f8f.5.1716929065260; Tue, 28 May 2024 13:44:25 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:b316:c62f:d434:bcd]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-35579f9649esm12635228f8f.29.2024.05.28.13.44.24 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 13:44:24 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 4/4] runqueue: Improve rehash get_unihash parallelism Date: Tue, 28 May 2024 21:44:20 +0100 Message-Id: <20240528204420.2845521-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240528204420.2845521-1-richard.purdie@linuxfoundation.org> References: <20240528204420.2845521-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 28 May 2024 20:44:34 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16275 Improve the rehash code to query unihashes in parallel since this is more efficient on slower links. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 999868dd72..47f48304ea 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2578,13 +2578,21 @@ class RunQueueExecute: while next: current = next.copy() next = set() + ready = {} for tid in current: if self.rqdata.runtaskentries[p].depends and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): continue + # get_taskhash for a given tid *must* be called before get_unihash* below + ready[tid] = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches) + + unihashes = bb.parse.siggen.get_unihashes(ready.keys()) + + for tid in ready: orighash = self.rqdata.runtaskentries[tid].hash - newhash = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches) + newhash = ready[tid] origuni = self.rqdata.runtaskentries[tid].unihash - newuni = bb.parse.siggen.get_unihash(tid) + newuni = unihashes[tid] + # FIXME, need to check it can come from sstate at all for determinism? remapped = False if newuni == origuni: