From patchwork Sat Jun 1 12:27:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 44493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CCC6C27C51 for ; Sat, 1 Jun 2024 12:27:37 +0000 (UTC) Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by mx.groups.io with SMTP id smtpd.web11.36149.1717244852521394385 for ; Sat, 01 Jun 2024 05:27:32 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=OABGK2UU; spf=softfail (domain: sakoman.com, ip: 209.85.210.174, mailfrom: steve@sakoman.com) Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-70249c5fb36so1219802b3a.2 for ; Sat, 01 Jun 2024 05:27:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1717244852; x=1717849652; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=JUYcg6Sv5Gn3fQQaDwdpurPht4QtDDI+R+iPtcbdmaw=; b=OABGK2UUt94EV2gwOOn8fQHvC0LS6cd/7oPwJFfTQtckCvjUV/TzaoL/x1c+7o3BdN bVzX+yM6hwbZ6qU3qspq3u8ozDRG2DvvvulT2qdu+FFzAnD4cyqtOunDuxDk4X0OzJKh I1mPYriIgNNKJVfXvIaD7fN1boqp3XYJjVu/J+PfmrNt6l6GxMbduhD55yc3BushBeFt qBtNgm0CXeSBuxnbLHuerKnklzInh3OLvoeSZKK0wPR/RZBnphzqnuXKmPqG4OcyUwZ6 ZuRVBl2ld6d97GGyrtZtCfPwiMapANbmCp0DtcIHgdA6/1+JC9C/MpgTLBOuHiLMRopX y8mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717244852; x=1717849652; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JUYcg6Sv5Gn3fQQaDwdpurPht4QtDDI+R+iPtcbdmaw=; b=ICMnRLRyXVG04QVg3dp+lK52bJmh/L9+PgaO7fFDQDZnC9fVvwjeMIlyApn4d27uIT Evm+uXsIyoPBOPKrzvAeyN8LTNqhraZWe7G6BvwxW9h4arsmKqoK0tpydNIo6by/XNlT KxEIGgDV5zl03184sFWZGzNfPzju9e+KlLJvNKfht3yrLNQ4MXHbtjyy+/yYCwMEm1h8 2C649UWYuam3tBw9y0Z95OJRBfbJalcUrQrHqLVrIptU73iH5w1vsfpgz8VIIyIA8i7u k9HHhtKFbi1GmLLOl51ZvihdGnP/dkvm5KtRGaBrVmhn+RsY71dxlJ7ghm9NFVt0GIP0 KIQQ== X-Gm-Message-State: AOJu0YwF1nCq9gUNfQBKnlY9iifyumXSJHQX5Ui/uAcforv7TNeyndU8 7/p9Rr2hpfUD4t+XJ5XLx7JxpT+xrg4grG8R0mpRDRWKKo/allgYgswKg5Ly2thJ8/pBBEmtNb3 o X-Google-Smtp-Source: AGHT+IFdMn3jlPWBoScjxVORiS7PqHLUT1aXuCAxCP/SmOo+GTG9NGllZWoTs1EucG7OK+Me+Y4wjA== X-Received: by 2002:a05:6a20:320d:b0:1b1:f7a1:df91 with SMTP id adf61e73a8af0-1b26f296e46mr4027853637.61.1717244851659; Sat, 01 Jun 2024 05:27:31 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242b30013sm2809362b3a.211.2024.06.01.05.27.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 01 Jun 2024 05:27:31 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 1/8] runqueue: Add timing warnings around slow loops Date: Sat, 1 Jun 2024 05:27:16 -0700 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 01 Jun 2024 12:27:37 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16305 From: Richard Purdie With hashserve enabled, there are two slow paths/loops, one at initial runqueue generation and also during the rehash process when new outhashes are found. Add timing information at the hashserve log level for when these loops take longer than 30s or 60s overall. This will leave evidence in the logs when things are running particularly slowly. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index bc7e18175..beec1e046 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1273,6 +1273,9 @@ class RunQueueData: bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids) + starttime = time.time() + lasttime = starttime + # Iterate over the task list and call into the siggen code dealtwith = set() todeal = set(self.runtaskentries) @@ -1284,6 +1287,14 @@ class RunQueueData: self.prepare_task_hash(tid) bb.event.check_for_interrupts(self.cooker.data) + if time.time() > (lasttime + 30): + lasttime = time.time() + hashequiv_logger.verbose("Initial setup loop progress: %s of %s in %s" % (len(todeal), len(self.runtaskentries), lasttime - starttime)) + + endtime = time.time() + if (endtime-starttime > 60): + hashequiv_logger.verbose("Initial setup loop took: %s" % (endtime-starttime)) + bb.parse.siggen.writeout_file_checksum_cache() #self.dump_data() @@ -2556,6 +2567,9 @@ class RunQueueExecute: elif self.rqdata.runtaskentries[p].depends.isdisjoint(total): next.add(p) + starttime = time.time() + lasttime = starttime + # When an item doesn't have dependencies in total, we can process it. Drop items from total when handled while next: current = next.copy() @@ -2588,6 +2602,14 @@ class RunQueueExecute: total.remove(tid) next.intersection_update(total) + if time.time() > (lasttime + 30): + lasttime = time.time() + hashequiv_logger.verbose("Rehash loop slow progress: %s in %s" % (len(total), lasttime - starttime)) + + endtime = time.time() + if (endtime-starttime > 60): + hashequiv_logger.verbose("Rehash loop took more than 60s: %s" % (endtime-starttime)) + if changed: for mc in self.rq.worker: RunQueue.send_pickled_data(self.rq.worker[mc].process, bb.parse.siggen.get_taskhashes(), "newtaskhashes") From patchwork Sat Jun 1 12:27:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 44490 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE8B0C41513 for ; Sat, 1 Jun 2024 12:27:36 +0000 (UTC) Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by mx.groups.io with SMTP id smtpd.web10.36210.1717244853856897433 for ; Sat, 01 Jun 2024 05:27:33 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=GdV5c3rt; spf=softfail (domain: sakoman.com, ip: 209.85.210.176, mailfrom: steve@sakoman.com) Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-7024cd9dd3dso1092280b3a.3 for ; Sat, 01 Jun 2024 05:27:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1717244853; x=1717849653; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=uBWIzfMaiAueC2Mb5ysKX3hbHYpeRnoedi3yHA423KA=; b=GdV5c3rt5tY5TxZDAj8VWwws5dL752ynKOadHPyaM1hg5CXE/43WyWCmrDa/kxQEKq Bp4Xa7mpeJuKHE2U800jUTrR+lllYiIWO3e9vlPScsCjX/zilf2BtSDampDGjfzE0gym /27BTm4e6voMJjvGMFqa0tjY8Pdkib+R2Io8p12mozYFEfoHHMYkP2VdNYmtJvUb6oT5 njAyV8DhB0q1lkn5UOKRGU6rKD9t27N1mj5j7MzOfkFkvrGrpVLnJ3JF4h8DjL7f+Ko2 /qtFiYy20gftd4Ld2OQ0sD0WKJHEgws2LVIToR9ByjGGzG1y4aFIAJ3obVT6y90WF7Xt 2Jcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717244853; x=1717849653; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uBWIzfMaiAueC2Mb5ysKX3hbHYpeRnoedi3yHA423KA=; b=ccwGkzwMr8EdbajTVHvkKS/HSeU6MOmUPlZ02eb1DlSCwmBfLN07jGJelsrpHcBKPH 9Qz1+97sW/ZItBtlxCMtmLx4vEi+1XzvbM9MlwCcciyJczkRdS0IE9ev1AKQ+Y2D8UJb 6Y4QJGyfEtJjAPBaTkXIuL3aJRU8hC706HXPHUJu09vN547voHS76D9X0yTwEBLCJRVT 64FdXGbHPMcdVsuADQKiIJC8BcJadVXTVIcjQBMSeMo38RPbraCWcXKusuP0fNrUa4O5 LuzxQKYl9SDfee72d4ZMq8jal8w6pI7ZqIuYE63VnOSZLlcbpisWI17FO7Za472lSGtA GtnQ== X-Gm-Message-State: AOJu0YxrtWedYpYIbL2OTMRrNYxx9IR2qorweY3bneRp+Bg9FeHHluOm ZvTkzY0riA2dETgo/wvanX+5UW1PCS/ZwLWKFgL8PWhWhYmUnTsXeYUfD0ReAXLz/RL/PxIE429 i X-Google-Smtp-Source: AGHT+IG8MtVWENjNVD0ekyMCjErTi5cNR8o8f5k4+qdZhOxhJdDPPTJrdJNbagnsxo+456sg/qrIqg== X-Received: by 2002:a05:6a00:21c4:b0:6ec:faff:ec1f with SMTP id d2e1a72fcca58-702477fd1dcmr4933119b3a.11.1717244853082; Sat, 01 Jun 2024 05:27:33 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242b30013sm2809362b3a.211.2024.06.01.05.27.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 01 Jun 2024 05:27:32 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 2/8] runqueue: Allow rehash loop to exit in case of interrupts Date: Sat, 1 Jun 2024 05:27:17 -0700 Message-Id: <80f5b5a7617e1883d05fc9c9d42b81026fab6c37.1717244760.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 01 Jun 2024 12:27:36 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16306 From: Richard Purdie The initial hash serve loop exits in the case where interrupts are present but probably checks a bit too often. Tweak that and also allow the slow rehash loop to break on interrupt, improving bitbake Ctrl+C response. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index beec1e046..85c3581f9 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1285,7 +1285,8 @@ class RunQueueData: dealtwith.add(tid) todeal.remove(tid) self.prepare_task_hash(tid) - bb.event.check_for_interrupts(self.cooker.data) + + bb.event.check_for_interrupts(self.cooker.data) if time.time() > (lasttime + 30): lasttime = time.time() @@ -2601,6 +2602,7 @@ class RunQueueExecute: next |= self.rqdata.runtaskentries[tid].revdeps total.remove(tid) next.intersection_update(total) + bb.event.check_for_interrupts(self.cooker.data) if time.time() > (lasttime + 30): lasttime = time.time() From patchwork Sat Jun 1 12:27:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 44494 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA295C27C4F for ; Sat, 1 Jun 2024 12:27:36 +0000 (UTC) Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by mx.groups.io with SMTP id smtpd.web11.36151.1717244855655188340 for ; Sat, 01 Jun 2024 05:27:35 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=vh/K8I8D; spf=softfail (domain: sakoman.com, ip: 209.85.210.177, mailfrom: steve@sakoman.com) Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-701ae8698d8so2335629b3a.0 for ; Sat, 01 Jun 2024 05:27:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1717244855; x=1717849655; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Bx+Yn2wOks02Dj9JWTkNVpwMnTd8zjR0QVVk50YnPco=; b=vh/K8I8Del8xyCARs6inyBBoLpSwT99+wnAFBPFgoh+gZdzJ03ESJK6lkCycvtJXbN TonGncAguAXvf1UIovo82QQn6SgtUhF7wnlCjvqs46T/ZABZc+2+xMMYZJ2O7n7X+8fh a/a6hOjB+gxW2a79aXeqYfh+F4E9akrXQVs+7Irzbp5HK0Qp3HZmeeKj5PZlEWoeSL/J 8W1MT7NwrC3G/cEZOwlx/U2RokDes890U7QEEDIWFLqSsjmZPmpuXrv02h6+cNNyg28n 07qiDAwQCkKxGE1v3GfiBDGRN/0BasjABmNVclK/tAsVdiKfMKWK88mHSxWMqCH0vXlu LvXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717244855; x=1717849655; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Bx+Yn2wOks02Dj9JWTkNVpwMnTd8zjR0QVVk50YnPco=; b=EV1JWsT7YJRlypGnBALD8WOfzbX1unZ+ppgaf/637z2gzj2HA9KleH3TvufSC/0HNq cphktxdQcB/0Az1Y8Jxmtzzfl+31kYssvesOHYm3eM+QoWCFHgFUs5W3trZ4xUoxtetV xhweUgvzWSxorb9gi9QFeiuiXyzK4BL5jmZcrBF09S5+SWApnvkj88+7mg6JDd8R1k4e Oa0jX4KJb5qthBK1sJ32HaNPFO5as7qsHLA6zU2T6oeWA1IIyKP4NBWdZ/Pjc33DP7UQ 7UzsEqr4Qn8OaWVhuUossL44ouSYa2vrd1oTecTNvnCChLgu29qyQ+Aqnbe8hAjamfie U7hw== X-Gm-Message-State: AOJu0YxFnwyLYrcXZ5V1c0ZKC388e/wKJ8xbf/CHHZWOvuh22/gUv/YI /t3lsVc57wJy9CYkY4jMLfi6Y09Oc7XoiUynbSBxSSMS12NWpYPVxHzH4bRmBRO53Q7//QyzRFc S X-Google-Smtp-Source: AGHT+IGHVRiO3xhQWveJl3F9Hj1rO6YaOmFuqETmo3pf9RCHi66uW7ttO+EW0tvHD4h0TMPC9OmreQ== X-Received: by 2002:a05:6a00:10cc:b0:6f3:ea8a:f974 with SMTP id d2e1a72fcca58-702477be39amr4787770b3a.1.1717244854383; Sat, 01 Jun 2024 05:27:34 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242b30013sm2809362b3a.211.2024.06.01.05.27.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 01 Jun 2024 05:27:34 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 3/8] siggen/runqueue: Report which dependencies affect the taskhash Date: Sat, 1 Jun 2024 05:27:18 -0700 Message-Id: <4d4998ac7719945a69544bbef343978db80da305.1717244760.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 01 Jun 2024 12:27:36 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16307 From: joshua Watt Report which task dependencies in BB_TASKDEPDATA are included in the taskhash. This allows tasks to identify which tasks dependencies may change without the task re-running. Knowing this information is important for tasks that want to transfer information from dependencies (such as SPDX) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 8 +++++--- lib/bb/siggen.py | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 85c3581f9..84a6f4172 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1302,7 +1302,7 @@ class RunQueueData: return len(self.runtaskentries) def prepare_task_hash(self, tid): - bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) + self.runtaskentries[tid].taskhash_deps = bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid) @@ -2457,7 +2457,8 @@ class RunQueueExecute: unihash = self.rqdata.runtaskentries[task].unihash deps = self.filtermcdeps(task, mc, deps) hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn] - taskdepdata_cache[task] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn] + taskhash_deps = self.rqdata.runtaskentries[task].taskhash_deps + taskdepdata_cache[task] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn, taskhash_deps] self.taskdepdata_cache = taskdepdata_cache @@ -2836,7 +2837,8 @@ class RunQueueExecute: taskhash = self.rqdata.runtaskentries[revdep].hash unihash = self.rqdata.runtaskentries[revdep].unihash hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn] - taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn] + taskhash_deps = self.rqdata.runtaskentries[revdep].taskhash_deps + taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn, taskhash_deps] for revdep2 in deps: if revdep2 not in taskdepdata: additional.append(revdep2) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 8ab08ec96..03dfda6f3 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -381,7 +381,7 @@ class SignatureGeneratorBasic(SignatureGenerator): self.taints[tid] = taint logger.warning("%s is tainted from a forced run" % tid) - return + return set(dep for _, dep in self.runtaskdeps[tid]) def get_taskhash(self, tid, deps, dataCaches): From patchwork Sat Jun 1 12:27:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 44492 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC17EC27C44 for ; Sat, 1 Jun 2024 12:27:36 +0000 (UTC) Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by mx.groups.io with SMTP id smtpd.web10.36211.1717244856482737449 for ; Sat, 01 Jun 2024 05:27:36 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=Hwbz4wxs; spf=softfail (domain: sakoman.com, ip: 209.85.210.176, mailfrom: steve@sakoman.com) Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-702342c60dfso1872576b3a.2 for ; Sat, 01 Jun 2024 05:27:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1717244856; x=1717849656; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=xGHus0hctalZvqZUVmpgCRabsjS9XcjL4CPRIMjvmAk=; b=Hwbz4wxszc7TS1GjIvndoEJep/DY5tIJlAHS2FJRnD2uf0SZ4iwu/ZL4UbLw2R+geV ws6ZOIMZsuBzpEVKtYsKwM0RSMLAPW+Xl3PUZ5YZSsmc5ovo65lFGQGRsvARHDQeEFn7 q21PIb/yN/8XWlG6Vgk7SfwtA1c+sNRS7pJWSUCqN8KOEplLbO21KhGjsPukGOjBviKF p4XM8rju4+Dk6WDBD6RSuc45tLTdM+4O95oYyZwTWUJ37cSmJVaJEB62Xs9ee4MNcICl PXO9uUIsXL/W6D+lVqs4+wFytKFMrPe5Ppjnmpclo5rp9VUFIrn7d5jNbKNTmYjzlVup 1CrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717244856; x=1717849656; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xGHus0hctalZvqZUVmpgCRabsjS9XcjL4CPRIMjvmAk=; b=ehwM946/i1Eus81qAtY+gth7rmrQqzdQ9D2DIzIcyCfAVJUPXXKvOYrZFr700+HhW2 Tdxcf1rnhSaL8jTB4KzRKJaQPuzNGk11CPPR5HUt5OOZC2yfrzUZXA1Vh5MVSXscy2YX 8lxsAUU8xAZugwXdC9z2lzDN4zxZyJUDYlQXpH3kjW8p/Bon2B6i7m4GTkh6Hwu/EMgc R/nbfj7OfnlkTojYTRucOaQ7xxy73MBkeffwuiTaoLF5fwYpyAUHCG8gmIwP+JrUuHh6 7Xjv0mL7HFs0rz3lK5VKbdx4ua9FsTN1NWwniUNINEmu0am7QUXQIpf30ZqREQ33VGTd y+uQ== X-Gm-Message-State: AOJu0YweZ6IAHPRm+i3ihXhLBfMV5+1hS5httDvy5xI3tfpZYq1zc4au R9eiZS5tHLVwslabmP1pnl7N7/UGCKum0Ur5V3WtJfbZMkLB/sUu4AxeEs2mEHtMuVoaH29CAAY 3 X-Google-Smtp-Source: AGHT+IERU7LkfTwnZjs7EjLAo9NhmG/Gh4IgDNz5nFGEaCCe9ioVNPFPT8Bmrr9NtbWgHzJdW2tvyQ== X-Received: by 2002:a05:6a00:a94:b0:702:2675:fffd with SMTP id d2e1a72fcca58-702478c7436mr5227558b3a.34.1717244855738; Sat, 01 Jun 2024 05:27:35 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242b30013sm2809362b3a.211.2024.06.01.05.27.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 01 Jun 2024 05:27:35 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 4/8] runqueue: Process unihashes in parallel at init Date: Sat, 1 Jun 2024 05:27:19 -0700 Message-Id: <33686c80347491a39cbbd7a926eed5452e9aa168.1717244760.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 01 Jun 2024 12:27:36 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16308 From: Richard Purdie Improve the runqueue init code to call unihash queries in parallel since this is faster and more efficient, particularly on slower links with longer round trip times. The call to the function from cooker is unneeded since that function calls prepare() and hence this functionality will already have run, so drop that obsolete call. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 1 - lib/bb/runqueue.py | 18 ++++++++++-------- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 939a99997..6318ef4a8 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1459,7 +1459,6 @@ class BBCooker: if t in task or getAllTaskSignatures: try: - rq.rqdata.prepare_task_hash(tid) sig.append([pn, t, rq.rqdata.get_task_unihash(tid)]) except KeyError: sig.append(self.getTaskSignatures(target, [t])[0]) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 84a6f4172..999868dd7 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1280,11 +1280,18 @@ class RunQueueData: dealtwith = set() todeal = set(self.runtaskentries) while todeal: + ready = set() for tid in todeal.copy(): if not (self.runtaskentries[tid].depends - dealtwith): - dealtwith.add(tid) - todeal.remove(tid) - self.prepare_task_hash(tid) + self.runtaskentries[tid].taskhash_deps = bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) + # get_taskhash for a given tid *must* be called before get_unihash* below + self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) + ready.add(tid) + unihashes = bb.parse.siggen.get_unihashes(ready) + for tid in ready: + dealtwith.add(tid) + todeal.remove(tid) + self.runtaskentries[tid].unihash = unihashes[tid] bb.event.check_for_interrupts(self.cooker.data) @@ -1301,11 +1308,6 @@ class RunQueueData: #self.dump_data() return len(self.runtaskentries) - def prepare_task_hash(self, tid): - self.runtaskentries[tid].taskhash_deps = bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) - self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches) - self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid) - def dump_data(self): """ Dump some debug information on the internal data structures From patchwork Sat Jun 1 12:27:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 44495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07D98C25B76 for ; Sat, 1 Jun 2024 12:27:47 +0000 (UTC) Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by mx.groups.io with SMTP id smtpd.web11.36153.1717244857739931887 for ; Sat, 01 Jun 2024 05:27:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=Q+h+Lkmt; spf=softfail (domain: sakoman.com, ip: 209.85.215.181, mailfrom: steve@sakoman.com) Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-6c8c880f526so178370a12.3 for ; Sat, 01 Jun 2024 05:27:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1717244857; x=1717849657; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=WQ0ieBxiPaIdymhADC8YdicpJmdMNn4nMX++jkAN9Lw=; b=Q+h+LkmtyfNR5Wpcev8yatVIIeqWe1DSJgfF+rx5YJhkVgfgkmQsC7B5M+iAbpEsDb 4/2jrI1/Gczdg5CHOfnVo8VLPlqTjTZvO0ip2gyGxCRx+ttNIn+nTvXuaIBuS8N5lTJw reudfO9/BYNtHAI3GkHm/tF0rjJml1uhc6raxVqfhm0hceCCLIhCCM/9znhu6oykLmSj N1Dp/Ak6lKagE4nizilIMGzQr1T+KbAj3jGNY3TSnHJR2OhJU9snRpR4LDHBgxfrSgqf u0VTt0B6FQsi2C6o06P9If6oKgKsytKmLo4l3P59HNUCF/eHfO99THn2TMVzgpZOcR5D yOWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717244857; x=1717849657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WQ0ieBxiPaIdymhADC8YdicpJmdMNn4nMX++jkAN9Lw=; b=MRsyUo7TRuOLDrNbcGq8PChxX31Kw89qVW9C1jqW1zDvgGWIIZTT7atFKn0EyNXE4u OMuuJecVDMgs6dte69XYZG42ahWccXXRDA9rUAw0f94AobWoCS5KXipYc/EHTEGraxcC oljBJob3qJ06dM9cfCIWMBjrN4u5COIECFZjjpjis7wgfPtibhdnCdyufoAgNoE4bNVW phCVrVicBCwTgzzpfcXF4pk+GuDYMKLhk73mxUD4vh2tOpxghZB1R9n2VuT7zvrrpMQv OBe1X+i0UnTXgt8VlGO44bhXIdqtkJOnCuIufir4J35ocp+Hsqso9aie5UA9Gy+eOeLZ 5EVA== X-Gm-Message-State: AOJu0YyNymKh5nTAMVSLPhR+sVbHfZ3U4FC0kDDDCv7FbYelwXeF32Mv LYUaiUUgEBGVsuwsdjehfwyXM5RpZ+YjZGIRIbIXsBnZm9T4ljUn/NHpNm/niVAIl19gTAwyuhK g X-Google-Smtp-Source: AGHT+IF3WtO0Sc5zsxVsglsGHO9wcXI3/fpw7gyGZyu4K9u/5nLIXFcrbvKV8W0YqPCdr2fBf/JFDw== X-Received: by 2002:a05:6a20:6a21:b0:1ae:4264:a324 with SMTP id adf61e73a8af0-1b26f0f1b35mr5474654637.11.1717244857019; Sat, 01 Jun 2024 05:27:37 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242b30013sm2809362b3a.211.2024.06.01.05.27.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 01 Jun 2024 05:27:36 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 5/8] runqueue: Improve rehash get_unihash parallelism Date: Sat, 1 Jun 2024 05:27:20 -0700 Message-Id: <37a94ab4ebdb34463394d714ca0b349547392b41.1717244760.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 01 Jun 2024 12:27:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16309 From: Richard Purdie Improve the rehash code to query unihashes in parallel since this is more efficient on slower links. Signed-off-by: Richard Purdie --- lib/bb/runqueue.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 999868dd7..47f48304e 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2578,13 +2578,21 @@ class RunQueueExecute: while next: current = next.copy() next = set() + ready = {} for tid in current: if self.rqdata.runtaskentries[p].depends and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total): continue + # get_taskhash for a given tid *must* be called before get_unihash* below + ready[tid] = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches) + + unihashes = bb.parse.siggen.get_unihashes(ready.keys()) + + for tid in ready: orighash = self.rqdata.runtaskentries[tid].hash - newhash = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches) + newhash = ready[tid] origuni = self.rqdata.runtaskentries[tid].unihash - newuni = bb.parse.siggen.get_unihash(tid) + newuni = unihashes[tid] + # FIXME, need to check it can come from sstate at all for determinism? remapped = False if newuni == origuni: From patchwork Sat Jun 1 12:27:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 44498 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 210BFC27C44 for ; Sat, 1 Jun 2024 12:27:47 +0000 (UTC) Received: from mail-il1-f176.google.com (mail-il1-f176.google.com [209.85.166.176]) by mx.groups.io with SMTP id smtpd.web10.36212.1717244859485399346 for ; Sat, 01 Jun 2024 05:27:39 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=wIASLQew; spf=softfail (domain: sakoman.com, ip: 209.85.166.176, mailfrom: steve@sakoman.com) Received: by mail-il1-f176.google.com with SMTP id e9e14a558f8ab-37492089404so2023265ab.0 for ; Sat, 01 Jun 2024 05:27:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1717244858; x=1717849658; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=FmLPh5lZGEas9O753DK7UraYdzUmvqvthmtRqxqNArc=; b=wIASLQewvMCU+lp7UqphCLgyLe2Hbr+k+41BeECxdDB197zVId1yRJuPPvPOgcf0RS xSUzlneHu7PYNKwGX3PO2eftPsqj7R+kHfmoafy4RcwsLPylyqDKArdfAzeNcrWPPtyN Va6d77JqeuEbsAu1zhGYZAm4jOR5QWfGo7t1Wq2ymTmSAH1CGf9QRGN6QaHsyqq9sjy+ Numi+H5RdKE772Vf08JP2iTFT3SmbGuqJySpyoZVAsfPhPcx26jv77Xn2csHkuC6R+Co ZXRWkkjjhX6KWOScf+zwPnHaMOf/0QoDYBHS+af57RrB8ZNczAyp+sCi88Up/barWOs9 V5MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717244858; x=1717849658; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FmLPh5lZGEas9O753DK7UraYdzUmvqvthmtRqxqNArc=; b=TLJbYndb5CqlYXmmAjiO2F4PKuVbjNxIBLBDCT90KdpSHWU6WhpxGglw0dq/gr/lkN VmtMY1g93P0PbWkFapwVjo6rE4MmsJ1VZPF1wgbNOw8LmFQxq6YojMfM69Uoj3mLe8Wp 9dBuesoxk6E2UT+KuvD8ETU8A6Yu3lZe4RSsK5wLRMUton/aZbuGyhGapt9ZV6J7UKw0 oZaRHJTvmLF/GoXbawsGT2pHIMm5pYbhN5qSqmTYZpyXGTHz284zk5shM2hSiT9cra9m E4JyN3209bZuASQDxfe94azZ5Q7K3IGBVwl+8IOuzS9GLW0t920yX/8SRMpANH/xaK7G QVpg== X-Gm-Message-State: AOJu0YwoH+hHbMU/Ed9obhOQIGAcLClpqIE8IzFkZf/JwScpnitwoeig /c/GfUqbkJ87Hh7jiulBnZmTifmOuom7fqhOjT2MAhTZWPROZvj7npBb8Tx6PgrtmJ4p4HwQHnD A X-Google-Smtp-Source: AGHT+IE7Ft13xcbbTXEzCroLX5SqrRwLxgJO4wFQHdOWIrHCuJ3+isRaX9vQT7PFMiJBiaIC2s+6Xg== X-Received: by 2002:a05:6602:168d:b0:7d5:c00a:7d30 with SMTP id ca18e2360f4ac-7eaffe8cae7mr558673039f.8.1717244858263; Sat, 01 Jun 2024 05:27:38 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242b30013sm2809362b3a.211.2024.06.01.05.27.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 01 Jun 2024 05:27:38 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 6/8] bb: Use namedtuple for Task data Date: Sat, 1 Jun 2024 05:27:21 -0700 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 01 Jun 2024 12:27:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16310 From: Joshua Watt Task dependency data is becoming unwieldy with the number of indices it contains. Convert it to use a named tuple instead, which allows members to be indexed by a named property or an index (which allows it to retain backward compatibility). Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie --- lib/bb/__init__.py | 12 ++++++++++++ lib/bb/runqueue.py | 45 +++++++++++++++++++++++++++------------------ 2 files changed, 39 insertions(+), 18 deletions(-) diff --git a/lib/bb/__init__.py b/lib/bb/__init__.py index eef45fe4e..cdec9e4d6 100644 --- a/lib/bb/__init__.py +++ b/lib/bb/__init__.py @@ -36,6 +36,7 @@ class BBHandledException(Exception): import os import logging +from collections import namedtuple class NullHandler(logging.Handler): @@ -227,3 +228,14 @@ def deprecate_import(current, modulename, fromlist, renames = None): setattr(sys.modules[current], newname, newobj) +TaskData = namedtuple("TaskData", [ + "pn", + "taskname", + "fn", + "deps", + "provides", + "taskhash", + "unihash", + "hashfn", + "taskhash_deps", +]) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 47f48304e..93079a977 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -2452,15 +2452,17 @@ class RunQueueExecute: taskdepdata_cache = {} for task in self.rqdata.runtaskentries: (mc, fn, taskname, taskfn) = split_tid_mcfn(task) - pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn] - deps = self.rqdata.runtaskentries[task].depends - provides = self.rqdata.dataCaches[mc].fn_provides[taskfn] - taskhash = self.rqdata.runtaskentries[task].hash - unihash = self.rqdata.runtaskentries[task].unihash - deps = self.filtermcdeps(task, mc, deps) - hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn] - taskhash_deps = self.rqdata.runtaskentries[task].taskhash_deps - taskdepdata_cache[task] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn, taskhash_deps] + taskdepdata_cache[task] = bb.TaskData( + pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn], + taskname = taskname, + fn = fn, + deps = self.filtermcdeps(task, mc, self.rqdata.runtaskentries[task].depends), + provides = self.rqdata.dataCaches[mc].fn_provides[taskfn], + taskhash = self.rqdata.runtaskentries[task].hash, + unihash = self.rqdata.runtaskentries[task].unihash, + hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn], + taskhash_deps = self.rqdata.runtaskentries[task].taskhash_deps, + ) self.taskdepdata_cache = taskdepdata_cache @@ -2475,9 +2477,11 @@ class RunQueueExecute: while next: additional = [] for revdep in next: - self.taskdepdata_cache[revdep][6] = self.rqdata.runtaskentries[revdep].unihash + self.taskdepdata_cache[revdep] = self.taskdepdata_cache[revdep]._replace( + unihash=self.rqdata.runtaskentries[revdep].unihash + ) taskdepdata[revdep] = self.taskdepdata_cache[revdep] - for revdep2 in self.taskdepdata_cache[revdep][3]: + for revdep2 in self.taskdepdata_cache[revdep].deps: if revdep2 not in taskdepdata: additional.append(revdep2) next = additional @@ -2841,14 +2845,19 @@ class RunQueueExecute: additional = [] for revdep in next: (mc, fn, taskname, taskfn) = split_tid_mcfn(revdep) - pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn] deps = getsetscenedeps(revdep) - provides = self.rqdata.dataCaches[mc].fn_provides[taskfn] - taskhash = self.rqdata.runtaskentries[revdep].hash - unihash = self.rqdata.runtaskentries[revdep].unihash - hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn] - taskhash_deps = self.rqdata.runtaskentries[revdep].taskhash_deps - taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn, taskhash_deps] + + taskdepdata[revdep] = bb.TaskData( + pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn], + taskname = taskname, + fn = fn, + deps = deps, + provides = self.rqdata.dataCaches[mc].fn_provides[taskfn], + taskhash = self.rqdata.runtaskentries[revdep].hash, + unihash = self.rqdata.runtaskentries[revdep].unihash, + hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn], + taskhash_deps = self.rqdata.runtaskentries[revdep].taskhash_deps, + ) for revdep2 in deps: if revdep2 not in taskdepdata: additional.append(revdep2) From patchwork Sat Jun 1 12:27:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 44496 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07DC1C41513 for ; Sat, 1 Jun 2024 12:27:47 +0000 (UTC) Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by mx.groups.io with SMTP id smtpd.web10.36213.1717244860558195149 for ; Sat, 01 Jun 2024 05:27:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=zpo1o0UN; spf=softfail (domain: sakoman.com, ip: 209.85.210.179, mailfrom: steve@sakoman.com) Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-7025b84c0daso175785b3a.2 for ; Sat, 01 Jun 2024 05:27:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1717244860; x=1717849660; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=VrBUtbmTzASjMhuNVF9aO5s2VrhJltfN2ABj7IPmhaE=; b=zpo1o0UNxvuZOiEu+k1wA+9dfywxi3hYmH+oPFtGrYkYoSE+A88Py1uXVYWTXLBN0Z nagcPr/PYx8oXzYn8LW759WhnZmc3AFZwvLePtpnXfUVZdgrWUsss1kNVmyR88ur73/o 1jyYAgiSqvlmqvyiaFDtxZYJ2LY+qjQDnOSQ10v+0o+b7iTg/aNXVRTS+O3M4LsZlq+j VpoBetfs30maf/y2CnxpxJWhoEsnP6eD9SC+6JYr2ANFD7mRz3f2Js9jqlaWWNTauM3S QqkBkj/tTg2MwEBN6KyPQUl0bFnwKQ7vTKJXcwloD27uId+MYrOFMIKzNJUTjP0YWW0Z vKXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717244860; x=1717849660; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VrBUtbmTzASjMhuNVF9aO5s2VrhJltfN2ABj7IPmhaE=; b=V4tCXjZvv9oDKWYTyMLs7BhuyyFMfgMeW/XeVXJMW88GcmtqIiaav37E8CvZM1Z/eG vkyERKYjC9D13lBeN5DFPb39KjYvJAlS3v5pvNmBCM0C/zkVu84ADmEqKmkYYjy9DFO0 z0w+oAfBlSPMK7Rqfa6oGykOMR4QHbx2iExAj+0NvjDONPoMg5o9r3UDj7dECD1pf1/e paT8ZYi4mwa/S+jdgt+rXFydxotCg1O/msLguUGiynVVUCyy2GrWuBtc11p/vfauJFRk bucdBMLXqqUGI0cDHQ0v7UGfitMwIML8N/JUfMP+BvQqobUslNHtAQmmevlJrO6UsHdG fl5w== X-Gm-Message-State: AOJu0Yx2vuhgzgVjItaDGTdz5YLu02k1XYAZejuIrS79WjMwwc4EBNH6 Cj4vs3CHHKIx7IShPi5Zgt6IWkUnUXX9uQlNspbsfWn0nAYb0L7B3Jpj+mhQDOKeo2P5UdVKVXi 7 X-Google-Smtp-Source: AGHT+IHk33wDgGc1XhupvsjFI3Gc+xJulfC0ekqTXWtK8zVhNwIL9PvHcvXiJYeDMcXH9APWFAB5FA== X-Received: by 2002:a05:6a00:1a91:b0:702:2749:6097 with SMTP id d2e1a72fcca58-702477bc078mr4574617b3a.1.1717244859593; Sat, 01 Jun 2024 05:27:39 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242b30013sm2809362b3a.211.2024.06.01.05.27.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 01 Jun 2024 05:27:39 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 7/8] hashserv: client: Add batch stream API Date: Sat, 1 Jun 2024 05:27:22 -0700 Message-Id: <3599293793816a7c5c9d0f296fb4bb96ea266f8d.1717244760.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 01 Jun 2024 12:27:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16311 From: Joshua Watt Changes the stream mode to do "batch" processing. This means that the sending and reciving of messages is done simultaneously so that messages can be sent as fast as possible without having to wait for each reply. This allows multiple messages to be in flight at once, reducing the effect of the round trip latency from the server. Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie --- lib/hashserv/client.py | 106 +++++++++++++++++++++++++++++++++++++---- lib/hashserv/tests.py | 75 +++++++++++++++++++++++++++++ 2 files changed, 172 insertions(+), 9 deletions(-) diff --git a/lib/hashserv/client.py b/lib/hashserv/client.py index 0b254bedd..775faf935 100644 --- a/lib/hashserv/client.py +++ b/lib/hashserv/client.py @@ -5,6 +5,7 @@ import logging import socket +import asyncio import bb.asyncrpc import json from . import create_async_client @@ -13,6 +14,66 @@ from . import create_async_client logger = logging.getLogger("hashserv.client") +class Batch(object): + def __init__(self): + self.done = False + self.cond = asyncio.Condition() + self.pending = [] + self.results = [] + self.sent_count = 0 + + async def recv(self, socket): + while True: + async with self.cond: + await self.cond.wait_for(lambda: self.pending or self.done) + + if not self.pending: + if self.done: + return + continue + + r = await socket.recv() + self.results.append(r) + + async with self.cond: + self.pending.pop(0) + + async def send(self, socket, msgs): + try: + # In the event of a restart due to a reconnect, all in-flight + # messages need to be resent first to keep to result count in sync + for m in self.pending: + await socket.send(m) + + for m in msgs: + # Add the message to the pending list before attempting to send + # it so that if the send fails it will be retried + async with self.cond: + self.pending.append(m) + self.cond.notify() + self.sent_count += 1 + + await socket.send(m) + + finally: + async with self.cond: + self.done = True + self.cond.notify() + + async def process(self, socket, msgs): + await asyncio.gather( + self.recv(socket), + self.send(socket, msgs), + ) + + if len(self.results) != self.sent_count: + raise ValueError( + f"Expected result count {len(self.results)}. Expected {self.sent_count}" + ) + + return self.results + + class AsyncClient(bb.asyncrpc.AsyncClient): MODE_NORMAL = 0 MODE_GET_STREAM = 1 @@ -36,11 +97,27 @@ class AsyncClient(bb.asyncrpc.AsyncClient): if become: await self.become_user(become) - async def send_stream(self, mode, msg): + async def send_stream_batch(self, mode, msgs): + """ + Does a "batch" process of stream messages. This sends the query + messages as fast as possible, and simultaneously attempts to read the + messages back. This helps to mitigate the effects of latency to the + hash equivalence server be allowing multiple queries to be "in-flight" + at once + + The implementation does more complicated tracking using a count of sent + messages so that `msgs` can be a generator function (i.e. its length is + unknown) + + """ + + b = Batch() + async def proc(): + nonlocal b + await self._set_mode(mode) - await self.socket.send(msg) - return await self.socket.recv() + return await b.process(self.socket, msgs) return await self._send_wrapper(proc) @@ -89,10 +166,15 @@ class AsyncClient(bb.asyncrpc.AsyncClient): self.mode = new_mode async def get_unihash(self, method, taskhash): - r = await self.send_stream(self.MODE_GET_STREAM, "%s %s" % (method, taskhash)) - if not r: - return None - return r + r = await self.get_unihash_batch([(method, taskhash)]) + return r[0] + + async def get_unihash_batch(self, args): + result = await self.send_stream_batch( + self.MODE_GET_STREAM, + (f"{method} {taskhash}" for method, taskhash in args), + ) + return [r if r else None for r in result] async def report_unihash(self, taskhash, method, outhash, unihash, extra={}): m = extra.copy() @@ -115,8 +197,12 @@ class AsyncClient(bb.asyncrpc.AsyncClient): ) async def unihash_exists(self, unihash): - r = await self.send_stream(self.MODE_EXIST_STREAM, unihash) - return r == "true" + r = await self.unihash_exists_batch([unihash]) + return r[0] + + async def unihash_exists_batch(self, unihashes): + result = await self.send_stream_batch(self.MODE_EXIST_STREAM, unihashes) + return [r == "true" for r in result] async def get_outhash(self, method, outhash, taskhash, with_unihash=True): return await self.invoke( @@ -237,10 +323,12 @@ class Client(bb.asyncrpc.Client): "connect_tcp", "connect_websocket", "get_unihash", + "get_unihash_batch", "report_unihash", "report_unihash_equiv", "get_taskhash", "unihash_exists", + "unihash_exists_batch", "get_outhash", "get_stats", "reset_stats", diff --git a/lib/hashserv/tests.py b/lib/hashserv/tests.py index 0809453cf..5349cd586 100644 --- a/lib/hashserv/tests.py +++ b/lib/hashserv/tests.py @@ -594,6 +594,43 @@ class HashEquivalenceCommonTests(object): 7: None, }) + def test_get_unihash_batch(self): + TEST_INPUT = ( + # taskhash outhash unihash + ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98512d0105985dffc9'), + # Duplicated taskhash with multiple output hashes and unihashes. + ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'), + # Equivalent hash + ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e816454ed46894bb3aab36"), + ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb422c855286e18a57d"), + ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2cd'), + ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2ce'), + ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca677e8ad852365c538'), + ) + EXTRA_QUERIES = ( + "6b6be7a84ab179b4240c4302518dc3f6", + ) + + for taskhash, outhash, unihash in TEST_INPUT: + self.client.report_unihash(taskhash, self.METHOD, outhash, unihash) + + + result = self.client.get_unihash_batch( + [(self.METHOD, data[0]) for data in TEST_INPUT] + + [(self.METHOD, e) for e in EXTRA_QUERIES] + ) + + self.assertListEqual(result, [ + "218e57509998197d570e2c98512d0105985dffc9", + "218e57509998197d570e2c98512d0105985dffc9", + "218e57509998197d570e2c98512d0105985dffc9", + "3b5d3d83f07f259e9086fcb422c855286e18a57d", + "f46d3fbb439bd9b921095da657a4de906510d2cd", + "f46d3fbb439bd9b921095da657a4de906510d2cd", + "05d2a63c81e32f0a36542ca677e8ad852365c538", + None, + ]) + def test_client_pool_unihash_exists(self): TEST_INPUT = ( # taskhash outhash unihash @@ -636,6 +673,44 @@ class HashEquivalenceCommonTests(object): result = client_pool.unihashes_exist(query) self.assertDictEqual(result, expected) + def test_unihash_exists_batch(self): + TEST_INPUT = ( + # taskhash outhash unihash + ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98512d0105985dffc9'), + # Duplicated taskhash with multiple output hashes and unihashes. + ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'), + # Equivalent hash + ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e816454ed46894bb3aab36"), + ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb422c855286e18a57d"), + ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2cd'), + ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2ce'), + ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca677e8ad852365c538'), + ) + EXTRA_QUERIES = ( + "6b6be7a84ab179b4240c4302518dc3f6", + ) + + result_unihashes = set() + + + for taskhash, outhash, unihash in TEST_INPUT: + result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash) + result_unihashes.add(result["unihash"]) + + query = [] + expected = [] + + for _, _, unihash in TEST_INPUT: + query.append(unihash) + expected.append(unihash in result_unihashes) + + + for unihash in EXTRA_QUERIES: + query.append(unihash) + expected.append(False) + + result = self.client.unihash_exists_batch(query) + self.assertListEqual(result, expected) def test_auth_read_perms(self): admin_client = self.start_auth_server() From patchwork Sat Jun 1 12:27:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 44497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13027C27C4F for ; Sat, 1 Jun 2024 12:27:47 +0000 (UTC) Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by mx.groups.io with SMTP id smtpd.web11.36154.1717244861963294790 for ; Sat, 01 Jun 2024 05:27:42 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=G8mBULPH; spf=softfail (domain: sakoman.com, ip: 209.85.210.173, mailfrom: steve@sakoman.com) Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-702548b056aso547584b3a.1 for ; Sat, 01 Jun 2024 05:27:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1717244861; x=1717849661; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=LRuJsD8VW+HVILuMIQ834MAEvuQQvTJju5ut59L8ymg=; b=G8mBULPHLi/qrtRn/Px1GMTutRP/vRBCktqC06oA83kwoKvrnuTEvF2TSEp/WFGS+R ExXa8Tk1568naRBVqgMye2oO4pYteB2l4QvkOuR5WT78ggaHnUslF9Tdsh5pHbw48FLR vlRZPRcUXf+NJpW3vwNSFeL4TmFOgckwvzTXF1AzQT6rHVgP2n9scXfjWrTU+BdfV2sG pQpY2SzELFHlIK/6toW1noo5ejeMpz+Z1LR0jiefi3TxEa9x3rWQfVOuKtV5aUT71Pj7 KHvInwEHvL1zEYPCr1QTC6TWf3rRVRXUDTlbmbtuZeNPvsFbhZ8rVPm+M8HSA9rdmLO5 +P8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717244861; x=1717849661; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LRuJsD8VW+HVILuMIQ834MAEvuQQvTJju5ut59L8ymg=; b=ISTGOBxPmKHqzw4Lr3lpSxJsWZhu/M5n8CUluk4dqLn4yhDtJ9gGBTAzAAapQNYCOz bq0JiZ2HNzER77ltEyPx3cvfuY5xG7OY16A34S/NaOj4eib1T8eDQbFNZywTidDPn9Oe psHu1oq3YABj0IB87w0PvP+I+bKOHj/4aQBHMM5CV3+iOIz2TlBF8MAoxDBh8dCJ+z7Y eQOEIHNJk8dj5aR29mEhtr4PAUbaYPczKCpLeF0rY018JG7ybv+lpqA6rVtgjKWEizdz ElETTL2UDIX9lnG2moJ9OAzfi2Ka40H0j+pzXOX7CwDQrO4aaCpTzw/+MEAgrkR2IyHA 5Yaw== X-Gm-Message-State: AOJu0Yxqbxz3NAaBmyru7EiLybppLT31k8j6vnOzycE0rRpcy2Hn5RRg 7Fzby7rlHpwlfm6lS2h99Y6P5GICLhIdgBpKbCV37j4lnCQ9df3cc/VwNQYkJG0N/N8+PaAJEzG U X-Google-Smtp-Source: AGHT+IFq30s0C2ecMhfwb9DawdoSzqr38WhqQig7PJ6Smf4qIOE7nDpzpScIhu7RDudcspjv111n0A== X-Received: by 2002:a05:6a20:96ca:b0:1b1:f6a9:6b0a with SMTP id adf61e73a8af0-1b26f275a5fmr3959907637.47.1717244860835; Sat, 01 Jun 2024 05:27:40 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242b30013sm2809362b3a.211.2024.06.01.05.27.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 01 Jun 2024 05:27:40 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 8/8] siggen: Enable batching of unihash queries Date: Sat, 1 Jun 2024 05:27:23 -0700 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 01 Jun 2024 12:27:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16312 From: Joshua Watt Uses the batching API of the client to reduce the effect of latency when making multiple queries to the server Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie --- lib/bb/siggen.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 03dfda6f3..65ca0811d 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -726,10 +726,13 @@ class SignatureGeneratorUniHashMixIn(object): return result if self.max_parallel <= 1 or len(queries) <= 1: - # No parallelism required. Make the query serially with the single client + # No parallelism required. Make the query using a single client with self.client() as client: - for tid, args in queries.items(): - query_result[tid] = client.get_unihash(*args) + keys = list(queries.keys()) + unihashes = client.get_unihash_batch(queries[k] for k in keys) + + for idx, k in enumerate(keys): + query_result[k] = unihashes[idx] else: with self.client_pool() as client_pool: query_result = client_pool.get_unihashes(queries)