From patchwork Thu May 30 15:41:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 44405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6810BC41513 for ; Thu, 30 May 2024 15:41:37 +0000 (UTC) Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) by mx.groups.io with SMTP id smtpd.web10.13086.1717083695713565671 for ; Thu, 30 May 2024 08:41:35 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=ceiosMOa; spf=pass (domain: gmail.com, ip: 209.85.161.43, mailfrom: jpewhacker@gmail.com) Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-5b2d065559cso446987eaf.3 for ; Thu, 30 May 2024 08:41:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717083694; x=1717688494; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=WsJOBQXRdmDN5AQEbaM+NDrwk7AMcOzT3uFdplwXzjM=; b=ceiosMOaT5gKAMZjCkf7DGabFGyMvzzao1wtNs0q2lI7CZOu+sqODtsbiHzgqlfs71 Kac9sJTCuGYL8Y/1DoJdjSYkGfwKTaZifUBbZ9RnLfqXhChh2pSk15szc7cSm6R2OJiB Asv4EolK5wHRG+IHCCsPbl3SRyBGibuKeqlizIyut+wP8bvIvp0Tn1uRwRjBBnkHiCK5 47do2A6qz7AxEug+jlrxIx/IA8eDla2ugWPy3ZjSG3IUQx25w3YTQDQlzDZpsgZTUok0 5zk7HIztwzyJBx+aKrH6t6/0nBZOulXJ8oKr1q/FRVMFpXc42sqWlRp0ltg17kaygtGJ tFww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717083694; x=1717688494; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WsJOBQXRdmDN5AQEbaM+NDrwk7AMcOzT3uFdplwXzjM=; b=DEqZPm5WprBm8ur2rbsvR2Jfq8IEcZPjGuXXb2PN5+tBdcxSo/BhYH16/LqS/zhTSg JpZom9cfCPSZ8oJDAJgiUlWJeaJPP/o+nDLLvAxG5+so2jidbmMJH6e+6u2VLqV8GSoi EffLYL5+Wt/r1vS2Az9BtH1JB1ocI6s+Aecx/ykqQADcj1CJyCpQDCgW8tsx9hnHHy5H 1JsdWjOUvvoCTqM7/Jwn0/gP0Gwi/JXkQQ6TzzMJnFBJynvZ06Tro0kn/BpIeEgiHnSy L/cEftEe3MDonxu4hlu4EocZrw86mzKe9UKM/2d8+aEkGqKafH7ybLgff6B5D4oyD8+e VxAA== X-Gm-Message-State: AOJu0YyGlk2HHnDjqUU5pK3jtmKUg6byQYchdEj2KlWJokXHJPQUuHcG Ew/QdUk1IcwBJ+gJxjyS0LWdwv+XL/Iy/i7ipnNZEl5Rct2AJAoiXo1fbA== X-Google-Smtp-Source: AGHT+IHCOvP8BKijKFp68sLP+FxT8ohqiHGzldRHdPyzhlhKGHW6TgBhEWFh0Esxd5L34fwJk4NI0A== X-Received: by 2002:a05:6820:1b90:b0:5b2:565b:b6de with SMTP id 006d021491bc7-5b9ec5bcee2mr2862539eaf.3.1717083693830; Thu, 30 May 2024 08:41:33 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::125b]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5b96c6d60f1sm3184813eaf.45.2024.05.30.08.41.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 May 2024 08:41:33 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: bitbake-devel@lists.openembedded.org Cc: Joshua Watt Subject: [bitbake-devel][PATCH 1/3] siggen: Drop client pool support Date: Thu, 30 May 2024 09:41:25 -0600 Message-ID: <20240530154127.792582-1-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.43.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 30 May 2024 15:41:37 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16289 Drops support for client pools, since batching support in the client code has proven to be much more effective Signed-off-by: Joshua Watt --- bitbake/lib/bb/siggen.py | 53 ++++++++++++---------------------------- 1 file changed, 15 insertions(+), 38 deletions(-) diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index 65ca0811d58..79f347db303 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -540,7 +540,7 @@ class SignatureGeneratorUniHashMixIn(object): def __init__(self, data): self.extramethod = {} # NOTE: The cache only tracks hashes that exist. Hashes that don't - # exist are always queries from the server since it is possible for + # exist are always queried from the server since it is possible for # hashes to appear over time, but much less likely for them to # disappear self.unihash_exists_cache = set() @@ -558,11 +558,11 @@ class SignatureGeneratorUniHashMixIn(object): super().__init__(data) def get_taskdata(self): - return (self.server, self.method, self.extramethod, self.max_parallel, self.username, self.password, self.env) + super().get_taskdata() + return (self.server, self.method, self.extramethod, self.username, self.password, self.env) + super().get_taskdata() def set_taskdata(self, data): - self.server, self.method, self.extramethod, self.max_parallel, self.username, self.password, self.env = data[:7] - super().set_taskdata(data[7:]) + self.server, self.method, self.extramethod, self.username, self.password, self.env = data[:6] + super().set_taskdata(data[6:]) def get_hashserv_creds(self): if self.username and self.password: @@ -595,13 +595,6 @@ class SignatureGeneratorUniHashMixIn(object): self._client = hashserv.create_client(self.server, **self.get_hashserv_creds()) yield self._client - @contextmanager - def client_pool(self): - with self._client_env(): - if getattr(self, '_client_pool', None) is None: - self._client_pool = hashserv.client.ClientPool(self.server, self.max_parallel, **self.get_hashserv_creds()) - yield self._client_pool - def reset(self, data): self.__close_clients() return super().reset(data) @@ -686,15 +679,10 @@ class SignatureGeneratorUniHashMixIn(object): else: uncached_query[key] = unihash - if self.max_parallel <= 1 or len(uncached_query) <= 1: - # No parallelism required. Make the query serially with the single client - with self.client() as client: - uncached_result = { - key: client.unihash_exists(value) for key, value in uncached_query.items() - } - else: - with self.client_pool() as client_pool: - uncached_result = client_pool.unihashes_exist(uncached_query) + with self.client() as client: + uncached_result = { + key: client.unihash_exists(value) for key, value in uncached_query.items() + } for key, exists in uncached_result.items(): if exists: @@ -712,32 +700,20 @@ class SignatureGeneratorUniHashMixIn(object): unihash """ result = {} - queries = {} - query_result = {} + query_tids = [] for tid in tids: unihash = self.get_cached_unihash(tid) if unihash: result[tid] = unihash else: - queries[tid] = (self._get_method(tid), self.taskhash[tid]) - - if len(queries) == 0: - return result + query_tids.append(tid) - if self.max_parallel <= 1 or len(queries) <= 1: - # No parallelism required. Make the query using a single client + if query_tids: with self.client() as client: - keys = list(queries.keys()) - unihashes = client.get_unihash_batch(queries[k] for k in keys) + unihashes = client.get_unihash_batch((self._get_method(tid), self.taskhash[tid]) for tid in query_tids) - for idx, k in enumerate(keys): - query_result[k] = unihashes[idx] - else: - with self.client_pool() as client_pool: - query_result = client_pool.get_unihashes(queries) - - for tid, unihash in query_result.items(): + for idx, tid in enumerate(query_tids): # In the absence of being able to discover a unique hash from the # server, make it be equivalent to the taskhash. The unique "hash" only # really needs to be a unique string (not even necessarily a hash), but @@ -752,6 +728,8 @@ class SignatureGeneratorUniHashMixIn(object): # to the server, there is a better chance that they will agree on # the unique hash. taskhash = self.taskhash[tid] + unihash = unihashes[idx] + if unihash: # A unique hash equal to the taskhash is not very interesting, # so it is reported it at debug level 2. If they differ, that @@ -898,7 +876,6 @@ class SignatureGeneratorTestEquivHash(SignatureGeneratorUniHashMixIn, SignatureG super().init_rundepcheck(data) self.server = data.getVar('BB_HASHSERVE') self.method = "sstate_output_hash" - self.max_parallel = 1 def clean_checksum_file_path(file_checksum_tuple): f, cs = file_checksum_tuple From patchwork Thu May 30 15:41:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 44404 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 661B0C27C43 for ; Thu, 30 May 2024 15:41:37 +0000 (UTC) Received: from mail-oo1-f54.google.com (mail-oo1-f54.google.com [209.85.161.54]) by mx.groups.io with SMTP id smtpd.web11.12851.1717083696724468519 for ; Thu, 30 May 2024 08:41:36 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=W2VZyJ21; spf=pass (domain: gmail.com, ip: 209.85.161.54, mailfrom: jpewhacker@gmail.com) Received: by mail-oo1-f54.google.com with SMTP id 006d021491bc7-5b9776123a3so508125eaf.0 for ; Thu, 30 May 2024 08:41:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717083695; x=1717688495; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iRkSUlybHhZzSokGJQV+ZwFY5HQKXm8VcUurYZHK+ZE=; b=W2VZyJ21Dkm0/JgI2YgxeM5Pvopb6ZlvLtEshPQ30C6GFXySy/oVUIvWmlZOgmP4DT 3Y+Z8j82a2GB66Q3DHGSNPgGTsRxDn1MpXgqiFchP5DdY7jxklu1tI42LxNoqwsec2l6 +EXUbQNz5m0WRTFfGXqoDBitmstjLqUqRhBTVlzIUqUaMPnXFTUkz1lh4t19VHXFXFFy FaR+tsIKTLqFs/A33iNj/T+4CkZL4fSlwcSpUDS2LnrDlQyrzaPOXCeWpTDG6/9tJF0d Caf/ckEgpTCCvbzTfGz17QkmZL59ansWoxutZRjNB1CQXojXC/DuwceVAmyTfivwbG82 s04w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717083695; x=1717688495; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iRkSUlybHhZzSokGJQV+ZwFY5HQKXm8VcUurYZHK+ZE=; b=m+LL2JcIw2v1uHRQqkQopnUuzK2PWdImRoAx73iYTXDMK+HqOoS5Re8zVSMVbAptyB RzDOdeAW9ipmsz/HOAqbfsNvv+Ge0uJCmDIN36Z3tgr8IA+yOkvQgN/Bz6HNT5sveYTg DtBkTCgwX++3tdpyzifZaNPDSnbnXaIH/OxL71CpytO4SqBBRMeeZn88q58RiRI7Yr/E urP0P+HCT453zPdU3kWHnfagQ1IcesJEbvAC6VHgpii2a6avOHuMvDKRhzmxXryn20g6 wyuW/0d/9AVmIr7A9F8wBQjRbGYGYIkmVzcb+EP6ZphjZeEnpp0M93BiSOwHhz6F1Ot9 TDmw== X-Gm-Message-State: AOJu0YwolfxEZnZqTMD5U5W9m5ZnwbQtZoYwbToOsWYt6LBCw0mTMwRP 52qrvadvco5TgJofc1IezISMvR/j6VH03dBbJGL+SAA4CiNjAIIiaY1srw== X-Google-Smtp-Source: AGHT+IH2iJZO7Z1wlzLq+Zmj9qbIMeZHG6exEDtua1JdCT9VaGRqIRPNlVUKSzmtt5Nxj3fxTzxuQw== X-Received: by 2002:a05:6820:1b8c:b0:5b9:8a06:c47 with SMTP id 006d021491bc7-5b9ec83b401mr2817982eaf.8.1717083694793; Thu, 30 May 2024 08:41:34 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::125b]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5b96c6d60f1sm3184813eaf.45.2024.05.30.08.41.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 May 2024 08:41:34 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: bitbake-devel@lists.openembedded.org Cc: Joshua Watt Subject: [bitbake-devel][PATCH 2/3] asyncrpc: Remove ClientPool Date: Thu, 30 May 2024 09:41:26 -0600 Message-ID: <20240530154127.792582-2-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240530154127.792582-1-JPEWhacker@gmail.com> References: <20240530154127.792582-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 30 May 2024 15:41:37 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16290 Batching support on the client side has proven to be a much more effective way of dealing with server latency than multiple client connections and is also much nicer on the server, so drop the client pool support from asyncrpc and the hash server Signed-off-by: Joshua Watt --- bitbake/lib/bb/asyncrpc/__init__.py | 2 +- bitbake/lib/bb/asyncrpc/client.py | 76 +------------------------- bitbake/lib/hashserv/client.py | 80 ---------------------------- bitbake/lib/hashserv/tests.py | 82 ----------------------------- 4 files changed, 2 insertions(+), 238 deletions(-) diff --git a/bitbake/lib/bb/asyncrpc/__init__.py b/bitbake/lib/bb/asyncrpc/__init__.py index 639e1607f8e..a4371643d74 100644 --- a/bitbake/lib/bb/asyncrpc/__init__.py +++ b/bitbake/lib/bb/asyncrpc/__init__.py @@ -5,7 +5,7 @@ # -from .client import AsyncClient, Client, ClientPool +from .client import AsyncClient, Client from .serv import AsyncServer, AsyncServerConnection from .connection import DEFAULT_MAX_CHUNK from .exceptions import ( diff --git a/bitbake/lib/bb/asyncrpc/client.py b/bitbake/lib/bb/asyncrpc/client.py index b49de993131..746bbefdd87 100644 --- a/bitbake/lib/bb/asyncrpc/client.py +++ b/bitbake/lib/bb/asyncrpc/client.py @@ -29,6 +29,7 @@ WEBSOCKETS_MIN_VERSION = (9, 1) if sys.version_info >= (3, 10, 0): WEBSOCKETS_MIN_VERSION = (10, 0) + def parse_address(addr): if addr.startswith(UNIX_PREFIX): return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX) :],)) @@ -260,78 +261,3 @@ class Client(object): def __exit__(self, exc_type, exc_value, traceback): self.close() return False - - -class ClientPool(object): - def __init__(self, max_clients): - self.avail_clients = [] - self.num_clients = 0 - self.max_clients = max_clients - self.loop = None - self.client_condition = None - - @abc.abstractmethod - async def _new_client(self): - raise NotImplementedError("Must be implemented in derived class") - - def close(self): - if self.client_condition: - self.client_condition = None - - if self.loop: - self.loop.run_until_complete(self.__close_clients()) - self.loop.run_until_complete(self.loop.shutdown_asyncgens()) - self.loop.close() - self.loop = None - - def run_tasks(self, tasks): - if not self.loop: - self.loop = asyncio.new_event_loop() - - thread = Thread(target=self.__thread_main, args=(tasks,)) - thread.start() - thread.join() - - @contextlib.asynccontextmanager - async def get_client(self): - async with self.client_condition: - if self.avail_clients: - client = self.avail_clients.pop() - elif self.num_clients < self.max_clients: - self.num_clients += 1 - client = await self._new_client() - else: - while not self.avail_clients: - await self.client_condition.wait() - client = self.avail_clients.pop() - - try: - yield client - finally: - async with self.client_condition: - self.avail_clients.append(client) - self.client_condition.notify() - - def __thread_main(self, tasks): - async def process_task(task): - async with self.get_client() as client: - await task(client) - - asyncio.set_event_loop(self.loop) - if not self.client_condition: - self.client_condition = asyncio.Condition() - tasks = [process_task(t) for t in tasks] - self.loop.run_until_complete(asyncio.gather(*tasks)) - - async def __close_clients(self): - for c in self.avail_clients: - await c.close() - self.avail_clients = [] - self.num_clients = 0 - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc_value, traceback): - self.close() - return False diff --git a/bitbake/lib/hashserv/client.py b/bitbake/lib/hashserv/client.py index 775faf935a8..d415617b206 100644 --- a/bitbake/lib/hashserv/client.py +++ b/bitbake/lib/hashserv/client.py @@ -352,83 +352,3 @@ class Client(bb.asyncrpc.Client): def _get_async_client(self): return AsyncClient(self.username, self.password) - - -class ClientPool(bb.asyncrpc.ClientPool): - def __init__( - self, - address, - max_clients, - *, - username=None, - password=None, - become=None, - ): - super().__init__(max_clients) - self.address = address - self.username = username - self.password = password - self.become = become - - async def _new_client(self): - client = await create_async_client( - self.address, - username=self.username, - password=self.password, - ) - if self.become: - await client.become_user(self.become) - return client - - def _run_key_tasks(self, queries, call): - results = {key: None for key in queries.keys()} - - def make_task(key, args): - async def task(client): - nonlocal results - unihash = await call(client, args) - results[key] = unihash - - return task - - def gen_tasks(): - for key, args in queries.items(): - yield make_task(key, args) - - self.run_tasks(gen_tasks()) - return results - - def get_unihashes(self, queries): - """ - Query multiple unihashes in parallel. - - The queries argument is a dictionary with arbitrary key. The values - must be a tuple of (method, taskhash). - - Returns a dictionary with a corresponding key for each input key, and - the value is the queried unihash (which might be none if the query - failed) - """ - - async def call(client, args): - method, taskhash = args - return await client.get_unihash(method, taskhash) - - return self._run_key_tasks(queries, call) - - def unihashes_exist(self, queries): - """ - Query multiple unihash existence checks in parallel. - - The queries argument is a dictionary with arbitrary key. The values - must be a unihash. - - Returns a dictionary with a corresponding key for each input key, and - the value is True or False if the unihash is known by the server (or - None if there was a failure) - """ - - async def call(client, unihash): - return await client.unihash_exists(unihash) - - return self._run_key_tasks(queries, call) diff --git a/bitbake/lib/hashserv/tests.py b/bitbake/lib/hashserv/tests.py index 5349cd58677..43c3c36f045 100644 --- a/bitbake/lib/hashserv/tests.py +++ b/bitbake/lib/hashserv/tests.py @@ -8,7 +8,6 @@ from . import create_server, create_client from .server import DEFAULT_ANON_PERMS, ALL_PERMISSIONS from bb.asyncrpc import InvokeError -from .client import ClientPool import hashlib import logging import multiprocessing @@ -555,45 +554,6 @@ class HashEquivalenceCommonTests(object): # shares a taskhash with Task 2 self.assertClientGetHash(self.client, taskhash2, unihash2) - - def test_client_pool_get_unihashes(self): - TEST_INPUT = ( - # taskhash outhash unihash - ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98512d0105985dffc9'), - # Duplicated taskhash with multiple output hashes and unihashes. - ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'), - # Equivalent hash - ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e816454ed46894bb3aab36"), - ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb422c855286e18a57d"), - ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2cd'), - ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2ce'), - ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca677e8ad852365c538'), - ) - EXTRA_QUERIES = ( - "6b6be7a84ab179b4240c4302518dc3f6", - ) - - with ClientPool(self.server_address, 10) as client_pool: - for taskhash, outhash, unihash in TEST_INPUT: - self.client.report_unihash(taskhash, self.METHOD, outhash, unihash) - - query = {idx: (self.METHOD, data[0]) for idx, data in enumerate(TEST_INPUT)} - for idx, taskhash in enumerate(EXTRA_QUERIES): - query[idx + len(TEST_INPUT)] = (self.METHOD, taskhash) - - result = client_pool.get_unihashes(query) - - self.assertDictEqual(result, { - 0: "218e57509998197d570e2c98512d0105985dffc9", - 1: "218e57509998197d570e2c98512d0105985dffc9", - 2: "218e57509998197d570e2c98512d0105985dffc9", - 3: "3b5d3d83f07f259e9086fcb422c855286e18a57d", - 4: "f46d3fbb439bd9b921095da657a4de906510d2cd", - 5: "f46d3fbb439bd9b921095da657a4de906510d2cd", - 6: "05d2a63c81e32f0a36542ca677e8ad852365c538", - 7: None, - }) - def test_get_unihash_batch(self): TEST_INPUT = ( # taskhash outhash unihash @@ -631,48 +591,6 @@ class HashEquivalenceCommonTests(object): None, ]) - def test_client_pool_unihash_exists(self): - TEST_INPUT = ( - # taskhash outhash unihash - ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98512d0105985dffc9'), - # Duplicated taskhash with multiple output hashes and unihashes. - ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'), - # Equivalent hash - ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e816454ed46894bb3aab36"), - ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb422c855286e18a57d"), - ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2cd'), - ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2ce'), - ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca677e8ad852365c538'), - ) - EXTRA_QUERIES = ( - "6b6be7a84ab179b4240c4302518dc3f6", - ) - - result_unihashes = set() - - - with ClientPool(self.server_address, 10) as client_pool: - for taskhash, outhash, unihash in TEST_INPUT: - result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash) - result_unihashes.add(result["unihash"]) - - query = {} - expected = {} - - for _, _, unihash in TEST_INPUT: - idx = len(query) - query[idx] = unihash - expected[idx] = unihash in result_unihashes - - - for unihash in EXTRA_QUERIES: - idx = len(query) - query[idx] = unihash - expected[idx] = False - - result = client_pool.unihashes_exist(query) - self.assertDictEqual(result, expected) - def test_unihash_exists_batch(self): TEST_INPUT = ( # taskhash outhash unihash From patchwork Thu May 30 15:41:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 44406 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 465BBC25B74 for ; Thu, 30 May 2024 15:41:47 +0000 (UTC) Received: from mail-oo1-f46.google.com (mail-oo1-f46.google.com [209.85.161.46]) by mx.groups.io with SMTP id smtpd.web11.12852.1717083697269613631 for ; Thu, 30 May 2024 08:41:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=EjUFYv9D; spf=pass (domain: gmail.com, ip: 209.85.161.46, mailfrom: jpewhacker@gmail.com) Received: by mail-oo1-f46.google.com with SMTP id 006d021491bc7-5b9735d7ab4so589894eaf.2 for ; Thu, 30 May 2024 08:41:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717083696; x=1717688496; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Wnu2FFgad3A1NOtOtZwU0Yzer+un4KYFB23wV7AVhpw=; b=EjUFYv9DYzmiCMW0YYYY2tWLdgLf5+DRSl6TadzKuhyKgD/y+PjL0vd+Sljb9zj1JM s8CMTIfVI/+XBajJAyaSHs+dmf8Z0LL0/0RMqKXI4ZMEc8FA6Wm2oP6NWFnfOpyEtIxQ VTbfYMGssHJ4lX+peHRAT1tfpY6kFD8Nv2Of4FCCjkW/VwZETieZPe6dsTgCbgQIqzrI buC8tF0ZOsGd2oVTEtGTd7Kii1DjGN6+Mv+PQGOMy2M9Xct2ZsIVtI8hsdYO25iuLgIB zY28lIC3B7yz2VONO5rwA+xfC39JOiw2PReQ5DDIxK/CuBHltx82zdwtBG4szL56Er8h 33nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717083696; x=1717688496; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Wnu2FFgad3A1NOtOtZwU0Yzer+un4KYFB23wV7AVhpw=; b=n4AmKBeJW6GcGO8bSyZd8II7vnGDRVtGqvHPvgCpS5od6zMBltmkObaE6PQLQPYf1X LZwHUTQCVlDTzMei+CQTzOdmqU/XO5tkPU+1XeQaiwAVFGxXh7SB/3hDvXzL06l+Mejd WqxQIYyH8a4XBIgvBIG0y/Vw64ZZjpiupFJ/hPtzZpf+mEjvdrthSvqVWN54Pfji3plD dy+BuXNnayvx7W7wqV9i7ErwmoSu3bFNi04fd60iEiCoYkKWa+wx9zLXlhy4Jq5SHtd1 bOqoQGMEjX3cqMeZCL+8UDN9hD5Z6QhSLTAdCi4gFLDUkphmcVB1BP0csbMxm4FnI9MU Z4Bw== X-Gm-Message-State: AOJu0YwfzZptsqVLKa3pm9ofWWN8sAP0OpTWWiUKumQbrXsGyiiZlzJX Zz6HjTgXuK7u4TJk25BPGOkeePRdq3wTUfsKxQwFwHez61O2YFAOe3CyLQ== X-Google-Smtp-Source: AGHT+IHCG0WYeaRnmOVf2Bq1UG8AjUM42gOaJa1xG8DnalOtVCN8RWRVPE6MmCodWtsX+rwZ1YnmnQ== X-Received: by 2002:a05:6820:1c8e:b0:5b9:d8ab:2e34 with SMTP id 006d021491bc7-5b9ec722c63mr2503026eaf.5.1717083695789; Thu, 30 May 2024 08:41:35 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::125b]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5b96c6d60f1sm3184813eaf.45.2024.05.30.08.41.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 May 2024 08:41:35 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: bitbake-devel@lists.openembedded.org Cc: Joshua Watt Subject: [bitbake-devel][PATCH 3/3] siggen: Batch unihash_exists checks Date: Thu, 30 May 2024 09:41:27 -0600 Message-ID: <20240530154127.792582-3-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240530154127.792582-1-JPEWhacker@gmail.com> References: <20240530154127.792582-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 30 May 2024 15:41:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16291 Similar to looking up unihashes, use the batch API when checking if a unihash exists to speed up lookups Signed-off-by: Joshua Watt --- bitbake/lib/bb/siggen.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index 79f347db303..92066da00c9 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -671,20 +671,20 @@ class SignatureGeneratorUniHashMixIn(object): if len(query) == 0: return {} - uncached_query = {} + query_keys = [] result = {} for key, unihash in query.items(): if unihash in self.unihash_exists_cache: result[key] = True else: - uncached_query[key] = unihash + query_keys.append(key) - with self.client() as client: - uncached_result = { - key: client.unihash_exists(value) for key, value in uncached_query.items() - } + if query_keys: + with self.client() as client: + query_result = client.unihash_exists_batch(query[k] for k in query_keys) - for key, exists in uncached_result.items(): + for idx, key in enumerate(query_keys): + exists = query_result[idx] if exists: self.unihash_exists_cache.add(query[key]) result[key] = exists