From patchwork Thu Jan 5 11:51:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 191F7C53210 for ; Thu, 5 Jan 2023 11:52:01 +0000 (UTC) Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by mx.groups.io with SMTP id smtpd.web11.10981.1672919519611793843 for ; Thu, 05 Jan 2023 03:52:00 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=dCMBvJlt; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.52, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f52.google.com with SMTP id k22-20020a05600c1c9600b003d1ee3a6289so1108843wms.2 for ; Thu, 05 Jan 2023 03:51:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=V6XHCO8udO82zSjxA1mD8t0o2seywuirYm+1+/fotRI=; b=dCMBvJltxBTZ8QUPp8Y3CBIYEAFz6n/wTS6BPyp6emcWRnGYLQLNBzmODTUTMfyqMB XWJ4sdH6NW00oHUPmM555Vl0ynIOZFOgH4KXVcpqGODzeyxwcsi+yzl65XVzB+W8VSBx 8vskK6I+v7eMyYMPzxkz/A3nMVAZpSIxMDbtY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V6XHCO8udO82zSjxA1mD8t0o2seywuirYm+1+/fotRI=; b=0C9OiPvSHxgI6jXD8RVCIWxu0021+gFjt03+YjrAayBdA7OM1BqZCcD6G8dWt5bBu6 0PFJq3SfRqsFidyTG/b8TskymE2SjIQTGQ02J9DahA6+uvBU8GxTEmUY/Xxl2lPeKqDE sXNWh2ss4M7o4xAfVIaeJybD8Jv7R+kruk0VAK1QFIlXp4S4Z3T/Cr3EWZSxszBgZKjb QcPzW3hVW/oc0EBPwrVdAuK12JSTk3MKbNbHDt+bdARYhdZ3OMj6emWYg+VNjsFC10Ff /+RQ69JX/STB8V2RzYWTLL9fDuNzLRBR7LpZzYmSrth86sJGDONxhMerttKxYD4DpEQu VW5g== X-Gm-Message-State: AFqh2koHLffnaXQFEOyvaZ168BYOcF5Ejit0CZrpsi/diOl2dgsjRwQj ZZw8OdD/W6MidVKqFxQCx7ci/ihDojeutm0n X-Google-Smtp-Source: AMrXdXuVgDdpSwG1h2IFMEGdEnrF6NZvcpdjVtgcZClV2wAEGsihd60KQtk8uIRV61kzW48mxnV9fQ== X-Received: by 2002:a05:600c:34d0:b0:3d6:b691:b80d with SMTP id d16-20020a05600c34d000b003d6b691b80dmr35747648wmq.21.1672919517395; Thu, 05 Jan 2023 03:51:57 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:990c:167e:ef8f:7586]) by smtp.gmail.com with ESMTPSA id v10-20020a05600c444a00b003d998412db6sm2363988wmn.28.2023.01.05.03.51.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:51:57 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/2] cooker: Rework the parsing results submission Date: Thu, 5 Jan 2023 11:51:55 +0000 Message-Id: <20230105115156.1212181-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 05 Jan 2023 11:52:01 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14289 The loop submitting the parser results was not efficient on many core systems as may of the parsers could block for a long time waiting to send the results. While a result was pending, the code wouldn't parse further jobs. Change the code to parse further jobs even if the results can't be submitted and lower the result submission timeout to keep the parsers busy. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 738849d189..1d347ddc52 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2107,25 +2107,29 @@ class Parser(multiprocessing.Process): multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, exitpriority=1) pending = [] + havejobs = True try: - while True: + while havejobs or pending: if self.quit.is_set(): break - if pending: - result = pending.pop() - else: - try: - job = self.jobs.pop() - except IndexError: - break + job = None + try: + job = self.jobs.pop() + except IndexError: + havejobs = False + if job: result = self.parse(*job) # Clear the siggen cache after parsing to control memory usage, its huge bb.parse.siggen.postparsing_clean_cache() - try: - self.results.put(result, timeout=0.25) - except queue.Full: pending.append(result) + + if pending: + try: + result = pending.pop() + self.results.put(result, timeout=0.05) + except queue.Full: + pending.append(result) finally: self.results.close() self.results.join_thread() From patchwork Thu Jan 5 11:51:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 17761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18693C3DA7D for ; Thu, 5 Jan 2023 11:52:01 +0000 (UTC) Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by mx.groups.io with SMTP id smtpd.web11.10982.1672919519735209988 for ; Thu, 05 Jan 2023 03:52:00 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=V4fFsTU3; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.45, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f45.google.com with SMTP id bg13-20020a05600c3c8d00b003d9712b29d2so1119197wmb.2 for ; Thu, 05 Jan 2023 03:51:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=s178e0ho5oi4zYnwmLT1VGJlaIUAqDM6uI6iDoIF6q4=; b=V4fFsTU3KPuxbtCentmczLcCGBl4cx+p/ABtYTKi0bf+fAILrTJUsa8dGUlN2G0RcQ kLPrjka79kTFDtYs8+D2/YOQBKttMw1fSJjYG6jcpCfllNYL87SyGBtQvGEYpkgNqFuh I5nXgcak7C7vHimnq5iy8+wuvjm19eO8XQZuc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s178e0ho5oi4zYnwmLT1VGJlaIUAqDM6uI6iDoIF6q4=; b=Ppoa4heggJedUw45pIyMVYU7KA+XLPuG7wd5WzOfRJXrUOaX7arArA+dKRqibv4J6v wO9DMzGcEItwkEUirUDmB1DyghGRIJqPpHBX6TVq64AXResm238aHq8L1DDc6AjaGCOY ki6Tri1tAY8JOJcJtVivWi/+OKALnVeEb2m3B4tYH+kpikFFqasJXZJxGovblnvGppn5 7nIHBf09g6zlVd0a/RWaV3RmYQaxuwLhfmcRdhun7VU9LvhG7YYVKF/nuxi9rtrNeQUI 1gJdLYpe4vzNmXCEGnhkjQ9csyctIEhH2l4QkLtBNhUYxJA46XJLO106+ALK3XmVjClt cxCg== X-Gm-Message-State: AFqh2kpCYEXzN0vhrvhzhwGBAPCCYqPiBqTSnsobidYP8G4o8lq4Lbne xEFfAJAFVnSQ3cA7t2KqEIR0mM4eM/bFx+Sv X-Google-Smtp-Source: AMrXdXvAAFCjOc0LC0O6KN82UBpKTqfVTd0450UISvF/nffx/nnu8itkUuITrMyDXTzuTRWuDWNlzw== X-Received: by 2002:a05:600c:3485:b0:3d0:761b:f86 with SMTP id a5-20020a05600c348500b003d0761b0f86mr35550682wmq.28.1672919517988; Thu, 05 Jan 2023 03:51:57 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:990c:167e:ef8f:7586]) by smtp.gmail.com with ESMTPSA id v10-20020a05600c444a00b003d998412db6sm2363988wmn.28.2023.01.05.03.51.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:51:57 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/2] cooker: Clean up inotify idle handler Date: Thu, 5 Jan 2023 11:51:56 +0000 Message-Id: <20230105115156.1212181-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230105115156.1212181-1-richard.purdie@linuxfoundation.org> References: <20230105115156.1212181-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 05 Jan 2023 11:52:01 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14288 We no longer need to abstract the inotify callback handler, remove the abstraction and simplify/clean up the code. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 6 ------ lib/bb/server/process.py | 5 +++-- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 1d347ddc52..d2c42c858d 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -190,12 +190,6 @@ class BBCooker: self.inotify_modified_files = [] - def _process_inotify_updates(server, cooker, halt): - cooker.process_inotify_updates() - return 1.0 - - self.idleCallBackRegister(_process_inotify_updates, self) - # TOSTOP must not be set or our children will hang when they output try: fd = sys.stdout.fileno() diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 78fdc6cf71..4422fd7311 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -158,8 +158,7 @@ class ProcessServer(): def wait_for_idle(self, timeout=30): # Wait for the idle loop to have cleared with self.idle_cond: - # FIXME - the 1 is the inotify processing in cooker which always runs - self.idle_cond.wait_for(lambda: len(self._idlefuns) <= 1, timeout) + self.idle_cond.wait_for(lambda: len(self._idlefuns) == 0, timeout) def main(self): self.cooker.pre_serve() @@ -389,6 +388,8 @@ class ProcessServer(): nextsleep = 0.1 fds = [] + self.cooker.process_inotify_updates() + with bb.utils.lock_timeout(self._idlefuncsLock): items = list(self._idlefuns.items())