From patchwork Sat Mar 26 20:34:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 5872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52BECC433FE for ; Sat, 26 Mar 2022 20:35:04 +0000 (UTC) Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by mx.groups.io with SMTP id smtpd.web09.7394.1648326903145747561 for ; Sat, 26 Mar 2022 13:35:03 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=JpEdz5nr; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f54.google.com with SMTP id r7so6227990wmq.2 for ; Sat, 26 Mar 2022 13:35:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=B7qKs0RXdT8tcnKNLNvHNimytnJmiwAooEs5nIYh7Dc=; b=JpEdz5nrgfwWKVYkWGmueNH/ykgW4QSy59TDiK4wD0XMCloMVl/FKa9q9YASuw3qbD EJbp+SoxrsKXOkzxn3BIMIpqJb3iMT4faEHZJ55pBAr5Tcsuf+kL1ulqLBryQQ7fOPFx hhuSRmHAPARyv7TVjNDQMDy2CFHkTMQULXIlA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=B7qKs0RXdT8tcnKNLNvHNimytnJmiwAooEs5nIYh7Dc=; b=O/T/kGwKlApzq0IpF6kSZg8j7VQS5pu+vOYv5ZaA/ZNCy16TUtKaAeGAO+m8a0HMWF qu1ninEFp3iFmmlLGdI/eJvUzncmLB5uHQ7ibaLM7ZccO9BD5rODJA6LaqD3rS9mCbOq AOWvP3iawMj+tWeUXlwYVqMvBNDILp36n4hhERbDc4TCpS/K6cuokury38IOsI6zFXXh rPx07UskpqpZjHE5pjJ1d7roDzTHKUQf1Uv60q9F2tpNFKY9eOsCWGKwy++IXqzbTaVB 7EW7fc7UtVPOqFcEHl9O7tnqoNllOjbADh8SiNtH729tVZbbitIYsSp4tU0siwtVfJAe 4PqQ== X-Gm-Message-State: AOAM530BXj9GjntunKEHrtU45epNvdXsRJSB02D3S/6kNKEOt/CkDkgg 3wyTzvO3MdyEsGd8YE8jHjkFe9tAlPqNoJvG X-Google-Smtp-Source: ABdhPJwHyzvfICXpR/PeipLQ1T+jAXlFTXc9PyLmnMCCfakUPYN1eS/OGiBK6/0y3ySG/VVn5UCB5g== X-Received: by 2002:a05:600c:4f87:b0:38c:adde:1d99 with SMTP id n7-20020a05600c4f8700b0038cadde1d99mr25638585wmq.16.1648326901105; Sat, 26 Mar 2022 13:35:01 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:3949:59d4:6f55:52f]) by smtp.gmail.com with ESMTPSA id a18-20020a05600c349200b0038ca453a887sm12242586wmq.19.2022.03.26.13.34.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 26 Mar 2022 13:35:00 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/6] cooker: Fix exception handling in parsers Date: Sat, 26 Mar 2022 20:34:53 +0000 Message-Id: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Mar 2022 20:35:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13513 We shouldn't be generating exception inside a generator. Rearrange the code to improve the handling of this. Also fix the misconverted code from when multiconfig was added and pass the exception as "result". Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index eac956aa97..c4d720a6b6 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2087,12 +2087,12 @@ class Parser(multiprocessing.Process): tb = sys.exc_info()[2] exc.recipe = filename exc.traceback = list(bb.exceptions.extract_traceback(tb, context=3)) - return True, exc + return True, None, exc # Need to turn BaseExceptions into Exceptions here so we gracefully shutdown # and for example a worker thread doesn't just exit on its own in response to # a SystemExit event for example. except BaseException as exc: - return True, ParsingFailure(exc, filename) + return True, None, ParsingFailure(exc, filename) finally: bb.event.LogHandler.filter = origfilter @@ -2252,11 +2252,7 @@ class CookerParser(object): pass else: empty = False - value = result[1] - if isinstance(value, BaseException): - raise value - else: - yield result + yield result if not (self.parsed >= self.toparse): raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None) @@ -2267,6 +2263,9 @@ class CookerParser(object): parsed = None try: parsed, mc, result = next(self.results) + if isinstance(result, BaseException): + # Turn exceptions back into exceptions + raise result except StopIteration: self.shutdown() return False From patchwork Sat Mar 26 20:34:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 5873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36662C433F5 for ; Sat, 26 Mar 2022 20:35:05 +0000 (UTC) Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by mx.groups.io with SMTP id smtpd.web10.7422.1648326903990791198 for ; Sat, 26 Mar 2022 13:35:04 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=ElKLF1vP; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.47, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f47.google.com with SMTP id r204-20020a1c44d5000000b0038ccb70e239so1408876wma.3 for ; Sat, 26 Mar 2022 13:35:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=sYndZs6A1eX88S/gUxvFN/QfdvLC6ew57JSoQWAy1vE=; b=ElKLF1vPxrHtL8aTgwB4+Et2fIYGxi3D8Ww+JF9swbx8D5sOL4XtJ3sdXALdmmdEV0 r9JWUv7G2UsFoYTw+pxBQZHGVenAAGqGtqnpAGrfaGGoyJawU4wB/BwEOgve3ToWXVLZ hAr8aoEZ/9rCeF5pBFs5b2NOnTEux7UmhqX4I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sYndZs6A1eX88S/gUxvFN/QfdvLC6ew57JSoQWAy1vE=; b=1VqYjqpg1GEfwbXX4Cqb4L/Vpn8EtIzCITQg/4O06KiIDfRS17F1qVb23Qw2PcsUhh C5BTgaF49YihBVPM+On2ti54eUucF2DvgrJx0HSKTGVntW0XkD+sGGwukcMEMfKBRRxQ VKKffkPuKAfb+gcEGxS8IOKdbIp7x16rtAHC+cSnC+jfVULMKOzeD/4jJLtqX0FhF/3Y D5lLE7pNsXUsM0HlhcD6XahCrNC0MaF1NixghcLkwlkn3jwTlCSOV3neNElWBDY2Fw80 Pj1snN/DPV9G0fvJS2cM/qtHaRQXXVwlWDTRSLT5VO+rAD9kp7tXH8UMGVg7DHOrvchy b5IA== X-Gm-Message-State: AOAM533BfTZe2mLchRSZn3oqJMizZlV2pjRMutSsUdH9l0cS01xfgXXx R4QRetLuM8IdNblOkEvUGH+ec6dti2y5p2bS X-Google-Smtp-Source: ABdhPJwNdukjn7OP8rHUVCy/4+PCry9VwgrSpA3IxWDja/7K2VBeNb9P++Vbc5cqVOqPyar3nz5low== X-Received: by 2002:a05:600c:3c9d:b0:37f:a5c3:fccf with SMTP id bg29-20020a05600c3c9d00b0037fa5c3fccfmr16684681wmb.13.1648326902367; Sat, 26 Mar 2022 13:35:02 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:3949:59d4:6f55:52f]) by smtp.gmail.com with ESMTPSA id a18-20020a05600c349200b0038ca453a887sm12242586wmq.19.2022.03.26.13.35.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 26 Mar 2022 13:35:01 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/6] cooker: Fix main loop starvation when parsing Date: Sat, 26 Mar 2022 20:34:54 +0000 Message-Id: <20220326203458.1391301-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> References: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Mar 2022 20:35:05 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13514 When parsing, the parser isn't servicing the main loop so a Ctrl+C in the UI won't be seen on the cooker/server side. Fix this by returning when queue timeouts occur. This helps where there is a hung or slow parsing thread. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index c4d720a6b6..fb71a968f2 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2249,7 +2249,7 @@ class CookerParser(object): result = self.result_queue.get(timeout=0.25) except queue.Empty: empty = True - pass + yield None, None, None else: empty = False yield result @@ -2266,6 +2266,10 @@ class CookerParser(object): if isinstance(result, BaseException): # Turn exceptions back into exceptions raise result + if parsed is None: + # Timeout, loop back through the main loop + return True + except StopIteration: self.shutdown() return False From patchwork Sat Mar 26 20:34:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 5874 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4065EC433FE for ; Sat, 26 Mar 2022 20:35:06 +0000 (UTC) Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by mx.groups.io with SMTP id smtpd.web09.7396.1648326905291493609 for ; Sat, 26 Mar 2022 13:35:05 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=OQWsSl76; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.47, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f47.google.com with SMTP id d7so15076925wrb.7 for ; Sat, 26 Mar 2022 13:35:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=JVbUVfXAYv56QT8+yrD8AwyJtTP1eOGwcft83b87yRo=; b=OQWsSl76la173OdHIkNgnaohgOQKZVsvgOhg1yLOA1PHn9RVkzyoe5HMELmZAn7qrk +WmagXaPjuUDKs7cSskW7PGigQV6uEc+XuMuUqw0MzjJyU7vlXltBhmZQSf4Su3gnP0O pHbABOwgA2iITDyYoJ0ZNhSZSiBogmKW3XM4w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JVbUVfXAYv56QT8+yrD8AwyJtTP1eOGwcft83b87yRo=; b=m+9X/0tOxRgOoVkDGhD2AZraBpTYcxxJCokgO5qOHyrEuNkc1qQdF226kvCmm9O5eJ utGYyhri0HauZAYEutTAmf0nGj2WkG/hjKTQNyM2QsXbHcfFnlSq3FR0MH8M32FVjVHK xKRSnA9dGdjSez4SoJIUU1hzh18eTvc0FhK0ZQVaG4d9+V0NtEwYCkz45Wap29do+tJC wIJVYdq6HaIXOE52t65ezziVoo2mBMT/3iU1ih4bwzmZaZOcmN/aoySj5Sv1a8so17/N /GgFjwgJWqqvimN890YcnRO1QH9tPv/xZazWqcwCX0aoYAwVkZtTNwSrHqwSD91x9bAx SegQ== X-Gm-Message-State: AOAM531bAH1LwUW+djxjw+KhwfTg517FzQI/mPk6AjjAH6hXds3eMAo4 btT7tz3J/8MMRVu2QIUP89F7DJ9MuObxnjVZ X-Google-Smtp-Source: ABdhPJwq9nRuYC74DdCIMES9KrTKdyEIpJTndY57Um8TJxn5JgDmQBSXSB7++xBbDsghGogRUVudqQ== X-Received: by 2002:adf:fc43:0:b0:203:d867:9fe5 with SMTP id e3-20020adffc43000000b00203d8679fe5mr14631924wrs.272.1648326903285; Sat, 26 Mar 2022 13:35:03 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:3949:59d4:6f55:52f]) by smtp.gmail.com with ESMTPSA id a18-20020a05600c349200b0038ca453a887sm12242586wmq.19.2022.03.26.13.35.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 26 Mar 2022 13:35:02 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/6] cooker: Improve exception handling in parsing process Date: Sat, 26 Mar 2022 20:34:55 +0000 Message-Id: <20220326203458.1391301-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> References: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Mar 2022 20:35:06 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13515 If an exception occurs in the parsing process, ensure the cleanup is called via all codepaths using a try/finally. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 48 ++++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index fb71a968f2..2264b18c54 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2041,32 +2041,32 @@ class Parser(multiprocessing.Process): self.init() pending = [] - while True: - try: - self.quit.get_nowait() - except queue.Empty: - pass - else: - self.results.close() - self.results.join_thread() - break - - if pending: - result = pending.pop() - else: + try: + while True: try: - job = self.jobs.pop() - except IndexError: - self.results.close() - self.results.join_thread() + self.quit.get_nowait() + except queue.Empty: + pass + else: break - result = self.parse(*job) - # Clear the siggen cache after parsing to control memory usage, its huge - bb.parse.siggen.postparsing_clean_cache() - try: - self.results.put(result, timeout=0.25) - except queue.Full: - pending.append(result) + + if pending: + result = pending.pop() + else: + try: + job = self.jobs.pop() + except IndexError: + break + result = self.parse(*job) + # Clear the siggen cache after parsing to control memory usage, its huge + bb.parse.siggen.postparsing_clean_cache() + try: + self.results.put(result, timeout=0.25) + except queue.Full: + pending.append(result) + finally: + self.results.close() + self.results.join_thread() def parse(self, mc, cache, filename, appends): try: From patchwork Sat Mar 26 20:34:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 5875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B340C433F5 for ; Sat, 26 Mar 2022 20:35:07 +0000 (UTC) Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by mx.groups.io with SMTP id smtpd.web11.7497.1648326905963267575 for ; Sat, 26 Mar 2022 13:35:06 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=UbDEmM6s; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.46, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f46.google.com with SMTP id r204-20020a1c44d5000000b0038ccb70e239so1408904wma.3 for ; Sat, 26 Mar 2022 13:35:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=sCJXQcib2xVfthx29oldgdvWkCp+F9uW4uWrO2ERy3o=; b=UbDEmM6sJnNpeu2w99e+8FWqBk9rjZ54m6ACgKwtp3dAxoZQxNpk2gmH1M6skoqKeN rbHfBMa6Nj+tJ7a2Mk3tAxWlHm8c6U9SJmYwH6nYBrufUBlfmNdQ45K+52lls1GDmmJy g1SBPvwroeFN6QQ7jMZYGRVLhZ2WDA0PLuuA0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sCJXQcib2xVfthx29oldgdvWkCp+F9uW4uWrO2ERy3o=; b=uyO2DuclH/kQ8oKsc/RDF9t55pPjzWnwwpxXKoQTomu7BaDlMXKLMpkJD44vtr/CcQ fCMBYAloO5giWvJXTXpFvqvboBGe5Wp3pIyRrxhvX3MICpR61HSyjJ7SRYSAuc54yz/w RmP/Qm1PUlD0JtnG0GibQ930iB3kN37sBW+j+Q0b9tiJFcJ0OeJinrTOqKKLIazMEKDt OjE9QpOL9MWAfwZxMB8ab1+LOZJ9w7QNaFJ90BCz8Nbg/0bjgWbPDtS0CMG6FP1Vunbv qfc+BCsCklPXc0b0itx95GdMYzOAdI3hbBkodRBp/HpfQ70tUDNA2G1iH3nt9FnMkDOW Qoyw== X-Gm-Message-State: AOAM533s/cHsgOHeuDUlT2EabrrP6sv+HOsF0Y6+lgC4F3lkC7PLVGfW 9G+6xCGQ6b+spRTxxhGjyDZo95GOW9zjznl6 X-Google-Smtp-Source: ABdhPJwvEt358Aw7ceWNIsaahtUIMGrRCkGqI97aWH8MwnKlXb9mI7eKQti444TtFEBwe7Y9kkSdjg== X-Received: by 2002:a7b:cb4b:0:b0:38c:8aa7:3f8a with SMTP id v11-20020a7bcb4b000000b0038c8aa73f8amr16930209wmj.37.1648326904232; Sat, 26 Mar 2022 13:35:04 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:3949:59d4:6f55:52f]) by smtp.gmail.com with ESMTPSA id a18-20020a05600c349200b0038ca453a887sm12242586wmq.19.2022.03.26.13.35.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 26 Mar 2022 13:35:03 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 4/6] server/process: Avoid hanging if a parser process is terminated Date: Sat, 26 Mar 2022 20:34:56 +0000 Message-Id: <20220326203458.1391301-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> References: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Mar 2022 20:35:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13516 If a parser process is terminated while holding a write lock, then it will lead to a deadlock (see https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate). With SIGTERM, we don't want to terminate holding the lock. We also don't want a SIGINT to cause a partial write to the event stream. Use signal masks to avoid this. Some ideas from Peter Kjellerstedt Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index efc3f04b4c..dc331b3957 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -20,6 +20,7 @@ import os import sys import time import select +import signal import socket import subprocess import errno @@ -739,8 +740,12 @@ class ConnectionWriter(object): def send(self, obj): obj = multiprocessing.reduction.ForkingPickler.dumps(obj) + # We must not terminate holding this lock. For SIGTERM, raising afterwards avoids this. + # For SIGINT, we don't want to have written partial data to the pipe. + signal.pthread_sigmask(signal.SIG_BLOCK, (signal.SIGINT, signal.SIGTERM)) with self.wlock: self.writer.send_bytes(obj) + signal.pthread_sigmask(signal.SIG_UNBLOCK, (signal.SIGINT, signal.SIGTERM)) def fileno(self): return self.writer.fileno() From patchwork Sat Mar 26 20:34:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 5876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3503DC433FE for ; Sat, 26 Mar 2022 20:35:08 +0000 (UTC) Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by mx.groups.io with SMTP id smtpd.web09.7398.1648326906979227936 for ; Sat, 26 Mar 2022 13:35:07 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Au4bgELc; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.41, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f41.google.com with SMTP id r7so13964978wrc.0 for ; Sat, 26 Mar 2022 13:35:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=UFVsoLpNIWkR4K/8j9+y3Z4oAAAD1Nsg1NnP9wX8mXw=; b=Au4bgELcVm0dFxstwqrTIxg0hwa1ywwqvwGDI8M+TS8Z7qAnJV1GgpKgHWld4C57xR wMrjr/Sr4HOIpX/MvPPqg5YSjOsmrEhPn/Ysn1B5mWnioM2fKRwTe1c3fASheAyCdF94 91o8oFIF2rA7ZkrA/SXfncgP9c18nnJjltYHo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UFVsoLpNIWkR4K/8j9+y3Z4oAAAD1Nsg1NnP9wX8mXw=; b=0WibSD2PIzhBmso9YfYvTPLL5O6L0mGzuze53YIM9npEYIrlRLtd8eIaAqbF/l4nze S/4mQJj56LhwqXnnDGBakFkvVqWTuFmPgGYf4k3tpShWENrqhHMgH14HNPvlxxzMWmwQ 9k89LI0ycXKmwRb7TSwQYljZiFwVrH8X2Z1caRcql1hxupTHRQA7HlSqLtfFEhEWB6Td gxLtO472QACydFBWPiUd0NMRUYbN2sm/iRs4KdNpvjmWkblrWSx3A/7dLICoTGiYCcb7 8OFOuxpzKvNnGXkbCQhqd8tA5XpR/8IrC8M/BUARaxTeXYL7Da/FiAqmjmDld5L4pRbD 9upQ== X-Gm-Message-State: AOAM533dJTUYf6FlzQFPCwJgCbh7v3WMPjzzFanhnT2Plz1usW/hiHZU 5tGIXIKXTO5ysblbb6LOaGWcYU+p3lNYgufW X-Google-Smtp-Source: ABdhPJwl+gJux3ZVa8l/oIt2kqSSRThwugEpHfwLL4ITHGVL8hCFMO6pyj0YQmvgawfM+7DvyflX5Q== X-Received: by 2002:adf:e748:0:b0:204:fbe2:932d with SMTP id c8-20020adfe748000000b00204fbe2932dmr15535819wrn.305.1648326905242; Sat, 26 Mar 2022 13:35:05 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:3949:59d4:6f55:52f]) by smtp.gmail.com with ESMTPSA id a18-20020a05600c349200b0038ca453a887sm12242586wmq.19.2022.03.26.13.35.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 26 Mar 2022 13:35:04 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 5/6] cooker: Ensure parsing processes have close called Date: Sat, 26 Mar 2022 20:34:57 +0000 Message-Id: <20220326203458.1391301-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> References: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Mar 2022 20:35:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13517 With pyhon 3.7 there is a close method which collects the process' return value and ensures zombines don't hang around. Add that call. Also join terminated processes before closing since the signal might not be instantaneous. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 2264b18c54..4b8b18a68d 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2198,8 +2198,10 @@ class CookerParser(object): if force: process.join(.1) process.terminate() - else: - process.join() + process.join() + # Added in 3.7, cleans up zombies + if hasattr(process, "close"): + process.close() self.parser_quit.close() # Allow data left in the cancel queue to be discarded From patchwork Sat Mar 26 20:34:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 5877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BBCCC433EF for ; Sat, 26 Mar 2022 20:35:09 +0000 (UTC) Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by mx.groups.io with SMTP id smtpd.web09.7399.1648326907765045724 for ; Sat, 26 Mar 2022 13:35:08 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=EAVKXjME; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.51, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f51.google.com with SMTP id r7so13965010wrc.0 for ; Sat, 26 Mar 2022 13:35:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ty+YthLjsrx5Q54ndd2odgUrNeaz/zyoQXuoqYS5K48=; b=EAVKXjME4LqgqjbbzQa+cvG8Ko+4jRRFx7SLAgA44WrFGb+SRtBfsy3Oz5kV7+bOSq U3g/VTs/ngkYv1UmRkz4zvW+EZVnCIgqayDNbCYe0R2vl9YtTktA7sMR0u+nYlGs6kGi WCZHRuj/dWGO6YUJ7KHXMXLoGRfwfz3KfUizQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ty+YthLjsrx5Q54ndd2odgUrNeaz/zyoQXuoqYS5K48=; b=k1VZz42usF+DoyGhdWvDjKQkMwryR6Ls9gp2bNm3Lrs+Fle3Sjsh9/yd/6iF63/1Dc pXeLGjpWXcucjnpZubpw5GWOMph7dl0b6az3oHoyNpyJikuXlMebINr30Zg8Me+YiZDE 28QV54DH/j5IhPSig5Vtnyy8cgiphBIr2rVprQ4s5ei6npmHIbU8PQplzm3jP4Q3AL07 kXZo3Y4kj4m5dtwpuzy+7hX1oi3bbSuf2AOMc1aeb8DHOVz8ZP2TC4kJ6SojVQXvm9Fv 44BpMWuI/rGXVr4KjaEzSX1PD8Y7FLGOuLGMd2SFXYlp4YWKnXRnWOq8Dy4uvcAg31Vp JM+w== X-Gm-Message-State: AOAM532InPJ7xTD41wrGsEfiq96Jv/JfbO9Imx43nQoP0xrhToBOv0Ih tgfRguy89BLOoUN+L3PQpnpxIMX4eodCVXzV X-Google-Smtp-Source: ABdhPJyHu16oyS39HQj/HS8KM7LXLHUOVtVw9fOlXOT1+VfNMWam8Ffvsf8VyebvzgUAkoTa1HHNYA== X-Received: by 2002:adf:f40c:0:b0:203:e768:ae54 with SMTP id g12-20020adff40c000000b00203e768ae54mr14581334wro.488.1648326906073; Sat, 26 Mar 2022 13:35:06 -0700 (PDT) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:3949:59d4:6f55:52f]) by smtp.gmail.com with ESMTPSA id a18-20020a05600c349200b0038ca453a887sm12242586wmq.19.2022.03.26.13.35.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 26 Mar 2022 13:35:05 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 6/6] cooker: Pass SIGINT to parsing processes at shutdown Date: Sat, 26 Mar 2022 20:34:58 +0000 Message-Id: <20220326203458.1391301-6-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> References: <20220326203458.1391301-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 26 Mar 2022 20:35:09 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13518 To try and improve shutdown, sent SIGINT to parsing processes now it is handled safely around locks, perhaps avoiding the need for the more risky SIGTERM. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 4b8b18a68d..bd7ab8ac3c 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2146,7 +2146,7 @@ class CookerParser(object): def init(): signal.signal(signal.SIGTERM, signal.SIG_DFL) signal.signal(signal.SIGHUP, signal.SIG_DFL) - signal.signal(signal.SIGINT, signal.SIG_IGN) + signal.signal(signal.SIGINT, signal.default_int_handler) bb.utils.set_process_name(multiprocessing.current_process().name) multiprocessing.util.Finalize(None, bb.codeparser.parser_cache_save, exitpriority=1) multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, exitpriority=1) @@ -2194,6 +2194,9 @@ class CookerParser(object): except queue.Empty: break + for process in self.processes: + os.kill(process.pid, signal.SIGINT) + for process in self.processes: if force: process.join(.1)