From patchwork Fri Mar 7 13:54:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 58478 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F9CBC282DE for ; Fri, 7 Mar 2025 13:55:19 +0000 (UTC) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by mx.groups.io with SMTP id smtpd.web10.11467.1741355702233806171 for ; Fri, 07 Mar 2025 05:55:02 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=wD2opB8w; spf=softfail (domain: sakoman.com, ip: 209.85.216.43, mailfrom: steve@sakoman.com) Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2ff64550991so2991970a91.0 for ; Fri, 07 Mar 2025 05:55:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1741355701; x=1741960501; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=1PYdLohszhNbI+iRiz8oj88WoX5DPTVskhsI5Ioe/UQ=; b=wD2opB8wwcMVn6hQN9tDZy8OBn3qzLIyfxI7M+DjFuiKvKm9TSboqr6FegTgo4qC1v zYvO/ygEXHzhC+8PphJ2BkmonpaVv24n/CyECkpc14KlV3BHGTiG6tu17dfSBkV7UbKX EpsX3yuHLeFB6HlQczmKHIkm+1l5Kt3saEIn9IsWM4p0+WbQ1bPqtg4dTL675N/vYWWy Fjg/ovFdaYlkamxP1G8Im0vPbWYbaoCCA1N3rGnNKNRkRTNTu9NsTqvaGYUPTJCIi0A8 G+BDJgiRov8xoMz7urhLvkPI3J+CDtpY92dtitO8E6jN4ltGZdnu/ZPveNK/AIF4BJim 7jtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741355701; x=1741960501; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1PYdLohszhNbI+iRiz8oj88WoX5DPTVskhsI5Ioe/UQ=; b=OrJoe/38eCvLad6wwOOQfz2f70m00/bD7Q7g3ADa36xcJKgpDnAyJ3TD8KDRWjdEwg YufJrsk6U/t7WyLOjArOYsVID6UmBQaUVoARPGimOychcN0iVCGeLqrSYypYM0rFUCeM RBv5/mtEQ2jLvTkvRFYg48ziylR8s0Foxg8ZkGZsltxDoxK4OyClq8kLc/v2JV/zMGRW uRtesmnOTC299H5hcbiK5IqeH6c4gXDePJYnLXUh5AZpBiSsGF48dl2wpPooORzOP8Nx aH/m20OZu0TPngnZ/AISIEuN0jh/eIU95dqC0OvXAO1vRVK1FIsfauR/nqVZXGO4lRiz Ex6Q== X-Gm-Message-State: AOJu0YwLANJLMhFGjjrqs9auxWHW1ntkzllkBr7U23ZLrIydcD3o2gOp Ae3zZe0wHinSzmKON+M+DC7q7mtox0NlzZ/csW3Q0MJoRsdK+ver+eNz8uLnnZ0z0T2vGqYmoj5 w X-Gm-Gg: ASbGncsc77IeFL10Ja8zoP49sYYK1vrwKg1k765fdoUiN+0gsS/Um5c1xlms3iy+XhI EPzzYnHpoHkrhsbG30hJezvuYBqlsNqYHhRgETnxett0ee76JraBnMm0gd21uEjK8J+dkWeA6BS G+Xbw0vEYC7JEOx8MRWekkQ6CDC/t5BmegWLSyvNTmvDCnIoaZO24OD5eW0HBxUF0mz6lF3rcTb K782IEc2PBBgZWSWy5xalbGwGn9T7rzinhFV36vNTPMWJKij+3fdXqU+AVSHMibURrvrmqNEQXb 2//2l83WTgV1ovZXA0SU0Q8BhD31JRBgbjGA X-Google-Smtp-Source: AGHT+IHZD8fi2bdmXPnMpvbkPFkv6YOjLmotwRNvjNy5gJqoGLW7ht1uJh35bm5AG/ivH/J+K1Nwug== X-Received: by 2002:a17:90b:2e90:b0:2ee:d7d3:3019 with SMTP id 98e67ed59e1d1-2ff7ce8e50amr6605386a91.12.1741355701417; Fri, 07 Mar 2025 05:55:01 -0800 (PST) Received: from hexa.. ([2602:feb4:3b:2100:cfcd:40b4:f918:b895]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2ff4e75edc9sm5488032a91.10.2025.03.07.05.55.00 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 05:55:01 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][styhead][2.10][PATCH 1/4] utils: Print information about lock issue before exiting Date: Fri, 7 Mar 2025 05:54:51 -0800 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 07 Mar 2025 13:55:19 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17408 From: Richard Purdie Signed-off-by: Richard Purdie (cherry picked from commit cdf6c51a064f8f335c3262b7f102618996f1a229) Signed-off-by: Steve Sakoman --- lib/bb/utils.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bb/utils.py b/lib/bb/utils.py index ebee65d3d..0dfe47dc5 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -1862,6 +1862,7 @@ def lock_timeout(lock): held = lock.acquire(timeout=5*60) try: if not held: + bb.server.process.serverlog("Couldn't get the lock for 5 mins, timed out, exiting.\n%s" % traceback.format_stack()) os._exit(1) yield held finally: From patchwork Fri Mar 7 13:54:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 58475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F669C19F32 for ; Fri, 7 Mar 2025 13:55:09 +0000 (UTC) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by mx.groups.io with SMTP id smtpd.web11.11368.1741355703627924033 for ; Fri, 07 Mar 2025 05:55:03 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=H3QcmhlB; spf=softfail (domain: sakoman.com, ip: 209.85.216.43, mailfrom: steve@sakoman.com) Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2f9b9c0088fso3280089a91.0 for ; Fri, 07 Mar 2025 05:55:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1741355703; x=1741960503; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=8QxXInYW13FxRcsNmm9q3w0pWz20qT7Xe+llWUdFlV4=; b=H3QcmhlBC9IFijF6/0eL4VgDqMsB0Nny6B3D5cUV3gR6Irfy369LsScsvtM9VGAqby 2Ku0HtuwUOchWNpMi4LX/iXnu9/RYg3UwXDJgX0S12ERFO0girklHu7FhDKRnyJrB8hh W0A8/49Xls4MqwGALKfUemNpGmbH83U+Ml32OlN5VbKqYfOESPpPjtPiztHXsm/ppFgB +SLzXqqSCFKU7gNgxlywf+HCDwCUxyPaxnNwRfyy6Vn9fsje31kt6y2Te9w2JxMU0VWD kP7PdKoPaCcJB/Liy7o9dhBbmRyteaiwccYhuM6RXoHAQdaUXho/3+CZwc+bihqXD/WR LA8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741355703; x=1741960503; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8QxXInYW13FxRcsNmm9q3w0pWz20qT7Xe+llWUdFlV4=; b=oTq8ArWAXDSzr+aYeXPqOeuDEagNXhVtOO2eok2I1GO0qWmxGl8oaSVD4YoxEQlhn5 x+JSDSOWfg43vwe9C6p2rDftvz4DNJpYowRs4Mi/NGgCv5ecte2KKBhVu3EAew3fKIXZ nGRQeLUyJ5qllhbWPGzm85WJAAJQ66L6+xdydrtLlmZAMPebl8OExy6JNc24llp4CRT7 pVx1n98UBvSqESq/PoMlr40U5MYfcCMJqh9Ewkds6QQFU7xaM2los/Q2Ar6caUH8rI2y r3vgJAf1hvO6Er5pvwvbpVMXBkOoCTyZXnE+xwGJ/JbYTbRlmVJ16l3x9pS5Yxe7lezj +XhA== X-Gm-Message-State: AOJu0Yw8p7tvQVChuuFU1uJjdfFszHs+z3sXVGKewIc/hDx41i4DncDA p+v/498oNf5IC4GzEyS9PwtX2lu0XqcSTUEzoNa/p6MxEgzvAW1gymZPhqOx/ueJszQlQ/UCULL m X-Gm-Gg: ASbGncu93W6oe+KZPTQUW0N1wsOYQQ62YbjYWbyjQUsRlGBkqxrcwVFf/0zunNP8Jf6 ImyrZ/SH/d2uhWHdV6IHV0w8J347cpcx1PT7KmQI4Qp+3pymBWO4WWDwHPFhY6WLxkiWSifPxeO hpNNfoOa6lHOH5dW4DrIKirjxKcDW2JuWriGNWFm68ZAUq45mzAAmoZfkmJZfb0IOOg91WnNZrJ gYYs9Ftf+K0zM9HyNa8a9oHYKgstgAs6QZ2YV7SF3e7xnrqcGVg6CbZbdpiMY0gJ5vWRaGWZU5K JSTssljYf5uJjuqhmoTlGJLZO8A7kNhnXvzB X-Google-Smtp-Source: AGHT+IH6GCFGuS1guYUlkwH/me5Fji1psAhyqpRfdCfKLc+omAPodgdeYEPwXnURsPkvPzgxu/QjYQ== X-Received: by 2002:a17:90a:d44c:b0:2fa:2133:bc87 with SMTP id 98e67ed59e1d1-2ff6175a466mr11905433a91.6.1741355702810; Fri, 07 Mar 2025 05:55:02 -0800 (PST) Received: from hexa.. ([2602:feb4:3b:2100:cfcd:40b4:f918:b895]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2ff4e75edc9sm5488032a91.10.2025.03.07.05.55.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 05:55:02 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][styhead][2.10][PATCH 2/4] utils: Tweak lock_timeout logic Date: Fri, 7 Mar 2025 05:54:52 -0800 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 07 Mar 2025 13:55:09 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17409 From: Richard Purdie We should really try and take the lock in the try/finally block so that in some rare cases such as badly timed interrupt/signal, we always release the lock. Signed-off-by: Richard Purdie (cherry picked from commit a9eb8bf7174b6962b5ba07192fe95b8c7112d9d2) Signed-off-by: Steve Sakoman --- lib/bb/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/utils.py b/lib/bb/utils.py index 0dfe47dc5..83e832c33 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -1859,8 +1859,8 @@ def path_is_descendant(descendant, ancestor): # we exit at some point than hang. 5 minutes with no progress means we're probably deadlocked. @contextmanager def lock_timeout(lock): - held = lock.acquire(timeout=5*60) try: + held = lock.acquire(timeout=5*60) if not held: bb.server.process.serverlog("Couldn't get the lock for 5 mins, timed out, exiting.\n%s" % traceback.format_stack()) os._exit(1) From patchwork Fri Mar 7 13:54:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 58479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99541C19F32 for ; Fri, 7 Mar 2025 13:55:19 +0000 (UTC) Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by mx.groups.io with SMTP id smtpd.web10.11470.1741355705029131927 for ; Fri, 07 Mar 2025 05:55:05 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=lZg9q3D5; spf=softfail (domain: sakoman.com, ip: 209.85.216.54, mailfrom: steve@sakoman.com) Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2ff6a98c638so3609605a91.0 for ; Fri, 07 Mar 2025 05:55:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1741355704; x=1741960504; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=gLXwYhqo7UMpagMu+O73c/oE62mOwy1Vn8DQXRT55zo=; b=lZg9q3D5dbDybjCxciU5pU0b6ahX/+QhF5DuCdNF5bvXQjhTmryrApb+cNIe7wYRLt SXKvT8i51XVS1HmhGyzMiIGTAzXT4JSSc/r7lZO7O+jntgPBdR59nnEF0uu90E7H6586 /BvVm+BIe0fdcXDE2O2G7dqUSoiD4Fzz3/m3z1VAfKfDOCLfli2BJs4Gu/32FQqXGlzg 3l8Biblme1bQE4wwTn3nluxWYWFTnzbBf+I6/mxppw8ddF8s6C5lYsuG16GU9KiueJW+ NrLi8kOtemlZAFnEhqlK9Lk2JtXL2doWos82lT31Ghdiq+F2Ev434gKs1b+YoFOyN7sL VFDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741355704; x=1741960504; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gLXwYhqo7UMpagMu+O73c/oE62mOwy1Vn8DQXRT55zo=; b=DdEENnyxUiZN78kes/+F9KI79S9KA4Y8SdtpY1k7khFbnpqRyBsD01qyUCMWLv1iV4 /n0Rcsbsl4B6RoxWIwAe0PSiT9yVMLh85c01BkoQjkPLouE8JY2/qTKn2i5hkJFeng/Z vBDTFIZULIc1l/4S1PdCishxq6L4LoB0es78187ZXCElJFz9Y0/jQVFyt5X1JkzG3KBY ysBV670sf4ioOQsn4YFhuSOiK7v3x0L6NFZSQbgZ5xEfRhl53UzTTVyq8bCmIHsTkEqf 62+m+3YXEAVxGqCoLZZCGUfopzDS4fywfoS9ehaNkIxbI6bYF7PrmhOV4iilw/1m3W0Z JXBw== X-Gm-Message-State: AOJu0YzmDju1uGmKYeOIigC7TegNx0i1ow6c3Ys7J6E57NdpdhmDPT5X nDXM/8DrB4sEpAEDQqkvlb74oD96roBCwNzoHLQyDvnI5nFTcdkbSyWHEZ7gCjAW3geaEH41PQw z X-Gm-Gg: ASbGncsJMjjKOkzkNQc6xN3d2wMm8zsSdYHumvIIILrdEyjNljzAeSQqgZt7rL0zl3e AxJLiFzbvh1bF5/YCtHq8qq5Z+tgm7/bmRHwGFSDh/YQwQuLr9JfTt/beWM2kT1WJypk1zzfFDc Ey1wnMqY5hy9wDmeqWajjTov9gQ6ldvVf2zLl3R/KWGZx8A5VUFjFwEeHWEhvp8U5krUKqOgcbS 7HEYwZ3DFRHDocemTBheqoqN+CFF5RRcpIme8P8QOmjeJVm27NOwALp7XRAigTo7RQktL5D3HoO equt2yRQozJ3dJtpRAVbv/gMfLDgEimj4t3k X-Google-Smtp-Source: AGHT+IHBCc2iW2stbho6/UKD6ndKJJieWgZOzgHfLp0KBllCTOSfbG5DhliDAqwWX0VB5r5hhz7p+Q== X-Received: by 2002:a17:90b:3887:b0:2f4:4003:f3ea with SMTP id 98e67ed59e1d1-2ff7cef99c2mr6485438a91.33.1741355704301; Fri, 07 Mar 2025 05:55:04 -0800 (PST) Received: from hexa.. ([2602:feb4:3b:2100:cfcd:40b4:f918:b895]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2ff4e75edc9sm5488032a91.10.2025.03.07.05.55.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 05:55:03 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][styhead][2.10][PATCH 3/4] utils: Add signal blocking for lock_timeout Date: Fri, 7 Mar 2025 05:54:53 -0800 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 07 Mar 2025 13:55:19 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17410 From: Richard Purdie We never want to exit whilst holding these locks as it deadlocks all python threads. Add signal blocking around the lock critical part so a signal shouldn't cause such an exit. Signed-off-by: Richard Purdie (cherry picked from commit a097755c671e2b530dea6200a94b39fa9dca246c) Signed-off-by: Steve Sakoman --- lib/bb/utils.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/bb/utils.py b/lib/bb/utils.py index 83e832c33..da026fe5b 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -1860,6 +1860,7 @@ def path_is_descendant(descendant, ancestor): @contextmanager def lock_timeout(lock): try: + s = signal.pthread_sigmask(signal.SIG_BLOCK, signal.valid_signals()) held = lock.acquire(timeout=5*60) if not held: bb.server.process.serverlog("Couldn't get the lock for 5 mins, timed out, exiting.\n%s" % traceback.format_stack()) @@ -1867,3 +1868,4 @@ def lock_timeout(lock): yield held finally: lock.release() + signal.pthread_sigmask(signal.SIG_SETMASK, s) From patchwork Fri Mar 7 13:54:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 58476 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3F30C28B25 for ; Fri, 7 Mar 2025 13:55:09 +0000 (UTC) Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by mx.groups.io with SMTP id smtpd.web10.11471.1741355706824741783 for ; Fri, 07 Mar 2025 05:55:06 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=2nRS/+eu; spf=softfail (domain: sakoman.com, ip: 209.85.216.52, mailfrom: steve@sakoman.com) Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2ff797f8f1bso1926837a91.3 for ; Fri, 07 Mar 2025 05:55:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1741355706; x=1741960506; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=GHg/XudC2kuB2xyXmHx8aKBZitT91K0qL8GvAR/4uFE=; b=2nRS/+euMSwphTSjWFE2x2RXHNJ6M7rT6alzNSUQIh8fIQuitgS/mr/aNacVCaazjt RoPFPeemV1PpEIqeMtbAXtVgZI0easPpghtYDe4IHN6umGOo0r2UdBCp1RZmPrJGoRm6 uAjPoa3xC9U9gw2ScviEEdWNtKiWAVsv8VXX3KiSLKiXBHHYE8jiHeA9Zkb+oE/uq6Cc mUxEdnYGDezHP16r9q2Dp6mtweYCPsxm4/xBFyePGjV4L55XRvCU3riuArBgNxzcWFA+ PPPL1kquFuCPw7reybiakGtJm/uPA50u81Oj90PIjqxOsNWyhDGQMTsvZ19I0+eMQh2q owfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741355706; x=1741960506; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GHg/XudC2kuB2xyXmHx8aKBZitT91K0qL8GvAR/4uFE=; b=Eu02odmBn5hm4cI7sG1QbSAXEPrm0nRqJ9iAH2lQmI+DH/SWA5DVPahFU84CnFgbFQ 09MK3XmUxt0XbzpEjyiTDLJnnuCqHm/OEGlGuO3wpjPi2NTdXIfuBvb6NuWklWcjY6tc FXpKnVZSxDy5IG9QtODK1/3T/lEUO4jQY+/oPu5PCotzB5uk94VFuw1N11E1RW+Z956c fzAQfsLZZbTL3L6sq++/3JfFJTj+zKAltd4OHoa1iFJdtnO78hItvSkyVdg3KQ5IwqBe Ft/j/gSRNYHDCfuKe6K3LS+V4Uwmqn94JdBx3DB6KWutQpSA4Hoz7rJJFzjod5WmJz95 6GBw== X-Gm-Message-State: AOJu0Yy1TnDwc79qb2O2QfpCHQTZdQOv8HwY3zDvzZlzooFNdf0KSWPp JihaJgHKvADaWGKIhrcuqqUb/f/yUmMoocuSFKcnS52HAx8LNg2tlXovTR+UzypR1evUCmGqpW5 6 X-Gm-Gg: ASbGnctbLra0FqA2dPvnC3ROmh0inCg3LrZsPCN434+LGI6dcYdXm0SkJ+QBh9Okk4W CygHfJN1sIzd3ru2YVlk4YptHMutywPSAxgZjHmkTE2kjqhDC3N2PpBeTXNEalyMZffahoaef+h S7CB9Xn4WcZHCAaewQv/Gh8KLj+bjy36rHJxLQ6Yh1WJgLP5L66KciJlkPkaHI+f+1TbLBrTctC RPfkBS3N4OLQFhRqJH3prEH3lCDL2wzG2Cz5L4tEDQJKHFvohy8euNeN9yF+pgJFtP9P5/23qlU y1DwHjmNjbyoWGNP/8+jy329Ycf4Ai2x+NcK X-Google-Smtp-Source: AGHT+IEZyierXhk2stsDw4z43A+2BN0Ffzxv9jbMl1NovM0go4jHyinusd5M6/uRpIqB03zG07iGag== X-Received: by 2002:a17:90b:2883:b0:2ff:6788:cc67 with SMTP id 98e67ed59e1d1-2ff7cf31b55mr4412708a91.34.1741355705884; Fri, 07 Mar 2025 05:55:05 -0800 (PST) Received: from hexa.. ([2602:feb4:3b:2100:cfcd:40b4:f918:b895]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2ff4e75edc9sm5488032a91.10.2025.03.07.05.55.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 05:55:05 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][styhead][2.10][PATCH 4/4] event/utils: Avoid deadlock from lock_timeout() and recursive events Date: Fri, 7 Mar 2025 05:54:54 -0800 Message-ID: <82b9f42126983579da03bdbb4e3ebf07346118a7.1741355508.git.steve@sakoman.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 07 Mar 2025 13:55:09 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17411 From: Richard Purdie We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which were problematic to debug. The failure was inside lock_timeout and once that was identified and the backtrace obtained, the problem becomes clearer: File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal retval = function(self, data, False) File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache self.parser = CookerParser(self, mcfilelist, total_masked) File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__ self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array) File "X/bitbake/lib/bb/cache.py", line 772, in __init__ loaded += c.prepare_cache(progress) File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache loaded = self.load_cachefile(progress) File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile progress(cachefile.tell() + previous_progress) File "X/bitbake/lib/bb/cache.py", line 751, in progress bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize), File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers _ui_handlers[h].event.send(event) File "X/bitbake/lib/bb/cooker.py", line 117, in send str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\') File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__ _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg sw(msg.message, msg.category, msg.filename, msg.lineno, File "X/bitbake/lib/bb/main.py", line 113, in _showwarning warnlog.warning(s) File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log self.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle self.callHandlers(record) File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers hdlr.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle self.emit(record) File "X/bitbake/lib/bb/event.py", line 778, in emit fire(record, None) File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers with bb.utils.lock_timeout(_thread_lock): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack()) or put in simpler terms, whilst sending an event(), an unrelated warning message happens to be triggered from asyncio: /usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport which triggers a second event() which can't be sent as we're already in the critcal section and already hold the lock. That warning is due to the version of asyncio used on Ubuntu 22.04 with python 3.10 and that comined with timing issues explains why we don't see it on other python versions or distros. We can't handle the second event as the lock is there to serialise the events. Instead, we queue the event and then process the queue later. Add a new version of lock_timeout which allows us to handle the situation more gracefully. Signed-off-by: Richard Purdie (cherry picked from commit 2c590ff1aff89d23b25ce808650f200013a1e6af) Signed-off-by: Steve Sakoman --- lib/bb/event.py | 10 +++++++++- lib/bb/utils.py | 15 +++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/lib/bb/event.py b/lib/bb/event.py index 952c85c0b..a12adbc93 100644 --- a/lib/bb/event.py +++ b/lib/bb/event.py @@ -194,7 +194,12 @@ def fire_ui_handlers(event, d): ui_queue.append(event) return - with bb.utils.lock_timeout(_thread_lock): + with bb.utils.lock_timeout_nocheck(_thread_lock) as lock: + if not lock: + # If we can't get the lock, we may be recursively called, queue and return + ui_queue.append(event) + return + errors = [] for h in _ui_handlers: #print "Sending event %s" % event @@ -213,6 +218,9 @@ def fire_ui_handlers(event, d): for h in errors: del _ui_handlers[h] + while ui_queue: + fire_ui_handlers(ui_queue.pop(), d) + def fire(event, d): """Fire off an Event""" diff --git a/lib/bb/utils.py b/lib/bb/utils.py index da026fe5b..67e22f438 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -1857,6 +1857,9 @@ def path_is_descendant(descendant, ancestor): # If we don't have a timeout of some kind and a process/thread exits badly (for example # OOM killed) and held a lock, we'd just hang in the lock futex forever. It is better # we exit at some point than hang. 5 minutes with no progress means we're probably deadlocked. +# This function can still deadlock python since it can't signal the other threads to exit +# (signals are handled in the main thread) and even os._exit() will wait on non-daemon threads +# to exit. @contextmanager def lock_timeout(lock): try: @@ -1869,3 +1872,15 @@ def lock_timeout(lock): finally: lock.release() signal.pthread_sigmask(signal.SIG_SETMASK, s) + +# A version of lock_timeout without the check that the lock was locked and a shorter timeout +@contextmanager +def lock_timeout_nocheck(lock): + try: + s = signal.pthread_sigmask(signal.SIG_BLOCK, signal.valid_signals()) + l = lock.acquire(timeout=10) + yield l + finally: + if l: + lock.release() + signal.pthread_sigmask(signal.SIG_SETMASK, s)