From patchwork Wed Mar 5 22:09:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 58390 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D0FBC19F32 for ; Wed, 5 Mar 2025 22:09:56 +0000 (UTC) Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by mx.groups.io with SMTP id smtpd.web11.6379.1741212587818363669 for ; Wed, 05 Mar 2025 14:09:47 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=s8uvVmPR; spf=softfail (domain: sakoman.com, ip: 209.85.214.181, mailfrom: steve@sakoman.com) Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-224019ad9edso21532125ad.1 for ; Wed, 05 Mar 2025 14:09:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1741212587; x=1741817387; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=1PYdLohszhNbI+iRiz8oj88WoX5DPTVskhsI5Ioe/UQ=; b=s8uvVmPR9R+GdxUeUxQv2i/1fTfeM4ffkUf/ULAEwj0+CSCd0CQ65O9yquQkvaLdKn uAzWMR2LLrPD5nfa65nuTAtCvxRCiYPZpjUES8wCch2xmOLFppZlgOCAWr8PlF67nShz ycn2DgnuIk5h97jDAIj1m52rhziWxTo/8TJNAjFfBB7vEHcoqALdcNInn9gbxiKCij0a NgDCQK9yuY90rasxKonlrhqh9pvzzNTFjMkDc+SsrHq77x58beJ+7a9USTLNP/fzImi6 cy4O3+/bLewiqXnSgp3Xm7apcxTtGFP8S6Z9EAqX8NV/tG9GcJSTwxdvDSHX4yDJWHi6 ig7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741212587; x=1741817387; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1PYdLohszhNbI+iRiz8oj88WoX5DPTVskhsI5Ioe/UQ=; b=XoDXog4IYv4W5YADqrDu3bzihQLn6cJJPM0UHnAbbYLqm3hV5S50tBe6nPFXCth0kn tG1sMsV3jSJd7nfAzRFNTq2JTqlz+vjpLK6teqnBkWDKTy9cSbXJD21ywVbVnfP/WZ+t NYjBRt//d0J8b82k38nNpnjuGhw+8Bs7hO7Im2Eb8w7htm6qb3aVmNm8BpawPPGwHGnE ZTN/k6bL2Ol48jg95+Uzq0ZOJBs2WVlUwMmr5W+2KnrzGFsrPrN5aQo2pfnvxj1UsHxm WPYHZSOq4xbOpenE8DXkQrygccxtYD+HmNxhQUmm+k68yU6+aayrrYJs+yckRnrdnrwQ FpEQ== X-Gm-Message-State: AOJu0YyoXimptqdeocjyKd11qwNX66OP9Iiz8SV3hcUYTBiN1edbyO5+ 2tBrwXAmalBq0IiozT9aqCJRsvHgvzMn2rWx52X3IiFjtNMjk3D3nTByPepIaedDGheVjNVTMBq F X-Gm-Gg: ASbGnct6stJcH6RUZTCouGJp3j+ME2GS3AjEsCGyajGqBGfwCPwmhzihmQB/G57iHB0 AaUB+rLnA4eX7bGG3YiEfTSBCL2z7wKIuin9vkPxD9yI96Z99nZDA/UUh2Co0cZ2QYQcVcaJXYQ NXtb6f5p8Yq8ESqm2/QcgeVUC7el/KKxXccN1qtowGFX4/lnOjnza147FxlHrx+XoMjj3Rl04W+ PtYTX68GEA9VEhNMMIhkfgWMGxTbuWY60EBIu92jp14nLXl1taPUnWZ+4/8kdBsrFY6z0TDevjE rzzJVmoiCq5DPOeoG3BvETxst9HmWp8yuhM= X-Google-Smtp-Source: AGHT+IGoz708QgEh8NwktG/2axF473f7Z4Lc9cMZ21xgLcdtUHnMcfQrA0KDS4UfFIEp5ja1+wy99Q== X-Received: by 2002:a17:902:d4c5:b0:220:ea90:1925 with SMTP id d9443c01a7336-223f1d20313mr82730475ad.35.1741212587025; Wed, 05 Mar 2025 14:09:47 -0800 (PST) Received: from hexa.. ([2602:feb4:3b:2100:2308:d28:2350:a5eb]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-223501d35a4sm118602405ad.35.2025.03.05.14.09.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Mar 2025 14:09:46 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 1/4] utils: Print information about lock issue before exiting Date: Wed, 5 Mar 2025 14:09:26 -0800 Message-ID: <8d40d8cc319674e77dcf4560243feeeccf53bc51.1741206585.git.steve@sakoman.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 05 Mar 2025 22:09:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17388 From: Richard Purdie Signed-off-by: Richard Purdie (cherry picked from commit cdf6c51a064f8f335c3262b7f102618996f1a229) Signed-off-by: Steve Sakoman --- lib/bb/utils.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bb/utils.py b/lib/bb/utils.py index ebee65d3d..0dfe47dc5 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -1862,6 +1862,7 @@ def lock_timeout(lock): held = lock.acquire(timeout=5*60) try: if not held: + bb.server.process.serverlog("Couldn't get the lock for 5 mins, timed out, exiting.\n%s" % traceback.format_stack()) os._exit(1) yield held finally: From patchwork Wed Mar 5 22:09:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 58387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DC6AC28B22 for ; Wed, 5 Mar 2025 22:09:56 +0000 (UTC) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by mx.groups.io with SMTP id smtpd.web11.6380.1741212589423710936 for ; Wed, 05 Mar 2025 14:09:49 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=VznjZF6h; spf=softfail (domain: sakoman.com, ip: 209.85.216.43, mailfrom: steve@sakoman.com) Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2feae794508so82247a91.0 for ; Wed, 05 Mar 2025 14:09:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1741212589; x=1741817389; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=8QxXInYW13FxRcsNmm9q3w0pWz20qT7Xe+llWUdFlV4=; b=VznjZF6h4zLuDbXNp2EvyqyUv1uIbGUdiFB3RYVdqkdTwli57XU1I5q0gR5oaeK8YF rEA8Q+aePNZBVHLwNxuL2i6NlStkvszc+7CBYh5kleyJGHTsZpGQzWKtjpj9zhZ5RBHT dokc46Bf7+DI5LGmPgm5wUN1f2P9AbwS9VUdFdSS8ygdf+cN3PAfvl5GbFbgtd6FRph6 rNFkMa08epiCUboqA6340ZoZI0gGnNVdad8B87yUHe6gfpAdkMdz+ncItYkXgv4WxTRM y2oJZb759OnDwCIY5ee5ybW2QqRqHSFlOEUxA9EBCAX2b2QBagFsKSzOmNEd+qQC5oDC EHjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741212589; x=1741817389; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8QxXInYW13FxRcsNmm9q3w0pWz20qT7Xe+llWUdFlV4=; b=ezwzNBLdNgs+h4YshM1b3kL/0P4yBnjB2nz0fhqH8YvKeB5T7TS+eRSnkXX9HI85pO RfMiDiTENqzDVgrrK++u/W1B1JLWtkLkBr0fkSASHxYDrDF9qPONmA+qA3HAIuQP/Ep5 l8KLUFg+gsNhr5H5zgklqatLySfCF7oN3By13Z9MgUIef6gXwsvREz+8cahn4RIzMDyc HQ9Dz6z5n4t7C7Jr3z38pcKMnsaSKYMfVAW4G4TSDKnuirZ+Sly0Iqv/1DLraWN1IQS8 K+eSkYlj0URzkAPuvWNJyFMn1ueMgftgYBxHggNeCLKLmu3Dc2fOuDpTfsjv8Oo45jMN kigg== X-Gm-Message-State: AOJu0Yzu+QkQlRiIA3OXy+VHIrTyTXX7IR90SfesdiwzW+f1/QE7oQi0 aEQoLhUREplbx7VmN9qvDmGYpWFL3rcIw+fr+vmvvi4CUowix4gM7n1gVcq3eBAc7aqiHvEnUm2 w X-Gm-Gg: ASbGncseiZfWWGNwNZQcmkXC8REkX5ypGrY5yMoMi2JAyIlDGQBeSwAV3oFenRI4NsR pYDRr3sNHeNHv4THAhfgz6ilgQdAagok9/GTmahZrqUisS1U3WhN8knAPs4iseFcxK4blf+CAEK WbCbnUfH6gpjo5mOD/9MuNCnrYtgRUIqWessssdeucUBzo2JGztqA8COY6jnhjxEVEF1H0CHMii GxaONLAF30usDY5Iw5n6BPq5xry4MC+A9jQI/1mMeWmq/hLDNOtFtQcKmOafZJF1deDtSKVTLri 4cp2sEfgwTJ1wEEVZGkhYJjH+lDTfgc74Zc= X-Google-Smtp-Source: AGHT+IFvqVH2sgTcHP3kbTfUnlnll30/G0MuVED/gOMKrB964dxvXQU4A5aNscMtF2JeEvdLB/vuKw== X-Received: by 2002:a17:90b:38c7:b0:2ea:3f34:f194 with SMTP id 98e67ed59e1d1-2ff497a94d1mr7968462a91.10.1741212588603; Wed, 05 Mar 2025 14:09:48 -0800 (PST) Received: from hexa.. ([2602:feb4:3b:2100:2308:d28:2350:a5eb]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-223501d35a4sm118602405ad.35.2025.03.05.14.09.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Mar 2025 14:09:48 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 2/4] utils: Tweak lock_timeout logic Date: Wed, 5 Mar 2025 14:09:27 -0800 Message-ID: <36e8e310faa4e5d572a13c0cf9ec82680cf8f5b9.1741206585.git.steve@sakoman.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 05 Mar 2025 22:09:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17389 From: Richard Purdie We should really try and take the lock in the try/finally block so that in some rare cases such as badly timed interrupt/signal, we always release the lock. Signed-off-by: Richard Purdie (cherry picked from commit a9eb8bf7174b6962b5ba07192fe95b8c7112d9d2) Signed-off-by: Steve Sakoman --- lib/bb/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/utils.py b/lib/bb/utils.py index 0dfe47dc5..83e832c33 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -1859,8 +1859,8 @@ def path_is_descendant(descendant, ancestor): # we exit at some point than hang. 5 minutes with no progress means we're probably deadlocked. @contextmanager def lock_timeout(lock): - held = lock.acquire(timeout=5*60) try: + held = lock.acquire(timeout=5*60) if not held: bb.server.process.serverlog("Couldn't get the lock for 5 mins, timed out, exiting.\n%s" % traceback.format_stack()) os._exit(1) From patchwork Wed Mar 5 22:09:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 58389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EEE9C282EC for ; Wed, 5 Mar 2025 22:09:56 +0000 (UTC) Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by mx.groups.io with SMTP id smtpd.web11.6382.1741212590807466945 for ; Wed, 05 Mar 2025 14:09:50 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=Y0Koq/bd; spf=softfail (domain: sakoman.com, ip: 209.85.214.169, mailfrom: steve@sakoman.com) Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2235189adaeso835285ad.0 for ; Wed, 05 Mar 2025 14:09:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1741212590; x=1741817390; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=gLXwYhqo7UMpagMu+O73c/oE62mOwy1Vn8DQXRT55zo=; b=Y0Koq/bdh+AIN/zEdyoKrxAS6pZ8kk8CwDBBAMxPeCrb/9pEdhudzsMrREaz2d+OB3 dR7TCoMyZ28OKXZ3omtwZH4/fcNqA1knN2kvSrWzFul3QHmjfztdfPgOe+BCCkaysU3B sCYmypN8DsgAQipMJjyT+XLQr23oZmezkgXwwbabSfGwYQGHLFGLOpxqxbgQTMl0yjVN D7xvVrLUK+xyPEvC8UZNw8vcqxtKYXlFnvEsCXjHhKBQ3WVy4uIgBid7G3dqGJAbSH5l wfHP0m8lDVJOr/UzPFnRKzbDCPqzyDqcHVpb93QvhPv0gkYjc061SrUFV6usM8NsgUE9 4/iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741212590; x=1741817390; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gLXwYhqo7UMpagMu+O73c/oE62mOwy1Vn8DQXRT55zo=; b=JMvll/0TOAHRH3koVREdcwH1w341X4sBh6Ti0NvG7mhK0OLPkzEUg7feFCu2GKlB9n nXxWccE4CRPHGYegidcGcO7DWhcCacnwiw12rGwjEcLseBfxgLgdLz7lSowzcHTAMb0w 504Fo1k/rNgPB/2cmTKR0+smzyt2rNoyazNgVlaMD431OsLigt3VGuHPaqSihWwfCiib bGw6FyNEafMdlFuxwVkVS5qKQ9sec84kg1CY0wLT7KwOE8Wo8yyFZ8KuO7tFtzK/65AP MigxjtBJMaS6qbfjG1Btqcc4bgGY/7PxgxCiJSOdcT7/SeVTXRQutkfHbaDEbGEeQWIw HaKg== X-Gm-Message-State: AOJu0YwAFkLmLLynNNDPjSmh5HVrgCKbgWxQfytXns/Oz3oAMTDYmPyT PYdN3ILDins7dwW3fxOIO6NNbxcph0yPjca2+2sqJLcuKiZw8+bG3aIvJoiccbS8E0M/rOQAzqD e X-Gm-Gg: ASbGncvrm/nRyZLWDD8fzvplEBc+3X12JMnVPXfCDM75/2qi6c1V3NMECygXaSx3CSI QFsTQ2IanFu/E9Qf8iWNr6zH+9UT1onj/K+WvNkoGAtIEQ1WXvLe4HQWRm9sxBYvBzk5UqdvYWy nvUIvJ0If2tf9syhPfaEm8NK9rw9biHACqxbQiLE6gwiraVAiAhMD38VBkynlIyl2XrqD+R08Ab uP/lZp+ZjkYIOC4WNotoPhaQK9jaunTangNus5X0s3CtyTZqEWhnvw1mfWGAvMJkiwyYnqqkcY1 +p8jxU9f7X8FOLlOUWfdBG3s4VDdgPRlST0= X-Google-Smtp-Source: AGHT+IEIk7f+D+Z7cFd0gO5OCNOEyI4sHq7pFuQYJJKY9NUuDFKT6dFhE6pSqWchn54C8owbDuNfHw== X-Received: by 2002:a17:903:2d0:b0:215:b1e3:c051 with SMTP id d9443c01a7336-224094410bdmr13783775ad.11.1741212590077; Wed, 05 Mar 2025 14:09:50 -0800 (PST) Received: from hexa.. ([2602:feb4:3b:2100:2308:d28:2350:a5eb]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-223501d35a4sm118602405ad.35.2025.03.05.14.09.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Mar 2025 14:09:49 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 3/4] utils: Add signal blocking for lock_timeout Date: Wed, 5 Mar 2025 14:09:28 -0800 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 05 Mar 2025 22:09:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17390 From: Richard Purdie We never want to exit whilst holding these locks as it deadlocks all python threads. Add signal blocking around the lock critical part so a signal shouldn't cause such an exit. Signed-off-by: Richard Purdie (cherry picked from commit a097755c671e2b530dea6200a94b39fa9dca246c) Signed-off-by: Steve Sakoman --- lib/bb/utils.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/bb/utils.py b/lib/bb/utils.py index 83e832c33..da026fe5b 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -1860,6 +1860,7 @@ def path_is_descendant(descendant, ancestor): @contextmanager def lock_timeout(lock): try: + s = signal.pthread_sigmask(signal.SIG_BLOCK, signal.valid_signals()) held = lock.acquire(timeout=5*60) if not held: bb.server.process.serverlog("Couldn't get the lock for 5 mins, timed out, exiting.\n%s" % traceback.format_stack()) @@ -1867,3 +1868,4 @@ def lock_timeout(lock): yield held finally: lock.release() + signal.pthread_sigmask(signal.SIG_SETMASK, s) From patchwork Wed Mar 5 22:09:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 58388 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74D0CC28B23 for ; Wed, 5 Mar 2025 22:09:56 +0000 (UTC) Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by mx.groups.io with SMTP id smtpd.web10.6333.1741212592627571511 for ; Wed, 05 Mar 2025 14:09:52 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=dtTa0pMC; spf=softfail (domain: sakoman.com, ip: 209.85.214.181, mailfrom: steve@sakoman.com) Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2234bec7192so137458855ad.2 for ; Wed, 05 Mar 2025 14:09:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1741212592; x=1741817392; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=GHg/XudC2kuB2xyXmHx8aKBZitT91K0qL8GvAR/4uFE=; b=dtTa0pMClWXnjHbbRiilrwEYTf7iS3sGEtuiJsQqe5zqNL3BUPTzQqGtF2Fv9r2wTg twmTKf9FxTVAdKkrWM8viYRN14iBN4WoF2YWYC0jzcD+cJGMKMJSYJS8K4aMMNXJF4YW ZzMxlRR03HJTFpb90ue4jHseEO4lAzdNjldlWGyfpBQM43DmFixeEx4Zi3Kk4EsHyr29 YMegTMqi5t0mLxkEh9FYXpgSgUb45+EMcCbDX4Op3ecCNip+MnkXgH2TeApzNdBlp9+3 lVZxlvJ7mgaa2yUEDTclqMmw4PoYKyyIDGI/zhxMiPF9DoAzCjAcoYOn48yzwx9EgrFa 5tYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741212592; x=1741817392; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GHg/XudC2kuB2xyXmHx8aKBZitT91K0qL8GvAR/4uFE=; b=k8qNqaE8a+nUz5F3DzCDLLgIiyB+KXk7wQxJlP5/OplxJ9IgBwlBx/WLWWzA3PyHTY 9ebEKilLHQhtlTFEr9eqm1uyfJHuUGsCnxiWqMNhOBm5yA6zv7oiLRCkYmmPNNEA/xPf LB0n0Tu4IE9i6Dq3N8xzwYfpWfICShh3+VGvsiA2603WQOCaRpuCFP4mmqzrxjxs+k5R bVl5x9Q3rIStYLp6vUn9HcEjulggVevjDJksRXVslcqfBh0I1CZwAxAHhxye/B5ObP5S SH4P1XaErnwQJfjhLEtHkLBswZrkdkgyrFonwubrwd0ylzrROQvPaklHKwBwxlBXBJa1 SSUg== X-Gm-Message-State: AOJu0Yz5nBd65+ew1F08epAS2cGyE+Wxby+L0Fp3lm9g4idfzPQ9Hrek 3LMxaK7IgU8we90ST0m44viVcyjDflnLA5y4yzDsuH968rD+xsCImohQIVAnwa9UWoHUXJaE9WK b X-Gm-Gg: ASbGnctkKyHIxU9xoar0YYFRgGCspXwe68wSCq9iOGf5S55CbJV05JlCM4cjDGR0gog hDNAdU+nohljTJNucfQWeaimNP6sH/U763WwgN7v1lojC34iKsipKgXHz6QF8fNHvM4QTCUQAmv B92WqpntaCRR1ecC96wcnfeJkml8OFDb1I1+Js3TTnneTJMWJxJMY9E2UtadfUjoM4tVygheF9e rfYOpPv9gHsrNDCGLX+55dTdSWKy1Uy9eJ8SH9A5vtSm0TtQtq9cW8mYeTS3yKg5awiJnAPvXzj rpZ1Oe334ImV97v7JcsixsIb4K8vgDlm+8E= X-Google-Smtp-Source: AGHT+IF8kJTqweLBKC3QHbP+Wt165Qus7skYJMHXU8OMUOTMNS80xZuEnOi+hrCCQWn59Ryx35Bdow== X-Received: by 2002:a17:902:d489:b0:220:e5be:29c7 with SMTP id d9443c01a7336-223f1cf42e6mr74132025ad.39.1741212591835; Wed, 05 Mar 2025 14:09:51 -0800 (PST) Received: from hexa.. ([2602:feb4:3b:2100:2308:d28:2350:a5eb]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-223501d35a4sm118602405ad.35.2025.03.05.14.09.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Mar 2025 14:09:51 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 4/4] event/utils: Avoid deadlock from lock_timeout() and recursive events Date: Wed, 5 Mar 2025 14:09:29 -0800 Message-ID: <706f13a41b438970ae13dac64fcfe67ed16040c2.1741206585.git.steve@sakoman.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 05 Mar 2025 22:09:56 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17391 From: Richard Purdie We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which were problematic to debug. The failure was inside lock_timeout and once that was identified and the backtrace obtained, the problem becomes clearer: File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal retval = function(self, data, False) File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache self.parser = CookerParser(self, mcfilelist, total_masked) File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__ self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array) File "X/bitbake/lib/bb/cache.py", line 772, in __init__ loaded += c.prepare_cache(progress) File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache loaded = self.load_cachefile(progress) File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile progress(cachefile.tell() + previous_progress) File "X/bitbake/lib/bb/cache.py", line 751, in progress bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize), File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers _ui_handlers[h].event.send(event) File "X/bitbake/lib/bb/cooker.py", line 117, in send str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\') File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__ _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg sw(msg.message, msg.category, msg.filename, msg.lineno, File "X/bitbake/lib/bb/main.py", line 113, in _showwarning warnlog.warning(s) File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log self.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle self.callHandlers(record) File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers hdlr.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle self.emit(record) File "X/bitbake/lib/bb/event.py", line 778, in emit fire(record, None) File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers with bb.utils.lock_timeout(_thread_lock): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack()) or put in simpler terms, whilst sending an event(), an unrelated warning message happens to be triggered from asyncio: /usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport which triggers a second event() which can't be sent as we're already in the critcal section and already hold the lock. That warning is due to the version of asyncio used on Ubuntu 22.04 with python 3.10 and that comined with timing issues explains why we don't see it on other python versions or distros. We can't handle the second event as the lock is there to serialise the events. Instead, we queue the event and then process the queue later. Add a new version of lock_timeout which allows us to handle the situation more gracefully. Signed-off-by: Richard Purdie (cherry picked from commit 2c590ff1aff89d23b25ce808650f200013a1e6af) Signed-off-by: Steve Sakoman --- lib/bb/event.py | 10 +++++++++- lib/bb/utils.py | 15 +++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/lib/bb/event.py b/lib/bb/event.py index 952c85c0b..a12adbc93 100644 --- a/lib/bb/event.py +++ b/lib/bb/event.py @@ -194,7 +194,12 @@ def fire_ui_handlers(event, d): ui_queue.append(event) return - with bb.utils.lock_timeout(_thread_lock): + with bb.utils.lock_timeout_nocheck(_thread_lock) as lock: + if not lock: + # If we can't get the lock, we may be recursively called, queue and return + ui_queue.append(event) + return + errors = [] for h in _ui_handlers: #print "Sending event %s" % event @@ -213,6 +218,9 @@ def fire_ui_handlers(event, d): for h in errors: del _ui_handlers[h] + while ui_queue: + fire_ui_handlers(ui_queue.pop(), d) + def fire(event, d): """Fire off an Event""" diff --git a/lib/bb/utils.py b/lib/bb/utils.py index da026fe5b..67e22f438 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -1857,6 +1857,9 @@ def path_is_descendant(descendant, ancestor): # If we don't have a timeout of some kind and a process/thread exits badly (for example # OOM killed) and held a lock, we'd just hang in the lock futex forever. It is better # we exit at some point than hang. 5 minutes with no progress means we're probably deadlocked. +# This function can still deadlock python since it can't signal the other threads to exit +# (signals are handled in the main thread) and even os._exit() will wait on non-daemon threads +# to exit. @contextmanager def lock_timeout(lock): try: @@ -1869,3 +1872,15 @@ def lock_timeout(lock): finally: lock.release() signal.pthread_sigmask(signal.SIG_SETMASK, s) + +# A version of lock_timeout without the check that the lock was locked and a shorter timeout +@contextmanager +def lock_timeout_nocheck(lock): + try: + s = signal.pthread_sigmask(signal.SIG_BLOCK, signal.valid_signals()) + l = lock.acquire(timeout=10) + yield l + finally: + if l: + lock.release() + signal.pthread_sigmask(signal.SIG_SETMASK, s)