From patchwork Sun Nov 28 22:02:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 90 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC9CCC433F5 for ; Sun, 28 Nov 2021 22:03:04 +0000 (UTC) Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by mx.groups.io with SMTP id smtpd.web11.52930.1638136984432556491 for ; Sun, 28 Nov 2021 14:03:04 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=ClZmq3qG; spf=softfail (domain: sakoman.com, ip: 209.85.216.44, mailfrom: steve@sakoman.com) Received: by mail-pj1-f44.google.com with SMTP id y14-20020a17090a2b4e00b001a5824f4918so13808523pjc.4 for ; Sun, 28 Nov 2021 14:03:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=tkvt/zmPhLXLF7jS4CFAN/rK39Vvl2tQAwBt0fAhdgA=; b=ClZmq3qGMB+2nEnLdxnj/2SA9CK5aLMLIF6i6foTOPHRsEFTZKmgPTb5yIW01IvUg+ GV/drJK2baprFGUpNQ9flLFWNIbwzNFSYNoZsc73TTYEIQ64OKYem7/JRWuttnANmN/1 YUkjo4E69kfJVreIzR18ijh9bEc5yUVN9FV29AymeKLJJkUKmM5tWzB4Yb6eFCWRey12 h2zbgQSojVTcpTkr318cPhUV7++1lX8b6JGl0PgKG/nn1kkxQnE9UJTTLgvCKR99lqU+ 7moG7qTQ3kCb/ZYXjFhmWJRP5V2z3eiy17gXJOd79VOsltm7KuykEXJJpUCGqIBpionO DgNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tkvt/zmPhLXLF7jS4CFAN/rK39Vvl2tQAwBt0fAhdgA=; b=6oXSN7TrH38nNjMdGnyZASXu/dk5Bnc3XSVkUW2c4S0tMcEV+hpWNBqrTL+sD5bFMM TyL2AtR0WtlZByYkK0yWd8Jk6Mw4WyqB6D0NMHADE5JFDSgUtxoM4kwVD52sWsli8EV3 e3nmEnG1p9bl/nDKMzflaSxOS5XrDDO6ECXwoOu9W2fAHD/pT0pWJdsY1iw1/tJ7jyne d+hAS5GfsboMLj13D8VQCRwTTz77U9bIUyqouvWtife7mdyrTYzWn+ic7ijBkkie9+TL HSl8jlucrFt3ghRyRnD3XPknKtNhwI1MVfvpkCwktNRIyIMizIMG44NV6GhdCSQEks05 RFyw== X-Gm-Message-State: AOAM5332pNYY+SjcwBeu7jkClllwpSzFbSSSLF+/qCsEwTAxIRcdE/el 6TTL4TkdMm7uyKdmQ96ndwkdAT4swmPUBHZj9w4= X-Google-Smtp-Source: ABdhPJwBOC20I631el+Pu7kLtKXlkBr0LBwdYQoWQbh6zRtpyO4TgcJTQmkuvVyrbuJny0SlyFV8Sw== X-Received: by 2002:a17:90b:1c81:: with SMTP id oo1mr33359301pjb.5.1638136983374; Sun, 28 Nov 2021 14:03:03 -0800 (PST) Received: from localhost.localdomain (rrcs-66-91-142-162.west.biz.rr.com. [66.91.142.162]) by smtp.gmail.com with ESMTPSA id d9sm19749139pjs.2.2021.11.28.14.03.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Nov 2021 14:03:02 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 1/3] command: Ensure exceptions inheriting from BBHandledException are visible Date: Sun, 28 Nov 2021 12:02:46 -1000 Message-Id: <80348b68a34b7ec45a0496a4af7f2ae0c26488f0.1638136822.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 28 Nov 2021 22:03:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13119 From: Richard Purdie Previous changes allowed BBHandledException to be detected but not exceptions which inherit from it. Fix this. The code really needs totally reworking to preserve the exceptions. [YOCTO #14054] Signed-off-by: Richard Purdie (cherry picked from commit ef762d92df6c2554c6248e80212f984d9ec4c651) Signed-off-by: Steve Sakoman --- lib/bb/command.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/bb/command.py b/lib/bb/command.py index 6abf3866..07128027 100644 --- a/lib/bb/command.py +++ b/lib/bb/command.py @@ -74,8 +74,12 @@ class Command: result = command_method(self, commandline) except CommandError as exc: return None, exc.args[0] - except (Exception, SystemExit): + except (Exception, SystemExit) as exc: import traceback + if isinstance(exc, bb.BBHandledException): + # We need to start returning real exceptions here. Until we do, we can't + # tell if an exception is an instance of bb.BBHandledException + return None, "bb.BBHandledException()\n" + traceback.format_exc() return None, traceback.format_exc() else: return result, None From patchwork Sun Nov 28 22:02:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 91 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDA00C433F5 for ; Sun, 28 Nov 2021 22:03:07 +0000 (UTC) Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by mx.groups.io with SMTP id smtpd.web11.52933.1638136986771488819 for ; Sun, 28 Nov 2021 14:03:06 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=hk9Greif; spf=softfail (domain: sakoman.com, ip: 209.85.216.44, mailfrom: steve@sakoman.com) Received: by mail-pj1-f44.google.com with SMTP id iq11so11014767pjb.3 for ; Sun, 28 Nov 2021 14:03:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=wEsBcQJSpPY36KdFWL3/WMM84JeXzts7LXdsXDFer6U=; b=hk9Greif9vGKauNyv79Q5NTTCaNW1Lj1CUPXRFcFrX44tbJTC0m2U+W9wrC1/PvM0V yotTv/paNRaAEGVMG6heWK3jCRmtrBthQ7EIj9JCzmyGQLDXDhJxYjiHVO9GbXELoy+v jw6KYRy/JCuDxMyH+Y+3EQmgKmbfrXx8zTmgdirqMlhZua9wODGFUNjGZvSMGzxeqHVU B7EWqiOB6MGu7dd0wbL3UYP3TlkbAyAnSWPPIxxTBlxWkhJSNgFwFCNJ5mCkgeCzPYv3 O2NtZ9jBO15ikMm7rH1aimvxVvJGBfUvuCacLcgOVUrr/+pNh3+kgYmh86EKtch4uktC /sGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wEsBcQJSpPY36KdFWL3/WMM84JeXzts7LXdsXDFer6U=; b=AnvoZVBm0xJeQQMX2/VINj5NGaNKyg+pDpaJhHqNTyrxc04io44aFlIo6X6qrQT8TN hqYQOVx6emGTX5E5TjUUHqAOciZMPNHt+ojShP/I12Jk9svZPSX3PPExOCVIT2ljKacB SK0ymhjbHMnwlO1hNNe/tKYbVUorBm1w9mTDAECJh35ICx6nGwnfAvG0FHWPMZyaiy3P MH41zFEOM5hiZpk0iYBLOeA7EwpEtf+c545IyqzlKEZZKgchcvXKJw6zl7XtRHxK/2Eb peSHcvfBv75z9jf8MgFI52vkwvOkrGW6DKoxecAFxTYiRHzXyzHfMXl3QxkmflI0sFxu A/yQ== X-Gm-Message-State: AOAM532HN5no/+bq34YLLFqcyqARtOIjTuHIDj3JdYvhtkxdT6pVPZEn W0C0oYCkyQj/0+Qv++et6ugd9n4IW2V+c0pNoOI= X-Google-Smtp-Source: ABdhPJwgI1Cak4PaZ7xcw8QEXI/kFRJkJ+pMiiAvqyKPy/+ryjFzQF3yOVUHMIGKOeH1j1JYhePVvQ== X-Received: by 2002:a17:902:d490:b0:141:fd0f:5316 with SMTP id c16-20020a170902d49000b00141fd0f5316mr54180858plg.14.1638136985664; Sun, 28 Nov 2021 14:03:05 -0800 (PST) Received: from localhost.localdomain (rrcs-66-91-142-162.west.biz.rr.com. [66.91.142.162]) by smtp.gmail.com with ESMTPSA id d9sm19749139pjs.2.2021.11.28.14.03.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Nov 2021 14:03:04 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 2/3] tinfoil: When sending commands we need to process events Date: Sun, 28 Nov 2021 12:02:47 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 28 Nov 2021 22:03:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13120 From: Richard Purdie The server may be displaying useful information for the user through log messages so we should display anything that has been sent. Its either this or expecting every UI to implement this code around every command call which isn't good API. [YOCTO #14054] Signed-off-by: Richard Purdie (cherry picked from commit 64ae9d7e2fad804dd9e12706c6d76b4b22f9586b) Signed-off-by: Steve Sakoman --- lib/bb/tinfoil.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py index 8c9b6b8c..ae690389 100644 --- a/lib/bb/tinfoil.py +++ b/lib/bb/tinfoil.py @@ -465,7 +465,16 @@ class Tinfoil: commandline = [command] if params: commandline.extend(params) - result = self.server_connection.connection.runCommand(commandline) + try: + result = self.server_connection.connection.runCommand(commandline) + finally: + while True: + event = self.wait_event() + if not event: + break + if isinstance(event, logging.LogRecord): + if event.taskpid == 0 or event.levelno > logging.INFO: + self.logger.handle(event) if result[1]: raise TinfoilCommandFailed(result[1]) return result[0] From patchwork Sun Nov 28 22:02:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 92 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2264C433F5 for ; Sun, 28 Nov 2021 22:03:09 +0000 (UTC) Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by mx.groups.io with SMTP id smtpd.web10.53529.1638136989463756874 for ; Sun, 28 Nov 2021 14:03:09 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=CD45nBQe; spf=softfail (domain: sakoman.com, ip: 209.85.214.170, mailfrom: steve@sakoman.com) Received: by mail-pl1-f170.google.com with SMTP id y7so10530147plp.0 for ; Sun, 28 Nov 2021 14:03:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=h/Hu1zUkIsQdr+Xl34NZBEuj3boBT0QuxND4ErFziEo=; b=CD45nBQe93dxYYSX0D44+K8ZX0iNvSL+jekMrGK0BWnwlDiEACmyka1ksK1VkKrFr4 hjMQP1CjiaUQR051DrQUxpMgCwweXsWZ3ZPGeBRwNEu15uYglDkum8fRmXeH7aGyuQJT xORY0l28OfXA7i6WVig5AcGeYxveBrXNUppld5Wdl3lBECfbyQMZ9k5sPRuurH2KcvNE MySf/HcFBU35sdBbVscjnJjRVY7v15dzFDHLdl7KSs2MdOU/UsVAmElk5XkL9CKSlxs6 Sbbq0SF0dYllTjNApuQDmeLdyHlUhIuPMBbBeRhHyzaPRAU+1gsJywxHoLKVFvIpSIuE ZFLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h/Hu1zUkIsQdr+Xl34NZBEuj3boBT0QuxND4ErFziEo=; b=62r7ZptBItC/8K4g3k6c2tgl9ZTacsi6KkZKphT7DK3E6ii/B+H/bUeXtxUowKe9J7 7uekJzGHVlYc6zSBCXLzRsHBWXS4V98S8zdpSA/l/P0QAqQABz4WxKp3iFciUjpkGwEo y3HBrLCuFKoLtNFcoOeWq6uBlDXuSKHtsxjck+G4rqMx0yILcgKK+U3wk+DjPui1Yeoi vwgEjUDVAjR57WhM38ks18ItWtHsRgcD4g+V4v9qGplSpNIcElhX+HT5ayzR/Hl1dpYG tZcdktSAH5HpZH3EE2gCXIBQ9eSqvGFqKEYugpmXhrQLqyTwz+zHko5hz1ls81LHcGp3 0gZA== X-Gm-Message-State: AOAM533wciAIr29pHlJJ6W0xUjVCfYqd4Uh1NjhIqiAPgWeB10+gnVDB zashjKgXTBCT8TiZGhcwnxA3UZ1E+26zaMxtDZ0= X-Google-Smtp-Source: ABdhPJyHzY1BTzcxXOq4EDldRM8wVNhzItpJnMQoWTl8kGuTnRLD5j8oGjeW4PCuVjhSZ3XbnhhAJg== X-Received: by 2002:a17:90b:2504:: with SMTP id ns4mr32689668pjb.175.1638136988302; Sun, 28 Nov 2021 14:03:08 -0800 (PST) Received: from localhost.localdomain (rrcs-66-91-142-162.west.biz.rr.com. [66.91.142.162]) by smtp.gmail.com with ESMTPSA id d9sm19749139pjs.2.2021.11.28.14.03.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Nov 2021 14:03:07 -0800 (PST) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][dunfell][1.46][PATCH 3/3] process/knotty: Improve early exception handling Date: Sun, 28 Nov 2021 12:02:48 -1000 Message-Id: <7fdd43c5cbde38daa013076de2fdedcf3c3d3107.1638136822.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 28 Nov 2021 22:03:09 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13121 From: Richard Purdie The new server startup code means exceptions can happen when we aren't setup to show them to the user correctly, leading to ugly tracebacks. Add in some special case handling of BBHandledException to at least ensure that common case doesn't traceback and the user sees meaningful output. In the future, the logging setup can likely be improved, as can the way runCommand handles exceptions, they all should likely become real exceptions again on the UI side. [YOCTO #14022] [YOCTO #14033] Signed-off-by: Richard Purdie (cherry picked from commit 6059d0e77f60ddb679049bd34478f41b1ab7995d) Signed-off-by: Steve Sakoman --- lib/bb/server/process.py | 7 ++++++- lib/bb/ui/knotty.py | 21 +++++++++++++++++---- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 43061eb3..7b135762 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -348,7 +348,12 @@ class ServerCommunicator(): logger.info("No reply from server in 30s") if not self.recv.poll(30): raise ProcessTimeout("Timeout while waiting for a reply from the bitbake server (60s)") - return self.recv.get() + ret, exc = self.recv.get() + # Should probably turn all exceptions in exc back into exceptions? + # For now, at least handle BBHandledException + if exc and "BBHandledException" in exc: + raise bb.BBHandledException() + return ret, exc def updateFeatureSet(self, featureset): _, error = self.runCommand(["setFeatures", featureset]) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index 87e873d6..e70c2464 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -380,14 +380,27 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent", "bb.build.TaskProgress", "bb.event.ProcessStarted", "bb.event.ProcessProgress", "bb.event.ProcessFinished"] +def drain_events_errorhandling(eventHandler): + # We don't have logging setup, we do need to show any events we see before exiting + event = True + logger = bb.msg.logger_create('bitbake', sys.stdout) + while event: + event = eventHandler.waitEvent(0) + if isinstance(event, logging.LogRecord): + logger.handle(event) + def main(server, eventHandler, params, tf = TerminalFilter): - if not params.observe_only: - params.updateToServer(server, os.environ.copy()) + try: + if not params.observe_only: + params.updateToServer(server, os.environ.copy()) - includelogs, loglines, consolelogfile, logconfigfile = _log_settings_from_server(server, params.observe_only) + includelogs, loglines, consolelogfile, logconfigfile = _log_settings_from_server(server, params.observe_only) - loglevel, _ = bb.msg.constructLogOptions() + loglevel, _ = bb.msg.constructLogOptions() + except bb.BBHandledException: + drain_events_errorhandling(eventHandler) + return 1 if params.options.quiet == 0: console_loglevel = loglevel