From patchwork Wed Feb 16 11:31:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 3643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 471FFC433FE for ; Wed, 16 Feb 2022 11:32:07 +0000 (UTC) Received: from mail-wr1-f45.google.com (mail-wr1-f45.google.com [209.85.221.45]) by mx.groups.io with SMTP id smtpd.web12.10613.1645011122472814549 for ; Wed, 16 Feb 2022 03:32:02 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Ae7PqeL4; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.45, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f45.google.com with SMTP id q7so2894213wrc.13 for ; Wed, 16 Feb 2022 03:32:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=+ij+CFMkEXOLBRZAsM07EfPSGEt2Tn8wIxhs86vjXXU=; b=Ae7PqeL4tSOZIFubzWXKcvsHZIhJ/ZiUhqonGrfCKb+tjVtKh/oUc5pYEuoasHan31 Mz3gOIgcTgLTjQt7HgWH3tBtjtIkGvB7gw4sy4Auf95wxRil7mdBHgjwMwP6NnN192H8 u3r1RzDRSAgjRZ+LKX/oSf2bghzTrgKMM/+v8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=+ij+CFMkEXOLBRZAsM07EfPSGEt2Tn8wIxhs86vjXXU=; b=0PMHViu7AKXTwPlZ3aN3h2fdJu7D/Krs+2PeoWq9v6V/CFvaS/Xc+x/cK5l1Hu304T 78bwEHp5aVIhhAtbBA2+cHuoCW73JJn3F1EJSv4q6YshyhadUFOweAkHQ12Y2CjhCpPj 7dN0eQ3lp7gvzwWd/qj5r7KRhuZNDRXLGWqpBp0AlbKJ4mxq+s1HafxP64pVIx/w7AtB A8FMpqsWCVXKN+7O80MvXUIx+h9fVOcg4QEsjSpw5rVo6r8MqDrZUa1tltbty9035Z4c Qk6hsdGM4pD1CU5IP4mvZ0T+f01mG8EELEgGdD4SVp4tOCpyT7gsBn8ivhIIAOfyd3ya bSGw== X-Gm-Message-State: AOAM531gg8auHI5iroAJAMd08UdRFImeE3c0K/t+Ur8Lt6QpjJvLU9bm MMyugSzIEzfxrjJDDbihhrbY1HvkMLcLDny8 X-Google-Smtp-Source: ABdhPJzwv8CQXmNCaaI2inKxMp20sSTiQ84TB96zTDr9iGnl1cG2KiSIq02V4MF04KK/Y2ybA3X5Ug== X-Received: by 2002:adf:e4c2:0:b0:1e3:3e5d:bd65 with SMTP id v2-20020adfe4c2000000b001e33e5dbd65mr2021891wrm.422.1645011120279; Wed, 16 Feb 2022 03:32:00 -0800 (PST) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:a680:61af:4a53:1ad3]) by smtp.gmail.com with ESMTPSA id x18sm35476238wrw.17.2022.02.16.03.31.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 03:31:59 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/5] data_smart: Fix overrides file/line message additions Date: Wed, 16 Feb 2022 11:31:54 +0000 Message-Id: <20220216113158.2899964-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 16 Feb 2022 11:32:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13351 The overrides warning message is meant to show filename and line numbers but the variable names are incorrect and this wasn't working. Fix it. Signed-off-by: Richard Purdie --- lib/bb/data_smart.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index 543372d153..242476be0e 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -494,10 +494,10 @@ class DataSmart(MutableMapping): if not var.startswith("__anon_") and ("_append" in var or "_prepend" in var or "_remove" in var): info = "%s" % var - if "filename" in loginfo: - info += " file: %s" % loginfo[filename] - if "lineno" in loginfo: - info += " line: %s" % loginfo[lineno] + if "file" in loginfo: + info += " file: %s" % loginfo["file"] + if "line" in loginfo: + info += " line: %s" % loginfo["line"] bb.fatal("Variable %s contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake." % info) self.expand_cache = {} From patchwork Wed Feb 16 11:31:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 3645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42132C4332F for ; Wed, 16 Feb 2022 11:32:07 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.web12.10614.1645011123351027335 for ; Wed, 16 Feb 2022 03:32:03 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=DzA10kxZ; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f50.google.com with SMTP id x3-20020a05600c21c300b0037c01ad715bso1377202wmj.2 for ; Wed, 16 Feb 2022 03:32:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=0vNaQVkcsQcQAXLW4sZiAVr2DYmpK530NHp6vGPxSWQ=; b=DzA10kxZjhUAUzbomJtMsexPHaaF31uyAhW9lJFb+QV00eAyRcfOv5nz+6NWFkMlhI YcKY0bU0L/w1rF9bbIev6ZNV6g3oWxD066fE/X9K6hkRVj/mjTIZ+Gds2NMeTzdDGO+u TubQTx5fd1BKiZOlj1JHwreZKzhIs+D34GVYQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0vNaQVkcsQcQAXLW4sZiAVr2DYmpK530NHp6vGPxSWQ=; b=rb/Xg25yjZkdXxpREAb7Fj9hWUAhv5mES1jy/aJhJhsaJW4vx8hvesu/z/fYDqXq84 X8NRZT4lLdqqZxXJeTB1vJHsRdrAqNcjzDGwsT3wCo+G/VpD28Fiq5jLu5IFnMr+IC0A qUm+IfKU3SgOhsfNK/7gRQxU/TbVKOezZV3KftGFXT+ZBlnLuvrVHh3Goe0qjfyq+5dD IDLtQnncSIQjjnyzhL4+C1Ji4Q3I52Ci0MmtkhBIBNwSN78He8vpx7qnuTrT//wNe+Sf YJfdvnRSYe+YddjXtjyCjKTHoT3IAzo75OxCLob7j53MwkXcnNeLanRE8wFSuko2ttQO P/rw== X-Gm-Message-State: AOAM532Qsu7DZSJ3jofB0XUoi3oDAps5IISs2OAPLb8SumnvE1XOgBPQ RjsZ+r8igfO5g3aB6Igzxj2F8cxMGcpl4AOP X-Google-Smtp-Source: ABdhPJz6fTr69RuT+pM6ZLBcXRRVukCIndmTepATowafFcjpOdmrBpABGJQqelSp/k0NJjEv8HKmaw== X-Received: by 2002:a05:600c:49a9:b0:349:bf22:468b with SMTP id h41-20020a05600c49a900b00349bf22468bmr1210625wmp.136.1645011121391; Wed, 16 Feb 2022 03:32:01 -0800 (PST) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:a680:61af:4a53:1ad3]) by smtp.gmail.com with ESMTPSA id x18sm35476238wrw.17.2022.02.16.03.32.00 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 03:32:00 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/5] cooker: Improve parsing failure from handled exception usability Date: Wed, 16 Feb 2022 11:31:55 +0000 Message-Id: <20220216113158.2899964-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216113158.2899964-1-richard.purdie@linuxfoundation.org> References: <20220216113158.2899964-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 16 Feb 2022 11:32:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13352 When a recipe raises a BBHandledException, it means the error was already shown to the user. Adding an additional one here isn't helpful. What is helpful is to mention that parsing was halted. Tweak the code to do this with improves the messages the user sees and helps understand what happened. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 08e45e79d0..d1d4e32595 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2167,6 +2167,8 @@ class CookerParser(object): self.total) bb.event.fire(event, self.cfgdata) + else: + bb.error("Parsing halted due to errors") for process in self.processes: self.parser_quit.put(None) @@ -2257,7 +2259,7 @@ class CookerParser(object): return False except bb.BBHandledException as exc: self.error += 1 - logger.error('Failed to parse recipe: %s' % exc.recipe) + logger.debug('Failed to parse recipe: %s' % exc.recipe) self.shutdown(clean=False, force=True) return False except ParsingFailure as exc: From patchwork Wed Feb 16 11:31:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 3641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7659CC433EF for ; Wed, 16 Feb 2022 11:32:05 +0000 (UTC) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by mx.groups.io with SMTP id smtpd.web10.10660.1645011124395446328 for ; Wed, 16 Feb 2022 03:32:04 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=emWoe5st; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f44.google.com with SMTP id u1so2918180wrg.11 for ; Wed, 16 Feb 2022 03:32:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=mf9cRxB2jcbipoDLPACEU6B9nlQVyv01TBp2ijmDQiU=; b=emWoe5stP09Q++W3V5Dl8s1uYt2GTGlaHQgwi3vNjbQRkjwaVah0PfUX31fwHqsNKV pcupYXuFqtd+1HzMDJRLIsGQon95Y1jUmFzEAuAvusYMCsDzF/GUbESyZELWSIEYVYDx fPDRZZVnqC7B4FpW4UyMVOVn7lwTn3NsOkN2Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mf9cRxB2jcbipoDLPACEU6B9nlQVyv01TBp2ijmDQiU=; b=fnbelarT3PU/cv4boOWAS5Cq8Ml2DvSB8WPF8foCzlFYucwipfja0mcUEvD06Nyx82 WF/fWAx+1mVvEGYbv91/6TYseXIQh8fDYFarsTdkWp1juQW51WNvbwiP9oRd2i5w5Z+9 qut0Gaky6HaqN26gIfO3VGVKCat+zTEpH8rQKW/XwnGmYeXrUXKZrmz9v7Qn/luJjcSl R5pSb0GtHFEPlfYuoOVexny91QWrJ478ywQ/mf+cIRl4Te4oB4iwvSG8KmIG7SgrCW9K lU83ERhTQx8/ZNs9GBANe7J6vCVrvDxyvdAdVsgZ/AdCbBnhkPVUK1BA1RbkpqM9wnak cuNQ== X-Gm-Message-State: AOAM531K3VO7jt0eDJjSttPUKkAmf286ugqiRnFEUUH7vk7SesgaRa3C ws+Z2Tff+8AlEfMNWGwJSMbsJ6BgJlqDgN10 X-Google-Smtp-Source: ABdhPJwuhMc/z88C3VvJ7cK9cMkci4hI2ut5/MAr8ZES0fpucFQfOCQvb413CnH+EloGOHL1tuTaJw== X-Received: by 2002:adf:c385:0:b0:1e3:3ac:5b1b with SMTP id p5-20020adfc385000000b001e303ac5b1bmr2020529wrf.545.1645011122356; Wed, 16 Feb 2022 03:32:02 -0800 (PST) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:a680:61af:4a53:1ad3]) by smtp.gmail.com with ESMTPSA id x18sm35476238wrw.17.2022.02.16.03.32.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 03:32:01 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/5] msg: Add a bb.warnonce() log method Date: Wed, 16 Feb 2022 11:31:56 +0000 Message-Id: <20220216113158.2899964-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216113158.2899964-1-richard.purdie@linuxfoundation.org> References: <20220216113158.2899964-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 16 Feb 2022 11:32:05 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13353 This adds a log level and logging function call to use it where the warning will only be displayed once, regardless of how many times the message is logged. This has to be done either in the cooker or on the UI side. I've opted for the UI side since display control is really a UI issue but it uses a common library filter function to enable it which can be reused elsewhere. The knotty message displayed as the build summary is tweaked to make sense when the numbers won't match since it will still count the number of times it was logged and this is probably helpful for debugging in some cases so I've deliberately left it that way. Signed-off-by: Richard Purdie --- lib/bb/__init__.py | 6 ++++++ lib/bb/msg.py | 15 +++++++++++++++ lib/bb/ui/knotty.py | 17 ++++++++++------- 3 files changed, 31 insertions(+), 7 deletions(-) diff --git a/lib/bb/__init__.py b/lib/bb/__init__.py index e01b8d5256..4b6ad5cbc8 100644 --- a/lib/bb/__init__.py +++ b/lib/bb/__init__.py @@ -71,6 +71,9 @@ class BBLoggerMixin(object): def verbnote(self, msg, *args, **kwargs): return self.log(logging.INFO + 2, msg, *args, **kwargs) + def warnonce(self, msg, *args, **kwargs): + return self.log(logging.WARNING - 1, msg, *args, **kwargs) + Logger = logging.getLoggerClass() class BBLogger(Logger, BBLoggerMixin): def __init__(self, name, *args, **kwargs): @@ -157,6 +160,9 @@ def verbnote(*args): def warn(*args): mainlogger.warning(''.join(args)) +def warnonce(*args): + mainlogger.warnonce(''.join(args)) + def error(*args, **kwargs): mainlogger.error(''.join(args), extra=kwargs) diff --git a/lib/bb/msg.py b/lib/bb/msg.py index 291b38ff7f..ae079c4596 100644 --- a/lib/bb/msg.py +++ b/lib/bb/msg.py @@ -31,6 +31,7 @@ class BBLogFormatter(logging.Formatter): VERBNOTE = logging.INFO + 2 ERROR = logging.ERROR WARNING = logging.WARNING + WARNONCE = logging.WARNING - 1 CRITICAL = logging.CRITICAL levelnames = { @@ -42,6 +43,7 @@ class BBLogFormatter(logging.Formatter): PLAIN : '', VERBNOTE: 'NOTE', WARNING : 'WARNING', + WARNONCE : 'WARNING', ERROR : 'ERROR', CRITICAL: 'ERROR', } @@ -58,6 +60,7 @@ class BBLogFormatter(logging.Formatter): PLAIN : BASECOLOR, VERBNOTE: BASECOLOR, WARNING : YELLOW, + WARNONCE : YELLOW, ERROR : RED, CRITICAL: RED, } @@ -121,6 +124,18 @@ class BBLogFilter(object): return True return False +class LogFilterWarnOnce(logging.Filter): + def __init__(self): + self.seen_warnings = set() + + def filter(self, record): + msg = record.msg + if record.levelno == bb.msg.BBLogFormatter.WARNONCE: + if record.msg in self.seen_warnings: + return False + self.seen_warnings.add(record.msg) + return True + class LogFilterGEQLevel(logging.Filter): def __init__(self, level): self.strlevel = str(level) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index 484545a684..1c46ec843b 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -418,7 +418,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): "formatter": "BitBake.consoleFormatter", "level": console_loglevel, "stream": "ext://sys.stdout", - "filters": ["BitBake.stdoutFilter"], + "filters": ["BitBake.warnonceFilter", "BitBake.stdoutFilter"], ".": { "is_console": True, }, @@ -428,7 +428,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): "formatter": "BitBake.consoleFormatter", "level": loglevel, "stream": "ext://sys.stderr", - "filters": ["BitBake.stderrFilter"], + "filters": ["BitBake.warnonceFilter", "BitBake.stderrFilter"], ".": { "is_console": True, }, @@ -443,7 +443,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): "formatter": "BitBake.consoleFormatter", "level": 1, "stream": "ext://sys.stdout", - "filters": ["BitBake.verbconsoleFilter"], + "filters": ["BitBake.warnonceFilter", "BitBake.verbconsoleFilter"], ".": { "is_console": True, }, @@ -476,6 +476,9 @@ def main(server, eventHandler, params, tf = TerminalFilter): "()": "bb.msg.LogFilterLTLevel", "level": console_loglevel }, + "BitBake.warnonceFilter": { + "()": "bb.msg.LogFilterWarnOnce", + }, }, "loggers": { "BitBake": { @@ -661,10 +664,10 @@ def main(server, eventHandler, params, tf = TerminalFilter): continue # Prefix task messages with recipe/task - if event.taskpid in helper.pidmap and event.levelno != bb.msg.BBLogFormatter.PLAIN: + if event.taskpid in helper.pidmap and event.levelno != bb.msg.BBLogFormatter.PLAIN and event.levelno != bb.msg.BBLogFormatter.WARNONCE: taskinfo = helper.running_tasks[helper.pidmap[event.taskpid]] event.msg = taskinfo['title'] + ': ' + event.msg - if hasattr(event, 'fn'): + if hasattr(event, 'fn') and event.levelno != bb.msg.BBLogFormatter.WARNONCE: event.msg = event.fn + ': ' + event.msg logging.getLogger(event.name).handle(event) continue @@ -875,8 +878,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): for failure in taskfailures: summary += "\n %s" % failure if warnings: - summary += pluralise("\nSummary: There was %s WARNING message shown.", - "\nSummary: There were %s WARNING messages shown.", warnings) + summary += pluralise("\nSummary: There was %s WARNING message.", + "\nSummary: There were %s WARNING messages.", warnings) if return_value and errors: summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.", "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors) From patchwork Wed Feb 16 11:31:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 3642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46E78C433F5 for ; Wed, 16 Feb 2022 11:32:06 +0000 (UTC) Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by mx.groups.io with SMTP id smtpd.web11.10584.1645011125474752576 for ; Wed, 16 Feb 2022 03:32:06 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=iGeV9QxU; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f50.google.com with SMTP id q7so2894428wrc.13 for ; Wed, 16 Feb 2022 03:32:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=0nQL8md43KDy3DnfI2XVd9tLIfVIy83dPdA6nNxZ1vg=; b=iGeV9QxUKU5w6TCE1JbCMdeiXrrt216MMHSzlc/HCWR5v+x1y69rCb+Qor5vogYOm6 8Xi3y2rWb3whnI5E28hXh4EFasCYrmRxAxixVug4MgAPSrank8z4ES2ahuOa7Q/Kz/mU h+d8xu0ItoTyrb7gCTfa2uAbbCe4BCfKvb6Iw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0nQL8md43KDy3DnfI2XVd9tLIfVIy83dPdA6nNxZ1vg=; b=5MoRRhhoRb/oBvSq0+cA1+xuGB7YOvINCid7Eyr3BYVfIk9ugE8ezGCqt8t8sDyM0Y LOU/fJHxNKD8+tPiOIoGGiiT8WCeUI2W+B6ichZs/dlamkVDkDJqQgjWyd3XEZd6LORR yDloST+4Z0BW9gnOfgT7jxA3wvbBB2S6PAbRaHcY/ciHifazoKjUfJjD8JNlzWW4eQUq 6tRhUGWoGHqsQCYQ73nSiKUtez8mx3wTjzxyMX2G8YK8BsTeebUXO3ir3CEPhvMjch7q ccKuRYot8KUij53JKOQ8trM4nlTdmPVfALBfxs17vNHMmRYLwK7U63vxxefpDeA8VTt4 pd8A== X-Gm-Message-State: AOAM530MDGKrhUqKDYBs8LmKKnn85vUJ/nICtVXJBKoGNPQVC/AIocrA kGoWZvXJzaGEMiERqD0+Ni6kfRZCCs5zTLzZ X-Google-Smtp-Source: ABdhPJwSgY1rv589w7LK2ttiOPI6b3lwS+v0WIfcwQ/g0G9NFG8V9IHYdTG8yYNnBCoRW0mkrKinXA== X-Received: by 2002:a5d:4bc9:0:b0:1e5:a66b:df03 with SMTP id l9-20020a5d4bc9000000b001e5a66bdf03mr2036080wrt.610.1645011123398; Wed, 16 Feb 2022 03:32:03 -0800 (PST) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:a680:61af:4a53:1ad3]) by smtp.gmail.com with ESMTPSA id x18sm35476238wrw.17.2022.02.16.03.32.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 03:32:02 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 4/5] msg: Add bb.erroronce() to logging Date: Wed, 16 Feb 2022 11:31:57 +0000 Message-Id: <20220216113158.2899964-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216113158.2899964-1-richard.purdie@linuxfoundation.org> References: <20220216113158.2899964-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 16 Feb 2022 11:32:06 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13354 Similar to the warnonce addition in the previous commit, add a similarly behaving version for error messages. Tweak the log filter further rather than adding an additional one. Signed-off-by: Richard Purdie --- lib/bb/__init__.py | 7 +++++++ lib/bb/msg.py | 10 +++++++++- lib/bb/ui/knotty.py | 20 ++++++++++---------- 3 files changed, 26 insertions(+), 11 deletions(-) diff --git a/lib/bb/__init__.py b/lib/bb/__init__.py index 4b6ad5cbc8..269f65edce 100644 --- a/lib/bb/__init__.py +++ b/lib/bb/__init__.py @@ -74,6 +74,10 @@ class BBLoggerMixin(object): def warnonce(self, msg, *args, **kwargs): return self.log(logging.WARNING - 1, msg, *args, **kwargs) + def erroronce(self, msg, *args, **kwargs): + return self.log(logging.ERROR - 1, msg, *args, **kwargs) + + Logger = logging.getLoggerClass() class BBLogger(Logger, BBLoggerMixin): def __init__(self, name, *args, **kwargs): @@ -166,6 +170,9 @@ def warnonce(*args): def error(*args, **kwargs): mainlogger.error(''.join(args), extra=kwargs) +def erroronce(*args): + mainlogger.erroronce(''.join(args)) + def fatal(*args, **kwargs): mainlogger.critical(''.join(args), extra=kwargs) raise BBHandledException() diff --git a/lib/bb/msg.py b/lib/bb/msg.py index ae079c4596..c2bd8aaf40 100644 --- a/lib/bb/msg.py +++ b/lib/bb/msg.py @@ -30,6 +30,7 @@ class BBLogFormatter(logging.Formatter): PLAIN = logging.INFO + 1 VERBNOTE = logging.INFO + 2 ERROR = logging.ERROR + ERRORONCE = logging.ERROR - 1 WARNING = logging.WARNING WARNONCE = logging.WARNING - 1 CRITICAL = logging.CRITICAL @@ -45,6 +46,7 @@ class BBLogFormatter(logging.Formatter): WARNING : 'WARNING', WARNONCE : 'WARNING', ERROR : 'ERROR', + ERRORONCE : 'ERROR', CRITICAL: 'ERROR', } @@ -62,6 +64,7 @@ class BBLogFormatter(logging.Formatter): WARNING : YELLOW, WARNONCE : YELLOW, ERROR : RED, + ERRORONCE : RED, CRITICAL: RED, } @@ -124,9 +127,10 @@ class BBLogFilter(object): return True return False -class LogFilterWarnOnce(logging.Filter): +class LogFilterShowOnce(logging.Filter): def __init__(self): self.seen_warnings = set() + self.seen_errors = set() def filter(self, record): msg = record.msg @@ -134,6 +138,10 @@ class LogFilterWarnOnce(logging.Filter): if record.msg in self.seen_warnings: return False self.seen_warnings.add(record.msg) + if record.levelno == bb.msg.BBLogFormatter.ERRORONCE: + if record.msg in self.seen_errors: + return False + self.seen_errors.add(record.msg) return True class LogFilterGEQLevel(logging.Filter): diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index 1c46ec843b..93eb4b56cb 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -418,7 +418,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): "formatter": "BitBake.consoleFormatter", "level": console_loglevel, "stream": "ext://sys.stdout", - "filters": ["BitBake.warnonceFilter", "BitBake.stdoutFilter"], + "filters": ["BitBake.showonceFilter", "BitBake.stdoutFilter"], ".": { "is_console": True, }, @@ -428,7 +428,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): "formatter": "BitBake.consoleFormatter", "level": loglevel, "stream": "ext://sys.stderr", - "filters": ["BitBake.warnonceFilter", "BitBake.stderrFilter"], + "filters": ["BitBake.showonceFilter", "BitBake.stderrFilter"], ".": { "is_console": True, }, @@ -443,7 +443,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): "formatter": "BitBake.consoleFormatter", "level": 1, "stream": "ext://sys.stdout", - "filters": ["BitBake.warnonceFilter", "BitBake.verbconsoleFilter"], + "filters": ["BitBake.showonceFilter", "BitBake.verbconsoleFilter"], ".": { "is_console": True, }, @@ -476,8 +476,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): "()": "bb.msg.LogFilterLTLevel", "level": console_loglevel }, - "BitBake.warnonceFilter": { - "()": "bb.msg.LogFilterWarnOnce", + "BitBake.showonceFilter": { + "()": "bb.msg.LogFilterShowOnce", }, }, "loggers": { @@ -650,7 +650,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): if isinstance(event, logging.LogRecord): lastprint = time.time() printinterval = 5000 - if event.levelno >= bb.msg.BBLogFormatter.ERROR: + if event.levelno >= bb.msg.BBLogFormatter.ERRORONCE: errors = errors + 1 return_value = 1 elif event.levelno == bb.msg.BBLogFormatter.WARNING: @@ -664,10 +664,10 @@ def main(server, eventHandler, params, tf = TerminalFilter): continue # Prefix task messages with recipe/task - if event.taskpid in helper.pidmap and event.levelno != bb.msg.BBLogFormatter.PLAIN and event.levelno != bb.msg.BBLogFormatter.WARNONCE: + if event.taskpid in helper.pidmap and event.levelno not in [bb.msg.BBLogFormatter.PLAIN, bb.msg.BBLogFormatter.WARNONCE, bb.msg.BBLogFormatter.ERRORONCE]: taskinfo = helper.running_tasks[helper.pidmap[event.taskpid]] event.msg = taskinfo['title'] + ': ' + event.msg - if hasattr(event, 'fn') and event.levelno != bb.msg.BBLogFormatter.WARNONCE: + if hasattr(event, 'fn') and event.levelno not in [bb.msg.BBLogFormatter.WARNONCE, bb.msg.BBLogFormatter.ERRORONCE]: event.msg = event.fn + ': ' + event.msg logging.getLogger(event.name).handle(event) continue @@ -881,8 +881,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): summary += pluralise("\nSummary: There was %s WARNING message.", "\nSummary: There were %s WARNING messages.", warnings) if return_value and errors: - summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.", - "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors) + summary += pluralise("\nSummary: There was %s ERROR message, returning a non-zero exit code.", + "\nSummary: There were %s ERROR messages, returning a non-zero exit code.", errors) if summary and params.options.quiet == 0: print(summary) From patchwork Wed Feb 16 11:31:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 3644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47B5FC43219 for ; Wed, 16 Feb 2022 11:32:07 +0000 (UTC) Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by mx.groups.io with SMTP id smtpd.web12.10615.1645011126119857005 for ; Wed, 16 Feb 2022 03:32:06 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=fq1+t2YE; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.51, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f51.google.com with SMTP id u2so1808694wrw.1 for ; Wed, 16 Feb 2022 03:32:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=U67L4WQ3xSjVipoAMgRt0JRoAgI5A+rg5eoYeq5BOZg=; b=fq1+t2YE7b7MDbK6Pwjx8zGWCdmXfbbUKXdDApnt0w0U1SS0IhJdcxcxemB9CRFCat D1M9hDkOJmfJujLN9yf7gnLmUuLlERcfIR1UZ/jSXaMNi23W0HmYyG4B3AfksaMGAmTV D6uneUpGR082dMQHIiaBH/0tuh1WzPhHiQ/6w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U67L4WQ3xSjVipoAMgRt0JRoAgI5A+rg5eoYeq5BOZg=; b=p0oWF+8dingG2twwfBfce1qE8xlv0kbRm6Q1n8IbWLwSxHu7MiE0WihhvIktpP5xvK K3S6VH94qyUNXfragMhRqqEwofEPdnpCMYYZxEN32QdL/ZjjATBq9WWIAMl2FPjyhLHG tKz3LMTHnMoIQSexvE70Ivfv6Qhr8OvUdY3peZzHBcbWDz5YD66DleBkAp939Zera70W EJbTLDPEKc0fPU34RAS8Hajc/Fjxsybf8i6SG7SsYJR46Fh2tVr1UZbIf5e75imEZQWL lD4U4Ycg0WBYTCRvWY7oG6ml5q4DpNG4FDqAMv/qj6LX1MoN1N6idOPzqwOURBnk9ba7 M83Q== X-Gm-Message-State: AOAM532/DMTBwyI2ta2L40wJVHkkozszAp59BO72C22z5CNlh9HS1UEE 5/0ENz5dYAGCRX7D6c6Qp+CVd/d2sylEzikB X-Google-Smtp-Source: ABdhPJwjmXmSDNh0jav2nuZxXaUuankGWVJzizLo57KdVhFHnsC5fVg6NhEbcXbpTVM6QzQJabx1gA== X-Received: by 2002:adf:fb06:0:b0:1e6:8b27:f1ea with SMTP id c6-20020adffb06000000b001e68b27f1eamr1924091wrr.353.1645011124343; Wed, 16 Feb 2022 03:32:04 -0800 (PST) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:a680:61af:4a53:1ad3]) by smtp.gmail.com with ESMTPSA id x18sm35476238wrw.17.2022.02.16.03.32.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 03:32:03 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 5/5] data_smart: Add hasOverrides method to public datastore API Date: Wed, 16 Feb 2022 11:31:58 +0000 Message-Id: <20220216113158.2899964-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216113158.2899964-1-richard.purdie@linuxfoundation.org> References: <20220216113158.2899964-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 16 Feb 2022 11:32:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13355 There are some cases where the metadata needs to check if a variable has any overrides set, even if they are currently inactive. That code currently pokes into datastore internals. Add API instead to replace and avoid that. Signed-off-by: Richard Purdie --- lib/bb/data_smart.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index 242476be0e..cb2d4b620f 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -488,6 +488,8 @@ class DataSmart(MutableMapping): else: self.initVar(var) + def hasOverrides(self, var): + return var in self.overridedata def setVar(self, var, value, **loginfo): #print("var=" + str(var) + " val=" + str(value))