From patchwork Mon Nov 22 02:24:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 66 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DBF8C433F5 for ; Mon, 22 Nov 2021 02:24:58 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web10.1441.1637547896753547701 for ; Sun, 21 Nov 2021 18:24:57 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10175"; a="320933684" X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="320933684" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:24:57 -0800 X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="588848656" Received: from kkleong1-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.239.135]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:24:55 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.50][PATCH 1/8] bitbake: correct the collections vs collections.abc deprecation Date: Mon, 22 Nov 2021 10:24:42 +0800 Message-Id: <8e76cfcd9a88329fcbf80cfe4bbd839c32c9187f.1637547674.git.anuj.mittal@intel.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Nov 2021 02:24:58 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13085 From: Alexander Kanavin This becomes a hard error in python 3.10. Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie (cherry picked from commit ae219e1f7460077f4492b31ac91cef4cf9b17277) Signed-off-by: Anuj Mittal --- lib/bb/cache.py | 3 ++- lib/bb/data_smart.py | 2 +- lib/bb/persist_data.py | 5 +++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/bb/cache.py b/lib/bb/cache.py index 27eb2717..5f9c0a77 100644 --- a/lib/bb/cache.py +++ b/lib/bb/cache.py @@ -19,7 +19,8 @@ import os import logging import pickle -from collections import defaultdict, Mapping +from collections import defaultdict +from collections.abc import Mapping import bb.utils from bb import PrefixLoggerAdapter import re diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index 8291ca65..aa9ac2c8 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -17,7 +17,7 @@ BitBake build tools. # Based on functions from the base bb module, Copyright 2003 Holger Schurig import copy, re, sys, traceback -from collections import MutableMapping +from collections.abc import MutableMapping import logging import hashlib import bb, bb.codeparser diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py index c6a209fb..6f32d81a 100644 --- a/lib/bb/persist_data.py +++ b/lib/bb/persist_data.py @@ -12,6 +12,7 @@ currently, providing a key/value store accessed by 'domain'. # import collections +import collections.abc import contextlib import functools import logging @@ -19,7 +20,7 @@ import os.path import sqlite3 import sys import warnings -from collections import Mapping +from collections.abc import Mapping sqlversion = sqlite3.sqlite_version_info if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): @@ -29,7 +30,7 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): logger = logging.getLogger("BitBake.PersistData") @functools.total_ordering -class SQLTable(collections.MutableMapping): +class SQLTable(collections.abc.MutableMapping): class _Decorators(object): @staticmethod def retry(*, reconnect=True): From patchwork Mon Nov 22 02:24:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 67 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D46CC433FE for ; Mon, 22 Nov 2021 02:24:59 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web10.1441.1637547896753547701 for ; Sun, 21 Nov 2021 18:24:58 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10175"; a="320933687" X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="320933687" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:24:58 -0800 X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="588848664" Received: from kkleong1-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.239.135]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:24:57 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.50][PATCH 2/8] bitbake: correct deprecation warning in process.py Date: Mon, 22 Nov 2021 10:24:43 +0800 Message-Id: <4ce36c7ffbe518051f1a2b1979f58498d3c217e3.1637547674.git.anuj.mittal@intel.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Nov 2021 02:24:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13086 From: Alexander Kanavin Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie (cherry picked from commit aff52fe21a0b27f6302555c1e52a864550eb46ce) Signed-off-by: Anuj Mittal --- lib/bb/server/process.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 07bb785a..fcdce197 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -659,7 +659,7 @@ class BBUIEventQueue: self.reader = ConnectionReader(readfd) self.t = threading.Thread() - self.t.setDaemon(True) + self.t.daemon = True self.t.run = self.startCallbackHandler self.t.start() From patchwork Mon Nov 22 02:24:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 68 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CA1EC433F5 for ; Mon, 22 Nov 2021 02:25:00 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web10.1441.1637547896753547701 for ; Sun, 21 Nov 2021 18:25:00 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10175"; a="320933694" X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="320933694" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:00 -0800 X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="588848669" Received: from kkleong1-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.239.135]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:24:58 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.50][PATCH 3/8] bitbake: adjust parser error check for python 3.10 compatibility Date: Mon, 22 Nov 2021 10:24:44 +0800 Message-Id: <62e396e0899eb0d295229a8e49e35716b24f6038.1637547674.git.anuj.mittal@intel.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Nov 2021 02:25:00 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13087 From: Alexander Kanavin The change was introduced in https://github.com/python/cpython/commit/a698d52c3975c80b45b139b2f08402ec514dce75 Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie (cherry picked from commit 8d3c6cbbe6ee734495713ae3b99c609527842506) Signed-off-by: Anuj Mittal --- lib/bb/data_smart.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index aa9ac2c8..65857a9c 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -403,7 +403,7 @@ class DataSmart(MutableMapping): s = __expand_python_regexp__.sub(varparse.python_sub, s) except SyntaxError as e: # Likely unmatched brackets, just don't expand the expression - if e.msg != "EOL while scanning string literal": + if e.msg != "EOL while scanning string literal" and not e.msg.startswith("unterminated string literal"): raise if s == olds: break From patchwork Mon Nov 22 02:24:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 69 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E18AC433F5 for ; Mon, 22 Nov 2021 02:25:02 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web10.1441.1637547896753547701 for ; Sun, 21 Nov 2021 18:25:01 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10175"; a="320933698" X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="320933698" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:01 -0800 X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="588848673" Received: from kkleong1-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.239.135]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:00 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.50][PATCH 4/8] hashserv: let asyncio discover the running loop Date: Mon, 22 Nov 2021 10:24:45 +0800 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Nov 2021 02:25:02 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13088 From: Justin Bronder >From 3.10 documentation [1]: Deprecated since version 3.8, removed in version 3.10: The loop parameter. This function has been implicitly getting the current running loop since 3.7 This is fixed in master as a side-effect of cf9bc0310b0092bf52b61057405aeb51c86ba137 which is more intrusive but likewise drops the loop parameter. 1. https://docs.python.org/3/library/asyncio-stream.html#asyncio.open_connection Signed-off-by: Justin Bronder Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie (cherry picked from commit 74a1e71b1e677a482fdedc685a71a1798ad63920) Signed-off-by: Anuj Mittal --- lib/hashserv/server.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/hashserv/server.py b/lib/hashserv/server.py index a0dc0c17..df0fa0a0 100644 --- a/lib/hashserv/server.py +++ b/lib/hashserv/server.py @@ -521,7 +521,7 @@ class Server(object): def start_tcp_server(self, host, port): self.server = self.loop.run_until_complete( - asyncio.start_server(self.handle_client, host, port, loop=self.loop) + asyncio.start_server(self.handle_client, host, port) ) for s in self.server.sockets: @@ -546,7 +546,7 @@ class Server(object): # Work around path length limits in AF_UNIX os.chdir(os.path.dirname(path)) self.server = self.loop.run_until_complete( - asyncio.start_unix_server(self.handle_client, os.path.basename(path), loop=self.loop) + asyncio.start_unix_server(self.handle_client, os.path.basename(path)) ) finally: os.chdir(cwd) From patchwork Mon Nov 22 02:24:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 70 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B9FEC433EF for ; Mon, 22 Nov 2021 02:25:03 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web10.1441.1637547896753547701 for ; Sun, 21 Nov 2021 18:25:03 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10175"; a="320933703" X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="320933703" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:02 -0800 X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="588848675" Received: from kkleong1-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.239.135]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:01 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.50][PATCH 5/8] cooker: Handle parse threads disappearing to avoid hangs Date: Mon, 22 Nov 2021 10:24:46 +0800 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Nov 2021 02:25:03 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13089 From: Richard Purdie If one of the parse threads disappears during parsing for some reason, bitbake currently hangs. Avoid this (and zombie threads hanging around) by joining() threads which have exited. Signed-off-by: Richard Purdie (cherry picked from commit dc86a533d951d13643ce446533370da804782afc) Signed-off-by: Anuj Mittal --- lib/bb/cooker.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index b041d2a0..f12f4caa 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2036,6 +2036,7 @@ class Parser(multiprocessing.Process): result = pending.pop() else: try: + time.sleep(0.25) job = self.jobs.pop() except IndexError: self.results.close() @@ -2214,7 +2215,7 @@ class CookerParser(object): yield not cached, mc, infos def parse_generator(self): - while True: + while self.processes: if self.parsed >= self.toparse: break @@ -2228,6 +2229,14 @@ class CookerParser(object): raise value else: yield result + for process in self.processes.copy(): + if not process.is_alive(): + process.join() + self.processes.remove(process) + + if not (self.parsed >= self.toparse): + raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None) + def parse_next(self): result = [] From patchwork Mon Nov 22 02:24:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 71 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 821D3C433EF for ; Mon, 22 Nov 2021 02:25:05 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web10.1441.1637547896753547701 for ; Sun, 21 Nov 2021 18:25:04 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10175"; a="320933704" X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="320933704" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:04 -0800 X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="588848695" Received: from kkleong1-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.239.135]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:03 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.50][PATCH 6/8] cooker: Remove debug code, oops :( Date: Mon, 22 Nov 2021 10:24:47 +0800 Message-Id: <099a71a5d2da605568c8d447aaed7e82241ccdbc.1637547674.git.anuj.mittal@intel.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Nov 2021 02:25:05 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13090 From: Richard Purdie Signed-off-by: Richard Purdie (cherry picked from commit 19291665fa8b6cc331290f2542af3e8e653203f1) Signed-off-by: Anuj Mittal --- lib/bb/cooker.py | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index f12f4caa..c952c574 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2036,7 +2036,6 @@ class Parser(multiprocessing.Process): result = pending.pop() else: try: - time.sleep(0.25) job = self.jobs.pop() except IndexError: self.results.close() From patchwork Mon Nov 22 02:24:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 72 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EB69C433FE for ; Mon, 22 Nov 2021 02:25:06 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web10.1441.1637547896753547701 for ; Sun, 21 Nov 2021 18:25:06 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10175"; a="320933709" X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="320933709" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:05 -0800 X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="588848709" Received: from kkleong1-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.239.135]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:04 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.50][PATCH 7/8] cooker: Handle parsing results queue race Date: Mon, 22 Nov 2021 10:24:48 +0800 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Nov 2021 02:25:06 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13091 From: Richard Purdie The previous fix introduced a race where the queue might not be empty but all the parser processes have exited. Handle this correctly to avoid occasional errors. Signed-off-by: Richard Purdie (cherry picked from commit 8e7f2b6500e26610f52d128b48ca0a09bf6fb2cb) Signed-off-by: Anuj Mittal --- lib/bb/cooker.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index c952c574..e2a5dc43 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2214,24 +2214,28 @@ class CookerParser(object): yield not cached, mc, infos def parse_generator(self): - while self.processes: + empty = False + while self.processes or not empty: + for process in self.processes.copy(): + if not process.is_alive(): + process.join() + self.processes.remove(process) + if self.parsed >= self.toparse: break try: result = self.result_queue.get(timeout=0.25) except queue.Empty: + empty = True pass else: + empty = False value = result[1] if isinstance(value, BaseException): raise value else: yield result - for process in self.processes.copy(): - if not process.is_alive(): - process.join() - self.processes.remove(process) if not (self.parsed >= self.toparse): raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None) From patchwork Mon Nov 22 02:24:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 73 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D0EEC433F5 for ; Mon, 22 Nov 2021 02:25:08 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web10.1441.1637547896753547701 for ; Sun, 21 Nov 2021 18:25:07 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: anuj.mittal@intel.com) X-IronPort-AV: E=McAfee;i="6200,9189,10175"; a="320933711" X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="320933711" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:07 -0800 X-IronPort-AV: E=Sophos;i="5.87,253,1631602800"; d="scan'208";a="588848716" Received: from kkleong1-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.215.239.135]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2021 18:25:06 -0800 From: Anuj Mittal To: bitbake-devel@lists.openembedded.org Subject: [1.50][PATCH 8/8] cooker: Fix task-depends.dot for multiconfig targets Date: Mon, 22 Nov 2021 10:24:49 +0800 Message-Id: <0d3a46619da95260b5d0bbcc148660d8f6a707d0.1637547674.git.anuj.mittal@intel.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 22 Nov 2021 02:25:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/13092 From: Richard Purdie The right hand side of dependencies in the task dependency file generated by bitbake -g was missing multiconfig prefixes, corrupting the data. Fix this. [YOCTO #14621] Signed-off-by: Richard Purdie (cherry picked from commit 1d5ca721040c5e39aefa11219f62710de6587701) Signed-off-by: Anuj Mittal --- lib/bb/cooker.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index e2a5dc43..c946800a 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -815,7 +815,9 @@ class BBCooker: for dep in rq.rqdata.runtaskentries[tid].depends: (depmc, depfn, _, deptaskfn) = bb.runqueue.split_tid_mcfn(dep) deppn = self.recipecaches[depmc].pkg_fn[deptaskfn] - depend_tree["tdepends"][dotname].append("%s.%s" % (deppn, bb.runqueue.taskname_from_tid(dep))) + if depmc: + depmc = "mc:" + depmc + ":" + depend_tree["tdepends"][dotname].append("%s%s.%s" % (depmc, deppn, bb.runqueue.taskname_from_tid(dep))) if taskfn not in seen_fns: seen_fns.append(taskfn) packages = []