From patchwork Wed Dec 11 17:41:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "chris.laplante@agilent.com" X-Patchwork-Id: 53948 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35C77E7717D for ; Wed, 11 Dec 2024 17:41:45 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (NAM12-DM6-obe.outbound.protection.outlook.com [40.107.243.68]) by mx.groups.io with SMTP id smtpd.web10.810.1733938902166612255 for ; Wed, 11 Dec 2024 09:41:42 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@agilent.com header.s=selector1 header.b=tS4tBLns; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: agilent.com, ip: 40.107.243.68, mailfrom: chris.laplante@agilent.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xKa1m4HRZXciQaSavo7+S4sMjB3B5OoUXzYkfafpj7HFWsO147KNvcVgfC5z9mjw5fcEQSu3JZGJ7hMdQKSifUMWE+qV+07MoLxCIjErRiWLo2CuLP03WYX6v8MNle0t1UxotabWN/LUDSj0SXMKvA/y+3zX6cIvmOMvgd9xD6tgFugGjfd2cWIa+WKJNJxqvXZLW+YvYCIqk5pEmivIAGqqfJ2CPrcYevQPEYDqkQtYf7eoH+PHxGXOUGCCUaoYOszp96ZnNVZYvqBIGtV4911ohBNruUdi1TaoJDtrVmdiWrBh2M66e0sbbEBaH4V+irlD12NdHtfXKs4LF8nSpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NtLcYY7FQ2+svxc21upBP3qDHO6vg0zmnFzIibKU5N4=; b=YxJBXrfn3XJAQTQt4KoQS63Jhg3gVHVwkfmRTHcX65ZOKE9V9YmvmUvr4YVYbFgp+Jhv6+MNkG+BHi5MzWXjQ+XqKGFOWjkGTwiKXbfB/VLCYaVlNB9jbSvBojmEwOPChWppmvW/UArdPqBf7E1Zp5usYGtLC8cZvZZsSpMaZBaAUDJJkI61GbYFfZrDq80NBsvSsTBdhzZ5H7KoYMr3qjpE8at8MbvL/dwPJd96SM2olP6ysiOgKPPHj7aeIHoq6cFIbJ6dYTZyjYHczdDMIrpEE9ovhSs3u2pWftjL1xLx8eeFbxx0VgqwjM6H/fi4/V29GoSoOpvUWWZF/lMb9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.25.218.37) smtp.rcpttodomain=lists.openembedded.org smtp.mailfrom=agilent.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=agilent.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=agilent.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NtLcYY7FQ2+svxc21upBP3qDHO6vg0zmnFzIibKU5N4=; b=tS4tBLns/9uzRbbVjdFbQM58CqXJ1uEfoFamuxTJ9RqdDyj5xLIPf+uXqyXAy0pxaHB3DhB0OupR/3Jpk82H/7J23EYrSpApuQ3sVnlh7OZSgYzMUnChAEx0LW50j2yzuKbpWosZ1BLJ3XGdeG6jC5c6LUro6U8oSZA89L4zfMKa558ylRb5Q5yYyqWalyI/9Vp2Ic8Oj+fBwNqgAjlhQb50AMf9uXjHhLVlcvXIL7nLg1Iq4Kux1W8iX0qPyKqKUHFDGRnfNJOOpS4uqb8qESDbZ2HEOgY0zlelts1BrcyMhL2hKQdQ4UDF/Hbl66ThiEWLSuhz89Rqsr9t8Nl3hw== Received: from BN9PR03CA0074.namprd03.prod.outlook.com (2603:10b6:408:fc::19) by SN7PR12MB8059.namprd12.prod.outlook.com (2603:10b6:806:32b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.18; Wed, 11 Dec 2024 17:41:35 +0000 Received: from BN2PEPF00004FBE.namprd04.prod.outlook.com (2603:10b6:408:fc:cafe::19) by BN9PR03CA0074.outlook.office365.com (2603:10b6:408:fc::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8182.16 via Frontend Transport; Wed, 11 Dec 2024 17:41:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.25.218.37) smtp.mailfrom=agilent.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=agilent.com; Received-SPF: Pass (protection.outlook.com: domain of agilent.com designates 192.25.218.37 as permitted sender) receiver=protection.outlook.com; client-ip=192.25.218.37; helo=edgeappmail.agilent.com; pr=C Received: from edgeappmail.agilent.com (192.25.218.37) by BN2PEPF00004FBE.mail.protection.outlook.com (10.167.243.184) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8251.15 via Frontend Transport; Wed, 11 Dec 2024 17:41:35 +0000 Received: from laplante-B650.lfs.agilent.com (192.25.126.5) by edgeappmail.agilent.com (192.25.218.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Wed, 11 Dec 2024 10:41:17 -0700 From: To: CC: Chris Laplante Subject: [PATCH] Make cooker 'skiplist' per-multiconfig/mc Date: Wed, 11 Dec 2024 12:41:27 -0500 Message-ID: <20241211174127.609537-1-chris.laplante@agilent.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FBE:EE_|SN7PR12MB8059:EE_ X-MS-Office365-Filtering-Correlation-Id: 1852cbef-ac30-45a2-05b3-08dd1a0b0ed1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|82310400026; X-Microsoft-Antispam-Message-Info: 5Tlk+MhmcIHYhXYoMQ6g/GvDXcy3Ay9xtQ2vuPpjnhBbm5kfZPjt6xZjfkdQGzV8xHRKvZf6+thCDR5GLE1+DlBz+i4KspI+169O7FgDVOiRu46yt2fqpN5MP9mi6eNrjikDcb4o/GNsjEYLTdBjS+UlxUK+5cJLpHCl7o9IKAyCXic59jNk6utYqJZPdfaYPraZo3IyE+yH6tk4L/ZJU4DCTudqbg852mszw4POHy1r/ok4+Edg9yREQYOBpdmfiIM+orXXvR7oZLKJAmGkAIXtpX4lsZT0uBraCZsA6DlY4hKWPrBtjmux4UG/Kk1yrkvwlFhSW0ZNRsORD5WlG7q9DYwhaTQuKqoxw5KFkFf6O1cnn+GY6hS4kQEEvEXgFVGz4mM0qCpSQQykIxglNYdvZQr/1CAO0TXHPVJMNH51z0zzYv06FOhp4vuqcaOEajTuVOjQbYkTRE7fUWgj5dwhfEGug4uz5KBk0JTNi0aSVahldlO+TlhG/zzAILH4XHnVUGt7R8WLtFJumgRxZxNQfpxFEfL6bZqH04ZCwhswFLBvydnitaGkuj8/M2rIwcFyqkkowJMClToXUm+iaqOKksBpzFiN8l/+6xAkYU9FcTdX5WTzK6kfwqUpdrFTCCIJmkvxXNbnmINN4GYU0X0ZksIYi1HW7FAy3Q7yEXkIxpiY7J6S11A3SBHF//GOjvl0k8Cq2fQBGla3WgxvMyRK/k2Okbz1/JJ4PlE3NlRoc+V8/1+ym6/TXo5gN0pFWiajjJceDdLs7e0UV7Uyo7yba5/W78cSRdTYz/HPeh+znJBu6WQ8sffu8vRD0X6n61MI4rw/XyOXfvBVcaX0clSbzn8bl9GfDhmL0dIkqY6D4fKMQh4jWXh/XIR+bCK85iGEW6cqN2IFE52MEvbXZpZpjgDLOe6A/t4QGTdvdgrFfASTTBLA5SpQJOKiOubMLMHBRfuarkKPMjjUc46Ec28Yl4SG7jFtmlOBmPYH8x4j7FzKkFUuAz7A7SL6uTgwVggiWo/yfh0pWFH9rM2U1dnS5sEG1uLFgk7+lyx/N2VadVhFsxTtx9W74v8AJCOw1tTZ7tuwxkKJh04usT83cavYowGW3whORQ7zrRXOSSFJA2hc5SC81vTYF0FLI/Nci4ztu/IGdDxuuRfyTRXMcRbM1fUL2SVSnHFOmco4KIfSTOgcLYEMlf1LYn+bowwO/q6Zr16XL8yy6HMoVth53GhxHuqSALCp1wNkj212ixrZ6aL45qWRIteFPZRfc2fthJUyOKnsI2G/p2jsf7x6vpMTIo4ZyZt3u8+U8TBGFdVIxO7EFwhE99HCbmWKDBEjyzfVa8D7X3ZqtUMvMrdLZwOrgyaU+xgIEaxzS1KJ87eQ2Wq4AdR3Y9uIKpl8kfYmBWrDOI9yM8c3+8NfF7ERDUUAle5RNxYMmZYdTRmLMdKrQMrx3VhBDG0FoiQXCWyv X-Forefront-Antispam-Report: CIP:192.25.218.37;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:edgeappmail.agilent.com;PTR:wpcosapexchedg1.windmz.agilent.com;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: agilent.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Dec 2024 17:41:35.0926 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1852cbef-ac30-45a2-05b3-08dd1a0b0ed1 X-MS-Exchange-CrossTenant-Id: a9c0bc09-8b46-4206-9351-2ba12fb4a5c0 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a9c0bc09-8b46-4206-9351-2ba12fb4a5c0;Ip=[192.25.218.37];Helo=[edgeappmail.agilent.com] X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: TreatMessagesAsInternal-BN2PEPF00004FBE.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8059 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 11 Dec 2024 17:41:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16910 From: Chris Laplante Previously, the cooker skiplist was shared across multiconfigs (including default ''). If you had a recipe that was incompatible with several multiconfigs for different reasons, then the displayed reason (i.e. the "ERROR: Nothing PROVIDES" and "* was skipped" messages) might vary across invocations of bitbake. This was caused by the random order in which recipes are parsed under different multiconfig contexts, with each skip reason overwriting the previously assigned reason. I hit this specificially when using COMPATIBLE_MACHINE, but COMPATIBLE_HOST (or anything using bb.parse.SkipRecipe) would have done it too. Signed-off-by: Chris Laplante --- lib/bb/command.py | 21 ++++++++++++++++++--- lib/bb/cooker.py | 11 ++++++----- lib/bb/tinfoil.py | 16 ++++++++++++---- lib/bblayers/query.py | 14 +++++++------- 4 files changed, 43 insertions(+), 19 deletions(-) diff --git a/lib/bb/command.py b/lib/bb/command.py index 7944faf981..baa7cbade1 100644 --- a/lib/bb/command.py +++ b/lib/bb/command.py @@ -421,15 +421,30 @@ class CommandsSync: return command.cooker.recipecaches[mc].pkg_dp getDefaultPreference.readonly = True + def getSkippedRecipes(self, command, params): + """ + Get the map of skipped recipes for the specified multiconfig/mc name (`params[0]`). + + Invoked by `bb.tinfoil.Tinfoil.get_skipped_recipes` + + :param command: Internally used parameter. + :param params: Parameter array. params[0] is multiconfig/mc name. If not given, then default mc '' is assumed. + :return: Dict whose keys are virtualfns and values are `bb.cooker.SkippedPackage` + """ + try: + mc = params[0] + except IndexError: + mc = '' + # Return list sorted by reverse priority order import bb.cache def sortkey(x): vfn, _ = x - realfn, _, mc = bb.cache.virtualfn2realfn(vfn) - return (-command.cooker.collections[mc].calc_bbfile_priority(realfn)[0], vfn) + realfn, _, item_mc = bb.cache.virtualfn2realfn(vfn) + return -command.cooker.collections[item_mc].calc_bbfile_priority(realfn)[0], vfn - skipdict = OrderedDict(sorted(command.cooker.skiplist.items(), key=sortkey)) + skipdict = OrderedDict(sorted(command.cooker.skiplist_by_mc[mc].items(), key=sortkey)) return list(skipdict.items()) getSkippedRecipes.readonly = True diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index ceaaac11ee..5b885cddd7 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -133,7 +133,8 @@ class BBCooker: self.baseconfig_valid = False self.parsecache_valid = False self.eventlog = None - self.skiplist = {} + # The skiplists, one per multiconfig + self.skiplist_by_mc = defaultdict(dict) self.featureset = CookerFeatures() if featureSet: for f in featureSet: @@ -614,8 +615,8 @@ class BBCooker: localdata = {} for mc in self.multiconfigs: - taskdata[mc] = bb.taskdata.TaskData(halt, skiplist=self.skiplist, allowincomplete=allowincomplete) - localdata[mc] = data.createCopy(self.databuilder.mcdata[mc]) + taskdata[mc] = bb.taskdata.TaskData(halt, skiplist=self.skiplist_by_mc[mc], allowincomplete=allowincomplete) + localdata[mc] = bb.data.createCopy(self.databuilder.mcdata[mc]) bb.data.expandKeys(localdata[mc]) current = 0 @@ -936,7 +937,7 @@ class BBCooker: for mc in self.multiconfigs: # First get list of recipes, including skipped recipefns = list(self.recipecaches[mc].pkg_fn.keys()) - recipefns.extend(self.skiplist.keys()) + recipefns.extend(self.skiplist_by_mc[mc].keys()) # Work out list of bbappends that have been applied applied_appends = [] @@ -2355,7 +2356,7 @@ class CookerParser(object): for virtualfn, info_array in result: if info_array[0].skipped: self.skipped += 1 - self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0]) + self.cooker.skiplist_by_mc[mc][virtualfn] = SkippedPackage(info_array[0]) self.bb_caches[mc].add_info(virtualfn, info_array, self.cooker.recipecaches[mc], parsed=parsed, watcher = self.cooker.add_filewatch) return True diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py index 0e937fba36..13b05cec2d 100644 --- a/lib/bb/tinfoil.py +++ b/lib/bb/tinfoil.py @@ -188,11 +188,19 @@ class TinfoilCookerAdapter: self._cache[name] = attrvalue return attrvalue + class TinfoilSkiplistByMcAdapter: + def __init__(self, tinfoil): + self.tinfoil = tinfoil + + def __getitem__(self, mc): + return self.tinfoil.get_skipped_recipes(mc) + def __init__(self, tinfoil): self.tinfoil = tinfoil self.multiconfigs = [''] + (tinfoil.config_data.getVar('BBMULTICONFIG') or '').split() self.collections = {} self.recipecaches = {} + self.skiplist_by_mc = self.TinfoilSkiplistByMcAdapter(tinfoil) for mc in self.multiconfigs: self.collections[mc] = self.TinfoilCookerCollectionAdapter(tinfoil, mc) self.recipecaches[mc] = self.TinfoilRecipeCacheAdapter(tinfoil, mc) @@ -201,8 +209,6 @@ class TinfoilCookerAdapter: # Grab these only when they are requested since they aren't always used if name in self._cache: return self._cache[name] - elif name == 'skiplist': - attrvalue = self.tinfoil.get_skipped_recipes() elif name == 'bbfile_config_priorities': ret = self.tinfoil.run_command('getLayerPriorities') bbfile_config_priorities = [] @@ -514,12 +520,12 @@ class Tinfoil: """ return defaultdict(list, self.run_command('getOverlayedRecipes', mc)) - def get_skipped_recipes(self): + def get_skipped_recipes(self, mc=''): """ Find recipes which were skipped (i.e. SkipRecipe was raised during parsing). """ - return OrderedDict(self.run_command('getSkippedRecipes')) + return OrderedDict(self.run_command('getSkippedRecipes', mc)) def get_all_providers(self, mc=''): return defaultdict(list, self.run_command('allProviders', mc)) @@ -533,6 +539,7 @@ class Tinfoil: def get_runtime_providers(self, rdep): return self.run_command('getRuntimeProviders', rdep) + # TODO: teach this method about mc def get_recipe_file(self, pn): """ Get the file name for the specified recipe/target. Raises @@ -541,6 +548,7 @@ class Tinfoil: """ best = self.find_best_provider(pn) if not best or (len(best) > 3 and not best[3]): + # TODO: pass down mc skiplist = self.get_skipped_recipes() taskdata = bb.taskdata.TaskData(None, skiplist=skiplist) skipreasons = taskdata.get_reasons(pn) diff --git a/lib/bblayers/query.py b/lib/bblayers/query.py index bfc18a7593..eb7cb465b4 100644 --- a/lib/bblayers/query.py +++ b/lib/bblayers/query.py @@ -142,10 +142,10 @@ skipped recipes will also be listed, with a " (skipped)" suffix. # Ensure we list skipped recipes # We are largely guessing about PN, PV and the preferred version here, # but we have no choice since skipped recipes are not fully parsed - skiplist = list(self.tinfoil.cooker.skiplist.keys()) - mcspec = 'mc:%s:' % mc + skiplist = list(self.tinfoil.cooker.skiplist_by_mc[mc].keys()) + if mc: - skiplist = [s[len(mcspec):] for s in skiplist if s.startswith(mcspec)] + skiplist = [s.removeprefix(f'mc:{mc}:') for s in skiplist] for fn in skiplist: recipe_parts = os.path.splitext(os.path.basename(fn))[0].split('_') @@ -162,7 +162,7 @@ skipped recipes will also be listed, with a " (skipped)" suffix. def print_item(f, pn, ver, layer, ispref): if not selected_layer or layer == selected_layer: if not bare and f in skiplist: - skipped = ' (skipped: %s)' % self.tinfoil.cooker.skiplist[f].skipreason + skipped = ' (skipped: %s)' % self.tinfoil.cooker.skiplist_by_mc[mc][f].skipreason else: skipped = '' if show_filenames: @@ -301,7 +301,7 @@ Lists recipes with the bbappends that apply to them as subitems. if self.show_appends_for_pn(pn, cooker_data, args.mc): appends = True - if not args.pnspec and self.show_appends_for_skipped(): + if not args.pnspec and self.show_appends_for_skipped(args.mc): appends = True if not appends: @@ -317,9 +317,9 @@ Lists recipes with the bbappends that apply to them as subitems. return self.show_appends_output(filenames, best_filename) - def show_appends_for_skipped(self): + def show_appends_for_skipped(self, mc): filenames = [os.path.basename(f) - for f in self.tinfoil.cooker.skiplist.keys()] + for f in self.tinfoil.cooker.skiplist_by_mc[mc].keys()] return self.show_appends_output(filenames, None, " (skipped)") def show_appends_output(self, filenames, best_filename, name_suffix = ''):