From patchwork Wed Aug 28 12:06:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 48396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E328C54EE3 for ; Wed, 28 Aug 2024 12:06:54 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.web11.11605.1724846810347639039 for ; Wed, 28 Aug 2024 05:06:50 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=gEMsRV2a; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-428243f928fso73270895e9.0 for ; Wed, 28 Aug 2024 05:06:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1724846808; x=1725451608; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=wWIZS6gp9qoX7fTbVOTp6kBRZEiQf4OEZggge0nukpg=; b=gEMsRV2aL25xWdMEbo89IzayvYbEUJpIAxtPo/WpbDfsACKzkLkJQj7rJ0AdJPTwvh O7qg/S20Ndm7tMeVM7l5F7EJMChRokQ6njoYbp/0vgVE3Z8TXnVCWZu3HPpaoEEqMYkC q1nD3QIJLpBOa8ZWyigVk3qL+JsqnxKqrIUTM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724846808; x=1725451608; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wWIZS6gp9qoX7fTbVOTp6kBRZEiQf4OEZggge0nukpg=; b=FNokoBIms0IR1BmOB6d2x9nKG1EAzTcsOPiFqS0pL7uW/H1agAoHLWcclQjDIoOLW8 7ZJfhhT/l5Z2pjcsV+Oj+ZHLtCmzu7u/Re2cL+yZ9/mVcN7qUnSMb94sxQGsGMneiFOd HNWMqsXLhSGXoPNe4D7m3Qhqa+mdbFJrVyWX/glRO3Y/9jNK+lLK8CaIWZuwhEP52G1x iRyGsrzLkazUZYxHO8hXSOT3iXDkbHlFJy7qayaX2NfQqmsmhYFpn0QcqG4slyjRRGnp se55pPey/f8JBfSf5vjNvXq+gKQVlGSeELGLtisgD6/fHWNGe+GbgOoj7VMBCSg0WlQk JOGw== X-Gm-Message-State: AOJu0YzRqvpj6UxaE/M3t/CECrwrN7Rv7NY394ACJBzphAwLnBI9us6S HKo18GroSQwe8csyyYzgves/Tvff34sdOQSh2MxtBFDDh1ZqczxLWRsx1CdhM5HlbfFe2/dLaD7 J X-Google-Smtp-Source: AGHT+IHS2+a7RQXmcckIQSNJ6qRXMMOh8bR8E7iD41hjCQ3tUZo5281AjhBg/E2GRCV/8N9p4Z9+PA== X-Received: by 2002:a05:600c:358d:b0:426:63f1:9a1b with SMTP id 5b1f17b1804b1-42acd5e7508mr124821795e9.33.1724846808303; Wed, 28 Aug 2024 05:06:48 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:354e:532a:804e:264e]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42ba639643esm20102415e9.1.2024.08.28.05.06.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Aug 2024 05:06:47 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH] coreparser: Allow code visitor expressions to be declared in metadata Date: Wed, 28 Aug 2024 13:06:46 +0100 Message-ID: <20240828120646.1458872-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 28 Aug 2024 12:06:54 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16523 Allow the metadata to define code visitor expressions which mean that custom dependencies can be handled in function libraries. An example is the qa.handle_error function in OE which can set something like: handle_error.visitorcode = """ if isinstance(args[0], ast.Constant) and isinstance(args[0].value, str): for i in ["ERROR_QA", "WARN_QA"]: if i not in self.contains: self.contains[i] = set() self.contains[i].add(args[0].value) else: self.warn(node.func, args[0]) self.execs.add(name) """ Meaning that it can have contains optimisations on ERROR and WARN_QA instead of hard dependencies. One drawback to this solution is the parsing order. Functions with visitorcode need to be defined before anything else references them or the visitor code will not function for the earlier references. Signed-off-by: Richard Purdie --- lib/bb/cache.py | 2 +- lib/bb/codeparser.py | 14 ++++++++++---- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/lib/bb/cache.py b/lib/bb/cache.py index c48feb7138..958652e0e3 100644 --- a/lib/bb/cache.py +++ b/lib/bb/cache.py @@ -28,7 +28,7 @@ import shutil logger = logging.getLogger("BitBake.Cache") -__cache_version__ = "155" +__cache_version__ = "156" def getCacheFile(path, filename, mc, data_hash): mcspec = '' diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py index b25a2133d2..01134edd4e 100644 --- a/lib/bb/codeparser.py +++ b/lib/bb/codeparser.py @@ -87,14 +87,17 @@ def add_module_functions(fn, functions, namespace): if e in functions: execs.remove(e) execs.add(namespace + "." + e) - modulecode_deps[name] = [parser.references.copy(), execs, parser.var_execs.copy(), parser.contains.copy(), parser.extra] + visitorcode = None + if hasattr(functions[f], 'visitorcode'): + visitorcode = compile(getattr(functions[f], "visitorcode"), f + ".visitorcode", 'exec') + modulecode_deps[name] = [parser.references.copy(), execs, parser.var_execs.copy(), parser.contains.copy(), parser.extra, visitorcode] #bb.warn("%s: %s\nRefs:%s Execs: %s %s %s" % (name, fn, parser.references, parser.execs, parser.var_execs, parser.contains)) def update_module_dependencies(d): for mod in modulecode_deps: excludes = set((d.getVarFlag(mod, "vardepsexclude") or "").split()) if excludes: - modulecode_deps[mod] = [modulecode_deps[mod][0] - excludes, modulecode_deps[mod][1] - excludes, modulecode_deps[mod][2] - excludes, modulecode_deps[mod][3], modulecode_deps[mod][4]] + modulecode_deps[mod] = [modulecode_deps[mod][0] - excludes, modulecode_deps[mod][1] - excludes, modulecode_deps[mod][2] - excludes, modulecode_deps[mod][3], modulecode_deps[mod][4], modulecode_deps[mod][5]] # A custom getstate/setstate using tuples is actually worth 15% cachesize by # avoiding duplication of the attribute names! @@ -161,7 +164,7 @@ class CodeParserCache(MultiProcessCache): # so that an existing cache gets invalidated. Additionally you'll need # to increment __cache_version__ in cache.py in order to ensure that old # recipe caches don't trigger "Taskhash mismatch" errors. - CACHE_VERSION = 12 + CACHE_VERSION = 14 def __init__(self): MultiProcessCache.__init__(self) @@ -261,7 +264,10 @@ class PythonParser(): def visit_Call(self, node): name = self.called_node_name(node.func) - if name and (name.endswith(self.getvars) or name.endswith(self.getvarflags) or name in self.containsfuncs or name in self.containsanyfuncs): + if name and name in modulecode_deps and modulecode_deps[name][5]: + args = node.args + exec(modulecode_deps[name][5]) + elif name and (name.endswith(self.getvars) or name.endswith(self.getvarflags) or name in self.containsfuncs or name in self.containsanyfuncs): if isinstance(node.args[0], ast.Constant) and isinstance(node.args[0].value, str): varname = node.args[0].value if name in self.containsfuncs and isinstance(node.args[1], ast.Constant):