From patchwork Fri Jun 27 13:48:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70746C7EE31 for ; Fri, 27 Jun 2025 13:48:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.14150.1751032133765073720 for ; Fri, 27 Jun 2025 06:48:53 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 11C9F1A00 for ; Fri, 27 Jun 2025 06:48:36 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DCFB43F66E for ; Fri, 27 Jun 2025 06:48:52 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 1/9] go-mod.bbclass: Calculate GO_MOD_CACHE_DIR relative to ${UNPACKDIR} Date: Fri, 27 Jun 2025 14:48:42 +0100 Message-ID: <20250627134850.152269-1-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:48:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219413 From: Peter Kjellerstedt Adapt to the recent move of S from ${WORKDIR} to ${UNPACKDIR}. Signed-off-by: Peter Kjellerstedt --- meta/classes-recipe/go-mod.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes-recipe/go-mod.bbclass b/meta/classes-recipe/go-mod.bbclass index 93ae72235ff..a15dda8f0ee 100644 --- a/meta/classes-recipe/go-mod.bbclass +++ b/meta/classes-recipe/go-mod.bbclass @@ -23,7 +23,7 @@ GOBUILDFLAGS:append = " -modcacherw" inherit go export GOMODCACHE = "${S}/pkg/mod" -GO_MOD_CACHE_DIR = "${@os.path.relpath(d.getVar('GOMODCACHE'), d.getVar('WORKDIR'))}" +GO_MOD_CACHE_DIR = "${@os.path.relpath(d.getVar('GOMODCACHE'), d.getVar('UNPACKDIR'))}" do_unpack[cleandirs] += "${GOMODCACHE}" GO_WORKDIR ?= "${GO_IMPORT}" From patchwork Fri Jun 27 13:48:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92A6BC83010 for ; Fri, 27 Jun 2025 13:48:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.14152.1751032134481443817 for ; Fri, 27 Jun 2025 06:48:54 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CFBE41A00 for ; Fri, 27 Jun 2025 06:48:36 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A6C193F66E for ; Fri, 27 Jun 2025 06:48:53 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 2/9] recipetool: create: Support creating extra files named after the recipe Date: Fri, 27 Jun 2025 14:48:43 +0100 Message-ID: <20250627134850.152269-2-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134850.152269-1-ross.burton@arm.com> References: <20250627134850.152269-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:48:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219414 From: Peter Kjellerstedt Signed-off-by: Peter Kjellerstedt --- scripts/lib/recipetool/create.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index edb64671037..b65e59de6a0 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -824,7 +824,8 @@ def create_recipe(args): extraoutdir = os.path.join(os.path.dirname(outfile), pn) bb.utils.mkdirhier(extraoutdir) for destfn, extrafile in extrafiles.items(): - shutil.move(extrafile, os.path.join(extraoutdir, destfn)) + fn = destfn.format(pn=pn, pv=realpv) + shutil.move(extrafile, os.path.join(extraoutdir, fn)) lines = lines_before lines_before = [] From patchwork Fri Jun 27 13:48:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75DD1C7EE32 for ; Fri, 27 Jun 2025 13:48:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.14126.1751032135251455371 for ; Fri, 27 Jun 2025 06:48:55 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 90C3D1A00 for ; Fri, 27 Jun 2025 06:48:37 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 70A8E3F66E for ; Fri, 27 Jun 2025 06:48:54 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 3/9] recipetool: allow recipe create handlers to specify bitbake tasks to run Date: Fri, 27 Jun 2025 14:48:44 +0100 Message-ID: <20250627134850.152269-3-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134850.152269-1-ross.burton@arm.com> References: <20250627134850.152269-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:48:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219415 When creating a recipe there can be cases where there is a class that does some of the recipe creation (such as cargo-update-recipe-crates). To avoid duplication of code, look for run_task assignments in the extravalues dictionary returned by the handler, and if it is set then call that task after writing the recipe. Signed-off-by: Ross Burton --- scripts/lib/recipetool/create.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index b65e59de6a0..5d7fcbbb988 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -764,6 +764,7 @@ def create_recipe(args): extrafiles = extravalues.pop('extrafiles', {}) extra_pn = extravalues.pop('PN', None) extra_pv = extravalues.pop('PV', None) + run_tasks = extravalues.pop('run_tasks', "").split() if extra_pv and not realpv: realpv = extra_pv @@ -918,6 +919,10 @@ def create_recipe(args): log_info_cond('Recipe %s has been created; further editing may be required to make it fully functional' % outfile, args.devtool) tinfoil.modified_files() + for task in run_tasks: + logger.info("Running task %s" % task) + tinfoil.build_file_sync(outfile, task) + if tempsrc: if args.keep_temp: logger.info('Preserving temporary directory %s' % tempsrc) From patchwork Fri Jun 27 13:48:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65744 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CC71C77B7F for ; Fri, 27 Jun 2025 13:48:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.14129.1751032136066456558 for ; Fri, 27 Jun 2025 06:48:56 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 75FB51A00 for ; Fri, 27 Jun 2025 06:48:38 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 327113F66E for ; Fri, 27 Jun 2025 06:48:55 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 4/9] recipetool: create_go: Use gomod fetcher instead of go mod vendor Date: Fri, 27 Jun 2025 14:48:45 +0100 Message-ID: <20250627134850.152269-4-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134850.152269-1-ross.burton@arm.com> References: <20250627134850.152269-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:48:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219416 From: Christian Lindeberg Use the go-mod bbclass together with the gomod fetcher instead of the go-vendor bbclass. Signed-off-by: Christian Lindeberg --- scripts/lib/recipetool/create_go.py | 731 ++++------------------------ 1 file changed, 104 insertions(+), 627 deletions(-) diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py index 5cc53931f00..3e9fc857842 100644 --- a/scripts/lib/recipetool/create_go.py +++ b/scripts/lib/recipetool/create_go.py @@ -10,48 +10,31 @@ # -from collections import namedtuple -from enum import Enum -from html.parser import HTMLParser from recipetool.create import RecipeHandler, handle_license_vars -from recipetool.create import find_licenses, tidy_licenses, fixup_license -from recipetool.create import determine_from_url -from urllib.error import URLError, HTTPError +from recipetool.create import find_licenses import bb.utils import json import logging import os import re -import subprocess import sys -import shutil import tempfile import urllib.parse import urllib.request -GoImport = namedtuple('GoImport', 'root vcs url suffix') logger = logging.getLogger('recipetool') -CodeRepo = namedtuple( - 'CodeRepo', 'path codeRoot codeDir pathMajor pathPrefix pseudoMajor') tinfoil = None -# Regular expression to parse pseudo semantic version -# see https://go.dev/ref/mod#pseudo-versions -re_pseudo_semver = re.compile( - r"^v[0-9]+\.(0\.0-|\d+\.\d+-([^+]*\.)?0\.)(?P\d{14})-(?P[A-Za-z0-9]+)(\+[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?$") -# Regular expression to parse semantic version -re_semver = re.compile( - r"^v(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)(?:-(?P(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$") - def tinfoil_init(instance): global tinfoil tinfoil = instance + class GoRecipeHandler(RecipeHandler): """Class to handle the go recipe creation""" @@ -83,577 +66,96 @@ class GoRecipeHandler(RecipeHandler): return bindir - def __resolve_repository_static(self, modulepath): - """Resolve the repository in a static manner - - The method is based on the go implementation of - `repoRootFromVCSPaths` in - https://github.com/golang/go/blob/master/src/cmd/go/internal/vcs/vcs.go - """ - - url = urllib.parse.urlparse("https://" + modulepath) - req = urllib.request.Request(url.geturl()) - - try: - resp = urllib.request.urlopen(req) - # Some modulepath are just redirects to github (or some other vcs - # hoster). Therefore, we check if this modulepath redirects to - # somewhere else - if resp.geturl() != url.geturl(): - bb.debug(1, "%s is redirectred to %s" % - (url.geturl(), resp.geturl())) - url = urllib.parse.urlparse(resp.geturl()) - modulepath = url.netloc + url.path - - except URLError as url_err: - # This is probably because the module path - # contains the subdir and major path. Thus, - # we ignore this error for now - logger.debug( - 1, "Failed to fetch page from [%s]: %s" % (url, str(url_err))) - - host, _, _ = modulepath.partition('/') - - class vcs(Enum): - pathprefix = "pathprefix" - regexp = "regexp" - type = "type" - repo = "repo" - check = "check" - schemelessRepo = "schemelessRepo" - - # GitHub - vcsGitHub = {} - vcsGitHub[vcs.pathprefix] = "github.com" - vcsGitHub[vcs.regexp] = re.compile( - r'^(?Pgithub\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/(?P[A-Za-z0-9_.\-]+))*$') - vcsGitHub[vcs.type] = "git" - vcsGitHub[vcs.repo] = "https://\\g" - - # Bitbucket - vcsBitbucket = {} - vcsBitbucket[vcs.pathprefix] = "bitbucket.org" - vcsBitbucket[vcs.regexp] = re.compile( - r'^(?Pbitbucket\.org/(?P[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+))(/(?P[A-Za-z0-9_.\-]+))*$') - vcsBitbucket[vcs.type] = "git" - vcsBitbucket[vcs.repo] = "https://\\g" - - # IBM DevOps Services (JazzHub) - vcsIBMDevOps = {} - vcsIBMDevOps[vcs.pathprefix] = "hub.jazz.net/git" - vcsIBMDevOps[vcs.regexp] = re.compile( - r'^(?Phub\.jazz\.net/git/[a-z0-9]+/[A-Za-z0-9_.\-]+)(/(?P[A-Za-z0-9_.\-]+))*$') - vcsIBMDevOps[vcs.type] = "git" - vcsIBMDevOps[vcs.repo] = "https://\\g" - - # Git at Apache - vcsApacheGit = {} - vcsApacheGit[vcs.pathprefix] = "git.apache.org" - vcsApacheGit[vcs.regexp] = re.compile( - r'^(?Pgit\.apache\.org/[a-z0-9_.\-]+\.git)(/(?P[A-Za-z0-9_.\-]+))*$') - vcsApacheGit[vcs.type] = "git" - vcsApacheGit[vcs.repo] = "https://\\g" - - # Git at OpenStack - vcsOpenStackGit = {} - vcsOpenStackGit[vcs.pathprefix] = "git.openstack.org" - vcsOpenStackGit[vcs.regexp] = re.compile( - r'^(?Pgit\.openstack\.org/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(\.git)?(/(?P[A-Za-z0-9_.\-]+))*$') - vcsOpenStackGit[vcs.type] = "git" - vcsOpenStackGit[vcs.repo] = "https://\\g" - - # chiselapp.com for fossil - vcsChiselapp = {} - vcsChiselapp[vcs.pathprefix] = "chiselapp.com" - vcsChiselapp[vcs.regexp] = re.compile( - r'^(?Pchiselapp\.com/user/[A-Za-z0-9]+/repository/[A-Za-z0-9_.\-]+)$') - vcsChiselapp[vcs.type] = "fossil" - vcsChiselapp[vcs.repo] = "https://\\g" - - # General syntax for any server. - # Must be last. - vcsGeneralServer = {} - vcsGeneralServer[vcs.regexp] = re.compile( - "(?P(?P([a-z0-9.\\-]+\\.)+[a-z0-9.\\-]+(:[0-9]+)?(/~?[A-Za-z0-9_.\\-]+)+?)\\.(?Pbzr|fossil|git|hg|svn))(/~?(?P[A-Za-z0-9_.\\-]+))*$") - vcsGeneralServer[vcs.schemelessRepo] = True - - vcsPaths = [vcsGitHub, vcsBitbucket, vcsIBMDevOps, - vcsApacheGit, vcsOpenStackGit, vcsChiselapp, - vcsGeneralServer] - - if modulepath.startswith("example.net") or modulepath == "rsc.io": - logger.warning("Suspicious module path %s" % modulepath) - return None - if modulepath.startswith("http:") or modulepath.startswith("https:"): - logger.warning("Import path should not start with %s %s" % - ("http", "https")) - return None - - rootpath = None - vcstype = None - repourl = None - suffix = None - - for srv in vcsPaths: - m = srv[vcs.regexp].match(modulepath) - if vcs.pathprefix in srv: - if host == srv[vcs.pathprefix]: - rootpath = m.group('root') - vcstype = srv[vcs.type] - repourl = m.expand(srv[vcs.repo]) - suffix = m.group('suffix') - break - elif m and srv[vcs.schemelessRepo]: - rootpath = m.group('root') - vcstype = m[vcs.type] - repourl = m[vcs.repo] - suffix = m.group('suffix') - break - - return GoImport(rootpath, vcstype, repourl, suffix) - - def __resolve_repository_dynamic(self, modulepath): - """Resolve the repository root in a dynamic manner. - - The method is based on the go implementation of - `repoRootForImportDynamic` in - https://github.com/golang/go/blob/master/src/cmd/go/internal/vcs/vcs.go - """ - url = urllib.parse.urlparse("https://" + modulepath) - - class GoImportHTMLParser(HTMLParser): - - def __init__(self): - super().__init__() - self.__srv = {} - - def handle_starttag(self, tag, attrs): - if tag == 'meta' and list( - filter(lambda a: (a[0] == 'name' and a[1] == 'go-import'), attrs)): - content = list( - filter(lambda a: (a[0] == 'content'), attrs)) - if content: - srv = content[0][1].split() - self.__srv[srv[0]] = srv - - def go_import(self, modulepath): - if modulepath in self.__srv: - srv = self.__srv[modulepath] - return GoImport(srv[0], srv[1], srv[2], None) - return None - - url = url.geturl() + "?go-get=1" - req = urllib.request.Request(url) - - try: - body = urllib.request.urlopen(req).read() - except HTTPError as http_err: - logger.warning( - "Unclean status when fetching page from [%s]: %s", url, str(http_err)) - body = http_err.fp.read() - except URLError as url_err: - logger.warning( - "Failed to fetch page from [%s]: %s", url, str(url_err)) - return None - - parser = GoImportHTMLParser() - parser.feed(body.decode('utf-8')) - parser.close() - - return parser.go_import(modulepath) - - def __resolve_from_golang_proxy(self, modulepath, version): - """ - Resolves repository data from golang proxy - """ - url = urllib.parse.urlparse("https://proxy.golang.org/" - + modulepath - + "/@v/" - + version - + ".info") - - # Transform url to lower case, golang proxy doesn't like mixed case - req = urllib.request.Request(url.geturl().lower()) - - try: - resp = urllib.request.urlopen(req) - except URLError as url_err: - logger.warning( - "Failed to fetch page from [%s]: %s", url, str(url_err)) - return None - - golang_proxy_res = resp.read().decode('utf-8') - modinfo = json.loads(golang_proxy_res) - - if modinfo and 'Origin' in modinfo: - origin = modinfo['Origin'] - _root_url = urllib.parse.urlparse(origin['URL']) - - # We normalize the repo URL since we don't want the scheme in it - _subdir = origin['Subdir'] if 'Subdir' in origin else None - _root, _, _ = self.__split_path_version(modulepath) - if _subdir: - _root = _root[:-len(_subdir)].strip('/') - - _commit = origin['Hash'] - _vcs = origin['VCS'] - return (GoImport(_root, _vcs, _root_url.geturl(), None), _commit) - - return None - - def __resolve_repository(self, modulepath): - """ - Resolves src uri from go module-path - """ - repodata = self.__resolve_repository_static(modulepath) - if not repodata or not repodata.url: - repodata = self.__resolve_repository_dynamic(modulepath) - if not repodata or not repodata.url: - logger.error( - "Could not resolve repository for module path '%s'" % modulepath) - # There is no way to recover from this - sys.exit(14) - if repodata: - logger.debug(1, "Resolved download path for import '%s' => %s" % ( - modulepath, repodata.url)) - return repodata - - def __split_path_version(self, path): - i = len(path) - dot = False - for j in range(i, 0, -1): - if path[j - 1] < '0' or path[j - 1] > '9': - break - if path[j - 1] == '.': - dot = True - break - i = j - 1 - - if i <= 1 or i == len( - path) or path[i - 1] != 'v' or path[i - 2] != '/': - return path, "", True - - prefix, pathMajor = path[:i - 2], path[i - 2:] - if dot or len( - pathMajor) <= 2 or pathMajor[2] == '0' or pathMajor == "/v1": - return path, "", False - - return prefix, pathMajor, True - - def __get_path_major(self, pathMajor): - if not pathMajor: - return "" - - if pathMajor[0] != '/' and pathMajor[0] != '.': - logger.error( - "pathMajor suffix %s passed to PathMajorPrefix lacks separator", pathMajor) - - if pathMajor.startswith(".v") and pathMajor.endswith("-unstable"): - pathMajor = pathMajor[:len("-unstable") - 2] - - return pathMajor[1:] - - def __build_coderepo(self, repo, path): - codedir = "" - pathprefix, pathMajor, _ = self.__split_path_version(path) - if repo.root == path: - pathprefix = path - elif path.startswith(repo.root): - codedir = pathprefix[len(repo.root):].strip('/') - - pseudoMajor = self.__get_path_major(pathMajor) - - logger.debug("root='%s', codedir='%s', prefix='%s', pathMajor='%s', pseudoMajor='%s'", - repo.root, codedir, pathprefix, pathMajor, pseudoMajor) - - return CodeRepo(path, repo.root, codedir, - pathMajor, pathprefix, pseudoMajor) - - def __resolve_version(self, repo, path, version): - hash = None - coderoot = self.__build_coderepo(repo, path) - - def vcs_fetch_all(): - tmpdir = tempfile.mkdtemp() - clone_cmd = "%s clone --bare %s %s" % ('git', repo.url, tmpdir) - bb.process.run(clone_cmd) - log_cmd = "git log --all --pretty='%H %d' --decorate=short" - output, _ = bb.process.run( - log_cmd, shell=True, stderr=subprocess.PIPE, cwd=tmpdir) - bb.utils.prunedir(tmpdir) - return output.strip().split('\n') - - def vcs_fetch_remote(tag): - # add * to grab ^{} - refs = {} - ls_remote_cmd = "git ls-remote -q --tags {} {}*".format( - repo.url, tag) - output, _ = bb.process.run(ls_remote_cmd) - output = output.strip().split('\n') - for line in output: - f = line.split(maxsplit=1) - if len(f) != 2: - continue - - for prefix in ["HEAD", "refs/heads/", "refs/tags/"]: - if f[1].startswith(prefix): - refs[f[1][len(prefix):]] = f[0] - - for key, hash in refs.items(): - if key.endswith(r"^{}"): - refs[key.strip(r"^{}")] = hash - - return refs[tag] - - m_pseudo_semver = re_pseudo_semver.match(version) - - if m_pseudo_semver: - remote_refs = vcs_fetch_all() - short_commit = m_pseudo_semver.group('commithash') - for l in remote_refs: - r = l.split(maxsplit=1) - sha1 = r[0] if len(r) else None - if not sha1: - logger.error( - "Ups: could not resolve abbref commit for %s" % short_commit) - - elif sha1.startswith(short_commit): - hash = sha1 - break - else: - m_semver = re_semver.match(version) - if m_semver: - - def get_sha1_remote(re): - rsha1 = None - for line in remote_refs: - # Split lines of the following format: - # 22e90d9b964610628c10f673ca5f85b8c2a2ca9a (tag: sometag) - lineparts = line.split(maxsplit=1) - sha1 = lineparts[0] if len(lineparts) else None - refstring = lineparts[1] if len( - lineparts) == 2 else None - if refstring: - # Normalize tag string and split in case of multiple - # regs e.g. (tag: speech/v1.10.0, tag: orchestration/v1.5.0 ...) - refs = refstring.strip('(), ').split(',') - for ref in refs: - if re.match(ref.strip()): - rsha1 = sha1 - return rsha1 - - semver = "v" + m_semver.group('major') + "."\ - + m_semver.group('minor') + "."\ - + m_semver.group('patch') \ - + (("-" + m_semver.group('prerelease')) - if m_semver.group('prerelease') else "") - - tag = os.path.join( - coderoot.codeDir, semver) if coderoot.codeDir else semver - - # probe tag using 'ls-remote', which is faster than fetching - # complete history - hash = vcs_fetch_remote(tag) - if not hash: - # backup: fetch complete history - remote_refs = vcs_fetch_all() - hash = get_sha1_remote( - re.compile(fr"(tag:|HEAD ->) ({tag})")) - - logger.debug( - "Resolving commit for tag '%s' -> '%s'", tag, hash) - return hash - - def __generate_srcuri_inline_fcn(self, path, version, replaces=None): - """Generate SRC_URI functions for go imports""" - - logger.info("Resolving repository for module %s", path) - # First try to resolve repo and commit from golang proxy - # Most info is already there and we don't have to go through the - # repository or even perform the version resolve magic - golang_proxy_info = self.__resolve_from_golang_proxy(path, version) - if golang_proxy_info: - repo = golang_proxy_info[0] - commit = golang_proxy_info[1] - else: - # Fallback - # Resolve repository by 'hand' - repo = self.__resolve_repository(path) - commit = self.__resolve_version(repo, path, version) - - url = urllib.parse.urlparse(repo.url) - repo_url = url.netloc + url.path - - coderoot = self.__build_coderepo(repo, path) - - inline_fcn = "${@go_src_uri(" - inline_fcn += f"'{repo_url}','{version}'" - if repo_url != path: - inline_fcn += f",path='{path}'" - if coderoot.codeDir: - inline_fcn += f",subdir='{coderoot.codeDir}'" - if repo.vcs != 'git': - inline_fcn += f",vcs='{repo.vcs}'" - if replaces: - inline_fcn += f",replaces='{replaces}'" - if coderoot.pathMajor: - inline_fcn += f",pathmajor='{coderoot.pathMajor}'" - inline_fcn += ")}" - - return inline_fcn, commit - - def __go_handle_dependencies(self, go_mod, srctree, localfilesdir, extravalues, d): - - import re - src_uris = [] - src_revs = [] - - def generate_src_rev(path, version, commithash): - src_rev = f"# {path}@{version} => {commithash}\n" - # Ups...maybe someone manipulated the source repository and the - # version or commit could not be resolved. This is a sign of - # a) the supply chain was manipulated (bad) - # b) the implementation for the version resolving didn't work - # anymore (less bad) - if not commithash: - src_rev += f"#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n" - src_rev += f"#!!! Could not resolve version !!!\n" - src_rev += f"#!!! Possible supply chain attack !!!\n" - src_rev += f"#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n" - src_rev += f"SRCREV_{path.replace('/', '.')} = \"{commithash}\"" - - return src_rev - - # we first go over replacement list, because we are essentialy - # interested only in the replaced path - if go_mod['Replace']: - for replacement in go_mod['Replace']: - oldpath = replacement['Old']['Path'] - path = replacement['New']['Path'] - version = '' - if 'Version' in replacement['New']: - version = replacement['New']['Version'] - - if os.path.exists(os.path.join(srctree, path)): - # the module refers to the local path, remove it from requirement list - # because it's a local module - go_mod['Require'][:] = [v for v in go_mod['Require'] if v.get('Path') != oldpath] - else: - # Replace the path and the version, so we don't iterate replacement list anymore - for require in go_mod['Require']: - if require['Path'] == oldpath: - require.update({'Path': path, 'Version': version}) - break - - for require in go_mod['Require']: - path = require['Path'] - version = require['Version'] - - inline_fcn, commithash = self.__generate_srcuri_inline_fcn( - path, version) - src_uris.append(inline_fcn) - src_revs.append(generate_src_rev(path, version, commithash)) - - # strip version part from module URL /vXX - baseurl = re.sub(r'/v(\d+)$', '', go_mod['Module']['Path']) - pn, _ = determine_from_url(baseurl) - go_mods_basename = "%s-modules.inc" % pn - - go_mods_filename = os.path.join(localfilesdir, go_mods_basename) - with open(go_mods_filename, "w") as f: - # We introduce this indirection to make the tests a little easier - f.write("SRC_URI += \"${GO_DEPENDENCIES_SRC_URI}\"\n") - f.write("GO_DEPENDENCIES_SRC_URI = \"\\\n") - for uri in src_uris: - f.write(" " + uri + " \\\n") - f.write("\"\n\n") - for rev in src_revs: - f.write(rev + "\n") - - extravalues['extrafiles'][go_mods_basename] = go_mods_filename - - def __go_run_cmd(self, cmd, cwd, d): - return bb.process.run(cmd, env=dict(os.environ, PATH=d.getVar('PATH')), - shell=True, cwd=cwd) - - def __go_native_version(self, d): - stdout, _ = self.__go_run_cmd("go version", None, d) - m = re.match(r".*\sgo((\d+).(\d+).(\d+))\s([\w\/]*)", stdout) - major = int(m.group(2)) - minor = int(m.group(3)) - patch = int(m.group(4)) - - return major, minor, patch - - def __go_mod_patch(self, srctree, localfilesdir, extravalues, d): - - patchfilename = "go.mod.patch" - go_native_version_major, go_native_version_minor, _ = self.__go_native_version( - d) - self.__go_run_cmd("go mod tidy -go=%d.%d" % - (go_native_version_major, go_native_version_minor), srctree, d) - stdout, _ = self.__go_run_cmd("go mod edit -json", srctree, d) - - # Create patch in order to upgrade go version - self.__go_run_cmd("git diff go.mod > %s" % (patchfilename), srctree, d) - # Restore original state - self.__go_run_cmd("git checkout HEAD go.mod go.sum", srctree, d) - - go_mod = json.loads(stdout) - tmpfile = os.path.join(localfilesdir, patchfilename) - shutil.move(os.path.join(srctree, patchfilename), tmpfile) - - extravalues['extrafiles'][patchfilename] = tmpfile - - return go_mod, patchfilename - - def __go_mod_vendor(self, go_mod, srctree, localfilesdir, extravalues, d): - # Perform vendoring to retrieve the correct modules.txt - tmp_vendor_dir = tempfile.mkdtemp() - - # -v causes to go to print modules.txt to stderr - _, stderr = self.__go_run_cmd( - "go mod vendor -v -o %s" % (tmp_vendor_dir), srctree, d) - - modules_txt_basename = "modules.txt" - modules_txt_filename = os.path.join(localfilesdir, modules_txt_basename) - with open(modules_txt_filename, "w") as f: - f.write(stderr) - - extravalues['extrafiles'][modules_txt_basename] = modules_txt_filename - - licenses = [] + @staticmethod + def __unescape_path(path): + """Unescape capital letters using exclamation points.""" + return re.sub(r'!([a-z])', lambda m: m.group(1).upper(), path) + + @staticmethod + def __fold_uri(uri): + """Fold URI for sorting shorter module paths before longer.""" + return uri.replace(';', ' ').replace('/', '!') + + @staticmethod + def __go_run_cmd(cmd, cwd, d): + env = dict(os.environ, PATH=d.getVar('PATH'), GOMODCACHE=d.getVar('GOMODCACHE')) + return bb.process.run(cmd, env=env, shell=True, cwd=cwd) + + def __go_mod(self, go_mod, srctree, localfilesdir, extravalues, d): + moddir = d.getVar('GOMODCACHE') + + # List main packages and their dependencies with the go list command. + stdout, _ = self.__go_run_cmd(f"go list -json=Dir,Module -deps {go_mod['Module']['Path']}/...", srctree, d) + pkgs = json.loads('[' + stdout.replace('}\n{', '},\n{') + ']') + + # Collect licenses for the dependencies. + licenses = set() lic_files_chksum = [] - licvalues = find_licenses(tmp_vendor_dir, d) - shutil.rmtree(tmp_vendor_dir) + lic_files = {} + for pkg in pkgs: + # TODO: If the package is in a subdirectory with its own license + # files then report those istead of the license files found in the + # module root directory. + mod = pkg.get('Module', None) + if not mod or mod.get('Main', False): + continue + path = os.path.relpath(mod['Dir'], moddir) + for lic in find_licenses(mod['Dir'], d): + lic_files[os.path.join(path, lic[1])] = (lic[0], lic[2]) - if licvalues: - for licvalue in licvalues: - license = licvalue[0] - lics = tidy_licenses(fixup_license(license)) - lics = [lic for lic in lics if lic not in licenses] - if len(lics): - licenses.extend(lics) - lic_files_chksum.append( - 'file://src/${GO_IMPORT}/vendor/%s;md5=%s' % (licvalue[1], licvalue[2])) + for lic_file in lic_files: + licenses.add(lic_files[lic_file][0]) + lic_files_chksum.append( + f'file://pkg/mod/{lic_file};md5={lic_files[lic_file][1]}') - # strip version part from module URL /vXX - baseurl = re.sub(r'/v(\d+)$', '', go_mod['Module']['Path']) - pn, _ = determine_from_url(baseurl) - licenses_basename = "%s-licenses.inc" % pn + # Collect the module cache files downloaded by the go list command as + # the go list command knows best what the go list command needs and it + # needs more files in the module cache than the go install command as + # it doesn't do the dependency pruning mentioned in the Go module + # reference, https://go.dev/ref/mod, for go 1.17 or higher. + src_uris = [] + downloaddir = os.path.join(moddir, 'cache', 'download') + for dirpath, _, filenames in os.walk(downloaddir): + path, base = os.path.split(os.path.relpath(dirpath, downloaddir)) + if base != '@v': + continue + path = self.__unescape_path(path) + zipver = None + for name in filenames: + ver, ext = os.path.splitext(name) + if ext == '.zip': + chksum = bb.utils.sha256_file(os.path.join(dirpath, name)) + src_uris.append(f'gomod://{path};version={ver};sha256sum={chksum}') + zipver = ver + break + for name in filenames: + ver, ext = os.path.splitext(name) + if ext == '.mod' and ver != zipver: + chksum = bb.utils.sha256_file(os.path.join(dirpath, name)) + src_uris.append(f'gomod://{path};version={ver};mod=1;sha256sum={chksum}') + self.__go_run_cmd("go clean -modcache", srctree, d) + + licenses_basename = "{pn}-licenses.inc" licenses_filename = os.path.join(localfilesdir, licenses_basename) with open(licenses_filename, "w") as f: - f.write("GO_MOD_LICENSES = \"%s\"\n\n" % - ' & '.join(sorted(licenses, key=str.casefold))) - # We introduce this indirection to make the tests a little easier - f.write("LIC_FILES_CHKSUM += \"${VENDORED_LIC_FILES_CHKSUM}\"\n") - f.write("VENDORED_LIC_FILES_CHKSUM = \"\\\n") - for lic in lic_files_chksum: - f.write(" " + lic + " \\\n") - f.write("\"\n") + f.write(f'GO_MOD_LICENSES = "{" & ".join(sorted(licenses))}"\n\n') + f.write('LIC_FILES_CHKSUM += "\\\n') + for lic in sorted(lic_files_chksum, key=self.__fold_uri): + f.write(' ' + lic + ' \\\n') + f.write('"\n') - extravalues['extrafiles'][licenses_basename] = licenses_filename + extravalues['extrafiles'][f"../{licenses_basename}"] = licenses_filename + + go_mods_basename = "{pn}-go-mods.inc" + go_mods_filename = os.path.join(localfilesdir, go_mods_basename) + with open(go_mods_filename, "w") as f: + f.write('SRC_URI += "\\\n') + for uri in sorted(src_uris, key=self.__fold_uri): + f.write(' ' + uri + ' \\\n') + f.write('"\n') + + extravalues['extrafiles'][f"../{go_mods_basename}"] = go_mods_filename def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): @@ -672,56 +174,30 @@ class GoRecipeHandler(RecipeHandler): d.prependVar('PATH', '%s:' % go_bindir) handled.append('buildsystem') - classes.append("go-vendor") + classes.append("go-mod") + + tmp_mod_dir = tempfile.mkdtemp(prefix='go-mod-') + d.setVar('GOMODCACHE', tmp_mod_dir) stdout, _ = self.__go_run_cmd("go mod edit -json", srctree, d) - go_mod = json.loads(stdout) - go_import = go_mod['Module']['Path'] - go_version_match = re.match("([0-9]+).([0-9]+)", go_mod['Go']) - go_version_major = int(go_version_match.group(1)) - go_version_minor = int(go_version_match.group(2)) - src_uris = [] + go_import = re.sub(r'/v([0-9]+)$', '', go_mod['Module']['Path']) localfilesdir = tempfile.mkdtemp(prefix='recipetool-go-') extravalues.setdefault('extrafiles', {}) - # Use an explicit name determined from the module name because it - # might differ from the actual URL for replaced modules - # strip version part from module URL /vXX - baseurl = re.sub(r'/v(\d+)$', '', go_mod['Module']['Path']) - pn, _ = determine_from_url(baseurl) - - # go.mod files with version < 1.17 may not include all indirect - # dependencies. Thus, we have to upgrade the go version. - if go_version_major == 1 and go_version_minor < 17: - logger.warning( - "go.mod files generated by Go < 1.17 might have incomplete indirect dependencies.") - go_mod, patchfilename = self.__go_mod_patch(srctree, localfilesdir, - extravalues, d) - src_uris.append( - "file://%s;patchdir=src/${GO_IMPORT}" % (patchfilename)) - - # Check whether the module is vendored. If so, we have nothing to do. - # Otherwise we gather all dependencies and add them to the recipe - if not os.path.exists(os.path.join(srctree, "vendor")): - - # Write additional $BPN-modules.inc file - self.__go_mod_vendor(go_mod, srctree, localfilesdir, extravalues, d) - lines_before.append("LICENSE += \" & ${GO_MOD_LICENSES}\"") - lines_before.append("require %s-licenses.inc" % (pn)) - - self.__rewrite_src_uri(lines_before, ["file://modules.txt"]) - - self.__go_handle_dependencies(go_mod, srctree, localfilesdir, extravalues, d) - lines_before.append("require %s-modules.inc" % (pn)) + # Write the ${BPN}-licenses.inc and ${BPN}-go-mods.inc files + self.__go_mod(go_mod, srctree, localfilesdir, extravalues, d) # Do generic license handling handle_license_vars(srctree, lines_before, handled, extravalues, d) - self.__rewrite_lic_uri(lines_before) + self.__rewrite_lic_vars(lines_before) - lines_before.append("GO_IMPORT = \"{}\"".format(baseurl)) - lines_before.append("SRCREV_FORMAT = \"${BPN}\"") + self.__rewrite_src_uri(lines_before) + + lines_before.append('require ${BPN}-licenses.inc') + lines_before.append('require ${BPN}-go-mods.inc') + lines_before.append(f'GO_IMPORT = "{go_import}"') def __update_lines_before(self, updated, newlines, lines_before): if updated: @@ -733,9 +209,11 @@ class GoRecipeHandler(RecipeHandler): lines_before.append(line) return updated - def __rewrite_lic_uri(self, lines_before): + def __rewrite_lic_vars(self, lines_before): def varfunc(varname, origvalue, op, newlines): + if varname == 'LICENSE': + return ' & '.join((origvalue, '${GO_MOD_LICENSES}')), None, -1, True if varname == 'LIC_FILES_CHKSUM': new_licenses = [] licenses = origvalue.split('\\') @@ -757,15 +235,14 @@ class GoRecipeHandler(RecipeHandler): return origvalue, None, 0, True updated, newlines = bb.utils.edit_metadata( - lines_before, ['LIC_FILES_CHKSUM'], varfunc) + lines_before, ['LICENSE', 'LIC_FILES_CHKSUM'], varfunc) return self.__update_lines_before(updated, newlines, lines_before) - def __rewrite_src_uri(self, lines_before, additional_uris = []): + def __rewrite_src_uri(self, lines_before): def varfunc(varname, origvalue, op, newlines): if varname == 'SRC_URI': - src_uri = ["git://${GO_IMPORT};destsuffix=git/src/${GO_IMPORT};nobranch=1;name=${BPN};protocol=https"] - src_uri.extend(additional_uris) + src_uri = ['git://${GO_IMPORT};protocol=https;nobranch=1;destsuffix=${GO_SRCURI_DESTSUFFIX}'] return src_uri, None, -1, True return origvalue, None, 0, True From patchwork Fri Jun 27 13:48:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81A83C7EE3A for ; Fri, 27 Jun 2025 13:48:57 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.14131.1751032136805510428 for ; Fri, 27 Jun 2025 06:48:56 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 367431A00 for ; Fri, 27 Jun 2025 06:48:39 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 172893F66E for ; Fri, 27 Jun 2025 06:48:55 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 5/9] oe/licenses: move tidy_licenses from recipetool Date: Fri, 27 Jun 2025 14:48:46 +0100 Message-ID: <20250627134850.152269-5-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134850.152269-1-ross.burton@arm.com> References: <20250627134850.152269-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:48:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219417 This function, to tidy a license string, is useful outside of recipetool so move it to oe.license. Signed-off-by: Ross Burton --- meta/lib/oe/license.py | 15 +++++++++++++++ scripts/lib/recipetool/create.py | 11 +---------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py index 6f882c3812c..6e55fa1e7f6 100644 --- a/meta/lib/oe/license.py +++ b/meta/lib/oe/license.py @@ -462,3 +462,18 @@ def skip_incompatible_package_licenses(d, pkgs): skipped_pkgs[pkg] = incompatible_lic return skipped_pkgs + +def tidy_licenses(value): + """ + Flat, split and sort licenses. + """ + from oe.license import flattened_licenses + + def _choose(a, b): + str_a, str_b = sorted((" & ".join(a), " & ".join(b)), key=str.casefold) + return ["(%s | %s)" % (str_a, str_b)] + + if not isinstance(value, str): + value = " & ".join(value) + + return sorted(list(set(flattened_licenses(value, _choose))), key=str.casefold) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index 5d7fcbbb988..ef0ba974a99 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -18,6 +18,7 @@ from urllib.parse import urlparse, urldefrag, urlsplit import hashlib import bb.fetch2 logger = logging.getLogger('recipetool') +from oe.license import tidy_licenses from oe.license_finder import find_licenses tinfoil = None @@ -950,16 +951,6 @@ def fixup_license(value): return '(' + value + ')' return value -def tidy_licenses(value): - """Flat, split and sort licenses""" - from oe.license import flattened_licenses - def _choose(a, b): - str_a, str_b = sorted((" & ".join(a), " & ".join(b)), key=str.casefold) - return ["(%s | %s)" % (str_a, str_b)] - if not isinstance(value, str): - value = " & ".join(value) - return sorted(list(set(flattened_licenses(value, _choose))), key=str.casefold) - def handle_license_vars(srctree, lines_before, handled, extravalues, d): lichandled = [x for x in handled if x[0] == 'license'] if lichandled: From patchwork Fri Jun 27 13:48:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B4EBC77B7F for ; Fri, 27 Jun 2025 13:49:07 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.14133.1751032137557528471 for ; Fri, 27 Jun 2025 06:48:57 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EBAEB1A00 for ; Fri, 27 Jun 2025 06:48:39 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id CB6DA3F66E for ; Fri, 27 Jun 2025 06:48:56 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 6/9] classes/go-mod-update-modules: add class to generate module list Date: Fri, 27 Jun 2025 14:48:47 +0100 Message-ID: <20250627134850.152269-6-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134850.152269-1-ross.burton@arm.com> References: <20250627134850.152269-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219418 Almost entirely based on the create_go.py module for recipetool by Christian Lindeberg , this instead has the logic inside a class that can be used to update the list of Go modules that are used, both SRC_URI and LICENSE. Integration with devtool upgrade will come shortly, but it needs a bit more work. Signed-off-by: Ross Burton --- .../go-mod-update-modules.bbclass | 152 ++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 meta/classes-recipe/go-mod-update-modules.bbclass diff --git a/meta/classes-recipe/go-mod-update-modules.bbclass b/meta/classes-recipe/go-mod-update-modules.bbclass new file mode 100644 index 00000000000..6ef7ab553b0 --- /dev/null +++ b/meta/classes-recipe/go-mod-update-modules.bbclass @@ -0,0 +1,152 @@ +addtask do_update_modules after do_configure +do_update_modules[nostamp] = "1" +do_update_modules[network] = "1" + +# This class maintains two files, BPN-go-mods.inc and BPN-licenses.inc. +# +# -go-mods.inc will append SRC_URI with all of the Go modules that are +# dependencies of this recipe. +# +# -licenses.inc will append LICENSE and LIC_FILES_CHKSUM with the found licenses +# in the modules. +# +# These files are machine-generated and should not be modified. + +python do_update_modules() { + import subprocess, tempfile, json, re, urllib.parse + from oe.license import tidy_licenses + from oe.license_finder import find_licenses + + def unescape_path(path): + """Unescape capital letters using exclamation points.""" + return re.sub(r'!([a-z])', lambda m: m.group(1).upper(), path) + + def fold_uri(uri): + """Fold URI for sorting shorter module paths before longer.""" + return uri.replace(';', ' ').replace('/', '!') + + def parse_existing_licenses(): + hashes = {} + for url in d.getVar("LIC_FILES_CHKSUM").split(): + (method, host, path, user, pswd, parm) = bb.fetch.decodeurl(url) + if "spdx" in parm and parm["spdx"] != "Unknown": + hashes[parm["md5"]] = urllib.parse.unquote_plus(parm["spdx"]) + return hashes + + bpn = d.getVar("BPN") + thisdir = d.getVar("THISDIR") + s_dir = d.getVar("S") + + with tempfile.TemporaryDirectory(prefix='go-mod-') as mod_cache_dir: + notice = """ +# This file has been generated by go-mod-update-modules.bbclass +# +# Do not modify it by hand, as the contents will be replaced when +# running the update-modules task. + +""" + + env = dict(os.environ, GOMODCACHE=mod_cache_dir) + + source = d.expand("${WORKDIR}/${GO_SRCURI_DESTSUFFIX}") + output = subprocess.check_output(("go", "mod", "edit", "-json"), cwd=source, env=env, text=True) + go_mod = json.loads(output) + + output = subprocess.check_output(("go", "list", "-json=Dir,Module", "-deps", f"{go_mod['Module']['Path']}/..."), cwd=source, env=env, text=True) + + # + # Licenses + # + + # load hashes from the existing licenses.inc + extra_hashes = parse_existing_licenses() + + # The output of this isn't actually valid JSON, but a series of dicts. + # Wrap in [] and join the dicts with , + # Very frustrating that the json parser in python can't repeatedly + # parse from a stream. + pkgs = json.loads('[' + output.replace('}\n{', '},\n{') + ']') + # Collect licenses for the dependencies. + licenses = set() + lic_files_chksum = [] + lic_files = {} + + for pkg in pkgs: + mod = pkg.get('Module', None) + if not mod or mod.get('Main', False): + continue + + mod_dir = mod['Dir'] + + if mod_dir.startswith(s_dir): + continue + + path = os.path.relpath(mod_dir, mod_cache_dir) + + for license_name, license_file, license_md5 in find_licenses(mod['Dir'], d, first_only=True, extra_hashes=extra_hashes): + lic_files[os.path.join(path, license_file)] = (license_name, license_md5) + + for lic_file in lic_files: + license_name, license_md5 = lic_files[lic_file] + if license_name == "Unknown": + bb.warn(f"Unknown license: {lic_file} {license_md5}") + + licenses.add(lic_files[lic_file][0]) + lic_files_chksum.append( + f'file://pkg/mod/{lic_file};md5={license_md5};spdx={urllib.parse.quote_plus(license_name)}') + + licenses_filename = os.path.join(thisdir, f"{bpn}-licenses.inc") + with open(licenses_filename, "w") as f: + f.write(notice) + f.write(f'LICENSE += "& {" & ".join(tidy_licenses(licenses))}"\n\n') + f.write('LIC_FILES_CHKSUM += "\\\n') + for lic in sorted(lic_files_chksum, key=fold_uri): + f.write(' ' + lic + ' \\\n') + f.write('"\n') + + # + # Sources + # + + # Collect the module cache files downloaded by the go list command as + # the go list command knows best what the go list command needs and it + # needs more files in the module cache than the go install command as + # it doesn't do the dependency pruning mentioned in the Go module + # reference, https://go.dev/ref/mod, for go 1.17 or higher. + src_uris = [] + downloaddir = os.path.join(mod_cache_dir, 'cache', 'download') + for dirpath, _, filenames in os.walk(downloaddir): + # We want to process files under @v directories + path, base = os.path.split(os.path.relpath(dirpath, downloaddir)) + if base != '@v': + continue + + path = unescape_path(path) + zipver = None + for name in filenames: + ver, ext = os.path.splitext(name) + if ext == '.zip': + chksum = bb.utils.sha256_file(os.path.join(dirpath, name)) + src_uris.append(f'gomod://{path};version={ver};sha256sum={chksum}') + zipver = ver + break + for name in filenames: + ver, ext = os.path.splitext(name) + if ext == '.mod' and ver != zipver: + chksum = bb.utils.sha256_file(os.path.join(dirpath, name)) + src_uris.append(f'gomod://{path};version={ver};mod=1;sha256sum={chksum}') + + + go_mods_filename = os.path.join(thisdir, f"{bpn}-go-mods.inc") + with open(go_mods_filename, "w") as f: + f.write(notice) + f.write('SRC_URI += "\\\n') + for uri in sorted(src_uris, key=fold_uri): + f.write(' ' + uri + ' \\\n') + f.write('"\n') + + subprocess.check_output(("go", "clean", "-modcache"), cwd=source, env=env, text=True) +} + +# This doesn't work as we need to wipe the inc files first so we don't try looking for LICENSE files that don't yet exist +# RECIPE_UPGRADE_EXTRA_TASKS += "do_update_modules" From patchwork Fri Jun 27 13:48:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B978C83010 for ; Fri, 27 Jun 2025 13:49:07 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.14156.1751032138294810782 for ; Fri, 27 Jun 2025 06:48:58 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ADD701A00 for ; Fri, 27 Jun 2025 06:48:40 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8E24C3F66E for ; Fri, 27 Jun 2025 06:48:57 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 7/9] recipetool/create_go: proxy module fetching to go-mod-update-modules Date: Fri, 27 Jun 2025 14:48:48 +0100 Message-ID: <20250627134850.152269-7-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134850.152269-1-ross.burton@arm.com> References: <20250627134850.152269-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219419 Now that the go-mod-update-modules class exists, this Go handler can create a stub recipe and then proxy the module handling to the class. Signed-off-by: Ross Burton --- .../go-mod-update-modules.bbclass | 4 +- scripts/lib/recipetool/create_go.py | 152 +++++------------- 2 files changed, 38 insertions(+), 118 deletions(-) diff --git a/meta/classes-recipe/go-mod-update-modules.bbclass b/meta/classes-recipe/go-mod-update-modules.bbclass index 6ef7ab553b0..5fccd0bb0d9 100644 --- a/meta/classes-recipe/go-mod-update-modules.bbclass +++ b/meta/classes-recipe/go-mod-update-modules.bbclass @@ -48,7 +48,7 @@ python do_update_modules() { env = dict(os.environ, GOMODCACHE=mod_cache_dir) - source = d.expand("${WORKDIR}/${GO_SRCURI_DESTSUFFIX}") + source = d.expand("${UNPACKDIR}/${GO_SRCURI_DESTSUFFIX}") output = subprocess.check_output(("go", "mod", "edit", "-json"), cwd=source, env=env, text=True) go_mod = json.loads(output) @@ -78,7 +78,7 @@ python do_update_modules() { mod_dir = mod['Dir'] - if mod_dir.startswith(s_dir): + if not mod_dir.startswith(mod_cache_dir): continue path = os.path.relpath(mod_dir, mod_cache_dir) diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py index 3e9fc857842..4b1fa39d13a 100644 --- a/scripts/lib/recipetool/create_go.py +++ b/scripts/lib/recipetool/create_go.py @@ -11,17 +11,15 @@ from recipetool.create import RecipeHandler, handle_license_vars -from recipetool.create import find_licenses import bb.utils import json import logging import os import re +import subprocess import sys import tempfile -import urllib.parse -import urllib.request logger = logging.getLogger('recipetool') @@ -66,97 +64,6 @@ class GoRecipeHandler(RecipeHandler): return bindir - @staticmethod - def __unescape_path(path): - """Unescape capital letters using exclamation points.""" - return re.sub(r'!([a-z])', lambda m: m.group(1).upper(), path) - - @staticmethod - def __fold_uri(uri): - """Fold URI for sorting shorter module paths before longer.""" - return uri.replace(';', ' ').replace('/', '!') - - @staticmethod - def __go_run_cmd(cmd, cwd, d): - env = dict(os.environ, PATH=d.getVar('PATH'), GOMODCACHE=d.getVar('GOMODCACHE')) - return bb.process.run(cmd, env=env, shell=True, cwd=cwd) - - def __go_mod(self, go_mod, srctree, localfilesdir, extravalues, d): - moddir = d.getVar('GOMODCACHE') - - # List main packages and their dependencies with the go list command. - stdout, _ = self.__go_run_cmd(f"go list -json=Dir,Module -deps {go_mod['Module']['Path']}/...", srctree, d) - pkgs = json.loads('[' + stdout.replace('}\n{', '},\n{') + ']') - - # Collect licenses for the dependencies. - licenses = set() - lic_files_chksum = [] - lic_files = {} - for pkg in pkgs: - # TODO: If the package is in a subdirectory with its own license - # files then report those istead of the license files found in the - # module root directory. - mod = pkg.get('Module', None) - if not mod or mod.get('Main', False): - continue - path = os.path.relpath(mod['Dir'], moddir) - for lic in find_licenses(mod['Dir'], d): - lic_files[os.path.join(path, lic[1])] = (lic[0], lic[2]) - - for lic_file in lic_files: - licenses.add(lic_files[lic_file][0]) - lic_files_chksum.append( - f'file://pkg/mod/{lic_file};md5={lic_files[lic_file][1]}') - - # Collect the module cache files downloaded by the go list command as - # the go list command knows best what the go list command needs and it - # needs more files in the module cache than the go install command as - # it doesn't do the dependency pruning mentioned in the Go module - # reference, https://go.dev/ref/mod, for go 1.17 or higher. - src_uris = [] - downloaddir = os.path.join(moddir, 'cache', 'download') - for dirpath, _, filenames in os.walk(downloaddir): - path, base = os.path.split(os.path.relpath(dirpath, downloaddir)) - if base != '@v': - continue - path = self.__unescape_path(path) - zipver = None - for name in filenames: - ver, ext = os.path.splitext(name) - if ext == '.zip': - chksum = bb.utils.sha256_file(os.path.join(dirpath, name)) - src_uris.append(f'gomod://{path};version={ver};sha256sum={chksum}') - zipver = ver - break - for name in filenames: - ver, ext = os.path.splitext(name) - if ext == '.mod' and ver != zipver: - chksum = bb.utils.sha256_file(os.path.join(dirpath, name)) - src_uris.append(f'gomod://{path};version={ver};mod=1;sha256sum={chksum}') - - self.__go_run_cmd("go clean -modcache", srctree, d) - - licenses_basename = "{pn}-licenses.inc" - licenses_filename = os.path.join(localfilesdir, licenses_basename) - with open(licenses_filename, "w") as f: - f.write(f'GO_MOD_LICENSES = "{" & ".join(sorted(licenses))}"\n\n') - f.write('LIC_FILES_CHKSUM += "\\\n') - for lic in sorted(lic_files_chksum, key=self.__fold_uri): - f.write(' ' + lic + ' \\\n') - f.write('"\n') - - extravalues['extrafiles'][f"../{licenses_basename}"] = licenses_filename - - go_mods_basename = "{pn}-go-mods.inc" - go_mods_filename = os.path.join(localfilesdir, go_mods_basename) - with open(go_mods_filename, "w") as f: - f.write('SRC_URI += "\\\n') - for uri in sorted(src_uris, key=self.__fold_uri): - f.write(' ' + uri + ' \\\n') - f.write('"\n') - - extravalues['extrafiles'][f"../{go_mods_basename}"] = go_mods_filename - def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): @@ -167,37 +74,52 @@ class GoRecipeHandler(RecipeHandler): if not files: return False - d = bb.data.createCopy(tinfoil.config_data) go_bindir = self.__ensure_go() if not go_bindir: sys.exit(14) - d.prependVar('PATH', '%s:' % go_bindir) handled.append('buildsystem') classes.append("go-mod") - tmp_mod_dir = tempfile.mkdtemp(prefix='go-mod-') - d.setVar('GOMODCACHE', tmp_mod_dir) + # Use go-mod-update-modules to set the full SRC_URI and LICENSE + classes.append("go-mod-update-modules") + extravalues["run_tasks"] = "update_modules" - stdout, _ = self.__go_run_cmd("go mod edit -json", srctree, d) - go_mod = json.loads(stdout) - go_import = re.sub(r'/v([0-9]+)$', '', go_mod['Module']['Path']) + with tempfile.TemporaryDirectory(prefix="go-mod-") as tmp_mod_dir: + env = dict(os.environ) + env["PATH"] += f":{go_bindir}" + env['GOMODCACHE'] = tmp_mod_dir - localfilesdir = tempfile.mkdtemp(prefix='recipetool-go-') - extravalues.setdefault('extrafiles', {}) + stdout = subprocess.check_output(["go", "mod", "edit", "-json"], cwd=srctree, env=env, text=True) + go_mod = json.loads(stdout) + go_import = re.sub(r'/v([0-9]+)$', '', go_mod['Module']['Path']) - # Write the ${BPN}-licenses.inc and ${BPN}-go-mods.inc files - self.__go_mod(go_mod, srctree, localfilesdir, extravalues, d) + localfilesdir = tempfile.mkdtemp(prefix='recipetool-go-') + extravalues.setdefault('extrafiles', {}) - # Do generic license handling - handle_license_vars(srctree, lines_before, handled, extravalues, d) - self.__rewrite_lic_vars(lines_before) + # Write the stub ${BPN}-licenses.inc and ${BPN}-go-mods.inc files + basename = "{pn}-licenses.inc" + filename = os.path.join(localfilesdir, basename) + with open(filename, "w") as f: + f.write("# FROM RECIPETOOL\n") + extravalues['extrafiles'][f"../{basename}"] = filename - self.__rewrite_src_uri(lines_before) + basename = "{pn}-go-mods.inc" + filename = os.path.join(localfilesdir, basename) + with open(filename, "w") as f: + f.write("# FROM RECIPETOOL\n") + extravalues['extrafiles'][f"../{basename}"] = filename - lines_before.append('require ${BPN}-licenses.inc') - lines_before.append('require ${BPN}-go-mods.inc') - lines_before.append(f'GO_IMPORT = "{go_import}"') + # Do generic license handling + d = bb.data.createCopy(tinfoil.config_data) + handle_license_vars(srctree, lines_before, handled, extravalues, d) + self.__rewrite_lic_vars(lines_before) + + self.__rewrite_src_uri(lines_before) + + lines_before.append('require ${BPN}-licenses.inc') + lines_before.append('require ${BPN}-go-mods.inc') + lines_before.append(f'GO_IMPORT = "{go_import}"') def __update_lines_before(self, updated, newlines, lines_before): if updated: @@ -210,10 +132,8 @@ class GoRecipeHandler(RecipeHandler): return updated def __rewrite_lic_vars(self, lines_before): - def varfunc(varname, origvalue, op, newlines): - if varname == 'LICENSE': - return ' & '.join((origvalue, '${GO_MOD_LICENSES}')), None, -1, True + import urllib.parse if varname == 'LIC_FILES_CHKSUM': new_licenses = [] licenses = origvalue.split('\\') @@ -235,7 +155,7 @@ class GoRecipeHandler(RecipeHandler): return origvalue, None, 0, True updated, newlines = bb.utils.edit_metadata( - lines_before, ['LICENSE', 'LIC_FILES_CHKSUM'], varfunc) + lines_before, ['LIC_FILES_CHKSUM'], varfunc) return self.__update_lines_before(updated, newlines, lines_before) def __rewrite_src_uri(self, lines_before): From patchwork Fri Jun 27 13:48:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84331C7EE31 for ; Fri, 27 Jun 2025 13:49:07 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.14134.1751032139090447382 for ; Fri, 27 Jun 2025 06:48:59 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 788E71A00 for ; Fri, 27 Jun 2025 06:48:41 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4F7013F66E for ; Fri, 27 Jun 2025 06:48:58 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 8/9] oeqa/sefltest/devtool: improve assignment matching in _test_recipe_contents Date: Fri, 27 Jun 2025 14:48:49 +0100 Message-ID: <20250627134850.152269-8-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134850.152269-1-ross.burton@arm.com> References: <20250627134850.152269-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219420 This function assumed that all assignments are done with just "=". However, being able to check += or ?= is also useful, so use a regex to split the line and be more flexible about what an assignment operator looks like. Signed-off-by: Ross Burton --- meta/lib/oeqa/selftest/cases/devtool.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index 74a7727cc00..05f228f03e7 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -154,7 +154,7 @@ class DevtoolTestCase(OESelftestTestCase): value = invalue invar = None elif '=' in line: - splitline = line.split('=', 1) + splitline = re.split(r"[?+:]*=[+]?", line, 1) var = splitline[0].rstrip() value = splitline[1].strip().strip('"') if value.endswith('\\'): From patchwork Fri Jun 27 13:48:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 65746 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FC51C7EE32 for ; Fri, 27 Jun 2025 13:49:07 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.14135.1751032139866257498 for ; Fri, 27 Jun 2025 06:49:00 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3B0DA1A00 for ; Fri, 27 Jun 2025 06:48:42 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 195143F66E for ; Fri, 27 Jun 2025 06:48:58 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 9/9] oeqa/selftest/devtool: update create_go test to match the new behaviour Date: Fri, 27 Jun 2025 14:48:50 +0100 Message-ID: <20250627134850.152269-9-ross.burton@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250627134850.152269-1-ross.burton@arm.com> References: <20250627134850.152269-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 27 Jun 2025 13:49:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/219421 Update the test now that the recipe uses go-mod-update-modules. Signed-off-by: Ross Burton --- meta/lib/oeqa/selftest/cases/recipetool.py | 33 +++++++--------------- 1 file changed, 10 insertions(+), 23 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index 2a91f6c7ae4..0bd724c8eef 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -757,13 +757,12 @@ class RecipetoolCreateTests(RecipetoolBase): def test_recipetool_create_go(self): # Basic test to check go recipe generation + self.maxDiff = None + temprecipe = os.path.join(self.tempdir, 'recipe') os.makedirs(temprecipe) recipefile = os.path.join(temprecipe, 'recipetool-go-test_git.bb') - deps_require_file = os.path.join(temprecipe, 'recipetool-go-test', 'recipetool-go-test-modules.inc') - lics_require_file = os.path.join(temprecipe, 'recipetool-go-test', 'recipetool-go-test-licenses.inc') - modules_txt_file = os.path.join(temprecipe, 'recipetool-go-test', 'modules.txt') srcuri = 'https://git.yoctoproject.org/recipetool-go-test.git' srcrev = "c3e213c01b6c1406b430df03ef0d1ae77de5d2f7" @@ -771,13 +770,11 @@ class RecipetoolCreateTests(RecipetoolBase): result = runCmd('recipetool create -o %s %s -S %s -B %s' % (temprecipe, srcuri, srcrev, srcbranch)) - self.maxDiff = None - inherits = ['go-vendor'] + inherits = ['go-mod', 'go-mod-update-modules'] checkvars = {} checkvars['GO_IMPORT'] = "git.yoctoproject.org/recipetool-go-test" - checkvars['SRC_URI'] = {'git://${GO_IMPORT};destsuffix=git/src/${GO_IMPORT};nobranch=1;name=${BPN};protocol=https', - 'file://modules.txt'} + checkvars['SRC_URI'] = {'git://${GO_IMPORT};protocol=https;nobranch=1;destsuffix=${GO_SRCURI_DESTSUFFIX}'} checkvars['LIC_FILES_CHKSUM'] = { 'file://src/${GO_IMPORT}/LICENSE;md5=4e3933dd47afbf115e484d11385fb3bd', 'file://src/${GO_IMPORT}/is/LICENSE;md5=62beaee5a116dd1e80161667b1df39ab' @@ -786,26 +783,16 @@ class RecipetoolCreateTests(RecipetoolBase): self._test_recipe_contents(recipefile, checkvars, inherits) self.assertNotIn('Traceback', result.output) + lics_require_file = os.path.join(temprecipe, 'recipetool-go-test-licenses.inc') + self.assertFileExists(lics_require_file) checkvars = {} - checkvars['VENDORED_LIC_FILES_CHKSUM'] = set( - ['file://src/${GO_IMPORT}/vendor/github.com/godbus/dbus/v5/LICENSE;md5=09042bd5c6c96a2b9e45ddf1bc517eed', - 'file://src/${GO_IMPORT}/vendor/github.com/matryer/is/LICENSE;md5=62beaee5a116dd1e80161667b1df39ab']) - self.assertTrue(os.path.isfile(lics_require_file)) + checkvars['LIC_FILES_CHKSUM'] = {'file://pkg/mod/github.com/godbus/dbus/v5@v5.1.0/LICENSE;md5=09042bd5c6c96a2b9e45ddf1bc517eed;spdx=BSD-2-Clause'} self._test_recipe_contents(lics_require_file, checkvars, []) - # make sure that dependencies don't mention local directory ./matryer/is - dependencies = \ - [ ('github.com/godbus/dbus','v5.1.0', 'github.com/godbus/dbus/v5', '/v5', ''), - ] - - src_uri = set() - for d in dependencies: - src_uri.add(self._go_urifiy(*d)) - + deps_require_file = os.path.join(temprecipe, 'recipetool-go-test-go-mods.inc') + self.assertFileExists(deps_require_file) checkvars = {} - checkvars['GO_DEPENDENCIES_SRC_URI'] = src_uri - - self.assertTrue(os.path.isfile(deps_require_file)) + checkvars['SRC_URI'] = {'gomod://github.com/godbus/dbus/v5;version=v5.1.0;sha256sum=03dfa8e71089a6f477310d15c4d3a036d82d028532881b50fee254358e782ad9'} self._test_recipe_contents(deps_require_file, checkvars, []) class RecipetoolTests(RecipetoolBase):