From patchwork Mon Aug 12 12:28:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 47681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB467C531DC for ; Mon, 12 Aug 2024 12:28:44 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web11.44996.1723465719439957078 for ; Mon, 12 Aug 2024 05:28:39 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=adLBueth; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id C1BF29C5F1F; Mon, 12 Aug 2024 08:28:38 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id H1AGxTbJbR-t; Mon, 12 Aug 2024 08:28:37 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id C199A9C5DB6; Mon, 12 Aug 2024 08:28:37 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com C199A9C5DB6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1723465717; bh=lbO0pa2Jn02xTo78mAckk0NTSUOre2snBO97BCSOUuQ=; h=From:To:Date:Message-Id:MIME-Version; b=adLBuethw3SINeCVcS5p1dh9snUI0NDKVHz9VPv7c9YH/KsvHSGozeadM1aQXl2qK HS2T6U/i3HBcawjL5aTQWmpSFaRCKzabHWyx3PndpxcaKbmMle6glnfRClnI/e6Y88 tvjruVS6T3DTA6s5zvKz4sqjigX/8ufFTTOIpt0eK/MAXTVWGYoZQBw4tY2fs/yRCu hlHqIfWnMl6DrfwgrWYyg2T63XytUV9fQAqxOIGiKrRW++3blWTlIr6nnOsiF1Vuah bEP0LEX2jMIA73vctiyrq8QnjJ3rz0HnS/fUjmETRj+904zHeG+SA71Omx62mzTWM9 kvLBq+U3+f4eg== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id RhOJXfBsdTF8; Mon, 12 Aug 2024 08:28:37 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id D31249C5E53; Mon, 12 Aug 2024 08:28:36 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, alexandre.belloni@bootlin.com, stefan.herbrechtsmeier@weidmueller.com, Enguerrand de Ribaucourt Subject: [OE-core][PATCH v4 1/4] npm: accept unspecified versions in package.json Date: Mon, 12 Aug 2024 14:28:26 +0200 Message-Id: <20240812122829.577515-2-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 12 Aug 2024 12:28:44 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16467 Our current emulation mandates that the package.json contains a version field. Some packages may not provide it when they are not published to the registry. The actual `npm pack` would allow such packages, so should we. This patch adds a default value to allow building such packages. This applies for instance to this package which doesn't declare a version: - https://github.com/cockpit-project/cockpit/blob/23701a555a5af13f998ee4c7526d27fdb5669d63/package.json#L2 v3: - Split bitbake npmsw.py modification in another commit Co-authored-by: Tanguy Raufflet Signed-off-by: Tanguy Raufflet Signed-off-by: Enguerrand de Ribaucourt --- meta/classes-recipe/npm.bbclass | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/meta/classes-recipe/npm.bbclass b/meta/classes-recipe/npm.bbclass index 91da3295f2..a73ff29be8 100644 --- a/meta/classes-recipe/npm.bbclass +++ b/meta/classes-recipe/npm.bbclass @@ -72,8 +72,10 @@ def npm_pack(env, srcdir, workdir): j = json.load(f) # base does not really matter and is for documentation purposes - # only. But the 'version' part must exist because other parts of + # only. But the 'version' part must exist because other parts of # the bbclass rely on it. + if 'version' not in j: + j['version'] = '0.0.0-unknown' base = j['name'].split('/')[-1] tarball = os.path.join(workdir, "%s-%s.tgz" % (base, j['version'])); From patchwork Mon Aug 12 12:28:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 47682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6C85C3DA7F for ; Mon, 12 Aug 2024 12:28:44 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web11.44997.1723465720175977804 for ; Mon, 12 Aug 2024 05:28:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=rmvBeRCX; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 7FB249C5F5C; Mon, 12 Aug 2024 08:28:39 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id b9PmrpylIGxw; Mon, 12 Aug 2024 08:28:38 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id D35419C5DB6; Mon, 12 Aug 2024 08:28:38 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com D35419C5DB6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1723465718; bh=EhzQ8wNaLqO2ae9g+r4pUxup5brrhibk6lif/pebQBQ=; h=From:To:Date:Message-Id:MIME-Version; b=rmvBeRCXNSvju1IoYiyDNMgR16Wduc0kMPmx0WHSDTgh+CnIh9zW6RdZdBNBunvMY Ezlj1aEkNNyi/F3yYGLRE2xGJlbA88ubETk7CoBlcx9ara16S/fM/vXZCNFib2fppc lJSLjFbyi6JLHPIHJOSqdtEHS22XvaamdPZsUWvwSgVlJfIR0mDpbFaWnGtBhGvenQ 7mtmlcStXn6U92CC5s4zXEmxbE/R1OlBrpuh2hlAtPfLpyCIIYjEogg2zQtfcne21Y NYeP1AKFbeSZFzd6Keot4kRjP389It/zfNwhDapDa1aSHRDxwlbg/vfqUUJ0VH766D jB3bqqGGFEiBA== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id o7cOodV7aS3P; Mon, 12 Aug 2024 08:28:38 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id E0BB69C5E53; Mon, 12 Aug 2024 08:28:37 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, alexandre.belloni@bootlin.com, stefan.herbrechtsmeier@weidmueller.com, Enguerrand de Ribaucourt Subject: [OE-core][PATCH v4 2/4] recipetool: create_npm: resolve licenses defined in package.json Date: Mon, 12 Aug 2024 14:28:27 +0200 Message-Id: <20240812122829.577515-3-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 12 Aug 2024 12:28:44 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16468 Some npm packages do not copy the LICENSE or COPY file into their git repository. They'll instead simply use SPDX identifiers in their package.json. A fallback for those repositories attempted to match the README file to a license file instead, which had a very low probability of success. This commit replaces this fallback with parsing the package.json and looking for the license in COMMON_LICENSE_DIR. If the license is not found, "Unknown" will still be produced. This also generates "Unknown" for packages which had no README file, which could silently not appear in the generated recipe. The user was more likely to miss them. Co-authored-by: Tanguy Raufflet Signed-off-by: Tanguy Raufflet Signed-off-by: Enguerrand de Ribaucourt --- scripts/lib/recipetool/create_npm.py | 57 ++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 16 deletions(-) diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py index 113a89f6a6..dd0ac01c3e 100644 --- a/scripts/lib/recipetool/create_npm.py +++ b/scripts/lib/recipetool/create_npm.py @@ -112,40 +112,52 @@ class NpmRecipeHandler(RecipeHandler): """Return the extra license files and the list of packages""" licfiles = [] packages = {} + # Licenses from package.json point to COMMON_LICENSE_DIR so we need + # to associate them explicitely for split_pkg_licenses() + fallback_licenses = dict() # Handle the parent package packages["${PN}"] = "" - def _licfiles_append_fallback_readme_files(destdir): - """Append README files as fallback to license files if a license files is missing""" + def _licfiles_append_fallback_package_files(destdir): + """Append package.json files as fallback to license files if a license files is missing""" + def _get_licenses_from_package_json(package_json): + with open(os.path.join(srctree, package_json), "r") as f: + data = json.load(f) + if "license" in data: + licenses = data["license"].split(" ") + licenses = [license.strip("()") for license in licenses if license != "OR" and license != "AND"] + return ["${COMMON_LICENSE_DIR}/" + license for license in licenses], licenses + else: + return [package_json], None fallback = True - readmes = [] basedir = os.path.join(srctree, destdir) for fn in os.listdir(basedir): upper = fn.upper() - if upper.startswith("README"): - fullpath = os.path.join(basedir, fn) - readmes.append(fullpath) if upper.startswith("COPYING") or "LICENCE" in upper or "LICENSE" in upper: fallback = False if fallback: - for readme in readmes: - licfiles.append(os.path.relpath(readme, srctree)) + pkg_json = os.path.join(basedir, "package.json") + return _get_licenses_from_package_json(pkg_json) + return [], None # Handle the dependencies def _handle_dependency(name, params, destdir): deptree = destdir.split('node_modules/') suffix = "-".join([npm_package(dep) for dep in deptree]) packages["${PN}" + suffix] = destdir - _licfiles_append_fallback_readme_files(destdir) + (fallback_licfiles, common_lics) = _licfiles_append_fallback_package_files(destdir) + licfiles.extend(fallback_licfiles) + if common_lics: + fallback_licenses["${PN}" + suffix] = common_lics with open(shrinkwrap_file, "r") as f: shrinkwrap = json.load(f) foreach_dependencies(shrinkwrap, _handle_dependency, dev) - return licfiles, packages + return licfiles, packages, fallback_licenses # Handle the peer dependencies def _handle_peer_dependency(self, shrinkwrap_file): @@ -266,18 +278,31 @@ class NpmRecipeHandler(RecipeHandler): fetcher.unpack(srctree) bb.note("Handling licences ...") - (licfiles, packages) = self._handle_licenses(srctree, shrinkwrap_file, dev) + (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev) def _guess_odd_license(licfiles): import bb md5sums = get_license_md5sums(d, linenumbers=True) + def _resolve_licfile(srctree, licfile): + match = re.search(r'\$\{COMMON_LICENSE_DIR\}/(.+)$', licfile) + if match: + license = match.group(1) + commonlicdir = d.getVar('COMMON_LICENSE_DIR') + return os.path.join(commonlicdir, license) + + return os.path.join(srctree, licfile) + chksums = [] licenses = [] + md5value = None for licfile in licfiles: - f = os.path.join(srctree, licfile) - md5value = bb.utils.md5_file(f) + f = _resolve_licfile(srctree, licfile) + try: + md5value = bb.utils.md5_file(f) + except FileNotFoundError: + logger.info("Could not determine license for '%s'" % licfile) (license, beginline, endline, md5) = md5sums.get(md5value, (None, "", "", "")) if not license: @@ -292,10 +317,10 @@ class NpmRecipeHandler(RecipeHandler): ";endline=%s" % (endline) if endline else "", md5 if md5 else md5value)) licenses.append((license, licfile, md5value)) - return (licenses, chksums) + return (licenses, chksums, fallback_licenses) - (licenses, extravalues["LIC_FILES_CHKSUM"]) = _guess_odd_license(licfiles) - split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, lines_after) + (licenses, extravalues["LIC_FILES_CHKSUM"], fallback_licenses) = _guess_odd_license(licfiles) + split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, lines_after, fallback_licenses) classes.append("npm") handled.append("buildsystem") From patchwork Mon Aug 12 12:28:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 47683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC82AC531DD for ; Mon, 12 Aug 2024 12:28:44 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web11.44998.1723465721292247190 for ; Mon, 12 Aug 2024 05:28:41 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=YxjOZmdj; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 9B3629C5D24; Mon, 12 Aug 2024 08:28:40 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id MyZfd5lWhrN9; Mon, 12 Aug 2024 08:28:39 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id D991B9C5F35; Mon, 12 Aug 2024 08:28:39 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com D991B9C5F35 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1723465719; bh=qDLER9r/6v2yuNogmpzJQCTRRxjRP0i1NCl8XefCBvg=; h=From:To:Date:Message-Id:MIME-Version; b=YxjOZmdjgpCduwVX7woNuYe7wRlCP7X/T90MtpbbL/g96Pf8C2hHzEfanmZIv6kpX Z2UdbV4JSOyYRS8MXjJLIcVVA2kE6GdoaVwS9kZ3DRbRj+cf6kCvYxyUMxFeNyySxZ WjvuGzZroFN2hhFW113EJW5LVpfelaw7MiHkYc52DNKT3lnF8Pe5cVRINoYdOVhwrg gnS3TT0ZUkC1MsNG+TzmyoAtzV7o5qfYEo7iIz7vu9BH0cHGzjhmQwfuk+olibRNum WU+v9QOvX9XWZsu/GTKmXQniVT+JXfqiE0V172h5DCZDh15Ney4C7TiVVdufDBdqHG DcXbESun6xN1w== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id XF_Yshg8659Q; Mon, 12 Aug 2024 08:28:39 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id EA0E79C5E53; Mon, 12 Aug 2024 08:28:38 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, alexandre.belloni@bootlin.com, stefan.herbrechtsmeier@weidmueller.com, Enguerrand de Ribaucourt Subject: [OE-core][PATCH v4 3/4] recipetool: create: split guess_license function Date: Mon, 12 Aug 2024 14:28:28 +0200 Message-Id: <20240812122829.577515-4-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 12 Aug 2024 12:28:44 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16469 The npm recipetool handler redefines the license code the could be unified. In order to do this refactoring, extract the bits we'll need into separate functions. guess_license() is renamed to find_licenses() and is split into find_license_files() and match_licenses(). Signed-off-by: Enguerrand de Ribaucourt --- v4: - fixed oe-selftest recipetool.RecipetoolTests.test_recipetool_handle_license_vars v3: - added commit --- meta/classes-global/license.bbclass | 8 ++--- meta/lib/oeqa/selftest/cases/recipetool.py | 1 + scripts/lib/recipetool/create.py | 34 ++++++++++++++-------- scripts/lib/recipetool/create_go.py | 4 +-- scripts/lib/recipetool/create_npm.py | 4 +-- 5 files changed, 31 insertions(+), 20 deletions(-) diff --git a/meta/classes-global/license.bbclass b/meta/classes-global/license.bbclass index b2e0d3faba..4e39ec1d44 100644 --- a/meta/classes-global/license.bbclass +++ b/meta/classes-global/license.bbclass @@ -39,7 +39,7 @@ python do_populate_lic() { } PSEUDO_IGNORE_PATHS .= ",${@','.join(((d.getVar('COMMON_LICENSE_DIR') or '') + ' ' + (d.getVar('LICENSE_PATH') or '') + ' ' + d.getVar('COREBASE') + '/meta/COPYING').split())}" -# it would be better to copy them in do_install:append, but find_license_filesa is python +# it would be better to copy them in do_install:append, but find_license_files is python python perform_packagecopy:prepend () { enabled = oe.data.typed_value('LICENSE_CREATE_PACKAGE', d) if d.getVar('CLASSOVERRIDE') == 'class-target' and enabled: @@ -149,14 +149,14 @@ def find_license_files(d): # and "with exceptions" being * # we'll just strip out the modifier and put # the base license. - find_license(node.s.replace("+", "").replace("*", "")) + find_licenses(node.s.replace("+", "").replace("*", "")) self.generic_visit(node) def visit_Constant(self, node): - find_license(node.value.replace("+", "").replace("*", "")) + find_licenses(node.value.replace("+", "").replace("*", "")) self.generic_visit(node) - def find_license(license_type): + def find_licenses(license_type): try: bb.utils.mkdirhier(gen_lic_dest) except: diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index 42202b7831..f742dd4d64 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -1068,6 +1068,7 @@ class RecipetoolTests(RecipetoolBase): d = DataConnectorCopy d.getVar = Mock(return_value=commonlicdir) + d.expand = Mock(side_effect=lambda x: x) srctree = tempfile.mkdtemp(prefix='recipetoolqa') self.track_for_cleanup(srctree) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index 066366e34f..c626844370 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -960,7 +960,7 @@ def handle_license_vars(srctree, lines_before, handled, extravalues, d): # Someone else has already handled the license vars, just return their value return lichandled[0][1] - licvalues = guess_license(srctree, d) + licvalues = find_licenses(srctree, d) licenses = [] lic_files_chksum = [] lic_unknown = [] @@ -1216,13 +1216,7 @@ def crunch_license(licfile): lictext = '' return md5val, lictext -def guess_license(srctree, d): - import bb - md5sums = get_license_md5sums(d) - - crunched_md5sums = crunch_known_licenses(d) - - licenses = [] +def find_license_files(srctree): licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*', 'e[dp]l-v10'] skip_extensions = (".html", ".js", ".json", ".svg", ".ts", ".go") licfiles = [] @@ -1235,11 +1229,22 @@ def guess_license(srctree, d): fullpath = os.path.join(root, fn) if not fullpath in licfiles: licfiles.append(fullpath) + + return licfiles + +def match_licenses(licfiles, srctree, d): + import bb + md5sums = get_license_md5sums(d) + + crunched_md5sums = crunch_known_licenses(d) + + licenses = [] for licfile in sorted(licfiles): - md5value = bb.utils.md5_file(licfile) + resolved_licfile = d.expand(licfile) + md5value = bb.utils.md5_file(resolved_licfile) license = md5sums.get(md5value, None) if not license: - crunched_md5, lictext = crunch_license(licfile) + crunched_md5, lictext = crunch_license(resolved_licfile) license = crunched_md5sums.get(crunched_md5, None) if lictext and not license: license = 'Unknown' @@ -1249,13 +1254,19 @@ def guess_license(srctree, d): if license: licenses.append((license, os.path.relpath(licfile, srctree), md5value)) + return licenses + +def find_licenses(srctree, d): + licfiles = find_license_files(srctree) + licenses = match_licenses(licfiles, srctree, d) + # FIXME should we grab at least one source file with a license header and add that too? return licenses def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=None, pn='${PN}'): """ - Given a list of (license, path, md5sum) as returned by guess_license(), + Given a list of (license, path, md5sum) as returned by match_licenses(), a dict of package name to path mappings, write out a set of package-specific LICENSE values. """ @@ -1418,4 +1429,3 @@ def register_commands(subparsers): parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS) parser_create.add_argument('--mirrors', action="store_true", help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by default).') parser_create.set_defaults(func=create_recipe) - diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py index a85a2f2786..5cc53931f0 100644 --- a/scripts/lib/recipetool/create_go.py +++ b/scripts/lib/recipetool/create_go.py @@ -14,7 +14,7 @@ from collections import namedtuple from enum import Enum from html.parser import HTMLParser from recipetool.create import RecipeHandler, handle_license_vars -from recipetool.create import guess_license, tidy_licenses, fixup_license +from recipetool.create import find_licenses, tidy_licenses, fixup_license from recipetool.create import determine_from_url from urllib.error import URLError, HTTPError @@ -624,7 +624,7 @@ class GoRecipeHandler(RecipeHandler): licenses = [] lic_files_chksum = [] - licvalues = guess_license(tmp_vendor_dir, d) + licvalues = find_licenses(tmp_vendor_dir, d) shutil.rmtree(tmp_vendor_dir) if licvalues: diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py index dd0ac01c3e..78dc248f31 100644 --- a/scripts/lib/recipetool/create_npm.py +++ b/scripts/lib/recipetool/create_npm.py @@ -17,7 +17,7 @@ from bb.fetch2.npm import npm_package from bb.fetch2.npmsw import foreach_dependencies from recipetool.create import RecipeHandler from recipetool.create import get_license_md5sums -from recipetool.create import guess_license +from recipetool.create import find_licenses from recipetool.create import split_pkg_licenses logger = logging.getLogger('recipetool') @@ -320,7 +320,7 @@ class NpmRecipeHandler(RecipeHandler): return (licenses, chksums, fallback_licenses) (licenses, extravalues["LIC_FILES_CHKSUM"], fallback_licenses) = _guess_odd_license(licfiles) - split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, lines_after, fallback_licenses) + split_pkg_licenses([*licenses, *find_licenses(srctree, d)], packages, lines_after, fallback_licenses) classes.append("npm") handled.append("buildsystem") From patchwork Mon Aug 12 12:28:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 47680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EC94C52D7F for ; Mon, 12 Aug 2024 12:28:44 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web10.44986.1723465722401830421 for ; Mon, 12 Aug 2024 05:28:42 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=NyR7vIaY; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id B3AEF9C5E53; Mon, 12 Aug 2024 08:28:41 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id 8WOPUz9Cyuli; Mon, 12 Aug 2024 08:28:41 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id E602F9C5FE4; Mon, 12 Aug 2024 08:28:40 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com E602F9C5FE4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1723465720; bh=7wEomm+AxXaSg/Du+ZTXIrZokRdsAL98CZxQNSAOW/8=; h=From:To:Date:Message-Id:MIME-Version; b=NyR7vIaYk/tzBuoQK237vc8pxlIWP/HidJW/PNQvSt1GIHybVJ7/AsU2KdfSOfmnu P82QfRXgpkeX7fwKapTQZRyE14TjYcLzWJOmwM+6kNZdpN5aFsgNmLl1W5s+Ao/HZP sK8MTcHPF7wiNPh5NO45BIq3B/NlhA57PozJIIJ4SSP6UbozcXimF/DE9wgq3DNipp X3dvJ+k1NRc5NinxnS9LZiNPkUYjj9Wb+TH4Dib4FexCiydNee8K8V98MhXvuf6MJB 0N17Z9wFKovwPsig3Wzi0cfqXBe2K+3BsiuCTorQ6Zp+BNWkENLsPBqtDxkb7WVhvz Mm8H/tYD0vutA== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id BcHHy4cuhOm3; Mon, 12 Aug 2024 08:28:40 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id F3E999C5E53; Mon, 12 Aug 2024 08:28:39 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, alexandre.belloni@bootlin.com, stefan.herbrechtsmeier@weidmueller.com, Enguerrand de Ribaucourt Subject: [OE-core][PATCH v4 4/4] recipetool: create_npm: reuse license utils Date: Mon, 12 Aug 2024 14:28:29 +0200 Message-Id: <20240812122829.577515-5-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 12 Aug 2024 12:28:44 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16470 create_npm.py duplicated the logic for matching licenses from files and also finding them. This patch refactors the code to reuse the license utils. This will make the code more maintainable and also align both behaviors. For instance, some licenses weren't matched properly because the duplicate logic did not support the difference in format in the md5 tables for COMMON_LICENSE_DIR and licenses.csv. This is also faster since the license files were being read twice. The result is slightly more accurate since the utils have better implementations, and I was able to reuse the logic for the root PN package, as well as the base LICENSE variable. I chose to extract generate_common_licenses_chksums into create.py since it can be considered a general utility function to allow other recipetool creators to refer to COMMON_LICENSE_DIR files. I updated the wording in the code when appropriate. v3: - added commit - this replaces the commit that added all the COMMON_LICENSE_DIR md5 to licenses.csv Signed-off-by: Enguerrand de Ribaucourt --- scripts/lib/recipetool/create.py | 8 +++ scripts/lib/recipetool/create_npm.py | 92 +++++++++------------------- 2 files changed, 36 insertions(+), 64 deletions(-) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index c626844370..ea2ef5be63 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -1295,6 +1295,14 @@ def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=None, pn outlicenses[pkgname] = licenses return outlicenses +def generate_common_licenses_chksums(common_licenses, d): + lic_files_chksums = [] + for license in tidy_licenses(common_licenses): + licfile = '${COMMON_LICENSE_DIR}/' + license + md5value = bb.utils.md5_file(d.expand(licfile)) + lic_files_chksums.append('file://%s;md5=%s' % (licfile, md5value)) + return lic_files_chksums + def read_pkgconfig_provides(d): pkgdatadir = d.getVar('PKGDATA_DIR') pkgmap = {} diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py index 78dc248f31..3363a0e7ee 100644 --- a/scripts/lib/recipetool/create_npm.py +++ b/scripts/lib/recipetool/create_npm.py @@ -16,8 +16,7 @@ from bb.fetch2.npm import NpmEnvironment from bb.fetch2.npm import npm_package from bb.fetch2.npmsw import foreach_dependencies from recipetool.create import RecipeHandler -from recipetool.create import get_license_md5sums -from recipetool.create import find_licenses +from recipetool.create import match_licenses, find_license_files, generate_common_licenses_chksums from recipetool.create import split_pkg_licenses logger = logging.getLogger('recipetool') @@ -112,51 +111,53 @@ class NpmRecipeHandler(RecipeHandler): """Return the extra license files and the list of packages""" licfiles = [] packages = {} - # Licenses from package.json point to COMMON_LICENSE_DIR so we need - # to associate them explicitely for split_pkg_licenses() + # Licenses from package.json will point to COMMON_LICENSE_DIR so we need + # to associate them explicitely to packages for split_pkg_licenses() fallback_licenses = dict() - # Handle the parent package - packages["${PN}"] = "" - - def _licfiles_append_fallback_package_files(destdir): - """Append package.json files as fallback to license files if a license files is missing""" + def _find_package_licenses(destdir): + """Either find license files, or use package.json metadata""" def _get_licenses_from_package_json(package_json): with open(os.path.join(srctree, package_json), "r") as f: data = json.load(f) if "license" in data: licenses = data["license"].split(" ") licenses = [license.strip("()") for license in licenses if license != "OR" and license != "AND"] - return ["${COMMON_LICENSE_DIR}/" + license for license in licenses], licenses + return [], licenses else: return [package_json], None - fallback = True basedir = os.path.join(srctree, destdir) - for fn in os.listdir(basedir): - upper = fn.upper() - if upper.startswith("COPYING") or "LICENCE" in upper or "LICENSE" in upper: - fallback = False - if fallback: + licfiles = find_license_files(basedir) + if len(licfiles) > 0: + return licfiles, None + else: + # A license wasn't found in the package directory, so we'll use the package.json metadata pkg_json = os.path.join(basedir, "package.json") return _get_licenses_from_package_json(pkg_json) - return [], None + + def _get_package_licenses(destdir, package): + (package_licfiles, package_licenses) = _find_package_licenses(destdir) + if package_licfiles: + licfiles.extend(package_licfiles) + else: + fallback_licenses[package] = package_licenses # Handle the dependencies def _handle_dependency(name, params, destdir): deptree = destdir.split('node_modules/') suffix = "-".join([npm_package(dep) for dep in deptree]) packages["${PN}" + suffix] = destdir - (fallback_licfiles, common_lics) = _licfiles_append_fallback_package_files(destdir) - licfiles.extend(fallback_licfiles) - if common_lics: - fallback_licenses["${PN}" + suffix] = common_lics + _get_package_licenses(destdir, "${PN}" + suffix) with open(shrinkwrap_file, "r") as f: shrinkwrap = json.load(f) - foreach_dependencies(shrinkwrap, _handle_dependency, dev) + # Handle the parent package + packages["${PN}"] = "" + _get_package_licenses(srctree, "${PN}") + return licfiles, packages, fallback_licenses # Handle the peer dependencies @@ -279,48 +280,11 @@ class NpmRecipeHandler(RecipeHandler): bb.note("Handling licences ...") (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev) - - def _guess_odd_license(licfiles): - import bb - - md5sums = get_license_md5sums(d, linenumbers=True) - - def _resolve_licfile(srctree, licfile): - match = re.search(r'\$\{COMMON_LICENSE_DIR\}/(.+)$', licfile) - if match: - license = match.group(1) - commonlicdir = d.getVar('COMMON_LICENSE_DIR') - return os.path.join(commonlicdir, license) - - return os.path.join(srctree, licfile) - - chksums = [] - licenses = [] - md5value = None - for licfile in licfiles: - f = _resolve_licfile(srctree, licfile) - try: - md5value = bb.utils.md5_file(f) - except FileNotFoundError: - logger.info("Could not determine license for '%s'" % licfile) - (license, beginline, endline, md5) = md5sums.get(md5value, - (None, "", "", "")) - if not license: - license = "Unknown" - logger.info("Please add the following line for '%s' to a " - "'lib/recipetool/licenses.csv' and replace `Unknown`, " - "`X`, `Y` and `MD5` with the license, begin line, " - "end line and partial MD5 checksum:\n" \ - "%s,Unknown,X,Y,MD5" % (licfile, md5value)) - chksums.append("file://%s%s%s;md5=%s" % (licfile, - ";beginline=%s" % (beginline) if beginline else "", - ";endline=%s" % (endline) if endline else "", - md5 if md5 else md5value)) - licenses.append((license, licfile, md5value)) - return (licenses, chksums, fallback_licenses) - - (licenses, extravalues["LIC_FILES_CHKSUM"], fallback_licenses) = _guess_odd_license(licfiles) - split_pkg_licenses([*licenses, *find_licenses(srctree, d)], packages, lines_after, fallback_licenses) + licvalues = match_licenses(licfiles, srctree, d) + split_pkg_licenses(licvalues, packages, lines_after, fallback_licenses) + fallback_licenses_flat = [license for sublist in fallback_licenses.values() for license in sublist] + extravalues["LIC_FILES_CHKSUM"] = generate_common_licenses_chksums(fallback_licenses_flat, d) + extravalues["LICENSE"] = fallback_licenses_flat classes.append("npm") handled.append("buildsystem")