From patchwork Mon Aug 12 12:28:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 47682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6C85C3DA7F for ; Mon, 12 Aug 2024 12:28:44 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web11.44997.1723465720175977804 for ; Mon, 12 Aug 2024 05:28:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=rmvBeRCX; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 7FB249C5F5C; Mon, 12 Aug 2024 08:28:39 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id b9PmrpylIGxw; Mon, 12 Aug 2024 08:28:38 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id D35419C5DB6; Mon, 12 Aug 2024 08:28:38 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com D35419C5DB6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1723465718; bh=EhzQ8wNaLqO2ae9g+r4pUxup5brrhibk6lif/pebQBQ=; h=From:To:Date:Message-Id:MIME-Version; b=rmvBeRCXNSvju1IoYiyDNMgR16Wduc0kMPmx0WHSDTgh+CnIh9zW6RdZdBNBunvMY Ezlj1aEkNNyi/F3yYGLRE2xGJlbA88ubETk7CoBlcx9ara16S/fM/vXZCNFib2fppc lJSLjFbyi6JLHPIHJOSqdtEHS22XvaamdPZsUWvwSgVlJfIR0mDpbFaWnGtBhGvenQ 7mtmlcStXn6U92CC5s4zXEmxbE/R1OlBrpuh2hlAtPfLpyCIIYjEogg2zQtfcne21Y NYeP1AKFbeSZFzd6Keot4kRjP389It/zfNwhDapDa1aSHRDxwlbg/vfqUUJ0VH766D jB3bqqGGFEiBA== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id o7cOodV7aS3P; Mon, 12 Aug 2024 08:28:38 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id E0BB69C5E53; Mon, 12 Aug 2024 08:28:37 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, alexandre.belloni@bootlin.com, stefan.herbrechtsmeier@weidmueller.com, Enguerrand de Ribaucourt Subject: [OE-core][PATCH v4 2/4] recipetool: create_npm: resolve licenses defined in package.json Date: Mon, 12 Aug 2024 14:28:27 +0200 Message-Id: <20240812122829.577515-3-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 12 Aug 2024 12:28:44 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16468 Some npm packages do not copy the LICENSE or COPY file into their git repository. They'll instead simply use SPDX identifiers in their package.json. A fallback for those repositories attempted to match the README file to a license file instead, which had a very low probability of success. This commit replaces this fallback with parsing the package.json and looking for the license in COMMON_LICENSE_DIR. If the license is not found, "Unknown" will still be produced. This also generates "Unknown" for packages which had no README file, which could silently not appear in the generated recipe. The user was more likely to miss them. Co-authored-by: Tanguy Raufflet Signed-off-by: Tanguy Raufflet Signed-off-by: Enguerrand de Ribaucourt --- scripts/lib/recipetool/create_npm.py | 57 ++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 16 deletions(-) diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py index 113a89f6a6..dd0ac01c3e 100644 --- a/scripts/lib/recipetool/create_npm.py +++ b/scripts/lib/recipetool/create_npm.py @@ -112,40 +112,52 @@ class NpmRecipeHandler(RecipeHandler): """Return the extra license files and the list of packages""" licfiles = [] packages = {} + # Licenses from package.json point to COMMON_LICENSE_DIR so we need + # to associate them explicitely for split_pkg_licenses() + fallback_licenses = dict() # Handle the parent package packages["${PN}"] = "" - def _licfiles_append_fallback_readme_files(destdir): - """Append README files as fallback to license files if a license files is missing""" + def _licfiles_append_fallback_package_files(destdir): + """Append package.json files as fallback to license files if a license files is missing""" + def _get_licenses_from_package_json(package_json): + with open(os.path.join(srctree, package_json), "r") as f: + data = json.load(f) + if "license" in data: + licenses = data["license"].split(" ") + licenses = [license.strip("()") for license in licenses if license != "OR" and license != "AND"] + return ["${COMMON_LICENSE_DIR}/" + license for license in licenses], licenses + else: + return [package_json], None fallback = True - readmes = [] basedir = os.path.join(srctree, destdir) for fn in os.listdir(basedir): upper = fn.upper() - if upper.startswith("README"): - fullpath = os.path.join(basedir, fn) - readmes.append(fullpath) if upper.startswith("COPYING") or "LICENCE" in upper or "LICENSE" in upper: fallback = False if fallback: - for readme in readmes: - licfiles.append(os.path.relpath(readme, srctree)) + pkg_json = os.path.join(basedir, "package.json") + return _get_licenses_from_package_json(pkg_json) + return [], None # Handle the dependencies def _handle_dependency(name, params, destdir): deptree = destdir.split('node_modules/') suffix = "-".join([npm_package(dep) for dep in deptree]) packages["${PN}" + suffix] = destdir - _licfiles_append_fallback_readme_files(destdir) + (fallback_licfiles, common_lics) = _licfiles_append_fallback_package_files(destdir) + licfiles.extend(fallback_licfiles) + if common_lics: + fallback_licenses["${PN}" + suffix] = common_lics with open(shrinkwrap_file, "r") as f: shrinkwrap = json.load(f) foreach_dependencies(shrinkwrap, _handle_dependency, dev) - return licfiles, packages + return licfiles, packages, fallback_licenses # Handle the peer dependencies def _handle_peer_dependency(self, shrinkwrap_file): @@ -266,18 +278,31 @@ class NpmRecipeHandler(RecipeHandler): fetcher.unpack(srctree) bb.note("Handling licences ...") - (licfiles, packages) = self._handle_licenses(srctree, shrinkwrap_file, dev) + (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev) def _guess_odd_license(licfiles): import bb md5sums = get_license_md5sums(d, linenumbers=True) + def _resolve_licfile(srctree, licfile): + match = re.search(r'\$\{COMMON_LICENSE_DIR\}/(.+)$', licfile) + if match: + license = match.group(1) + commonlicdir = d.getVar('COMMON_LICENSE_DIR') + return os.path.join(commonlicdir, license) + + return os.path.join(srctree, licfile) + chksums = [] licenses = [] + md5value = None for licfile in licfiles: - f = os.path.join(srctree, licfile) - md5value = bb.utils.md5_file(f) + f = _resolve_licfile(srctree, licfile) + try: + md5value = bb.utils.md5_file(f) + except FileNotFoundError: + logger.info("Could not determine license for '%s'" % licfile) (license, beginline, endline, md5) = md5sums.get(md5value, (None, "", "", "")) if not license: @@ -292,10 +317,10 @@ class NpmRecipeHandler(RecipeHandler): ";endline=%s" % (endline) if endline else "", md5 if md5 else md5value)) licenses.append((license, licfile, md5value)) - return (licenses, chksums) + return (licenses, chksums, fallback_licenses) - (licenses, extravalues["LIC_FILES_CHKSUM"]) = _guess_odd_license(licfiles) - split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, lines_after) + (licenses, extravalues["LIC_FILES_CHKSUM"], fallback_licenses) = _guess_odd_license(licfiles) + split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, lines_after, fallback_licenses) classes.append("npm") handled.append("buildsystem")