From patchwork Mon Jul 8 13:33:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 46085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24748C41513 for ; Mon, 8 Jul 2024 13:34:08 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web11.79800.1720445647692541029 for ; Mon, 08 Jul 2024 06:34:07 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=yG1vAjbc; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 0F2019C5BAC; Mon, 8 Jul 2024 09:34:07 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id cWbDnIiljbG1; Mon, 8 Jul 2024 09:34:05 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id B03F59C5D3D; Mon, 8 Jul 2024 09:34:05 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com B03F59C5D3D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1720445645; bh=EhzQ8wNaLqO2ae9g+r4pUxup5brrhibk6lif/pebQBQ=; h=From:To:Date:Message-Id:MIME-Version; b=yG1vAjbcorsW3taSmZuzEbNLszSfT0igQuPQ5flMCEQo/EnxmQ+/Gt8XR7SMwxo2S sDlX2euTKkgty+ajkR4XQ6wdtxd8djSqeR+rCuBFzP6CMCv6otgpvEZqvFNWOjjdYZ 5zF9kLXEMEq6VUjdHwcUpWIuoQ5hBTFngmgfzPc6jFtI/WmvZA6azdMdoSp3ozlxut 8BGBxOBjkwvqKnF7uZtVc1M7L3Oobi1zwAEoL6FeI2x65ce9LZRhCSgCTmaQwYDmF/ a1iLrMTibpePpEnrUnLT33z1/cBN+PL9P6Z/c96lA9Xftgm1GWsdzUjLzDVcZAtnEf OOp3prQWxefww== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id iE-Ws6GUxeb4; Mon, 8 Jul 2024 09:34:05 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id F19519C5D1A; Mon, 8 Jul 2024 09:34:04 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, Enguerrand de Ribaucourt Subject: [PATCH v2 3/4] recipetool: create_npm: resolve licenses defined in package.json Date: Mon, 8 Jul 2024 15:33:56 +0200 Message-Id: <20240708133357.306422-4-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240708133357.306422-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240708133357.306422-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 08 Jul 2024 13:34:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16404 Some npm packages do not copy the LICENSE or COPY file into their git repository. They'll instead simply use SPDX identifiers in their package.json. A fallback for those repositories attempted to match the README file to a license file instead, which had a very low probability of success. This commit replaces this fallback with parsing the package.json and looking for the license in COMMON_LICENSE_DIR. If the license is not found, "Unknown" will still be produced. This also generates "Unknown" for packages which had no README file, which could silently not appear in the generated recipe. The user was more likely to miss them. Co-authored-by: Tanguy Raufflet Signed-off-by: Tanguy Raufflet Signed-off-by: Enguerrand de Ribaucourt --- scripts/lib/recipetool/create_npm.py | 57 ++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 16 deletions(-) diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py index 113a89f6a6..dd0ac01c3e 100644 --- a/scripts/lib/recipetool/create_npm.py +++ b/scripts/lib/recipetool/create_npm.py @@ -112,40 +112,52 @@ class NpmRecipeHandler(RecipeHandler): """Return the extra license files and the list of packages""" licfiles = [] packages = {} + # Licenses from package.json point to COMMON_LICENSE_DIR so we need + # to associate them explicitely for split_pkg_licenses() + fallback_licenses = dict() # Handle the parent package packages["${PN}"] = "" - def _licfiles_append_fallback_readme_files(destdir): - """Append README files as fallback to license files if a license files is missing""" + def _licfiles_append_fallback_package_files(destdir): + """Append package.json files as fallback to license files if a license files is missing""" + def _get_licenses_from_package_json(package_json): + with open(os.path.join(srctree, package_json), "r") as f: + data = json.load(f) + if "license" in data: + licenses = data["license"].split(" ") + licenses = [license.strip("()") for license in licenses if license != "OR" and license != "AND"] + return ["${COMMON_LICENSE_DIR}/" + license for license in licenses], licenses + else: + return [package_json], None fallback = True - readmes = [] basedir = os.path.join(srctree, destdir) for fn in os.listdir(basedir): upper = fn.upper() - if upper.startswith("README"): - fullpath = os.path.join(basedir, fn) - readmes.append(fullpath) if upper.startswith("COPYING") or "LICENCE" in upper or "LICENSE" in upper: fallback = False if fallback: - for readme in readmes: - licfiles.append(os.path.relpath(readme, srctree)) + pkg_json = os.path.join(basedir, "package.json") + return _get_licenses_from_package_json(pkg_json) + return [], None # Handle the dependencies def _handle_dependency(name, params, destdir): deptree = destdir.split('node_modules/') suffix = "-".join([npm_package(dep) for dep in deptree]) packages["${PN}" + suffix] = destdir - _licfiles_append_fallback_readme_files(destdir) + (fallback_licfiles, common_lics) = _licfiles_append_fallback_package_files(destdir) + licfiles.extend(fallback_licfiles) + if common_lics: + fallback_licenses["${PN}" + suffix] = common_lics with open(shrinkwrap_file, "r") as f: shrinkwrap = json.load(f) foreach_dependencies(shrinkwrap, _handle_dependency, dev) - return licfiles, packages + return licfiles, packages, fallback_licenses # Handle the peer dependencies def _handle_peer_dependency(self, shrinkwrap_file): @@ -266,18 +278,31 @@ class NpmRecipeHandler(RecipeHandler): fetcher.unpack(srctree) bb.note("Handling licences ...") - (licfiles, packages) = self._handle_licenses(srctree, shrinkwrap_file, dev) + (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev) def _guess_odd_license(licfiles): import bb md5sums = get_license_md5sums(d, linenumbers=True) + def _resolve_licfile(srctree, licfile): + match = re.search(r'\$\{COMMON_LICENSE_DIR\}/(.+)$', licfile) + if match: + license = match.group(1) + commonlicdir = d.getVar('COMMON_LICENSE_DIR') + return os.path.join(commonlicdir, license) + + return os.path.join(srctree, licfile) + chksums = [] licenses = [] + md5value = None for licfile in licfiles: - f = os.path.join(srctree, licfile) - md5value = bb.utils.md5_file(f) + f = _resolve_licfile(srctree, licfile) + try: + md5value = bb.utils.md5_file(f) + except FileNotFoundError: + logger.info("Could not determine license for '%s'" % licfile) (license, beginline, endline, md5) = md5sums.get(md5value, (None, "", "", "")) if not license: @@ -292,10 +317,10 @@ class NpmRecipeHandler(RecipeHandler): ";endline=%s" % (endline) if endline else "", md5 if md5 else md5value)) licenses.append((license, licfile, md5value)) - return (licenses, chksums) + return (licenses, chksums, fallback_licenses) - (licenses, extravalues["LIC_FILES_CHKSUM"]) = _guess_odd_license(licfiles) - split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, lines_after) + (licenses, extravalues["LIC_FILES_CHKSUM"], fallback_licenses) = _guess_odd_license(licfiles) + split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, lines_after, fallback_licenses) classes.append("npm") handled.append("buildsystem")