From patchwork Thu Aug 8 14:48:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 47542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0448C52D7C for ; Thu, 8 Aug 2024 14:48:38 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web10.61816.1723128517423756557 for ; Thu, 08 Aug 2024 07:48:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=tLcOgMaJ; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id C338A9C5D64; Thu, 8 Aug 2024 10:48:36 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id Xdns-Ydj7EEF; Thu, 8 Aug 2024 10:48:36 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 2AD2D9C5F0C; Thu, 8 Aug 2024 10:48:36 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com 2AD2D9C5F0C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1723128516; bh=EhzQ8wNaLqO2ae9g+r4pUxup5brrhibk6lif/pebQBQ=; h=From:To:Date:Message-Id:MIME-Version; b=tLcOgMaJJ4RJxU0eK5LPAHGlv5l4DhJnqLutUbYWfnYygvCZHpqVBPwnendqy4YMv YZR+w3HfzZXGSe16ybHhKiQrMK6KY6fq2o0dUabs4S0ZkEy/fLwvqXTzHYFdsjFkeR 7fun0gN97QD98nN1fYyMgLMn/Z1qK4ClO0Wu212CUqSAx8ZS2vU61zBQwCu229zsJI PuiMSvsxlgr2y22fdVmGIagTTazhSm/XxgOvPu1r1BPvPLZ+4661tqtPFx41OiosLx wn49X28HUQp1KqnRE/AF+z2BQgfE2t9j7mRHal64i1vhgQA5Eo24TyCgCoMwswJ2PE G3bxcJKo0rS+g== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id TGrkLJTYE9Hk; Thu, 8 Aug 2024 10:48:36 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 358CA9C5FA1; Thu, 8 Aug 2024 10:48:35 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, alexandre.belloni@bootlin.com, stefan.herbrechtsmeier@weidmueller.com, Enguerrand de Ribaucourt Subject: [PATCH v3 4/6] recipetool: create_npm: resolve licenses defined in package.json Date: Thu, 8 Aug 2024 16:48:22 +0200 Message-Id: <20240808144824.543208-5-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240808144824.543208-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240808144824.543208-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 08 Aug 2024 14:48:38 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16457 Some npm packages do not copy the LICENSE or COPY file into their git repository. They'll instead simply use SPDX identifiers in their package.json. A fallback for those repositories attempted to match the README file to a license file instead, which had a very low probability of success. This commit replaces this fallback with parsing the package.json and looking for the license in COMMON_LICENSE_DIR. If the license is not found, "Unknown" will still be produced. This also generates "Unknown" for packages which had no README file, which could silently not appear in the generated recipe. The user was more likely to miss them. Co-authored-by: Tanguy Raufflet Signed-off-by: Tanguy Raufflet Signed-off-by: Enguerrand de Ribaucourt --- scripts/lib/recipetool/create_npm.py | 57 ++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 16 deletions(-) diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py index 113a89f6a6..dd0ac01c3e 100644 --- a/scripts/lib/recipetool/create_npm.py +++ b/scripts/lib/recipetool/create_npm.py @@ -112,40 +112,52 @@ class NpmRecipeHandler(RecipeHandler): """Return the extra license files and the list of packages""" licfiles = [] packages = {} + # Licenses from package.json point to COMMON_LICENSE_DIR so we need + # to associate them explicitely for split_pkg_licenses() + fallback_licenses = dict() # Handle the parent package packages["${PN}"] = "" - def _licfiles_append_fallback_readme_files(destdir): - """Append README files as fallback to license files if a license files is missing""" + def _licfiles_append_fallback_package_files(destdir): + """Append package.json files as fallback to license files if a license files is missing""" + def _get_licenses_from_package_json(package_json): + with open(os.path.join(srctree, package_json), "r") as f: + data = json.load(f) + if "license" in data: + licenses = data["license"].split(" ") + licenses = [license.strip("()") for license in licenses if license != "OR" and license != "AND"] + return ["${COMMON_LICENSE_DIR}/" + license for license in licenses], licenses + else: + return [package_json], None fallback = True - readmes = [] basedir = os.path.join(srctree, destdir) for fn in os.listdir(basedir): upper = fn.upper() - if upper.startswith("README"): - fullpath = os.path.join(basedir, fn) - readmes.append(fullpath) if upper.startswith("COPYING") or "LICENCE" in upper or "LICENSE" in upper: fallback = False if fallback: - for readme in readmes: - licfiles.append(os.path.relpath(readme, srctree)) + pkg_json = os.path.join(basedir, "package.json") + return _get_licenses_from_package_json(pkg_json) + return [], None # Handle the dependencies def _handle_dependency(name, params, destdir): deptree = destdir.split('node_modules/') suffix = "-".join([npm_package(dep) for dep in deptree]) packages["${PN}" + suffix] = destdir - _licfiles_append_fallback_readme_files(destdir) + (fallback_licfiles, common_lics) = _licfiles_append_fallback_package_files(destdir) + licfiles.extend(fallback_licfiles) + if common_lics: + fallback_licenses["${PN}" + suffix] = common_lics with open(shrinkwrap_file, "r") as f: shrinkwrap = json.load(f) foreach_dependencies(shrinkwrap, _handle_dependency, dev) - return licfiles, packages + return licfiles, packages, fallback_licenses # Handle the peer dependencies def _handle_peer_dependency(self, shrinkwrap_file): @@ -266,18 +278,31 @@ class NpmRecipeHandler(RecipeHandler): fetcher.unpack(srctree) bb.note("Handling licences ...") - (licfiles, packages) = self._handle_licenses(srctree, shrinkwrap_file, dev) + (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev) def _guess_odd_license(licfiles): import bb md5sums = get_license_md5sums(d, linenumbers=True) + def _resolve_licfile(srctree, licfile): + match = re.search(r'\$\{COMMON_LICENSE_DIR\}/(.+)$', licfile) + if match: + license = match.group(1) + commonlicdir = d.getVar('COMMON_LICENSE_DIR') + return os.path.join(commonlicdir, license) + + return os.path.join(srctree, licfile) + chksums = [] licenses = [] + md5value = None for licfile in licfiles: - f = os.path.join(srctree, licfile) - md5value = bb.utils.md5_file(f) + f = _resolve_licfile(srctree, licfile) + try: + md5value = bb.utils.md5_file(f) + except FileNotFoundError: + logger.info("Could not determine license for '%s'" % licfile) (license, beginline, endline, md5) = md5sums.get(md5value, (None, "", "", "")) if not license: @@ -292,10 +317,10 @@ class NpmRecipeHandler(RecipeHandler): ";endline=%s" % (endline) if endline else "", md5 if md5 else md5value)) licenses.append((license, licfile, md5value)) - return (licenses, chksums) + return (licenses, chksums, fallback_licenses) - (licenses, extravalues["LIC_FILES_CHKSUM"]) = _guess_odd_license(licfiles) - split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, lines_after) + (licenses, extravalues["LIC_FILES_CHKSUM"], fallback_licenses) = _guess_odd_license(licfiles) + split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, lines_after, fallback_licenses) classes.append("npm") handled.append("buildsystem")