From patchwork Mon Aug 12 12:28:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 47680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EC94C52D7F for ; Mon, 12 Aug 2024 12:28:44 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web10.44986.1723465722401830421 for ; Mon, 12 Aug 2024 05:28:42 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=NyR7vIaY; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id B3AEF9C5E53; Mon, 12 Aug 2024 08:28:41 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id 8WOPUz9Cyuli; Mon, 12 Aug 2024 08:28:41 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id E602F9C5FE4; Mon, 12 Aug 2024 08:28:40 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com E602F9C5FE4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1723465720; bh=7wEomm+AxXaSg/Du+ZTXIrZokRdsAL98CZxQNSAOW/8=; h=From:To:Date:Message-Id:MIME-Version; b=NyR7vIaYk/tzBuoQK237vc8pxlIWP/HidJW/PNQvSt1GIHybVJ7/AsU2KdfSOfmnu P82QfRXgpkeX7fwKapTQZRyE14TjYcLzWJOmwM+6kNZdpN5aFsgNmLl1W5s+Ao/HZP sK8MTcHPF7wiNPh5NO45BIq3B/NlhA57PozJIIJ4SSP6UbozcXimF/DE9wgq3DNipp X3dvJ+k1NRc5NinxnS9LZiNPkUYjj9Wb+TH4Dib4FexCiydNee8K8V98MhXvuf6MJB 0N17Z9wFKovwPsig3Wzi0cfqXBe2K+3BsiuCTorQ6Zp+BNWkENLsPBqtDxkb7WVhvz Mm8H/tYD0vutA== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id BcHHy4cuhOm3; Mon, 12 Aug 2024 08:28:40 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id F3E999C5E53; Mon, 12 Aug 2024 08:28:39 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, alexandre.belloni@bootlin.com, stefan.herbrechtsmeier@weidmueller.com, Enguerrand de Ribaucourt Subject: [OE-core][PATCH v4 4/4] recipetool: create_npm: reuse license utils Date: Mon, 12 Aug 2024 14:28:29 +0200 Message-Id: <20240812122829.577515-5-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240812122829.577515-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 12 Aug 2024 12:28:44 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16470 create_npm.py duplicated the logic for matching licenses from files and also finding them. This patch refactors the code to reuse the license utils. This will make the code more maintainable and also align both behaviors. For instance, some licenses weren't matched properly because the duplicate logic did not support the difference in format in the md5 tables for COMMON_LICENSE_DIR and licenses.csv. This is also faster since the license files were being read twice. The result is slightly more accurate since the utils have better implementations, and I was able to reuse the logic for the root PN package, as well as the base LICENSE variable. I chose to extract generate_common_licenses_chksums into create.py since it can be considered a general utility function to allow other recipetool creators to refer to COMMON_LICENSE_DIR files. I updated the wording in the code when appropriate. v3: - added commit - this replaces the commit that added all the COMMON_LICENSE_DIR md5 to licenses.csv Signed-off-by: Enguerrand de Ribaucourt --- scripts/lib/recipetool/create.py | 8 +++ scripts/lib/recipetool/create_npm.py | 92 +++++++++------------------- 2 files changed, 36 insertions(+), 64 deletions(-) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index c626844370..ea2ef5be63 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -1295,6 +1295,14 @@ def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=None, pn outlicenses[pkgname] = licenses return outlicenses +def generate_common_licenses_chksums(common_licenses, d): + lic_files_chksums = [] + for license in tidy_licenses(common_licenses): + licfile = '${COMMON_LICENSE_DIR}/' + license + md5value = bb.utils.md5_file(d.expand(licfile)) + lic_files_chksums.append('file://%s;md5=%s' % (licfile, md5value)) + return lic_files_chksums + def read_pkgconfig_provides(d): pkgdatadir = d.getVar('PKGDATA_DIR') pkgmap = {} diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py index 78dc248f31..3363a0e7ee 100644 --- a/scripts/lib/recipetool/create_npm.py +++ b/scripts/lib/recipetool/create_npm.py @@ -16,8 +16,7 @@ from bb.fetch2.npm import NpmEnvironment from bb.fetch2.npm import npm_package from bb.fetch2.npmsw import foreach_dependencies from recipetool.create import RecipeHandler -from recipetool.create import get_license_md5sums -from recipetool.create import find_licenses +from recipetool.create import match_licenses, find_license_files, generate_common_licenses_chksums from recipetool.create import split_pkg_licenses logger = logging.getLogger('recipetool') @@ -112,51 +111,53 @@ class NpmRecipeHandler(RecipeHandler): """Return the extra license files and the list of packages""" licfiles = [] packages = {} - # Licenses from package.json point to COMMON_LICENSE_DIR so we need - # to associate them explicitely for split_pkg_licenses() + # Licenses from package.json will point to COMMON_LICENSE_DIR so we need + # to associate them explicitely to packages for split_pkg_licenses() fallback_licenses = dict() - # Handle the parent package - packages["${PN}"] = "" - - def _licfiles_append_fallback_package_files(destdir): - """Append package.json files as fallback to license files if a license files is missing""" + def _find_package_licenses(destdir): + """Either find license files, or use package.json metadata""" def _get_licenses_from_package_json(package_json): with open(os.path.join(srctree, package_json), "r") as f: data = json.load(f) if "license" in data: licenses = data["license"].split(" ") licenses = [license.strip("()") for license in licenses if license != "OR" and license != "AND"] - return ["${COMMON_LICENSE_DIR}/" + license for license in licenses], licenses + return [], licenses else: return [package_json], None - fallback = True basedir = os.path.join(srctree, destdir) - for fn in os.listdir(basedir): - upper = fn.upper() - if upper.startswith("COPYING") or "LICENCE" in upper or "LICENSE" in upper: - fallback = False - if fallback: + licfiles = find_license_files(basedir) + if len(licfiles) > 0: + return licfiles, None + else: + # A license wasn't found in the package directory, so we'll use the package.json metadata pkg_json = os.path.join(basedir, "package.json") return _get_licenses_from_package_json(pkg_json) - return [], None + + def _get_package_licenses(destdir, package): + (package_licfiles, package_licenses) = _find_package_licenses(destdir) + if package_licfiles: + licfiles.extend(package_licfiles) + else: + fallback_licenses[package] = package_licenses # Handle the dependencies def _handle_dependency(name, params, destdir): deptree = destdir.split('node_modules/') suffix = "-".join([npm_package(dep) for dep in deptree]) packages["${PN}" + suffix] = destdir - (fallback_licfiles, common_lics) = _licfiles_append_fallback_package_files(destdir) - licfiles.extend(fallback_licfiles) - if common_lics: - fallback_licenses["${PN}" + suffix] = common_lics + _get_package_licenses(destdir, "${PN}" + suffix) with open(shrinkwrap_file, "r") as f: shrinkwrap = json.load(f) - foreach_dependencies(shrinkwrap, _handle_dependency, dev) + # Handle the parent package + packages["${PN}"] = "" + _get_package_licenses(srctree, "${PN}") + return licfiles, packages, fallback_licenses # Handle the peer dependencies @@ -279,48 +280,11 @@ class NpmRecipeHandler(RecipeHandler): bb.note("Handling licences ...") (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev) - - def _guess_odd_license(licfiles): - import bb - - md5sums = get_license_md5sums(d, linenumbers=True) - - def _resolve_licfile(srctree, licfile): - match = re.search(r'\$\{COMMON_LICENSE_DIR\}/(.+)$', licfile) - if match: - license = match.group(1) - commonlicdir = d.getVar('COMMON_LICENSE_DIR') - return os.path.join(commonlicdir, license) - - return os.path.join(srctree, licfile) - - chksums = [] - licenses = [] - md5value = None - for licfile in licfiles: - f = _resolve_licfile(srctree, licfile) - try: - md5value = bb.utils.md5_file(f) - except FileNotFoundError: - logger.info("Could not determine license for '%s'" % licfile) - (license, beginline, endline, md5) = md5sums.get(md5value, - (None, "", "", "")) - if not license: - license = "Unknown" - logger.info("Please add the following line for '%s' to a " - "'lib/recipetool/licenses.csv' and replace `Unknown`, " - "`X`, `Y` and `MD5` with the license, begin line, " - "end line and partial MD5 checksum:\n" \ - "%s,Unknown,X,Y,MD5" % (licfile, md5value)) - chksums.append("file://%s%s%s;md5=%s" % (licfile, - ";beginline=%s" % (beginline) if beginline else "", - ";endline=%s" % (endline) if endline else "", - md5 if md5 else md5value)) - licenses.append((license, licfile, md5value)) - return (licenses, chksums, fallback_licenses) - - (licenses, extravalues["LIC_FILES_CHKSUM"], fallback_licenses) = _guess_odd_license(licfiles) - split_pkg_licenses([*licenses, *find_licenses(srctree, d)], packages, lines_after, fallback_licenses) + licvalues = match_licenses(licfiles, srctree, d) + split_pkg_licenses(licvalues, packages, lines_after, fallback_licenses) + fallback_licenses_flat = [license for sublist in fallback_licenses.values() for license in sublist] + extravalues["LIC_FILES_CHKSUM"] = generate_common_licenses_chksums(fallback_licenses_flat, d) + extravalues["LICENSE"] = fallback_licenses_flat classes.append("npm") handled.append("buildsystem")