From patchwork Thu Aug 22 07:53:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 48084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59CDFC3DA4A for ; Thu, 22 Aug 2024 07:53:30 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web10.8453.1724313205115587184 for ; Thu, 22 Aug 2024 00:53:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=Noqq481m; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 69E4D9C60D3; Thu, 22 Aug 2024 03:53:24 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id q4OHqDFStX9l; Thu, 22 Aug 2024 03:53:24 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 00ED39C5E00; Thu, 22 Aug 2024 03:53:24 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com 00ED39C5E00 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1724313204; bh=zzxcbdv9xw6FetVB3J/GbPt7URvvc3TfqrxxCQNaFuY=; h=From:To:Date:Message-Id:MIME-Version; b=Noqq481mSwSSHDl2TgLVQEI5z+TR9N2fwP2lOH5AFISunavLqah6IQbv0ffVv8S7H 1ZXvWR+rx/M/vamB17RJgm8bAdRvUfmsIluXoOLpB6c09Xg2EGqpLK74M7ZbzhYKY0 YN/qOwT2SZ+h5AM9r6HC7aQm5WQE4MI3h5YuVy7gkHwUlBrRZke1gLiSirq7KptGVt XFdANH4kOmrY5zpGs1UitYna+XuCY/Ioaj9fTFS0Reqrv5GajKJdkob9NNbgN7DfnL i1JBxet9q61P2t49le8jA7Lr2tAoFGwzr7KPZ659zCQG0MY9s4G2y1IlLAUVpuAmkz iwYQANCURsXiQ== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id 14d4hP9tDUl5; Thu, 22 Aug 2024 03:53:23 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 11F599C5D86; Thu, 22 Aug 2024 03:53:22 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, alexandre.belloni@bootlin.com, stefan.herbrechtsmeier@weidmueller.com, Enguerrand de Ribaucourt Subject: [bitbake-devel][PATCH v5 1/2] bitbake: fetch2: npmsw: fix fetching git revisions not on master Date: Thu, 22 Aug 2024 09:53:14 +0200 Message-Id: <20240822075315.18570-2-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240822075315.18570-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240822075315.18570-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 22 Aug 2024 07:53:30 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16512 The NPM package.json documentation[1] states that git URLs may contain a commit-ish suffix to specify a specific revision. When running `npm install`, this revision will be looked for on any branch of the repository. The bitbake implementation however translates the URL stored in package.json into a git URL to be fetch by the bitbake git fetcher. The bitbake fetcher git.py, enforces the branch to be master by default. If the revision specified in the package.json is not on the master branch, the fetch will fail while the package.json is valid. To fix this, append the ";nobranch=1" suffix to the revision in the git URL to be fetched. This will make the bitbake git fetcher ignore the branch and respect the behavior of `npm install``. This can be tested with the following command: $ devtool add --npm-dev https://github.com/seapath/cockpit-cluster-dashboard.git -B version Which points to a project which has a package.json with a git URL: ```json "devDependencies": { "cockpit-repo": "git+https://github.com/cockpit-project/cockpit.git#d34cabacb8e5e1e028c7eea3d6e3b606d862b8ac" } ``` In this repo, the specified revision is on the "main" branch, which would fail without this fix. [1] https://docs.npmjs.com/cli/v10/configuring-npm/package-json#git-urls-as-dependencies Co-authored-by: Tanguy Raufflet Signed-off-by: Tanguy Raufflet Signed-off-by: Enguerrand de Ribaucourt --- bitbake/lib/bb/fetch2/npmsw.py | 1 + 1 file changed, 1 insertion(+) diff --git a/bitbake/lib/bb/fetch2/npmsw.py b/bitbake/lib/bb/fetch2/npmsw.py index b55e885d7b..d8ed9df327 100644 --- a/bitbake/lib/bb/fetch2/npmsw.py +++ b/bitbake/lib/bb/fetch2/npmsw.py @@ -184,6 +184,7 @@ class NpmShrinkWrap(FetchMethod): uri = URI("git://" + str(groups["url"])) uri.params["protocol"] = str(groups["protocol"]) uri.params["rev"] = str(groups["rev"]) + uri.params["nobranch"] = "1" uri.params["destsuffix"] = destsuffix url = str(uri) From patchwork Thu Aug 22 07:53:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Enguerrand de Ribaucourt X-Patchwork-Id: 48085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75464C5320E for ; Thu, 22 Aug 2024 07:53:30 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web10.8454.1724313206023665150 for ; Thu, 22 Aug 2024 00:53:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=avqkWQ6R; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: enguerrand.de-ribaucourt@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 647059C5D86; Thu, 22 Aug 2024 03:53:25 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id uBTrkUHzsebI; Thu, 22 Aug 2024 03:53:25 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 0D22F9C60D2; Thu, 22 Aug 2024 03:53:25 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com 0D22F9C60D2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1724313205; bh=K1Vlnwpi7XAIQBj3P7onMHllRZDIFgL5oUF4NRBLPJk=; h=From:To:Date:Message-Id:MIME-Version; b=avqkWQ6RZTB51XPlMRXLx3Lj5Ms5D3tkt2qIyj9aOWRcoS5gWyeczbJo5GzeuQNK5 Y7Jwd7XOJZdGF0Sw384+JCzllYIupmvVx6+bMnp/HbqEUhKHCjjG3aB5RPxse9TmNp rf7dpka0h1et8s5BNuMrIJ2VCdJ+m1JWcUrQZUI5KQ1ebC0sgG3AIF5L1gzZw+FJDr 05SHrb9s+lkALlTTC8paQKmcaBdLRobXBHujztzvRhNoLmnjzM0FogIetU4QPH59CV JRdyNVxx21+/+S2iHY+TCK7GTuGsGZLXoWOt5kw9p/4C5DeigrLDY9xSnXIFkJbhJp GPKavqyFg2hcQ== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id K3RjRCqaRhOR; Thu, 22 Aug 2024 03:53:24 -0400 (EDT) Received: from sfl-deribaucourt.rennes.sfl (80-15-101-118.ftth.fr.orangecustomers.net [80.15.101.118]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 1F2269C5D86; Thu, 22 Aug 2024 03:53:24 -0400 (EDT) From: Enguerrand de Ribaucourt To: bitbake-devel@lists.openembedded.org Cc: tanguy.raufflet@savoirfairelinux.com, richard.purdie@linuxfoundation.org, alexandre.belloni@bootlin.com, stefan.herbrechtsmeier@weidmueller.com, Enguerrand de Ribaucourt Subject: [bitbake-devel][PATCH v5 2/2] bitbake: fetch2: npmsw: allow packages not declaring a registry version Date: Thu, 22 Aug 2024 09:53:15 +0200 Message-Id: <20240822075315.18570-3-enguerrand.de-ribaucourt@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240822075315.18570-1-enguerrand.de-ribaucourt@savoirfairelinux.com> References: <20240822075315.18570-1-enguerrand.de-ribaucourt@savoirfairelinux.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 22 Aug 2024 07:53:30 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16514 We fetch npm dependencies from the npm-shrinkwrap.json file. They can point to a package on the NPM registry with a version field, or to a git/http/file URL with the resolved field. Such packages are allowed not to declare a registry version field because they may not have been published to the NPM registry. The previous implementation refuses to fetch such packages and throws an error. The resolved field contains the exact source, including the revision, wich we can use to pass as SRC_URI to the git/http/file fetcher. The integrity field is also mandatory for HTTP tarballs which will ensure reproducibility. So even if the version field is not present, we are still fetching a precise revision of the package. Another commit published along this stack is also required in the npm class to support these packages. v5: - improve commit message v3: - Split bitbake npmsw.py modification in another commit Co-authored-by: Tanguy Raufflet Signed-off-by: Tanguy Raufflet Signed-off-by: Enguerrand de Ribaucourt --- bitbake/lib/bb/fetch2/npmsw.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/npmsw.py b/bitbake/lib/bb/fetch2/npmsw.py index d8ed9df327..a5fa598deb 100644 --- a/bitbake/lib/bb/fetch2/npmsw.py +++ b/bitbake/lib/bb/fetch2/npmsw.py @@ -97,7 +97,7 @@ class NpmShrinkWrap(FetchMethod): integrity = params.get("integrity", None) resolved = params.get("resolved", None) - version = params.get("version", None) + version = params.get("version", resolved) # Handle registry sources if is_semver(version) and integrity: