| Message ID | 20250127142918.163-1-eric.meyers@arthrex.com |
|---|---|
| Headers | show |
| Series | NPM/NPM Shrinkwrap Basic Authentication Credential Support: | expand |
On Mon, 27 Jan 2025 at 15:29, eric.meyers15310 via lists.openembedded.org <eric.meyers15310=gmail.com@lists.openembedded.org> wrote: > Changes in this commit include: > - npm fetcher changes: > - Add “username=”, “password=”, and “namespace=” parameters to “npm://” SRC_URI > - These parameters will be used to create a “npmrc” file that is passed into the NpmEnvironment object - used for authentication when connecting to NPM registry You need to create the file with a separate step, then point the fetcher or the underlying executable to use it. We do not allow credentials in recipes or other metadata that gets committed to a layer repo. It's not different from placing ssh keys or http passwords into respective files in $HOME. Alex
On Mon, Jan 27, 2025 at 8:39 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > On Mon, 27 Jan 2025 at 15:29, eric.meyers15310 via > lists.openembedded.org > <eric.meyers15310=gmail.com@lists.openembedded.org> wrote: > > Changes in this commit include: > > - npm fetcher changes: > > - Add “username=”, “password=”, and “namespace=” parameters to “npm://” SRC_URI > > - These parameters will be used to create a “npmrc” file that is passed into the NpmEnvironment object - used for authentication when connecting to NPM registry > > You need to create the file with a separate step, then point the > fetcher or the underlying executable to use it. We do not allow > credentials in recipes or other metadata that gets committed to a > layer repo. It's not different from placing ssh keys or http passwords > into respective files in $HOME. > > Alex Are you suggesting that instead of username= and password= parameters in the npm/npmsw SRC_URI, it should be an "npmrc" parameter that effectively the user can point to the location of their .npmrc file for authentication on an NPM registry? Then, within a user's associated recipe, they would need to create the npmrc file in a do_fetch:prepend() step that that corresponds to the path passed into the SRC_URI? That seems like it'd be fine to retrieve the initial "npm view" output (https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/npm.py#n186), which the fetcher uses to resolve the tarball url. But then when subsequently passing the tarball url into the wget fetcher (https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/npm.py#n238), the wget SRC_URI will need to provide a username/password for basic authentication to access the artifact - that's kinda where I'm stuck and thought a "username=" and "password=" parameter were needed for the npm fetcher similar to the "user=" and "pswd=" parameters available in the wget fetcher. Please correct me if my understanding is wrong, I've been agonizing over this for a few weeks now and couldn't think of a better/more concise solution here, but may be overthinking things at this point.
On Mon, 27 Jan 2025 at 16:44, Eric Meyers <eric.meyers15310@gmail.com> wrote: > Are you suggesting that instead of username= and password= parameters > in the npm/npmsw SRC_URI, it should be an "npmrc" parameter that > effectively the user can point to the location of their .npmrc file > for authentication on an NPM registry? Then, within a user's > associated recipe, they would need to create the npmrc file in a > do_fetch:prepend() step that that corresponds to the path passed into > the SRC_URI? The only acceptable solution would be one that takes passwords written in plain text out of layers and local.conf altogether. And the way to do it is to write these things into ~/.npmrc and ~/.netrc in a secure manner (e.g. pass in to users via encrypted email with instructions that the file must not be readable by anyone but the user), and ensure that npm and wget are able to pick them up from those files (wget does this already, npm I'm not sure). I know we have the password parameter in wget and other 'old' fetchers, that's a bad mis-feature in my opinion and should be removed. For git fetcher patches to add the equivalent have been repeatedly rejected: https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/git.py#n823 Alex
On Mon, Jan 27, 2025 at 10:01 AM Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > On Mon, 27 Jan 2025 at 16:44, Eric Meyers <eric.meyers15310@gmail.com> > wrote: > > Are you suggesting that instead of username= and password= parameters > > in the npm/npmsw SRC_URI, it should be an "npmrc" parameter that > > effectively the user can point to the location of their .npmrc file > > for authentication on an NPM registry? Then, within a user's > > associated recipe, they would need to create the npmrc file in a > > do_fetch:prepend() step that that corresponds to the path passed into > > the SRC_URI? > > The only acceptable solution would be one that takes passwords written > in plain text out of layers and local.conf altogether. And the way to > do it is to write these things into ~/.npmrc and ~/.netrc in a secure > manner (e.g. pass in to users via encrypted email with instructions > that the file must not be readable by anyone but the user), and ensure > that npm and wget are able to pick them up from those files (wget does > this already, npm I'm not sure). > Could this also be solved by letting the end user decide, sort of like the way FETCHCMD_* works? A CREDENTIALCMD_npm variable in local.conf could be saddled with exporting the namespace, password, and username values into whatever environment needs them in a "Just in Time" fashion to lower (but not completely eliminate) exposure. An identity agent, like the 1P or vault CLI would be an obvious choice here, but one could just as easily tie it to a script that devolves down to looking in ~/.npmrc if they wanted to. ..Ch:W..
On Mon, Jan 27, 2025 at 3:06 PM Chuck Wolber <chuckwolber@gmail.com> wrote: > > > > On Mon, Jan 27, 2025 at 10:01 AM Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: >> >> On Mon, 27 Jan 2025 at 16:44, Eric Meyers <eric.meyers15310@gmail.com> wrote: >> > Are you suggesting that instead of username= and password= parameters >> > in the npm/npmsw SRC_URI, it should be an "npmrc" parameter that >> > effectively the user can point to the location of their .npmrc file >> > for authentication on an NPM registry? Then, within a user's >> > associated recipe, they would need to create the npmrc file in a >> > do_fetch:prepend() step that that corresponds to the path passed into >> > the SRC_URI? >> >> The only acceptable solution would be one that takes passwords written >> in plain text out of layers and local.conf altogether. And the way to >> do it is to write these things into ~/.npmrc and ~/.netrc in a secure >> manner (e.g. pass in to users via encrypted email with instructions >> that the file must not be readable by anyone but the user), and ensure >> that npm and wget are able to pick them up from those files (wget does >> this already, npm I'm not sure). > > > Could this also be solved by letting the end user decide, sort of like the way FETCHCMD_* > works? > > A CREDENTIALCMD_npm variable in local.conf could be saddled with exporting the > namespace, password, and username values into whatever environment needs them in a > "Just in Time" fashion to lower (but not completely eliminate) exposure. > > An identity agent, like the 1P or vault CLI would be an obvious choice here, but one could > just as easily tie it to a script that devolves down to looking in ~/.npmrc if they wanted to. > > ..Ch:W.. > > -- > "Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire Last week I had off, so I'm just getting around to addressing these comments this week. Chuck's suggestion is interesting and I will definitely consider it. So this amounts to two potential paths forward for this issue (1 by Alex and 1 by Chuck): 1. Take a similar approach to how the wget fetcher uses default netrc credentials within the user's home directory and use default credentials populated by the user within an npmrc file in their home directory. - Pros: Moves away from specifying username/password in recipe, using precedent set in wget fetcher, potentially less changes in fetcher (no parameter additions, no creation of npmrc file, etc) - Cons: Currently, the wget fetcher relies on default behavior of "netrc" python library - there is no analogous "npmrc" python library and we would need to add logic to check this file manually (shouldn't be too difficult but still would need to rework current NPM environment behavior in the fetcher). Might pose challenges in CI/CD environments where these files need to now be created if using private registries (again should be easy enough). 2. Take a similar approach to how FETCHCMD_npm is used and specify another variable named "FETCHCREDENTIALCMD_npm" that allows the user to point to a specific npmrc file to use in their environment. - Pros: Moves away from specifying username/password in recipe - Cons: Adding another variable to bitbake, lowers (but doesn't completely eliminate) credential exposure, might require more support within recipetool within poky I am going to take a stab at reworking this patch to match approach #1 outlined above, but if any hiccups come up, I'll post issues here and re-evaluate using another approach (potentially #2 above). #1 feels like it will be easier to implement and more straightforward than #2, but perhaps I'm missing something.
On Mon, 2025-02-03 at 09:33 -0600, Eric Meyers via lists.openembedded.org wrote: > On Mon, Jan 27, 2025 at 3:06 PM Chuck Wolber <chuckwolber@gmail.com> > wrote: > > > > > > > > On Mon, Jan 27, 2025 at 10:01 AM Alexander Kanavin via > > lists.openembedded.org > > <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > > > > > On Mon, 27 Jan 2025 at 16:44, Eric Meyers > > > <eric.meyers15310@gmail.com> wrote: > > > > Are you suggesting that instead of username= and password= > > > > parameters > > > > in the npm/npmsw SRC_URI, it should be an "npmrc" parameter > > > > that > > > > effectively the user can point to the location of their .npmrc > > > > file > > > > for authentication on an NPM registry? Then, within a user's > > > > associated recipe, they would need to create the npmrc file in > > > > a > > > > do_fetch:prepend() step that that corresponds to the path > > > > passed into > > > > the SRC_URI? > > > > > > The only acceptable solution would be one that takes passwords > > > written > > > in plain text out of layers and local.conf altogether. And the > > > way to > > > do it is to write these things into ~/.npmrc and ~/.netrc in a > > > secure > > > manner (e.g. pass in to users via encrypted email with > > > instructions > > > that the file must not be readable by anyone but the user), and > > > ensure > > > that npm and wget are able to pick them up from those files (wget > > > does > > > this already, npm I'm not sure). > > > > > > Could this also be solved by letting the end user decide, sort of > > like the way FETCHCMD_* > > works? > > > > A CREDENTIALCMD_npm variable in local.conf could be saddled with > > exporting the > > namespace, password, and username values into whatever environment > > needs them in a > > "Just in Time" fashion to lower (but not completely eliminate) > > exposure. > > > > An identity agent, like the 1P or vault CLI would be an obvious > > choice here, but one could > > just as easily tie it to a script that devolves down to looking in > > ~/.npmrc if they wanted to. > > > > ..Ch:W.. > > > > -- > > "Perfection must be reached by degrees; she requires the slow hand > > of time." - Voltaire > > Last week I had off, so I'm just getting around to addressing these > comments this week. Chuck's suggestion is interesting and I will > definitely consider it. So this amounts to two potential paths > forward > for this issue (1 by Alex and 1 by Chuck): > > 1. Take a similar approach to how the wget fetcher uses default netrc > credentials within the user's home directory and use default > credentials populated by the user within an npmrc file in their home > directory. > - Pros: Moves away from specifying username/password in recipe, > using precedent set in wget fetcher, potentially less changes in > fetcher (no parameter additions, no creation of npmrc file, etc) > - Cons: Currently, the wget fetcher relies on default behavior of > "netrc" python library - there is no analogous "npmrc" python library > and we would need to add logic to check this file manually (shouldn't > be too difficult but still would need to rework current NPM > environment behavior in the fetcher). Might pose challenges in CI/CD > environments where these files need to now be created if using > private > registries (again should be easy enough). > > 2. Take a similar approach to how FETCHCMD_npm is used and specify > another variable named "FETCHCREDENTIALCMD_npm" that allows the user > to point to a specific npmrc file to use in their environment. > - Pros: Moves away from specifying username/password in recipe > - Cons: Adding another variable to bitbake, lowers (but doesn't > completely eliminate) credential exposure, might require more support > within recipetool within poky > > I am going to take a stab at reworking this patch to match approach > #1 > outlined above, but if any hiccups come up, I'll post issues here and > re-evaluate using another approach (potentially #2 above). #1 feels > like it will be easier to implement and more straightforward than #2, > but perhaps I'm missing something. FWIW I think this is the right thing to do and #1 matches what we've done elsewhere. Having watched what happens in the real world with passwords in urls, we really don't want to support that. Cheers, Richard
Hi Eric, have you seen my RFC [1] to remove the npm dependency from the npm fetcher and use plain wget? I have a Work in Progress to parse the package-lock.json inside a task. This task generate plain https SRC_URIs and fetch everything with the wget fetcher. Furthermore I have eliminate the npm cache to speed up the build. I plan to post a first RFC with cargo, go mod and npm support in the next days. Regards Stefan [1] https://lists.openembedded.org/g/bitbake-devel/topic/110212697 Am 03.02.2025 um 16:33 schrieb Eric Meyers via lists.openembedded.org: > On Mon, Jan 27, 2025 at 3:06 PM Chuck Wolber <chuckwolber@gmail.com> wrote: >> >> >> On Mon, Jan 27, 2025 at 10:01 AM Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: >>> On Mon, 27 Jan 2025 at 16:44, Eric Meyers <eric.meyers15310@gmail.com> wrote: >>>> Are you suggesting that instead of username= and password= parameters >>>> in the npm/npmsw SRC_URI, it should be an "npmrc" parameter that >>>> effectively the user can point to the location of their .npmrc file >>>> for authentication on an NPM registry? Then, within a user's >>>> associated recipe, they would need to create the npmrc file in a >>>> do_fetch:prepend() step that that corresponds to the path passed into >>>> the SRC_URI? >>> The only acceptable solution would be one that takes passwords written >>> in plain text out of layers and local.conf altogether. And the way to >>> do it is to write these things into ~/.npmrc and ~/.netrc in a secure >>> manner (e.g. pass in to users via encrypted email with instructions >>> that the file must not be readable by anyone but the user), and ensure >>> that npm and wget are able to pick them up from those files (wget does >>> this already, npm I'm not sure). >> >> Could this also be solved by letting the end user decide, sort of like the way FETCHCMD_* >> works? >> >> A CREDENTIALCMD_npm variable in local.conf could be saddled with exporting the >> namespace, password, and username values into whatever environment needs them in a >> "Just in Time" fashion to lower (but not completely eliminate) exposure. >> >> An identity agent, like the 1P or vault CLI would be an obvious choice here, but one could >> just as easily tie it to a script that devolves down to looking in ~/.npmrc if they wanted to. >> >> ..Ch:W.. >> >> -- >> "Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire > Last week I had off, so I'm just getting around to addressing these > comments this week. Chuck's suggestion is interesting and I will > definitely consider it. So this amounts to two potential paths forward > for this issue (1 by Alex and 1 by Chuck): > > 1. Take a similar approach to how the wget fetcher uses default netrc > credentials within the user's home directory and use default > credentials populated by the user within an npmrc file in their home > directory. > - Pros: Moves away from specifying username/password in recipe, > using precedent set in wget fetcher, potentially less changes in > fetcher (no parameter additions, no creation of npmrc file, etc) > - Cons: Currently, the wget fetcher relies on default behavior of > "netrc" python library - there is no analogous "npmrc" python library > and we would need to add logic to check this file manually (shouldn't > be too difficult but still would need to rework current NPM > environment behavior in the fetcher). Might pose challenges in CI/CD > environments where these files need to now be created if using private > registries (again should be easy enough). > > 2. Take a similar approach to how FETCHCMD_npm is used and specify > another variable named "FETCHCREDENTIALCMD_npm" that allows the user > to point to a specific npmrc file to use in their environment. > - Pros: Moves away from specifying username/password in recipe > - Cons: Adding another variable to bitbake, lowers (but doesn't > completely eliminate) credential exposure, might require more support > within recipetool within poky > > I am going to take a stab at reworking this patch to match approach #1 > outlined above, but if any hiccups come up, I'll post issues here and > re-evaluate using another approach (potentially #2 above). #1 feels > like it will be easier to implement and more straightforward than #2, > but perhaps I'm missing something. > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#17122): https://lists.openembedded.org/g/bitbake-devel/message/17122 > Mute This Topic: https://lists.openembedded.org/mt/110839321/6374899 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [stefan.herbrechtsmeier-oss@weidmueller.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Tue, Feb 4, 2025 at 7:19 AM Stefan Herbrechtsmeier <stefan.herbrechtsmeier-oss@weidmueller.com> wrote: > > Hi Eric, > > have you seen my RFC [1] to remove the npm dependency from the npm > fetcher and use plain wget? I have a Work in Progress to parse the > package-lock.json inside a task. This task generate plain https SRC_URIs > and fetch everything with the wget fetcher. Furthermore I have eliminate > the npm cache to speed up the build. I plan to post a first RFC with > cargo, go mod and npm support in the next days. > > Regards > Stefan > > [1] https://lists.openembedded.org/g/bitbake-devel/topic/110212697 > Hi Stefan, No I was not aware of that change - thanks for bringing it to my attention. It looks interesting and I think would make my (eventual) patch resubmission obsolete if I’m understanding it correctly. A few questions: - Sounds like this change will remove the need for an npmrc file when using private npm registries - correct? (since npm altogether is being removed) - Do these changes impact devtool/recipetool within the upstream poky repo at all? I assume so if the parameters to npm SRC_URI have changed. Did you have any plans to make those changes by any chance? If not, then I will try and pick up this change in my environment to test alongside. - Our workflow here eventually is to run “devtool add “npm://<private_npm_registry>;params”” and have the recipe and shrinkwrap generated automatically. - Do you plan on backporting these changes over to scarthgap? - What commit are you based off of within bitbake master for your patchset? Thanks, -Eric
From: "eric.meyers@arthrex.com" <eric.meyers@arthrex.com> Changes in this commit include: - npm fetcher changes: - Add “username=”, “password=”, and “namespace=” parameters to “npm://” SRC_URI - These parameters will be used to create a “npmrc” file that is passed into the NpmEnvironment object - used for authentication when connecting to NPM registry - Username and password parameters will be forwarded along to wget fetcher after resolving the tarball_url in the npm view output. - npmsw fetcher changes: - Add “username=”, “password=”, “basic_auth_registry=” parameters to “npmsw://” SRC_URI - For each registry specified in the shrinkwrap file, the fetcher will attempt to match the basic_auth_registry argument to the registry URL in the shrinkwrap, and if they match, then it will forward the username and password parameters along to the wget fetcher - fetch test changes: - skimdb.npmjs.com was shut down, so switched alternate_registry test to arbitrarily use registry.yarnpkg. Eric Meyers (3): fetch2/npm: Add basic auth credential support for npm fetcher fetch2/npmsw: Add basic auth credential support for npmsw fetcher tests/fetch: Switch to working npm registry for registry_alternate test lib/bb/fetch2/npm.py | 64 +++++++++++++++++++++++++++++++++++++++++- lib/bb/fetch2/npmsw.py | 36 ++++++++++++++++++++++++ lib/bb/tests/fetch.py | 2 +- 3 files changed, 100 insertions(+), 2 deletions(-)