| Message ID | 20241224-sstate-mirror-user-password-v1-1-d33e424617be@bootlin.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series | SSTATE_MIRRORS/SOURCE_MIRROR_URL: add instructions for mirror authentication | expand |
On Tue Dec 24, 2024 at 4:35 PM CET, Antonin Godard wrote: > [ YOCTO #15218 ] > > There are different ways of configuring the build host when > authentication to a mirror (SSTATE_MIRRORS or SOURCE_MIRROR_URL) is > required. > > Document these methods, and begin with the preferred approach which is > to edit ~/.netrc appropriately. When that is not possible, specifying > the credentials from the URL is also possible, so document it as well. > > Reported-by: peter.zsifkovits@at.bosch.com > Tested-by: Antonin Godard <antonin.godard@bootlin.com> # Debian 12 container > Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> > --- > Tested in a Debian 12 container like so: > > $ mkdir downloads > $ cp .../{bash-5.2.21.tar.gz,bash-5.2.21.tar.gz.done} downloads/ > $ pip install sauth # python http server with basic auth > $ sauth username password 127.0.0.1 > > In local.conf: > > INHERIT += "own-mirrors" > SOURCE_MIRROR_URL = "http://127.0.0.1:8333/downloads" > BB_ALLOWED_NETWORKS = "127.0.0.1" > > In bitbake's shell: > > $ bitbake bash -c cleanall > $ bitbake bash -c fetch > > As expected, error in log.do_fetch: > > Connecting to 127.0.0.1:8333... connected. > HTTP request sent, awaiting response... 401 Unauthorized > > Now, configure the following in ~/.netrc: > > machine 127.0.0.1 > login username > password password Forgot to mention: I also chowned this file to 600 and 400, and it works.
Hi Antonin, On 12/24/24 4:35 PM, Antonin Godard via lists.yoctoproject.org wrote: > [ YOCTO #15218 ] > > There are different ways of configuring the build host when > authentication to a mirror (SSTATE_MIRRORS or SOURCE_MIRROR_URL) is > required. > > Document these methods, and begin with the preferred approach which is > to edit ~/.netrc appropriately. When that is not possible, specifying > the credentials from the URL is also possible, so document it as well. > > Reported-by: peter.zsifkovits@at.bosch.com > Tested-by: Antonin Godard <antonin.godard@bootlin.com> # Debian 12 container > Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> > --- > Tested in a Debian 12 container like so: > > $ mkdir downloads > $ cp .../{bash-5.2.21.tar.gz,bash-5.2.21.tar.gz.done} downloads/ > $ pip install sauth # python http server with basic auth > $ sauth username password 127.0.0.1 > > In local.conf: > > INHERIT += "own-mirrors" > SOURCE_MIRROR_URL = "http://127.0.0.1:8333/downloads" > BB_ALLOWED_NETWORKS = "127.0.0.1" > > In bitbake's shell: > > $ bitbake bash -c cleanall > $ bitbake bash -c fetch > > As expected, error in log.do_fetch: > > Connecting to 127.0.0.1:8333... connected. > HTTP request sent, awaiting response... 401 Unauthorized > > Now, configure the following in ~/.netrc: > > machine 127.0.0.1 > login username > password password > > Try again: "bitbake bash -c fetch" runs successfully. > > Remove ~/.netrc, replace in local.conf: > > SOURCE_MIRROR_URL = "http://127.0.0.1:8333/downloads;user=username;pswd=password" > > In bitbake's shell: > > $ bitbake bash -c cleanall > $ bitbake bash -c fetch > > "bitbake bash -c fetch" runs successfully. > --- > documentation/ref-manual/variables.rst | 53 ++++++++++++++++++++++++++++++++++ > 1 file changed, 53 insertions(+) > > diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst > index e8db89f8c9642b1e36c574e1369093a7a2ce30a4..88eda960d0f06307d483ecd9c2f6d1c9cc6920f4 100644 > --- a/documentation/ref-manual/variables.rst > +++ b/documentation/ref-manual/variables.rst > @@ -8032,6 +8032,31 @@ system and gives an overview of their function and contents. > > You can specify only a single URL in :term:`SOURCE_MIRROR_URL`. > > + .. note:: > + > + If the mirror is protected behind a username and password, the > + :term:`build host` needs to be configured so the :term:`build system > + <OpenEmbedded Build System>` is able to fetch from the mirror. > + > + The recommended way to do that is by setting the following parameters > + in ``$HOME/.netrc`` (``$HOME`` being the :term:`build host` home > + directory):: > + > + machine example.com > + login <user> > + password <password> > + > + This file requires permissions set to ``400`` or ``600`` to prevent > + other users from reading the file:: > + > + chmod 600 "$HOME/.netrc" > + > + Another method to configure the username and password is from the URL > + in :term:`SOURCE_MIRROR_URL` directly, with the ``user`` and ``pswd`` > + parameters:: > + > + SOURCE_MIRROR_URL = "http://example.com/my_source_mirror;user=<user>;pswd=<password>" > + > :term:`SPDX_ARCHIVE_PACKAGED` > This option allows to add to :term:`SPDX` output compressed archives > of the files in the generated target packages. > @@ -8439,6 +8464,34 @@ system and gives an overview of their function and contents. > file://.* https://someserver.tld/share/sstate/PATH;downloadfilename=PATH \ > file://.* file:///some-local-dir/sstate/PATH" > > + .. note:: > + > + If the mirror is protected behind a username and password, the > + :term:`build host` needs to be configured so the :term:`build system > + <OpenEmbedded Build System>` is able to download the sstate cache using > + authentication. > + > + The recommended way to do that is by setting the following parameters > + in ``$HOME/.netrc`` (``$HOME`` being the :term:`build host` home > + directory):: > + > + machine someserver.tld > + login <user> > + password <password> > + > + This file requires permissions set to ``400`` or ``600`` to prevent > + other users from reading the file:: > + > + chmod 600 "$HOME/.netrc" > + > + Another method to configure the username and password is from the > + URL in :term:`SSTATE_MIRRORS` directly, with the ``user`` and ``pswd`` > + parameters:: > + > + SSTATE_MIRRORS ?= "\ > + file://.* https://someserver.tld/share/sstate/PATH;user=<user>;pswd=<password>;downloadfilename=PATH \ > + " > + Considering most of the added text is common between SSTATE_MIRRORS and SOURCE_MIRROR_URL maybe we should use a substitution? c.f. https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions Looks good to me otherwise, Cheers, Quentin
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index e8db89f8c9642b1e36c574e1369093a7a2ce30a4..88eda960d0f06307d483ecd9c2f6d1c9cc6920f4 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -8032,6 +8032,31 @@ system and gives an overview of their function and contents. You can specify only a single URL in :term:`SOURCE_MIRROR_URL`. + .. note:: + + If the mirror is protected behind a username and password, the + :term:`build host` needs to be configured so the :term:`build system + <OpenEmbedded Build System>` is able to fetch from the mirror. + + The recommended way to do that is by setting the following parameters + in ``$HOME/.netrc`` (``$HOME`` being the :term:`build host` home + directory):: + + machine example.com + login <user> + password <password> + + This file requires permissions set to ``400`` or ``600`` to prevent + other users from reading the file:: + + chmod 600 "$HOME/.netrc" + + Another method to configure the username and password is from the URL + in :term:`SOURCE_MIRROR_URL` directly, with the ``user`` and ``pswd`` + parameters:: + + SOURCE_MIRROR_URL = "http://example.com/my_source_mirror;user=<user>;pswd=<password>" + :term:`SPDX_ARCHIVE_PACKAGED` This option allows to add to :term:`SPDX` output compressed archives of the files in the generated target packages. @@ -8439,6 +8464,34 @@ system and gives an overview of their function and contents. file://.* https://someserver.tld/share/sstate/PATH;downloadfilename=PATH \ file://.* file:///some-local-dir/sstate/PATH" + .. note:: + + If the mirror is protected behind a username and password, the + :term:`build host` needs to be configured so the :term:`build system + <OpenEmbedded Build System>` is able to download the sstate cache using + authentication. + + The recommended way to do that is by setting the following parameters + in ``$HOME/.netrc`` (``$HOME`` being the :term:`build host` home + directory):: + + machine someserver.tld + login <user> + password <password> + + This file requires permissions set to ``400`` or ``600`` to prevent + other users from reading the file:: + + chmod 600 "$HOME/.netrc" + + Another method to configure the username and password is from the + URL in :term:`SSTATE_MIRRORS` directly, with the ``user`` and ``pswd`` + parameters:: + + SSTATE_MIRRORS ?= "\ + file://.* https://someserver.tld/share/sstate/PATH;user=<user>;pswd=<password>;downloadfilename=PATH \ + " + The Yocto Project actually shares the cache data objects built by its autobuilder::