| Message ID | 20241121-docs-build-dockerfiles-v1-1-3b54e1237bf5@bootlin.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [yocto-docs] Add scripts to build the docs in containers | expand |
Hi Antonin, On 11/21/24 2:22 PM, Antonin Godard via lists.yoctoproject.org wrote: > Add two scripts for building a container image and building the > documentation in that image: build-docs-container and > container-build-docs. > > For now, the documentation/tools/dockerfiles directory contains two > Dockerfiles for building with Ubuntu 24.04 and Debian 12, but we could > extend this to the supported distros. > > It should be possible to build the full documentation with two commands: > > ./documentation/tools/build-docs-container ubuntu-24-04 > ./documentation/tools/build-docs ubuntu-24-04 > > The first command builds the container image by pulling the dependencies > listed in poky.yaml.in. The second command uses this image to build the > docs. > > CONTAINERCMD can be replaced by "podman" to build with podman. > > Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> > --- > documentation/tools/build-docs | 48 ++++++++++++++++++++ > documentation/tools/build-docs-container | 51 ++++++++++++++++++++++ > .../tools/dockerfiles/Dockerfile.debian-12 | 17 ++++++++ > .../tools/dockerfiles/Dockerfile.ubuntu-24-04 | 17 ++++++++ > 4 files changed, 133 insertions(+) > > diff --git a/documentation/tools/build-docs b/documentation/tools/build-docs > new file mode 100755 > index 0000000000000000000000000000000000000000..2fe8aff3a9834e4f5015e04d64b7462190278004 > --- /dev/null > +++ b/documentation/tools/build-docs > @@ -0,0 +1,48 @@ > +#!/usr/bin/env bash > +# > +# Build the documentation in a container built by build-docs-container. > +# > +# Usage: > +# > +# ./documentation/tools/build-docs <image> [<command>] > +# > +# Where <image> is one of the keys in > +# documentation/tools/build-docs-container's DEPS_KEYS_YQ. > +# And <command> an optional command to run, default being to run "make publish". > +# > +# E.g.: > +# > +# ./documentation/tools/build-docs ubuntu-24-04 > + > +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd) > +CONTAINERCMD=${CONTAINERCMD:-docker} > + > +DOCS_DIR="$SCRIPT_DIR/../.." > + > +main () > +{ > + local image="$1" > + shift > + local command="${*:-make -C documentation publish}" > + Should we only require "publish" to be passed instead of make -C documentation publish? I believe this should be doable with ENTRYPOINT? > + if [ "$CONTAINERCMD" = "docker" ]; then > + EXTRA_ARGS_RUN="--user $(id -u):$(id -g)" > + elif [ "$CONTAINERCMD" = "podman" ]; then > + # we need net access to fetch bitbake terms > + EXTRA_ARGS_RUN="\ > + --cap-add=NET_RAW \ > + --userns=keep-id" please add --security-opt label=disable here as well, for SELinux based systems such as Fedora. My understanding is that for anything that is to be used by either the host or the container should disable labeling. > + fi > + > + $CONTAINERCMD run \ > + --rm --interactive --tty \ > + --volume "$DOCS_DIR:/docs:rw" \ > + --workdir "/docs" \ > + $EXTRA_ARGS_RUN \ > + "yocto-docs-$image" \ please prepend the image name with localhost/ to be sure we don't end up downloading some yocto-docs-ubuntu from docker.io for example :) > + $command > +} > + > +set -eu -o pipefail > + > +main "$@" > diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container > new file mode 100755 > index 0000000000000000000000000000000000000000..8cfe0f7aa75bf5fcfbbc0e838d66e0ef10f20696 > --- /dev/null > +++ b/documentation/tools/build-docs-container > @@ -0,0 +1,51 @@ > +#!/usr/bin/env bash > +# > +# Build a container ready to build the documentation be reading the dependencies > +# listed in poky.yaml.in. > +# > +# Usage: > +# > +# ./documentation/tools/build-docs-container <image> > +# > +# Where <image> is one of the keys in DEPS_KEYS_YQ. E.g.: > +# > +# ./documentation/tools/build-docs-container ubuntu-24-04 > + > +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd) > +CONTAINERCMD=${CONTAINERCMD:-docker} > + > +# Keys used by yq to get the dependencies from poky.yaml.in > +declare -A DEPS_KEYS_YQ=( > + [ubuntu-24-04]=".UBUNTU_DEBIAN_HOST_PACKAGES_DOC .UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF" > + [debian-12]=".UBUNTU_DEBIAN_HOST_PACKAGES_DOC .UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF" > +) > + > +main () > +{ > + local image="$1" > + > + for cmd in $CONTAINERCMD yq; do > + if ! which "$cmd" >/dev/null 2>&1; then > + echo "The $cmd command was not found. Make sure you have $cmd installed." > + exit 1 > + fi > + done > + > + local dockername="Dockerfile.$image" > + > + # Put all the dependencies in the deps variable > + for dep_key in ${DEPS_KEYS_YQ[$image]}; do > + deps="$deps $(yq --raw-output "$dep_key" "$SCRIPT_DIR/../poky.yaml.in")" > + done > + > + ( cd "$SCRIPT_DIR/dockerfiles" \ > + && $CONTAINERCMD build \ > + --tag "yocto-docs-$image:latest" \ > + --file "$dockername" \ > + --build-arg DEPS="$deps" \ I am not sure this is future-proof. We'd be limited to the max length of parameters one can use in a terminal. What about creating a temporary file and using that temporary file from within the Dockerfile via COPY. Also, you really want COPY and not MOUNT because changes in files in MOUNT won't invalidate the layer IIRC. > + . ) Can't we just run docker build [...] -f $dockername "$SCRIPT_DIR/dockerfiles" to avoid having to cd into it in a subshell? > +} > + > +set -e -o pipefail > + set -u while we are at it :) > +main "$@" Question: why do we split the image creation from the running? > diff --git a/documentation/tools/dockerfiles/Dockerfile.debian-12 b/documentation/tools/dockerfiles/Dockerfile.debian-12 > new file mode 100644 > index 0000000000000000000000000000000000000000..18c9c45c3ccfc5d90a09057d874512c6b9826611 > --- /dev/null > +++ b/documentation/tools/dockerfiles/Dockerfile.debian-12 > @@ -0,0 +1,17 @@ > +FROM debian:12 Would be nice to pass the version number from the shell script as well I guess? There's a trick for that one IIRC, I'd have to check how I've done that and get back to you if you're interested. > + > +ENV DEBIAN_FRONTEND=noninteractive > + > +ARG DEPS > +ENV DEPS=${DEPS} > + > +RUN apt-get update \ > + && apt-get --yes --fix-broken upgrade \ That should not be necessary? > + && apt-get --yes --no-install-recommends install \ > + ${DEPS} \ > + && apt-get --yes autoremove \ > + && apt-get clean > + > +RUN LANG="en_US.UTF-8" locale-gen > + > +RUN git config --global --add safe.directory /docs > diff --git a/documentation/tools/dockerfiles/Dockerfile.ubuntu-24-04 b/documentation/tools/dockerfiles/Dockerfile.ubuntu-24-04 > new file mode 100644 > index 0000000000000000000000000000000000000000..6ac384c4f242b508027a92d2acad490da41dc374 > --- /dev/null > +++ b/documentation/tools/dockerfiles/Dockerfile.ubuntu-24-04 > @@ -0,0 +1,17 @@ > +FROM ubuntu:24.04 > + > +ENV DEBIAN_FRONTEND=noninteractive > + > +ARG DEPS > +ENV DEPS=${DEPS} > + > +RUN apt-get update \ > + && apt-get --yes --fix-broken upgrade \ > + && apt-get --yes --no-install-recommends install \ > + ${DEPS} \ > + && apt-get --yes autoremove \ > + && apt-get clean > + > +RUN LANG="en_US.UTF-8" locale-gen > + I believe this is not working if I remember correctly the thread with Guénaël? Otherwise, the Dockerfile are identical, so maybe parameterizing the FROM line would be better than duplicating? Cheers, Quentin
Hi Quentin, On Mon Nov 25, 2024 at 7:26 PM CET, Quentin Schulz wrote: > Hi Antonin, > > On 11/21/24 2:22 PM, Antonin Godard via lists.yoctoproject.org wrote: >> Add two scripts for building a container image and building the >> documentation in that image: build-docs-container and >> container-build-docs. >> >> For now, the documentation/tools/dockerfiles directory contains two >> Dockerfiles for building with Ubuntu 24.04 and Debian 12, but we could >> extend this to the supported distros. >> >> It should be possible to build the full documentation with two commands: >> >> ./documentation/tools/build-docs-container ubuntu-24-04 >> ./documentation/tools/build-docs ubuntu-24-04 >> >> The first command builds the container image by pulling the dependencies >> listed in poky.yaml.in. The second command uses this image to build the >> docs. >> >> CONTAINERCMD can be replaced by "podman" to build with podman. >> >> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> >> --- >> documentation/tools/build-docs | 48 ++++++++++++++++++++ >> documentation/tools/build-docs-container | 51 ++++++++++++++++++++++ >> .../tools/dockerfiles/Dockerfile.debian-12 | 17 ++++++++ >> .../tools/dockerfiles/Dockerfile.ubuntu-24-04 | 17 ++++++++ >> 4 files changed, 133 insertions(+) >> >> diff --git a/documentation/tools/build-docs b/documentation/tools/build-docs >> new file mode 100755 >> index 0000000000000000000000000000000000000000..2fe8aff3a9834e4f5015e04d64b7462190278004 >> --- /dev/null >> +++ b/documentation/tools/build-docs >> @@ -0,0 +1,48 @@ >> +#!/usr/bin/env bash >> +# >> +# Build the documentation in a container built by build-docs-container. >> +# >> +# Usage: >> +# >> +# ./documentation/tools/build-docs <image> [<command>] >> +# >> +# Where <image> is one of the keys in >> +# documentation/tools/build-docs-container's DEPS_KEYS_YQ. >> +# And <command> an optional command to run, default being to run "make publish". >> +# >> +# E.g.: >> +# >> +# ./documentation/tools/build-docs ubuntu-24-04 >> + >> +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd) >> +CONTAINERCMD=${CONTAINERCMD:-docker} >> + >> +DOCS_DIR="$SCRIPT_DIR/../.." >> + >> +main () >> +{ >> + local image="$1" >> + shift >> + local command="${*:-make -C documentation publish}" >> + > > Should we only require "publish" to be passed instead of make -C > documentation publish? I believe this should be doable with ENTRYPOINT? I agree. I don't see any other command we'd want to run instead of "make <something>". About the entrypoint, what would be the added value of having one? >> + if [ "$CONTAINERCMD" = "docker" ]; then >> + EXTRA_ARGS_RUN="--user $(id -u):$(id -g)" >> + elif [ "$CONTAINERCMD" = "podman" ]; then >> + # we need net access to fetch bitbake terms >> + EXTRA_ARGS_RUN="\ >> + --cap-add=NET_RAW \ >> + --userns=keep-id" > > please add > > --security-opt label=disable > > here as well, for SELinux based systems such as Fedora. My understanding > is that for anything that is to be used by either the host or the > container should disable labeling. Thanks for the tip. Will add (assuming this is not an issue for other distros). >> + fi >> + >> + $CONTAINERCMD run \ >> + --rm --interactive --tty \ >> + --volume "$DOCS_DIR:/docs:rw" \ >> + --workdir "/docs" \ >> + $EXTRA_ARGS_RUN \ >> + "yocto-docs-$image" \ > > please prepend the image name with localhost/ to be sure we don't end up > downloading some yocto-docs-ubuntu from docker.io for example :) Nice tip, didn't know about that one, thanks :) >> + $command >> +} >> + >> +set -eu -o pipefail >> + >> +main "$@" >> diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container >> new file mode 100755 >> index 0000000000000000000000000000000000000000..8cfe0f7aa75bf5fcfbbc0e838d66e0ef10f20696 >> --- /dev/null >> +++ b/documentation/tools/build-docs-container >> @@ -0,0 +1,51 @@ >> +#!/usr/bin/env bash >> +# >> +# Build a container ready to build the documentation be reading the dependencies >> +# listed in poky.yaml.in. >> +# >> +# Usage: >> +# >> +# ./documentation/tools/build-docs-container <image> >> +# >> +# Where <image> is one of the keys in DEPS_KEYS_YQ. E.g.: >> +# >> +# ./documentation/tools/build-docs-container ubuntu-24-04 >> + >> +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd) >> +CONTAINERCMD=${CONTAINERCMD:-docker} >> + >> +# Keys used by yq to get the dependencies from poky.yaml.in >> +declare -A DEPS_KEYS_YQ=( >> + [ubuntu-24-04]=".UBUNTU_DEBIAN_HOST_PACKAGES_DOC .UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF" >> + [debian-12]=".UBUNTU_DEBIAN_HOST_PACKAGES_DOC .UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF" >> +) >> + >> +main () >> +{ >> + local image="$1" >> + >> + for cmd in $CONTAINERCMD yq; do >> + if ! which "$cmd" >/dev/null 2>&1; then >> + echo "The $cmd command was not found. Make sure you have $cmd installed." >> + exit 1 >> + fi >> + done >> + >> + local dockername="Dockerfile.$image" >> + >> + # Put all the dependencies in the deps variable >> + for dep_key in ${DEPS_KEYS_YQ[$image]}; do >> + deps="$deps $(yq --raw-output "$dep_key" "$SCRIPT_DIR/../poky.yaml.in")" >> + done >> + >> + ( cd "$SCRIPT_DIR/dockerfiles" \ >> + && $CONTAINERCMD build \ >> + --tag "yocto-docs-$image:latest" \ >> + --file "$dockername" \ >> + --build-arg DEPS="$deps" \ > > I am not sure this is future-proof. We'd be limited to the max length of > parameters one can use in a terminal. What about creating a temporary > file and using that temporary file from within the Dockerfile via COPY. > Also, you really want COPY and not MOUNT because changes in files in > MOUNT won't invalidate the layer IIRC. Yes, I agree this would be better, I'll work on adding that instead. >> + . ) > > Can't we just run > > docker build [...] -f $dockername "$SCRIPT_DIR/dockerfiles" > > to avoid having to cd into it in a subshell? I don't remember why but I had trouble with this. For some reason I had to be in the same directory as the Dockerfile... I'll give it another shot. >> +} >> + >> +set -e -o pipefail >> + > > set -u > > while we are at it :) Forgot this one, thanks! >> +main "$@" > > Question: why do we split the image creation from the running? Good question. Initially it made sense to me to split both to avoid running "docker build" each time, but knowing it would get cached I hesitated. If checking for the cache is a low-cost operation (seems to be on my computer), I also vote for having a single script that runs docker build each time. That way we don't forget to rebuild when adding things to poky.yaml.in. >> diff --git a/documentation/tools/dockerfiles/Dockerfile.debian-12 b/documentation/tools/dockerfiles/Dockerfile.debian-12 >> new file mode 100644 >> index 0000000000000000000000000000000000000000..18c9c45c3ccfc5d90a09057d874512c6b9826611 >> --- /dev/null >> +++ b/documentation/tools/dockerfiles/Dockerfile.debian-12 >> @@ -0,0 +1,17 @@ >> +FROM debian:12 > > Would be nice to pass the version number from the shell script as well I > guess? > > There's a trick for that one IIRC, I'd have to check how I've done that > and get back to you if you're interested. Yes, that'd be an improvement. >> + >> +ENV DEBIAN_FRONTEND=noninteractive >> + >> +ARG DEPS >> +ENV DEPS=${DEPS} >> + >> +RUN apt-get update \ >> + && apt-get --yes --fix-broken upgrade \ > > That should not be necessary? Probably not. I picked that up from an older Dockerfile, and I don't remember the reason why I put that here. Will remove. >> + && apt-get --yes --no-install-recommends install \ >> + ${DEPS} \ >> + && apt-get --yes autoremove \ >> + && apt-get clean >> + >> +RUN LANG="en_US.UTF-8" locale-gen >> + >> +RUN git config --global --add safe.directory /docs >> diff --git a/documentation/tools/dockerfiles/Dockerfile.ubuntu-24-04 b/documentation/tools/dockerfiles/Dockerfile.ubuntu-24-04 >> new file mode 100644 >> index 0000000000000000000000000000000000000000..6ac384c4f242b508027a92d2acad490da41dc374 >> --- /dev/null >> +++ b/documentation/tools/dockerfiles/Dockerfile.ubuntu-24-04 >> @@ -0,0 +1,17 @@ >> +FROM ubuntu:24.04 >> + >> +ENV DEBIAN_FRONTEND=noninteractive >> + >> +ARG DEPS >> +ENV DEPS=${DEPS} >> + >> +RUN apt-get update \ >> + && apt-get --yes --fix-broken upgrade \ >> + && apt-get --yes --no-install-recommends install \ >> + ${DEPS} \ >> + && apt-get --yes autoremove \ >> + && apt-get clean >> + >> +RUN LANG="en_US.UTF-8" locale-gen >> + > > I believe this is not working if I remember correctly the thread with > Guénaël? You're right, actually this does nothing, and modifying /etc/locale.gen seems to be the proper way (for debian/ubuntu compatibility). > Otherwise, the Dockerfile are identical, so maybe parameterizing the > FROM line would be better than duplicating? Agreed, goes with the improvement/passing the version trick you mentioned above. I'll give a thought and see how I can do that. Thank you, Antonin
Hi Quentin, On Fri Nov 29, 2024 at 4:01 PM CET, Antonin Godard wrote: [...] >>> + fi >>> + >>> + $CONTAINERCMD run \ >>> + --rm --interactive --tty \ >>> + --volume "$DOCS_DIR:/docs:rw" \ >>> + --workdir "/docs" \ >>> + $EXTRA_ARGS_RUN \ >>> + "yocto-docs-$image" \ >> >> please prepend the image name with localhost/ to be sure we don't end up >> downloading some yocto-docs-ubuntu from docker.io for example :) > > Nice tip, didn't know about that one, thanks :) Hm, this doesn't seem to work for me, and I couldn't find anything on the net that shows this. Is there something I'm missing? Unable to find image 'localhost/yocto-docs-ubuntu-24-04:latest' locally docker: Error response from daemon: Get "http://localhost/v2/": dial tcp [::1]:80: connect: connection refused. Building an image named "localhost/..." will work, but I think it's just building an image with the name "localhost/...". Not really building something local. Antonin
Hi Antonin, On 12/3/24 11:48 AM, Antonin Godard wrote: > Hi Quentin, > > On Fri Nov 29, 2024 at 4:01 PM CET, Antonin Godard wrote: > [...] >>>> + fi >>>> + >>>> + $CONTAINERCMD run \ >>>> + --rm --interactive --tty \ >>>> + --volume "$DOCS_DIR:/docs:rw" \ >>>> + --workdir "/docs" \ >>>> + $EXTRA_ARGS_RUN \ >>>> + "yocto-docs-$image" \ >>> >>> please prepend the image name with localhost/ to be sure we don't end up >>> downloading some yocto-docs-ubuntu from docker.io for example :) >> >> Nice tip, didn't know about that one, thanks :) > > Hm, this doesn't seem to work for me, and I couldn't find anything on the net > that shows this. Is there something I'm missing? > > Unable to find image 'localhost/yocto-docs-ubuntu-24-04:latest' locally > docker: Error response from daemon: Get "http://localhost/v2/": dial tcp [::1]:80: connect: connection refused. > > > Building an image named "localhost/..." will work, but I think it's just > building an image with the name "localhost/...". Not really building something > local. > $ cat Containerfile FROM debian:bookworm-slim $ podman build -f Containerfile -t test-image:latest [...] $ podman image ls | grep test-image:latest localhost/test-image latest b2ac1dc43440 14 months ago 156 MB $ podman run -it --userns=keep-id localhost/test-image qschulz@e706c5542422:/$ $ cat Containerfile.from FROM localhost/test-image $ podman build -f Containerfile.from -t another-image:latest [...] $ podman run -it --userns=keep-id localhost/another-image qschulz@4e1f3229a196:/$ On Fedora 41. Could possibly be yet another difference with Docker? I couldn't find a way to force fully qualified names from the CLI (one can do that with podman on fedora by removing unqualified-search-registries in /etc/containers/registries.conf I believe). I guess maybe --block-registry '*' would work, but still not sure this is something supported by Docker either, /me shrugs. Cheers, Quentin
Hi Quentin, On Tue Dec 3, 2024 at 12:37 PM CET, Quentin Schulz wrote: > Hi Antonin, > > On 12/3/24 11:48 AM, Antonin Godard wrote: >> Hi Quentin, >> >> On Fri Nov 29, 2024 at 4:01 PM CET, Antonin Godard wrote: >> [...] >>>>> + fi >>>>> + >>>>> + $CONTAINERCMD run \ >>>>> + --rm --interactive --tty \ >>>>> + --volume "$DOCS_DIR:/docs:rw" \ >>>>> + --workdir "/docs" \ >>>>> + $EXTRA_ARGS_RUN \ >>>>> + "yocto-docs-$image" \ >>>> >>>> please prepend the image name with localhost/ to be sure we don't end up >>>> downloading some yocto-docs-ubuntu from docker.io for example :) >>> >>> Nice tip, didn't know about that one, thanks :) >> >> Hm, this doesn't seem to work for me, and I couldn't find anything on the net >> that shows this. Is there something I'm missing? >> >> Unable to find image 'localhost/yocto-docs-ubuntu-24-04:latest' locally >> docker: Error response from daemon: Get "http://localhost/v2/": dial tcp [::1]:80: connect: connection refused. >> >> >> Building an image named "localhost/..." will work, but I think it's just >> building an image with the name "localhost/...". Not really building something >> local. >> > > $ cat Containerfile > FROM debian:bookworm-slim > $ podman build -f Containerfile -t test-image:latest > [...] > $ podman image ls | grep test-image:latest > localhost/test-image > latest b2ac1dc43440 14 months ago 156 MB > $ podman run -it --userns=keep-id localhost/test-image > qschulz@e706c5542422:/$ > $ cat Containerfile.from > FROM localhost/test-image > $ podman build -f Containerfile.from -t another-image:latest > [...] > $ podman run -it --userns=keep-id localhost/another-image > qschulz@4e1f3229a196:/$ > > On Fedora 41. > > Could possibly be yet another difference with Docker? I think that's it. Supported by podman but not docker, as I had no issue doing this with podman in the end. > I couldn't find a way to force fully qualified names from the CLI (one > can do that with podman on fedora by removing > unqualified-search-registries in /etc/containers/registries.conf I > believe). I guess maybe --block-registry '*' would work, but still not > sure this is something supported by Docker either, /me shrugs. For simplicity and compatibility, I guess we can leave it as that (no localhost/). In the second iteration of my patch I will merge the two scripts into one (also for simplicity), so there should be no way to reach the run command without actually building the container prior. Thanks, Antonin
Hi Antonin, On 12/3/24 1:24 PM, Antonin Godard wrote: > Hi Quentin, > > On Tue Dec 3, 2024 at 12:37 PM CET, Quentin Schulz wrote: >> Hi Antonin, >> >> On 12/3/24 11:48 AM, Antonin Godard wrote: >>> Hi Quentin, >>> >>> On Fri Nov 29, 2024 at 4:01 PM CET, Antonin Godard wrote: >>> [...] >>>>>> + fi >>>>>> + >>>>>> + $CONTAINERCMD run \ >>>>>> + --rm --interactive --tty \ >>>>>> + --volume "$DOCS_DIR:/docs:rw" \ >>>>>> + --workdir "/docs" \ >>>>>> + $EXTRA_ARGS_RUN \ >>>>>> + "yocto-docs-$image" \ >>>>> >>>>> please prepend the image name with localhost/ to be sure we don't end up >>>>> downloading some yocto-docs-ubuntu from docker.io for example :) >>>> >>>> Nice tip, didn't know about that one, thanks :) >>> >>> Hm, this doesn't seem to work for me, and I couldn't find anything on the net >>> that shows this. Is there something I'm missing? >>> >>> Unable to find image 'localhost/yocto-docs-ubuntu-24-04:latest' locally >>> docker: Error response from daemon: Get "http://localhost/v2/": dial tcp [::1]:80: connect: connection refused. >>> >>> >>> Building an image named "localhost/..." will work, but I think it's just >>> building an image with the name "localhost/...". Not really building something >>> local. >>> >> >> $ cat Containerfile >> FROM debian:bookworm-slim >> $ podman build -f Containerfile -t test-image:latest >> [...] >> $ podman image ls | grep test-image:latest >> localhost/test-image >> latest b2ac1dc43440 14 months ago 156 MB >> $ podman run -it --userns=keep-id localhost/test-image >> qschulz@e706c5542422:/$ >> $ cat Containerfile.from >> FROM localhost/test-image >> $ podman build -f Containerfile.from -t another-image:latest >> [...] >> $ podman run -it --userns=keep-id localhost/another-image >> qschulz@4e1f3229a196:/$ >> >> On Fedora 41. >> >> Could possibly be yet another difference with Docker? > > I think that's it. Supported by podman but not docker, as I had no issue doing > this with podman in the end. > >> I couldn't find a way to force fully qualified names from the CLI (one >> can do that with podman on fedora by removing >> unqualified-search-registries in /etc/containers/registries.conf I >> believe). I guess maybe --block-registry '*' would work, but still not >> sure this is something supported by Docker either, /me shrugs. > > For simplicity and compatibility, I guess we can leave it as that (no > localhost/). In the second iteration of my patch I will merge the two scripts > into one (also for simplicity), so there should be no way to reach the run > command without actually building the container prior. > I think it could still be possible if one pulls an image from a registry which has the short name identical to the local image without localhost/ prefix. I tried with: 1. podman build -f Containerfile -t sean0x42/markdown-extract 2. podman run -it --userns=keep-id --user root sean0x42/markdown-extract 3. podman run -it --userns=keep-id --user root docker.io/sean0x42/markdown-extract 4. podman run -it --userns=keep-id --user root sean0x42/markdown-extract 5. podman build -f Containerfile -t sean0x42/markdown-extract 6. podman run -it --userns=keep-id --user root sean0x42/markdown-extract 7. podman run -it --userns=keep-id --user root localhost/sean0x42/markdown-extract 2. would use my local image 3. would use docker.io's 4. would use docker,io's 6. would use docker.io's 7. would use my local image Not sure we should care that much about it though. Cheers, Quentin
diff --git a/documentation/tools/build-docs b/documentation/tools/build-docs new file mode 100755 index 0000000000000000000000000000000000000000..2fe8aff3a9834e4f5015e04d64b7462190278004 --- /dev/null +++ b/documentation/tools/build-docs @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# +# Build the documentation in a container built by build-docs-container. +# +# Usage: +# +# ./documentation/tools/build-docs <image> [<command>] +# +# Where <image> is one of the keys in +# documentation/tools/build-docs-container's DEPS_KEYS_YQ. +# And <command> an optional command to run, default being to run "make publish". +# +# E.g.: +# +# ./documentation/tools/build-docs ubuntu-24-04 + +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd) +CONTAINERCMD=${CONTAINERCMD:-docker} + +DOCS_DIR="$SCRIPT_DIR/../.." + +main () +{ + local image="$1" + shift + local command="${*:-make -C documentation publish}" + + if [ "$CONTAINERCMD" = "docker" ]; then + EXTRA_ARGS_RUN="--user $(id -u):$(id -g)" + elif [ "$CONTAINERCMD" = "podman" ]; then + # we need net access to fetch bitbake terms + EXTRA_ARGS_RUN="\ + --cap-add=NET_RAW \ + --userns=keep-id" + fi + + $CONTAINERCMD run \ + --rm --interactive --tty \ + --volume "$DOCS_DIR:/docs:rw" \ + --workdir "/docs" \ + $EXTRA_ARGS_RUN \ + "yocto-docs-$image" \ + $command +} + +set -eu -o pipefail + +main "$@" diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container new file mode 100755 index 0000000000000000000000000000000000000000..8cfe0f7aa75bf5fcfbbc0e838d66e0ef10f20696 --- /dev/null +++ b/documentation/tools/build-docs-container @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# +# Build a container ready to build the documentation be reading the dependencies +# listed in poky.yaml.in. +# +# Usage: +# +# ./documentation/tools/build-docs-container <image> +# +# Where <image> is one of the keys in DEPS_KEYS_YQ. E.g.: +# +# ./documentation/tools/build-docs-container ubuntu-24-04 + +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd) +CONTAINERCMD=${CONTAINERCMD:-docker} + +# Keys used by yq to get the dependencies from poky.yaml.in +declare -A DEPS_KEYS_YQ=( + [ubuntu-24-04]=".UBUNTU_DEBIAN_HOST_PACKAGES_DOC .UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF" + [debian-12]=".UBUNTU_DEBIAN_HOST_PACKAGES_DOC .UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF" +) + +main () +{ + local image="$1" + + for cmd in $CONTAINERCMD yq; do + if ! which "$cmd" >/dev/null 2>&1; then + echo "The $cmd command was not found. Make sure you have $cmd installed." + exit 1 + fi + done + + local dockername="Dockerfile.$image" + + # Put all the dependencies in the deps variable + for dep_key in ${DEPS_KEYS_YQ[$image]}; do + deps="$deps $(yq --raw-output "$dep_key" "$SCRIPT_DIR/../poky.yaml.in")" + done + + ( cd "$SCRIPT_DIR/dockerfiles" \ + && $CONTAINERCMD build \ + --tag "yocto-docs-$image:latest" \ + --file "$dockername" \ + --build-arg DEPS="$deps" \ + . ) +} + +set -e -o pipefail + +main "$@" diff --git a/documentation/tools/dockerfiles/Dockerfile.debian-12 b/documentation/tools/dockerfiles/Dockerfile.debian-12 new file mode 100644 index 0000000000000000000000000000000000000000..18c9c45c3ccfc5d90a09057d874512c6b9826611 --- /dev/null +++ b/documentation/tools/dockerfiles/Dockerfile.debian-12 @@ -0,0 +1,17 @@ +FROM debian:12 + +ENV DEBIAN_FRONTEND=noninteractive + +ARG DEPS +ENV DEPS=${DEPS} + +RUN apt-get update \ + && apt-get --yes --fix-broken upgrade \ + && apt-get --yes --no-install-recommends install \ + ${DEPS} \ + && apt-get --yes autoremove \ + && apt-get clean + +RUN LANG="en_US.UTF-8" locale-gen + +RUN git config --global --add safe.directory /docs diff --git a/documentation/tools/dockerfiles/Dockerfile.ubuntu-24-04 b/documentation/tools/dockerfiles/Dockerfile.ubuntu-24-04 new file mode 100644 index 0000000000000000000000000000000000000000..6ac384c4f242b508027a92d2acad490da41dc374 --- /dev/null +++ b/documentation/tools/dockerfiles/Dockerfile.ubuntu-24-04 @@ -0,0 +1,17 @@ +FROM ubuntu:24.04 + +ENV DEBIAN_FRONTEND=noninteractive + +ARG DEPS +ENV DEPS=${DEPS} + +RUN apt-get update \ + && apt-get --yes --fix-broken upgrade \ + && apt-get --yes --no-install-recommends install \ + ${DEPS} \ + && apt-get --yes autoremove \ + && apt-get clean + +RUN LANG="en_US.UTF-8" locale-gen + +RUN git config --global --add safe.directory /docs
Add two scripts for building a container image and building the documentation in that image: build-docs-container and container-build-docs. For now, the documentation/tools/dockerfiles directory contains two Dockerfiles for building with Ubuntu 24.04 and Debian 12, but we could extend this to the supported distros. It should be possible to build the full documentation with two commands: ./documentation/tools/build-docs-container ubuntu-24-04 ./documentation/tools/build-docs ubuntu-24-04 The first command builds the container image by pulling the dependencies listed in poky.yaml.in. The second command uses this image to build the docs. CONTAINERCMD can be replaced by "podman" to build with podman. Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> --- documentation/tools/build-docs | 48 ++++++++++++++++++++ documentation/tools/build-docs-container | 51 ++++++++++++++++++++++ .../tools/dockerfiles/Dockerfile.debian-12 | 17 ++++++++ .../tools/dockerfiles/Dockerfile.ubuntu-24-04 | 17 ++++++++ 4 files changed, 133 insertions(+) --- base-commit: 4d833d0a5f3ee741bc7e603c6316786903df335e change-id: 20241120-docs-build-dockerfiles-07fc9f72cab8 prerequisite-change-id: 20241120-update-doc-deps-1d59abdb2119:v1 prerequisite-patch-id: 5377acf74f0873f407d74e05bb0865cc23a542da prerequisite-patch-id: a0858899556e86046ffad661eeb0fe9a89990e4d prerequisite-patch-id: ff7cb872932bdde94b7e1f23bbcaa2d5e9072762 prerequisite-patch-id: f5ae26c37c785e16bd052e6b4260da35b661d25e prerequisite-patch-id: 99cc275f70735f5c6ccbdc74998fe170b3cbf167 Best regards,