| Message ID | 20241217-instructions-shell-container-v2-2-ce687b0cfac2@cherry.de |
|---|---|
| State | Superseded |
| Headers | show |
| Series | add script for containers building the documentation | expand |
Hi Quentin, On Tue Dec 17, 2024 at 5:08 PM CET, Quentin Schulz wrote: > From: Quentin Schulz <quentin.schulz@cherry.de> > > This adds a script for building a container and building the > documentation within that new container image. > > The openSUSE instructions now require a --non-interactive flag otherwise > they fail to run. Sadly there doesn't seem to be a way to have this in > an environment variable à-la DEBIAN_FRONTEND=noninteractive. Hmm, this means that we end up instructing to pass "--non-interactive" in the docs? That's not ideal :/ I suggested a workaround below. I hope there won't be too many cases where we have to do things differently in the context of a containerfile in the future. Apart from this case I cannot think of anything else that could happen. Anything comes to mind for you? > Somehow tzdata package in Ubuntu doesn't respect > DEBIAN_FRONTEND=noninteractive hence why the timezone needs to be set by > hand. > > Note that only instructions for Debian Bookworm (12), Ubuntu 22.04 and > 24.04, and Fedora 38, 39 and 40, currently work. > > Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de> > --- > documentation/tools/Containerfile.almalinux | 1 + > documentation/tools/Containerfile.apt | 26 +++++ > documentation/tools/Containerfile.debian | 1 + > documentation/tools/Containerfile.dnf | 25 +++++ > documentation/tools/Containerfile.fedora | 1 + > documentation/tools/Containerfile.ubuntu | 1 + > documentation/tools/Containerfile.zypper | 24 +++++ > documentation/tools/build-docs-container | 113 +++++++++++++++++++++ > .../tools/host_packages_scripts/opensuse_docs.sh | 2 +- > .../host_packages_scripts/opensuse_docs_pdf.sh | 2 +- > .../host_packages_scripts/opensuse_essential.sh | 2 +- > 11 files changed, 195 insertions(+), 3 deletions(-) > > diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux > new file mode 120000 > index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 > --- /dev/null > +++ b/documentation/tools/Containerfile.almalinux > @@ -0,0 +1 @@ > +Containerfile.dnf > \ No newline at end of file > diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt > new file mode 100644 > index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517 > --- /dev/null > +++ b/documentation/tools/Containerfile.apt > @@ -0,0 +1,26 @@ > +ARG ARG_FROM=debian:12 # default value to avoid warning > +FROM $ARG_FROM > + > +ARG DOCS=ubuntu_docs.sh > +ARG DOCS_PDF=ubuntu_docs_pdf.sh > + > +ENV DEBIAN_FRONTEND=noninteractive > +ARG TZ=Europe/Vienna > + > +# relative to the location of the dockerfile > +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh > +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh > + > +RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \ > + && apt-get update \ > + && apt-get install -y sudo \ > + && yes | /temp/host_packages_docs.sh \ > + && yes | /temp/host_packages_docs_pdf.sh \ > + && apt-get --yes autoremove \ > + && apt-get clean \ > + && rm -rf /temp > + > +RUN git config --global --add safe.directory /docs > + > +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] > +CMD ["publish"] > diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian > new file mode 120000 > index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a > --- /dev/null > +++ b/documentation/tools/Containerfile.debian > @@ -0,0 +1 @@ > +Containerfile.apt > \ No newline at end of file > diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf > new file mode 100644 > index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886 > --- /dev/null > +++ b/documentation/tools/Containerfile.dnf > @@ -0,0 +1,25 @@ > +ARG ARG_FROM=fedora:40 # default value to avoid warning > +FROM $ARG_FROM > + > +ARG DOCS=fedora_docs.sh > +ARG DOCS_PDF=fedora_docs_pdf.sh > +ARG PIP3=pip3_docs.sh > + > +# relative to the location of the dockerfile > +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh > +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh > +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh > + > +RUN dnf update -y \ > + && dnf install -y sudo \ > + && yes | /temp/host_packages_docs.sh \ > + && yes | /temp/host_packages_docs_pdf.sh \ > + && yes | /temp/pip3_docs.sh \ > + && dnf autoremove -y \ > + && dnf clean all -y \ > + && rm -rf /temp > + > +RUN git config --global --add safe.directory /docs > + > +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] > +CMD ["publish"] > diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora > new file mode 120000 > index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 > --- /dev/null > +++ b/documentation/tools/Containerfile.fedora > @@ -0,0 +1 @@ > +Containerfile.dnf > \ No newline at end of file > diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu > new file mode 120000 > index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a > --- /dev/null > +++ b/documentation/tools/Containerfile.ubuntu > @@ -0,0 +1 @@ > +Containerfile.apt > \ No newline at end of file > diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper > new file mode 100644 > index 0000000000000000000000000000000000000000..bcda5261ae98e8df16a3bcef17a7bf204033da99 > --- /dev/null > +++ b/documentation/tools/Containerfile.zypper > @@ -0,0 +1,24 @@ > +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning > +FROM $ARG_FROM > + > +ARG DOCS=opensuse_docs.sh > +ARG DOCS_PDF=opensuse_docs_pdf.sh > +ARG PIP3=pip3_docs.sh > + > +# relative to the location of the dockerfile > +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh > +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh > +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh A bit of a hacky workaround, but maybe here run sed -i 's/zypper/zypper --non-interactive/ ...' on the scripts? So we don't need to provide this option in the doc. > + > +RUN zypper update -y \ > + && zypper install -y sudo \ > + && yes | /temp/host_packages_docs.sh \ > + && yes | /temp/host_packages_docs_pdf.sh \ > + && yes | /temp/pip3_docs.sh \ > + && zypper clean --all \ > + && rm -rf /temp > + > +RUN git config --global --add safe.directory /docs > + > +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] > +CMD ["publish"] > diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container > new file mode 100755 > index 0000000000000000000000000000000000000000..f382af75a7e89cfcc8a6080fdbc65fb68ff9ea8d > --- /dev/null > +++ b/documentation/tools/build-docs-container > @@ -0,0 +1,113 @@ > +#!/usr/bin/env bash > +# > +# Build a container ready to build the documentation be reading the dependencies > +# listed in shell scripts in documentation/tools/host_packages_scripts, and > +# start a documentation build in this container. > +# > +# Usage: > +# > +# ./documentation/tools/build-docs-container <image> [<make target>] > +# > +# e.g.: > +# > +# ./documentation/tools/build-docs-container ubuntu:24.04 html > +# > +# Will build the docs in an Ubuntu 24.04 container in html. > +# > +# The container engine can be selected by exporting CONTAINERCMD in the > +# environment. The default is docker, but podman can also be used. > + > +set -eu -o pipefail > + > +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) > +CONTAINERCMD=${CONTAINERCMD:-docker} > +DOCS_DIR="$SCRIPT_DIR/../.." > +SH_DIR="$SCRIPT_DIR/host_packages_scripts" > + > +main () > +{ I would be good to have a check here to avoid $ ./tools/build-docs-container ./tools/build-docs-container: line 29: $1: unbound variable You could probably copypaste what I did in my version of the script. > + local image="$1" > + shift > + > + OCI=$(which "$CONTAINERCMD") > + > + # docker build doesn't accept 2 colons, so "sanitize" the name > + local sanitized_dockername > + sanitized_dockername=$(echo "$image" | tr ':.' '-') > + > + local version > + version=$(echo "$image" | awk -F: '{print $NF}') > + > + case $image in > + almalinux*) > + containerfile=Containerfile.almalinux > + docs=almalinux_docs.sh > + docs_pdf=almalinux_docs_pdf.sh > + pip3=pip3_docs.sh > + ;; > + debian*) > + containerfile=Containerfile.debian > + docs=ubuntu_docs.sh > + docs_pdf=ubuntu_docs_pdf.sh > + ;; > + fedora*) > + containerfile=Containerfile.fedora > + docs=fedora_docs.sh > + docs_pdf=fedora_docs_pdf.sh > + pip3=pip3_docs.sh > + ;; > + opensuse* | leap*) > + image=opensuse/leap:$version > + containerfile=Containerfile.zypper > + docs=opensuse_docs.sh > + docs_pdf=opensuse_docs_pdf.sh > + pip3=pip3_docs.sh > + ;; > + ubuntu*) > + containerfile=Containerfile.ubuntu > + docs=ubuntu_docs.sh > + docs_pdf=ubuntu_docs_pdf.sh > + ;; > + *) > + echo "$image not supported" Can we have a way of listing images here so a user doesn't need to read the script and guess which one to use? > + exit 1 > + ;; > + esac > + > + $OCI build \ > + --tag "yocto-docs-$sanitized_dockername:latest" \ > + --build-arg ARG_FROM="docker.io/$image" \ > + --build-arg DOCS="$docs" \ > + --build-arg DOCS_PDF="$docs_pdf" \ > + --build-arg PIP3="${pip3:-}" \ > + --file "$SCRIPT_DIR/$containerfile" \ > + "$SH_DIR/" > + > + local -a args_run=( > + --rm > + --interactive > + --tty > + --volume="$DOCS_DIR:/docs:rw" > + --workdir=/docs > + --security-opt label=disable > + ) > + > + if [ "$OCI" = "docker" ]; then > + args_run+=( > + --user="$(id -u)":"$(id -g)" > + ) > + elif [ "$OCI" = "podman" ]; then > + # we need net access to fetch bitbake terms > + args_run+=( > + --cap-add=NET_RAW > + --userns=keep-id > + ) > + fi > + > + $OCI run \ > + "${args_run[@]}" \ > + "yocto-docs-$sanitized_dockername" \ > + "$@" > +} > + > +main "$@" > diff --git a/documentation/tools/host_packages_scripts/opensuse_docs.sh b/documentation/tools/host_packages_scripts/opensuse_docs.sh > index 7d36eb0f99668168dc4fed44d71181338bdc59a3..896ad52b5042a9679555a30e582fa32076c3a3e7 100644 > --- a/documentation/tools/host_packages_scripts/opensuse_docs.sh > +++ b/documentation/tools/host_packages_scripts/opensuse_docs.sh > @@ -1 +1 @@ > -sudo zypper install git glibc-i18ndata make python3-pip rsvg-convert which > +sudo zypper --non-interactive install git glibc-i18ndata make python3-pip rsvg-convert which > diff --git a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh > index ee9f07886ce1c0eac8e1d1118cda8ddc496f0139..f8a3b800d9790c0434f0c3f577ea1916a7fe1b28 100644 > --- a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh > +++ b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh > @@ -1 +1 @@ > -sudo zypper install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex > +sudo zypper --non-interactive install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex > diff --git a/documentation/tools/host_packages_scripts/opensuse_essential.sh b/documentation/tools/host_packages_scripts/opensuse_essential.sh > index a784f4a5dc05f3734177f08ed97663a04d20a978..19e85f4a736532b890fc55492721cf345c2b4083 100644 > --- a/documentation/tools/host_packages_scripts/opensuse_essential.sh > +++ b/documentation/tools/host_packages_scripts/opensuse_essential.sh > @@ -1,2 +1,2 @@ > -sudo zypper install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd > +sudo zypper --non-interactive install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd > sudo pip3 install GitPython Thanks, Antonin
Hi Antonin, On 12/26/24 11:41 AM, Antonin Godard wrote: > Hi Quentin, > > On Tue Dec 17, 2024 at 5:08 PM CET, Quentin Schulz wrote: >> From: Quentin Schulz <quentin.schulz@cherry.de> >> >> This adds a script for building a container and building the >> documentation within that new container image. >> >> The openSUSE instructions now require a --non-interactive flag otherwise >> they fail to run. Sadly there doesn't seem to be a way to have this in >> an environment variable à-la DEBIAN_FRONTEND=noninteractive. > > Hmm, this means that we end up instructing to pass "--non-interactive" in the > docs? That's not ideal :/ I suggested a workaround below. > > I hope there won't be too many cases where we have to do things differently in > the context of a containerfile in the future. Apart from this case I cannot > think of anything else that could happen. Anything comes to mind for you? > There are always interesting corner cases in unattended CI :) >> Somehow tzdata package in Ubuntu doesn't respect >> DEBIAN_FRONTEND=noninteractive hence why the timezone needs to be set by >> hand. >> >> Note that only instructions for Debian Bookworm (12), Ubuntu 22.04 and >> 24.04, and Fedora 38, 39 and 40, currently work. >> >> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de> >> --- >> documentation/tools/Containerfile.almalinux | 1 + >> documentation/tools/Containerfile.apt | 26 +++++ >> documentation/tools/Containerfile.debian | 1 + >> documentation/tools/Containerfile.dnf | 25 +++++ >> documentation/tools/Containerfile.fedora | 1 + >> documentation/tools/Containerfile.ubuntu | 1 + >> documentation/tools/Containerfile.zypper | 24 +++++ >> documentation/tools/build-docs-container | 113 +++++++++++++++++++++ >> .../tools/host_packages_scripts/opensuse_docs.sh | 2 +- >> .../host_packages_scripts/opensuse_docs_pdf.sh | 2 +- >> .../host_packages_scripts/opensuse_essential.sh | 2 +- >> 11 files changed, 195 insertions(+), 3 deletions(-) >> >> diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux >> new file mode 120000 >> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 >> --- /dev/null >> +++ b/documentation/tools/Containerfile.almalinux >> @@ -0,0 +1 @@ >> +Containerfile.dnf >> \ No newline at end of file >> diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt >> new file mode 100644 >> index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517 >> --- /dev/null >> +++ b/documentation/tools/Containerfile.apt >> @@ -0,0 +1,26 @@ >> +ARG ARG_FROM=debian:12 # default value to avoid warning >> +FROM $ARG_FROM >> + >> +ARG DOCS=ubuntu_docs.sh >> +ARG DOCS_PDF=ubuntu_docs_pdf.sh >> + >> +ENV DEBIAN_FRONTEND=noninteractive >> +ARG TZ=Europe/Vienna >> + >> +# relative to the location of the dockerfile >> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh >> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh >> + >> +RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \ >> + && apt-get update \ >> + && apt-get install -y sudo \ >> + && yes | /temp/host_packages_docs.sh \ >> + && yes | /temp/host_packages_docs_pdf.sh \ >> + && apt-get --yes autoremove \ >> + && apt-get clean \ >> + && rm -rf /temp >> + >> +RUN git config --global --add safe.directory /docs >> + >> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] >> +CMD ["publish"] >> diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian >> new file mode 120000 >> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a >> --- /dev/null >> +++ b/documentation/tools/Containerfile.debian >> @@ -0,0 +1 @@ >> +Containerfile.apt >> \ No newline at end of file >> diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf >> new file mode 100644 >> index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886 >> --- /dev/null >> +++ b/documentation/tools/Containerfile.dnf >> @@ -0,0 +1,25 @@ >> +ARG ARG_FROM=fedora:40 # default value to avoid warning >> +FROM $ARG_FROM >> + >> +ARG DOCS=fedora_docs.sh >> +ARG DOCS_PDF=fedora_docs_pdf.sh >> +ARG PIP3=pip3_docs.sh >> + >> +# relative to the location of the dockerfile >> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh >> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh >> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh >> + >> +RUN dnf update -y \ >> + && dnf install -y sudo \ >> + && yes | /temp/host_packages_docs.sh \ >> + && yes | /temp/host_packages_docs_pdf.sh \ >> + && yes | /temp/pip3_docs.sh \ >> + && dnf autoremove -y \ >> + && dnf clean all -y \ >> + && rm -rf /temp >> + >> +RUN git config --global --add safe.directory /docs >> + >> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] >> +CMD ["publish"] >> diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora >> new file mode 120000 >> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 >> --- /dev/null >> +++ b/documentation/tools/Containerfile.fedora >> @@ -0,0 +1 @@ >> +Containerfile.dnf >> \ No newline at end of file >> diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu >> new file mode 120000 >> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a >> --- /dev/null >> +++ b/documentation/tools/Containerfile.ubuntu >> @@ -0,0 +1 @@ >> +Containerfile.apt >> \ No newline at end of file >> diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper >> new file mode 100644 >> index 0000000000000000000000000000000000000000..bcda5261ae98e8df16a3bcef17a7bf204033da99 >> --- /dev/null >> +++ b/documentation/tools/Containerfile.zypper >> @@ -0,0 +1,24 @@ >> +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning >> +FROM $ARG_FROM >> + >> +ARG DOCS=opensuse_docs.sh >> +ARG DOCS_PDF=opensuse_docs_pdf.sh >> +ARG PIP3=pip3_docs.sh >> + >> +# relative to the location of the dockerfile >> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh >> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh >> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh > > A bit of a hacky workaround, but maybe here run > sed -i 's/zypper/zypper --non-interactive/ ...' on the scripts? So we don't need > to provide this option in the doc. > Another dirty hack is to have a shell zypper wrapper that calls /usr/bin/zypper --non-interactive. Up to you. >> + >> +RUN zypper update -y \ >> + && zypper install -y sudo \ >> + && yes | /temp/host_packages_docs.sh \ >> + && yes | /temp/host_packages_docs_pdf.sh \ >> + && yes | /temp/pip3_docs.sh \ >> + && zypper clean --all \ >> + && rm -rf /temp >> + >> +RUN git config --global --add safe.directory /docs >> + >> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] >> +CMD ["publish"] >> diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container >> new file mode 100755 >> index 0000000000000000000000000000000000000000..f382af75a7e89cfcc8a6080fdbc65fb68ff9ea8d >> --- /dev/null >> +++ b/documentation/tools/build-docs-container >> @@ -0,0 +1,113 @@ >> +#!/usr/bin/env bash >> +# >> +# Build a container ready to build the documentation be reading the dependencies >> +# listed in shell scripts in documentation/tools/host_packages_scripts, and >> +# start a documentation build in this container. >> +# >> +# Usage: >> +# >> +# ./documentation/tools/build-docs-container <image> [<make target>] >> +# >> +# e.g.: >> +# >> +# ./documentation/tools/build-docs-container ubuntu:24.04 html >> +# >> +# Will build the docs in an Ubuntu 24.04 container in html. >> +# >> +# The container engine can be selected by exporting CONTAINERCMD in the >> +# environment. The default is docker, but podman can also be used. >> + >> +set -eu -o pipefail >> + >> +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) >> +CONTAINERCMD=${CONTAINERCMD:-docker} >> +DOCS_DIR="$SCRIPT_DIR/../.." >> +SH_DIR="$SCRIPT_DIR/host_packages_scripts" >> + >> +main () >> +{ > > I would be good to have a check here to avoid > > $ ./tools/build-docs-container > ./tools/build-docs-container: line 29: $1: unbound variable > > You could probably copypaste what I did in my version of the script. > Will do. >> + local image="$1" >> + shift >> + >> + OCI=$(which "$CONTAINERCMD") >> + >> + # docker build doesn't accept 2 colons, so "sanitize" the name >> + local sanitized_dockername >> + sanitized_dockername=$(echo "$image" | tr ':.' '-') >> + >> + local version >> + version=$(echo "$image" | awk -F: '{print $NF}') >> + >> + case $image in >> + almalinux*) >> + containerfile=Containerfile.almalinux >> + docs=almalinux_docs.sh >> + docs_pdf=almalinux_docs_pdf.sh >> + pip3=pip3_docs.sh >> + ;; >> + debian*) >> + containerfile=Containerfile.debian >> + docs=ubuntu_docs.sh >> + docs_pdf=ubuntu_docs_pdf.sh >> + ;; >> + fedora*) >> + containerfile=Containerfile.fedora >> + docs=fedora_docs.sh >> + docs_pdf=fedora_docs_pdf.sh >> + pip3=pip3_docs.sh >> + ;; >> + opensuse* | leap*) >> + image=opensuse/leap:$version >> + containerfile=Containerfile.zypper >> + docs=opensuse_docs.sh >> + docs_pdf=opensuse_docs_pdf.sh >> + pip3=pip3_docs.sh >> + ;; >> + ubuntu*) >> + containerfile=Containerfile.ubuntu >> + docs=ubuntu_docs.sh >> + docs_pdf=ubuntu_docs_pdf.sh >> + ;; >> + *) >> + echo "$image not supported" > > Can we have a way of listing images here so a user doesn't need to read the > script and guess which one to use? > The ones in https://docs.yoctoproject.org/ref-manual/system-requirements.html#supported-linux-distributions? Though I guess Rocky and CentOS are still missing. Do we want the name of the distro or the name + the version? Basically we can end up with two places where this information will be, in the docs and in the container. Cheers, Quentin
Hi Quentin, On Mon Jan 13, 2025 at 5:43 PM CET, Quentin Schulz wrote: [...] >>> +# relative to the location of the dockerfile >>> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh >>> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh >>> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh >> >> A bit of a hacky workaround, but maybe here run >> sed -i 's/zypper/zypper --non-interactive/ ...' on the scripts? So we don't need >> to provide this option in the doc. >> > > Another dirty hack is to have a shell zypper wrapper that calls > /usr/bin/zypper --non-interactive. > > Up to you. Let's go for a sed in the Containerfile. [...] >> >> Can we have a way of listing images here so a user doesn't need to read the >> script and guess which one to use? >> > > The ones in > https://docs.yoctoproject.org/ref-manual/system-requirements.html#supported-linux-distributions? > Though I guess Rocky and CentOS are still missing. Do we want the name > of the distro or the name + the version? Basically we can end up with > two places where this information will be, in the docs and in the container. I was thinking of just listing what the script supports, not from the docs. I did that in my version of the script. Something like `./build-docs-container list`. In your case, it would probably extract that information from the Containerfiles. Antonin
diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux new file mode 120000 index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 --- /dev/null +++ b/documentation/tools/Containerfile.almalinux @@ -0,0 +1 @@ +Containerfile.dnf \ No newline at end of file diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt new file mode 100644 index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517 --- /dev/null +++ b/documentation/tools/Containerfile.apt @@ -0,0 +1,26 @@ +ARG ARG_FROM=debian:12 # default value to avoid warning +FROM $ARG_FROM + +ARG DOCS=ubuntu_docs.sh +ARG DOCS_PDF=ubuntu_docs_pdf.sh + +ENV DEBIAN_FRONTEND=noninteractive +ARG TZ=Europe/Vienna + +# relative to the location of the dockerfile +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh + +RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \ + && apt-get update \ + && apt-get install -y sudo \ + && yes | /temp/host_packages_docs.sh \ + && yes | /temp/host_packages_docs_pdf.sh \ + && apt-get --yes autoremove \ + && apt-get clean \ + && rm -rf /temp + +RUN git config --global --add safe.directory /docs + +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] +CMD ["publish"] diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian new file mode 120000 index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a --- /dev/null +++ b/documentation/tools/Containerfile.debian @@ -0,0 +1 @@ +Containerfile.apt \ No newline at end of file diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf new file mode 100644 index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886 --- /dev/null +++ b/documentation/tools/Containerfile.dnf @@ -0,0 +1,25 @@ +ARG ARG_FROM=fedora:40 # default value to avoid warning +FROM $ARG_FROM + +ARG DOCS=fedora_docs.sh +ARG DOCS_PDF=fedora_docs_pdf.sh +ARG PIP3=pip3_docs.sh + +# relative to the location of the dockerfile +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh + +RUN dnf update -y \ + && dnf install -y sudo \ + && yes | /temp/host_packages_docs.sh \ + && yes | /temp/host_packages_docs_pdf.sh \ + && yes | /temp/pip3_docs.sh \ + && dnf autoremove -y \ + && dnf clean all -y \ + && rm -rf /temp + +RUN git config --global --add safe.directory /docs + +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] +CMD ["publish"] diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora new file mode 120000 index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 --- /dev/null +++ b/documentation/tools/Containerfile.fedora @@ -0,0 +1 @@ +Containerfile.dnf \ No newline at end of file diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu new file mode 120000 index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a --- /dev/null +++ b/documentation/tools/Containerfile.ubuntu @@ -0,0 +1 @@ +Containerfile.apt \ No newline at end of file diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper new file mode 100644 index 0000000000000000000000000000000000000000..bcda5261ae98e8df16a3bcef17a7bf204033da99 --- /dev/null +++ b/documentation/tools/Containerfile.zypper @@ -0,0 +1,24 @@ +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning +FROM $ARG_FROM + +ARG DOCS=opensuse_docs.sh +ARG DOCS_PDF=opensuse_docs_pdf.sh +ARG PIP3=pip3_docs.sh + +# relative to the location of the dockerfile +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh + +RUN zypper update -y \ + && zypper install -y sudo \ + && yes | /temp/host_packages_docs.sh \ + && yes | /temp/host_packages_docs_pdf.sh \ + && yes | /temp/pip3_docs.sh \ + && zypper clean --all \ + && rm -rf /temp + +RUN git config --global --add safe.directory /docs + +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] +CMD ["publish"] diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container new file mode 100755 index 0000000000000000000000000000000000000000..f382af75a7e89cfcc8a6080fdbc65fb68ff9ea8d --- /dev/null +++ b/documentation/tools/build-docs-container @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# +# Build a container ready to build the documentation be reading the dependencies +# listed in shell scripts in documentation/tools/host_packages_scripts, and +# start a documentation build in this container. +# +# Usage: +# +# ./documentation/tools/build-docs-container <image> [<make target>] +# +# e.g.: +# +# ./documentation/tools/build-docs-container ubuntu:24.04 html +# +# Will build the docs in an Ubuntu 24.04 container in html. +# +# The container engine can be selected by exporting CONTAINERCMD in the +# environment. The default is docker, but podman can also be used. + +set -eu -o pipefail + +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) +CONTAINERCMD=${CONTAINERCMD:-docker} +DOCS_DIR="$SCRIPT_DIR/../.." +SH_DIR="$SCRIPT_DIR/host_packages_scripts" + +main () +{ + local image="$1" + shift + + OCI=$(which "$CONTAINERCMD") + + # docker build doesn't accept 2 colons, so "sanitize" the name + local sanitized_dockername + sanitized_dockername=$(echo "$image" | tr ':.' '-') + + local version + version=$(echo "$image" | awk -F: '{print $NF}') + + case $image in + almalinux*) + containerfile=Containerfile.almalinux + docs=almalinux_docs.sh + docs_pdf=almalinux_docs_pdf.sh + pip3=pip3_docs.sh + ;; + debian*) + containerfile=Containerfile.debian + docs=ubuntu_docs.sh + docs_pdf=ubuntu_docs_pdf.sh + ;; + fedora*) + containerfile=Containerfile.fedora + docs=fedora_docs.sh + docs_pdf=fedora_docs_pdf.sh + pip3=pip3_docs.sh + ;; + opensuse* | leap*) + image=opensuse/leap:$version + containerfile=Containerfile.zypper + docs=opensuse_docs.sh + docs_pdf=opensuse_docs_pdf.sh + pip3=pip3_docs.sh + ;; + ubuntu*) + containerfile=Containerfile.ubuntu + docs=ubuntu_docs.sh + docs_pdf=ubuntu_docs_pdf.sh + ;; + *) + echo "$image not supported" + exit 1 + ;; + esac + + $OCI build \ + --tag "yocto-docs-$sanitized_dockername:latest" \ + --build-arg ARG_FROM="docker.io/$image" \ + --build-arg DOCS="$docs" \ + --build-arg DOCS_PDF="$docs_pdf" \ + --build-arg PIP3="${pip3:-}" \ + --file "$SCRIPT_DIR/$containerfile" \ + "$SH_DIR/" + + local -a args_run=( + --rm + --interactive + --tty + --volume="$DOCS_DIR:/docs:rw" + --workdir=/docs + --security-opt label=disable + ) + + if [ "$OCI" = "docker" ]; then + args_run+=( + --user="$(id -u)":"$(id -g)" + ) + elif [ "$OCI" = "podman" ]; then + # we need net access to fetch bitbake terms + args_run+=( + --cap-add=NET_RAW + --userns=keep-id + ) + fi + + $OCI run \ + "${args_run[@]}" \ + "yocto-docs-$sanitized_dockername" \ + "$@" +} + +main "$@" diff --git a/documentation/tools/host_packages_scripts/opensuse_docs.sh b/documentation/tools/host_packages_scripts/opensuse_docs.sh index 7d36eb0f99668168dc4fed44d71181338bdc59a3..896ad52b5042a9679555a30e582fa32076c3a3e7 100644 --- a/documentation/tools/host_packages_scripts/opensuse_docs.sh +++ b/documentation/tools/host_packages_scripts/opensuse_docs.sh @@ -1 +1 @@ -sudo zypper install git glibc-i18ndata make python3-pip rsvg-convert which +sudo zypper --non-interactive install git glibc-i18ndata make python3-pip rsvg-convert which diff --git a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh index ee9f07886ce1c0eac8e1d1118cda8ddc496f0139..f8a3b800d9790c0434f0c3f577ea1916a7fe1b28 100644 --- a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh +++ b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh @@ -1 +1 @@ -sudo zypper install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex +sudo zypper --non-interactive install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex diff --git a/documentation/tools/host_packages_scripts/opensuse_essential.sh b/documentation/tools/host_packages_scripts/opensuse_essential.sh index a784f4a5dc05f3734177f08ed97663a04d20a978..19e85f4a736532b890fc55492721cf345c2b4083 100644 --- a/documentation/tools/host_packages_scripts/opensuse_essential.sh +++ b/documentation/tools/host_packages_scripts/opensuse_essential.sh @@ -1,2 +1,2 @@ -sudo zypper install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd +sudo zypper --non-interactive install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd sudo pip3 install GitPython