From patchwork Tue Dec 17 16:08:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Schulz X-Patchwork-Id: 54251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D6E3E7717F for ; Tue, 17 Dec 2024 16:09:09 +0000 (UTC) Received: from smtp-190a.mail.infomaniak.ch (smtp-190a.mail.infomaniak.ch [185.125.25.10]) by mx.groups.io with SMTP id smtpd.web10.85154.1734451748288042566 for ; Tue, 17 Dec 2024 08:09:08 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: 0leil.net, ip: 185.125.25.10, mailfrom: foss+yocto@0leil.net) Received: from smtp-3-0001.mail.infomaniak.ch (smtp-3-0001.mail.infomaniak.ch [10.4.36.108]) by smtp-3-3000.mail.infomaniak.ch (Postfix) with ESMTPS id 4YCMFG0x0DzXdw; Tue, 17 Dec 2024 17:09:06 +0100 (CET) Received: from unknown by smtp-3-0001.mail.infomaniak.ch (Postfix) with ESMTPA id 4YCMFF5BWvzMfG; Tue, 17 Dec 2024 17:09:05 +0100 (CET) From: Quentin Schulz Date: Tue, 17 Dec 2024 17:08:50 +0100 Subject: [PATCH RFC v2 1/2] docs: use literalinclude for system requirements MIME-Version: 1.0 Message-Id: <20241217-instructions-shell-container-v2-1-ce687b0cfac2@cherry.de> References: <20241217-instructions-shell-container-v2-0-ce687b0cfac2@cherry.de> In-Reply-To: <20241217-instructions-shell-container-v2-0-ce687b0cfac2@cherry.de> To: Quentin Schulz , docs@lists.yoctoproject.org X-Mailer: b4 0.14.2 X-Infomaniak-Routing: alpha List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 17 Dec 2024 16:09:09 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/5972 From: Quentin Schulz The YAML variables for the host dependencies are updated by hand and actually only used inside code blocks. Let's migrate all instructions into separate shell scripts that are then literalinclude'd into the Sphinx documentation. This allows a few things: - ability to run shellcheck on the scripts if we ever want to - manually calling the appropriate script from a supported distro to build stuff (distro or bitbake/yocto stuff) - use this script to create containers to do CI of documentation on different distros, to make sure our instructions are all up to date, Signed-off-by: Quentin Schulz --- documentation/poky.yaml.in | 234 --------------------- documentation/ref-manual/system-requirements.rst | 82 +++++--- .../tools/host_packages_scripts/almalinux_docs.sh | 1 + .../host_packages_scripts/almalinux_docs_pdf.sh | 1 + .../host_packages_scripts/almalinux_essential.sh | 5 + .../tools/host_packages_scripts/fedora_docs.sh | 1 + .../tools/host_packages_scripts/fedora_docs_pdf.sh | 1 + .../host_packages_scripts/fedora_essential.sh | 1 + .../tools/host_packages_scripts/opensuse_docs.sh | 1 + .../host_packages_scripts/opensuse_docs_pdf.sh | 1 + .../host_packages_scripts/opensuse_essential.sh | 2 + .../tools/host_packages_scripts/pip3_docs.sh | 1 + .../tools/host_packages_scripts/ubuntu_docs.sh | 1 + .../tools/host_packages_scripts/ubuntu_docs_pdf.sh | 1 + .../host_packages_scripts/ubuntu_essential.sh | 1 + 15 files changed, 65 insertions(+), 269 deletions(-) diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in index 0933cbbd351985899ecaea94ae42748bd7ba1753..d637952bbed13b22afac95bd77aef9f2e35fa630 100644 --- a/documentation/poky.yaml.in +++ b/documentation/poky.yaml.in @@ -25,237 +25,3 @@ MIN_DISK_SPACE : "90" MIN_DISK_SPACE_RM_WORK : "40" # RAM (Gbytes) needed to generate qemux86-64 core-image-sato on Ubuntu 22.04 (x86-64) on a 4 core system MIN_RAM : "8" - -# -# Dependencies -# - -# Shared between distros -PIP3_HOST_PACKAGES_DOC: sphinx sphinx_rtd_theme pyyaml - -UBUNTU_DEBIAN_HOST_PACKAGES_ESSENTIAL: >- - build-essential - chrpath - cpio - debianutils - diffstat - file - gawk - gcc - git - iputils-ping - libacl1 - liblz4-tool - locales - python3 - python3-git - python3-jinja2 - python3-pexpect - python3-pip - python3-subunit - socat - texinfo - unzip - wget - xz-utils - zstd - -UBUNTU_DEBIAN_HOST_PACKAGES_DOC: >- - git - librsvg2-bin - locales - make - python3-saneyaml - python3-sphinx-rtd-theme - sphinx - -UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF: >- - fonts-freefont-otf - latexmk - tex-gyre - texlive-fonts-extra - texlive-fonts-recommended - texlive-lang-all - texlive-latex-extra - texlive-latex-recommended - texlive-xetex - -FEDORA_HOST_PACKAGES_ESSENTIAL: >- - bzip2 - ccache - chrpath - cpio - cpp - diffstat - diffutils - file - findutils - gawk - gcc - gcc-c++ - git - glibc-devel - glibc-langpack-en - gzip - hostname - libacl - lz4 - make - patch - perl - perl-Data-Dumper - perl-File-Compare - perl-File-Copy - perl-FindBin - perl-Text-ParseWords - perl-Thread-Queue - perl-bignum - perl-locale - python - python3 - python3-GitPython - python3-jinja2 - python3-pexpect - python3-pip - rpcgen - socat - tar - texinfo - unzip - wget - which - xz - zstd - -FEDORA_HOST_PACKAGES_DOC: >- - git - glibc-locale-source - librsvg2-tools - make - python3-pip - which - -FEDORA_HOST_PACKAGES_DOC_PDF: >- - 'texlive-collection-lang*' - latexmk - texlive-collection-fontsextra - texlive-collection-fontsrecommended - texlive-collection-latex - texlive-collection-latexextra - texlive-collection-latexrecommended - texlive-collection-xetex - texlive-fncychap - texlive-gnu-freefont - texlive-tex-gyre - texlive-xetex - -OPENSUSE_HOST_PACKAGES_ESSENTIAL: >- - bzip2 - chrpath - diffstat - gcc - gcc-c++ - git - gzip - hostname - libacl1 - lz4 - make - makeinfo - patch - python - python-curses - python-xml - python3 - python3-Jinja2 - python3-curses - python3-pexpect - python3-pip - rpcgen - socat - tar - wget - which - xz - zstd - -OPENSUSE_PIP3_HOST_PACKAGES_ESSENTIAL: GitPython - -OPENSUSE_HOST_PACKAGES_DOC: >- - git - glibc-i18ndata - make - python3-pip - rsvg-convert - which - -OPENSUSE_HOST_PACKAGES_DOC_PDF: >- - 'texlive-collection-lang*' - texlive-collection-fontsextra - texlive-collection-fontsrecommended - texlive-collection-latex - texlive-collection-latexextra - texlive-collection-latexrecommended - texlive-collection-xetex - texlive-fncychap - texlive-gnu-freefont - texlive-latexmk - texlive-tex-gyre - texlive-xetex - -ALMALINUX_HOST_PACKAGES_ESSENTIAL: >- - bzip2 - ccache - chrpath - cpio - cpp - diffstat - diffutils - gawk - gcc - gcc-c++ - git - glibc-devel - glibc-langpack-en - gzip - libacl - lz4 - make - patch - perl - perl-Data-Dumper - perl-Text-ParseWords - perl-Thread-Queue - python3 - python3-GitPython - python3-jinja2 - python3-pexpect - python3-pip - rpcgen - socat - tar - texinfo - unzip - wget - which - xz - zstd - -ALMALINUX_HOST_PACKAGES_DOC: >- - git - glibc-locale-source - librsvg2-tools - make - python3-pip - which - -ALMALINUX_HOST_PACKAGES_DOC_PDF: >- - latexmk - texlive-collection-fontsrecommended - texlive-collection-latex - texlive-collection-latexrecommended - texlive-collection-xetex - texlive-fncychap - texlive-gnu-freefont - texlive-tex-gyre - texlive-xetex diff --git a/documentation/ref-manual/system-requirements.rst b/documentation/ref-manual/system-requirements.rst index b087d374d23a71104dc9c050e96e4a2c20f5abb0..93b95a7fb251c4a0192c2d29a1dbf365eb69871e 100644 --- a/documentation/ref-manual/system-requirements.rst +++ b/documentation/ref-manual/system-requirements.rst @@ -158,9 +158,10 @@ Ubuntu and Debian ----------------- Here are the packages needed to build an image on a headless system -with a supported Ubuntu or Debian Linux distribution:: +with a supported Ubuntu or Debian Linux distribution: - $ sudo apt install &UBUNTU_DEBIAN_HOST_PACKAGES_ESSENTIAL; +.. literalinclude:: ../tools/host_packages_scripts/ubuntu_essential.sh + :language: shell You also need to ensure you have the ``en_US.UTF-8`` locale enabled:: @@ -189,64 +190,71 @@ If this is not the case, you can reconfigure the ``locales`` package to add it $ sudo apt build-dep qemu $ sudo apt remove oss4-dev -Here are the packages needed to build Project documentation manuals:: +Here are the packages needed to build Project documentation manuals: - $ sudo apt install &UBUNTU_DEBIAN_HOST_PACKAGES_DOC; +.. literalinclude:: ../tools/host_packages_scripts/ubuntu_docs.sh + :language: shell In addition to the previous packages, here are the packages needed to build the -documentation in PDF format:: +documentation in PDF format: - $ sudo apt install &UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF; +.. literalinclude:: ../tools/host_packages_scripts/ubuntu_docs_pdf.sh + :language: shell Fedora Packages --------------- Here are the packages needed to build an image on a headless system -with a supported Fedora Linux distribution:: +with a supported Fedora Linux distribution: - $ sudo dnf install &FEDORA_HOST_PACKAGES_ESSENTIAL; +.. literalinclude:: ../tools/host_packages_scripts/fedora_essential.sh + :language: shell -Here are the packages needed to build Project documentation manuals:: +Here are the packages needed to build Project documentation manuals: - $ sudo dnf install &FEDORA_HOST_PACKAGES_DOC; - $ sudo pip3 install &PIP3_HOST_PACKAGES_DOC; +.. literalinclude:: ../tools/host_packages_scripts/fedora_docs.sh + :language: shell + +.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh + :language: shell In addition to the previous packages, here are the packages needed to build the -documentation in PDF format:: +documentation in PDF format: - $ sudo dnf install &FEDORA_HOST_PACKAGES_DOC_PDF; +.. literalinclude:: ../tools/host_packages_scripts/fedora_docs_pdf.sh + :language: shell openSUSE Packages ----------------- Here are the packages needed to build an image on a headless system -with a supported openSUSE distribution:: +with a supported openSUSE distribution: - $ sudo zypper install &OPENSUSE_HOST_PACKAGES_ESSENTIAL; - $ sudo pip3 install &OPENSUSE_PIP3_HOST_PACKAGES_ESSENTIAL; +.. literalinclude:: ../tools/host_packages_scripts/opensuse_essential.sh + :language: shell -Here are the packages needed to build Project documentation manuals:: +Here are the packages needed to build Project documentation manuals: - $ sudo zypper install &OPENSUSE_HOST_PACKAGES_DOC; - $ sudo pip3 install &PIP3_HOST_PACKAGES_DOC; +.. literalinclude:: ../tools/host_packages_scripts/opensuse_docs.sh + :language: shell + +.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh + :language: shell In addition to the previous packages, here are the packages needed to build the -documentation in PDF format:: - - $ sudo zypper install &OPENSUSE_HOST_PACKAGES_DOC_PDF; +documentation in PDF format: +.. literalinclude:: ../tools/host_packages_scripts/opensuse_docs_pdf.sh + :language: shell AlmaLinux Packages ------------------ Here are the packages needed to build an image on a headless system -with a supported AlmaLinux distribution:: +with a supported AlmaLinux distribution: - $ sudo dnf install -y epel-release - $ sudo yum install dnf-plugins-core - $ sudo dnf config-manager --set-enabled crb - $ sudo dnf makecache - $ sudo dnf install &ALMALINUX_HOST_PACKAGES_ESSENTIAL; +.. literalinclude:: ../tools/host_packages_scripts/almalinux_essential.sh + :language: shell .. note:: @@ -261,15 +269,13 @@ with a supported AlmaLinux distribution:: - The ``makecache`` command consumes additional Metadata from ``epel-release``. -Here are the packages needed to build Project documentation manuals:: +Here are the packages needed to build Project documentation manuals: - $ sudo dnf install &ALMALINUX_HOST_PACKAGES_DOC; - $ sudo pip3 install &PIP3_HOST_PACKAGES_DOC; +.. literalinclude:: ../tools/host_packages_scripts/almalinux_docs.sh + :language: shell -In addition to the previous packages, here are the packages needed to build the -documentation in PDF format:: - - $ sudo dnf install &ALMALINUX_HOST_PACKAGES_DOC_PDF; +.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh + :language: shell .. warning:: @@ -278,6 +284,12 @@ documentation in PDF format:: ``texlive-collection-latexextra``, so you may run into issues. These may be installed using `tlmgr `_. +In addition to the previous packages, here are the packages needed to build the +documentation in PDF format: + +.. literalinclude:: ../tools/host_packages_scripts/almalinux_docs_pdf.sh + :language: shell + .. _system-requirements-buildtools: Required Git, tar, Python, make and gcc Versions diff --git a/documentation/tools/host_packages_scripts/almalinux_docs.sh b/documentation/tools/host_packages_scripts/almalinux_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..8188d529a1e9b8034e5b14ed4e9e633e7ae0600b --- /dev/null +++ b/documentation/tools/host_packages_scripts/almalinux_docs.sh @@ -0,0 +1 @@ +sudo dnf install git glibc-locale-source librsvg2-tools make python3-pip which diff --git a/documentation/tools/host_packages_scripts/almalinux_docs_pdf.sh b/documentation/tools/host_packages_scripts/almalinux_docs_pdf.sh new file mode 100644 index 0000000000000000000000000000000000000000..8341eb8c252d50b8a5cd3ff09ff013b1d49249bf --- /dev/null +++ b/documentation/tools/host_packages_scripts/almalinux_docs_pdf.sh @@ -0,0 +1 @@ +sudo dnf install latexmk texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-tex-gyre texlive-xetex diff --git a/documentation/tools/host_packages_scripts/almalinux_essential.sh b/documentation/tools/host_packages_scripts/almalinux_essential.sh new file mode 100644 index 0000000000000000000000000000000000000000..16f25fa1f29c84d512df97988f527d1ea834bea7 --- /dev/null +++ b/documentation/tools/host_packages_scripts/almalinux_essential.sh @@ -0,0 +1,5 @@ +sudo dnf install -y epel-release +sudo yum install dnf-plugins-core +sudo dnf config-manager --set-enabled crb +sudo dnf makecache +sudo dnf install bzip2 ccache chrpath cpio cpp diffstat diffutils gawk gcc gcc-c++ git glibc-devel glibc-langpack-en gzip libacl lz4 make patch perl perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3 python3-GitPython python3-jinja2 python3-pexpect python3-pip rpcgen socat tar texinfo unzip wget which xz zstd diff --git a/documentation/tools/host_packages_scripts/fedora_docs.sh b/documentation/tools/host_packages_scripts/fedora_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..8188d529a1e9b8034e5b14ed4e9e633e7ae0600b --- /dev/null +++ b/documentation/tools/host_packages_scripts/fedora_docs.sh @@ -0,0 +1 @@ +sudo dnf install git glibc-locale-source librsvg2-tools make python3-pip which diff --git a/documentation/tools/host_packages_scripts/fedora_docs_pdf.sh b/documentation/tools/host_packages_scripts/fedora_docs_pdf.sh new file mode 100644 index 0000000000000000000000000000000000000000..816f1c36ec35e046782d341bb70dff1834bfd8a1 --- /dev/null +++ b/documentation/tools/host_packages_scripts/fedora_docs_pdf.sh @@ -0,0 +1 @@ +sudo dnf install 'texlive-collection-lang*' latexmk texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-tex-gyre texlive-xetex diff --git a/documentation/tools/host_packages_scripts/fedora_essential.sh b/documentation/tools/host_packages_scripts/fedora_essential.sh new file mode 100644 index 0000000000000000000000000000000000000000..4841853e1e224683b563c8d6112446f0633a081b --- /dev/null +++ b/documentation/tools/host_packages_scripts/fedora_essential.sh @@ -0,0 +1 @@ +sudo dnf install bzip2 ccache chrpath cpio cpp diffstat diffutils file findutils gawk gcc gcc-c++ git glibc-devel glibc-langpack-en gzip hostname libacl lz4 make patch perl perl-Data-Dumper perl-File-Compare perl-File-Copy perl-FindBin perl-Text-ParseWords perl-Thread-Queue perl-bignum perl-locale python python3 python3-GitPython python3-jinja2 python3-pexpect python3-pip rpcgen socat tar texinfo unzip wget which xz zstd diff --git a/documentation/tools/host_packages_scripts/opensuse_docs.sh b/documentation/tools/host_packages_scripts/opensuse_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..7d36eb0f99668168dc4fed44d71181338bdc59a3 --- /dev/null +++ b/documentation/tools/host_packages_scripts/opensuse_docs.sh @@ -0,0 +1 @@ +sudo zypper install git glibc-i18ndata make python3-pip rsvg-convert which diff --git a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh new file mode 100644 index 0000000000000000000000000000000000000000..ee9f07886ce1c0eac8e1d1118cda8ddc496f0139 --- /dev/null +++ b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh @@ -0,0 +1 @@ +sudo zypper install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex diff --git a/documentation/tools/host_packages_scripts/opensuse_essential.sh b/documentation/tools/host_packages_scripts/opensuse_essential.sh new file mode 100644 index 0000000000000000000000000000000000000000..a784f4a5dc05f3734177f08ed97663a04d20a978 --- /dev/null +++ b/documentation/tools/host_packages_scripts/opensuse_essential.sh @@ -0,0 +1,2 @@ +sudo zypper install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd +sudo pip3 install GitPython diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..fd6ad98053fb6aa1ff55669191dc99578c830455 --- /dev/null +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh @@ -0,0 +1 @@ +sudo pip3 install sphinx sphinx_rtd_theme pyyaml diff --git a/documentation/tools/host_packages_scripts/ubuntu_docs.sh b/documentation/tools/host_packages_scripts/ubuntu_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..d2a1832e279057519321a1fb7820efd60d886564 --- /dev/null +++ b/documentation/tools/host_packages_scripts/ubuntu_docs.sh @@ -0,0 +1 @@ +sudo apt install git librsvg2-bin locales make python3-saneyaml python3-saneyaml python3-sphinx-rtd-theme sphinx diff --git a/documentation/tools/host_packages_scripts/ubuntu_docs_pdf.sh b/documentation/tools/host_packages_scripts/ubuntu_docs_pdf.sh new file mode 100644 index 0000000000000000000000000000000000000000..d6310bd21deae81df32d0cb843aab4a5622ac4fa --- /dev/null +++ b/documentation/tools/host_packages_scripts/ubuntu_docs_pdf.sh @@ -0,0 +1 @@ +sudo apt install fonts-freefont-otf latexmk tex-gyre texlive-fonts-extra texlive-fonts-recommended texlive-lang-all texlive-latex-extra texlive-latex-recommended texlive-xetex diff --git a/documentation/tools/host_packages_scripts/ubuntu_essential.sh b/documentation/tools/host_packages_scripts/ubuntu_essential.sh new file mode 100644 index 0000000000000000000000000000000000000000..f3388c88c93d885e3b035651e434c669654ee241 --- /dev/null +++ b/documentation/tools/host_packages_scripts/ubuntu_essential.sh @@ -0,0 +1 @@ +sudo apt-get install build-essential chrpath cpio debianutils diffstat file gawk gcc git iputils-ping libacl1 liblz4-tool locales python3 python3-git python3-jinja2 python3-pexpect python3-pip python3-subunit socat texinfo unzip wget xz-utils zstd From patchwork Tue Dec 17 16:08:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Quentin Schulz X-Patchwork-Id: 54252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F0C4E77187 for ; Tue, 17 Dec 2024 16:09:09 +0000 (UTC) Received: from smtp-190e.mail.infomaniak.ch (smtp-190e.mail.infomaniak.ch [185.125.25.14]) by mx.groups.io with SMTP id smtpd.web10.85153.1734451747904427034 for ; Tue, 17 Dec 2024 08:09:08 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: 0leil.net, ip: 185.125.25.14, mailfrom: foss+yocto@0leil.net) Received: from smtp-3-0001.mail.infomaniak.ch (unknown [IPv6:2001:1600:4:17::246c]) by smtp-3-3000.mail.infomaniak.ch (Postfix) with ESMTPS id 4YCMFG3CKKzTCF; Tue, 17 Dec 2024 17:09:06 +0100 (CET) Received: from unknown by smtp-3-0001.mail.infomaniak.ch (Postfix) with ESMTPA id 4YCMFG0r0qzM9C; Tue, 17 Dec 2024 17:09:06 +0100 (CET) From: Quentin Schulz Date: Tue, 17 Dec 2024 17:08:51 +0100 Subject: [PATCH RFC v2 2/2] tools: add script for building documentation inside containers MIME-Version: 1.0 Message-Id: <20241217-instructions-shell-container-v2-2-ce687b0cfac2@cherry.de> References: <20241217-instructions-shell-container-v2-0-ce687b0cfac2@cherry.de> In-Reply-To: <20241217-instructions-shell-container-v2-0-ce687b0cfac2@cherry.de> To: Quentin Schulz , docs@lists.yoctoproject.org X-Mailer: b4 0.14.2 X-Infomaniak-Routing: alpha List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 17 Dec 2024 16:09:09 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/5971 From: Quentin Schulz This adds a script for building a container and building the documentation within that new container image. The openSUSE instructions now require a --non-interactive flag otherwise they fail to run. Sadly there doesn't seem to be a way to have this in an environment variable à-la DEBIAN_FRONTEND=noninteractive. Somehow tzdata package in Ubuntu doesn't respect DEBIAN_FRONTEND=noninteractive hence why the timezone needs to be set by hand. Note that only instructions for Debian Bookworm (12), Ubuntu 22.04 and 24.04, and Fedora 38, 39 and 40, currently work. Signed-off-by: Quentin Schulz --- documentation/tools/Containerfile.almalinux | 1 + documentation/tools/Containerfile.apt | 26 +++++ documentation/tools/Containerfile.debian | 1 + documentation/tools/Containerfile.dnf | 25 +++++ documentation/tools/Containerfile.fedora | 1 + documentation/tools/Containerfile.ubuntu | 1 + documentation/tools/Containerfile.zypper | 24 +++++ documentation/tools/build-docs-container | 113 +++++++++++++++++++++ .../tools/host_packages_scripts/opensuse_docs.sh | 2 +- .../host_packages_scripts/opensuse_docs_pdf.sh | 2 +- .../host_packages_scripts/opensuse_essential.sh | 2 +- 11 files changed, 195 insertions(+), 3 deletions(-) diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux new file mode 120000 index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 --- /dev/null +++ b/documentation/tools/Containerfile.almalinux @@ -0,0 +1 @@ +Containerfile.dnf \ No newline at end of file diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt new file mode 100644 index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517 --- /dev/null +++ b/documentation/tools/Containerfile.apt @@ -0,0 +1,26 @@ +ARG ARG_FROM=debian:12 # default value to avoid warning +FROM $ARG_FROM + +ARG DOCS=ubuntu_docs.sh +ARG DOCS_PDF=ubuntu_docs_pdf.sh + +ENV DEBIAN_FRONTEND=noninteractive +ARG TZ=Europe/Vienna + +# relative to the location of the dockerfile +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh + +RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \ + && apt-get update \ + && apt-get install -y sudo \ + && yes | /temp/host_packages_docs.sh \ + && yes | /temp/host_packages_docs_pdf.sh \ + && apt-get --yes autoremove \ + && apt-get clean \ + && rm -rf /temp + +RUN git config --global --add safe.directory /docs + +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] +CMD ["publish"] diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian new file mode 120000 index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a --- /dev/null +++ b/documentation/tools/Containerfile.debian @@ -0,0 +1 @@ +Containerfile.apt \ No newline at end of file diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf new file mode 100644 index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886 --- /dev/null +++ b/documentation/tools/Containerfile.dnf @@ -0,0 +1,25 @@ +ARG ARG_FROM=fedora:40 # default value to avoid warning +FROM $ARG_FROM + +ARG DOCS=fedora_docs.sh +ARG DOCS_PDF=fedora_docs_pdf.sh +ARG PIP3=pip3_docs.sh + +# relative to the location of the dockerfile +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh + +RUN dnf update -y \ + && dnf install -y sudo \ + && yes | /temp/host_packages_docs.sh \ + && yes | /temp/host_packages_docs_pdf.sh \ + && yes | /temp/pip3_docs.sh \ + && dnf autoremove -y \ + && dnf clean all -y \ + && rm -rf /temp + +RUN git config --global --add safe.directory /docs + +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] +CMD ["publish"] diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora new file mode 120000 index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 --- /dev/null +++ b/documentation/tools/Containerfile.fedora @@ -0,0 +1 @@ +Containerfile.dnf \ No newline at end of file diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu new file mode 120000 index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a --- /dev/null +++ b/documentation/tools/Containerfile.ubuntu @@ -0,0 +1 @@ +Containerfile.apt \ No newline at end of file diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper new file mode 100644 index 0000000000000000000000000000000000000000..bcda5261ae98e8df16a3bcef17a7bf204033da99 --- /dev/null +++ b/documentation/tools/Containerfile.zypper @@ -0,0 +1,24 @@ +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning +FROM $ARG_FROM + +ARG DOCS=opensuse_docs.sh +ARG DOCS_PDF=opensuse_docs_pdf.sh +ARG PIP3=pip3_docs.sh + +# relative to the location of the dockerfile +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh + +RUN zypper update -y \ + && zypper install -y sudo \ + && yes | /temp/host_packages_docs.sh \ + && yes | /temp/host_packages_docs_pdf.sh \ + && yes | /temp/pip3_docs.sh \ + && zypper clean --all \ + && rm -rf /temp + +RUN git config --global --add safe.directory /docs + +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] +CMD ["publish"] diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container new file mode 100755 index 0000000000000000000000000000000000000000..f382af75a7e89cfcc8a6080fdbc65fb68ff9ea8d --- /dev/null +++ b/documentation/tools/build-docs-container @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# +# Build a container ready to build the documentation be reading the dependencies +# listed in shell scripts in documentation/tools/host_packages_scripts, and +# start a documentation build in this container. +# +# Usage: +# +# ./documentation/tools/build-docs-container [] +# +# e.g.: +# +# ./documentation/tools/build-docs-container ubuntu:24.04 html +# +# Will build the docs in an Ubuntu 24.04 container in html. +# +# The container engine can be selected by exporting CONTAINERCMD in the +# environment. The default is docker, but podman can also be used. + +set -eu -o pipefail + +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) +CONTAINERCMD=${CONTAINERCMD:-docker} +DOCS_DIR="$SCRIPT_DIR/../.." +SH_DIR="$SCRIPT_DIR/host_packages_scripts" + +main () +{ + local image="$1" + shift + + OCI=$(which "$CONTAINERCMD") + + # docker build doesn't accept 2 colons, so "sanitize" the name + local sanitized_dockername + sanitized_dockername=$(echo "$image" | tr ':.' '-') + + local version + version=$(echo "$image" | awk -F: '{print $NF}') + + case $image in + almalinux*) + containerfile=Containerfile.almalinux + docs=almalinux_docs.sh + docs_pdf=almalinux_docs_pdf.sh + pip3=pip3_docs.sh + ;; + debian*) + containerfile=Containerfile.debian + docs=ubuntu_docs.sh + docs_pdf=ubuntu_docs_pdf.sh + ;; + fedora*) + containerfile=Containerfile.fedora + docs=fedora_docs.sh + docs_pdf=fedora_docs_pdf.sh + pip3=pip3_docs.sh + ;; + opensuse* | leap*) + image=opensuse/leap:$version + containerfile=Containerfile.zypper + docs=opensuse_docs.sh + docs_pdf=opensuse_docs_pdf.sh + pip3=pip3_docs.sh + ;; + ubuntu*) + containerfile=Containerfile.ubuntu + docs=ubuntu_docs.sh + docs_pdf=ubuntu_docs_pdf.sh + ;; + *) + echo "$image not supported" + exit 1 + ;; + esac + + $OCI build \ + --tag "yocto-docs-$sanitized_dockername:latest" \ + --build-arg ARG_FROM="docker.io/$image" \ + --build-arg DOCS="$docs" \ + --build-arg DOCS_PDF="$docs_pdf" \ + --build-arg PIP3="${pip3:-}" \ + --file "$SCRIPT_DIR/$containerfile" \ + "$SH_DIR/" + + local -a args_run=( + --rm + --interactive + --tty + --volume="$DOCS_DIR:/docs:rw" + --workdir=/docs + --security-opt label=disable + ) + + if [ "$OCI" = "docker" ]; then + args_run+=( + --user="$(id -u)":"$(id -g)" + ) + elif [ "$OCI" = "podman" ]; then + # we need net access to fetch bitbake terms + args_run+=( + --cap-add=NET_RAW + --userns=keep-id + ) + fi + + $OCI run \ + "${args_run[@]}" \ + "yocto-docs-$sanitized_dockername" \ + "$@" +} + +main "$@" diff --git a/documentation/tools/host_packages_scripts/opensuse_docs.sh b/documentation/tools/host_packages_scripts/opensuse_docs.sh index 7d36eb0f99668168dc4fed44d71181338bdc59a3..896ad52b5042a9679555a30e582fa32076c3a3e7 100644 --- a/documentation/tools/host_packages_scripts/opensuse_docs.sh +++ b/documentation/tools/host_packages_scripts/opensuse_docs.sh @@ -1 +1 @@ -sudo zypper install git glibc-i18ndata make python3-pip rsvg-convert which +sudo zypper --non-interactive install git glibc-i18ndata make python3-pip rsvg-convert which diff --git a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh index ee9f07886ce1c0eac8e1d1118cda8ddc496f0139..f8a3b800d9790c0434f0c3f577ea1916a7fe1b28 100644 --- a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh +++ b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh @@ -1 +1 @@ -sudo zypper install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex +sudo zypper --non-interactive install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex diff --git a/documentation/tools/host_packages_scripts/opensuse_essential.sh b/documentation/tools/host_packages_scripts/opensuse_essential.sh index a784f4a5dc05f3734177f08ed97663a04d20a978..19e85f4a736532b890fc55492721cf345c2b4083 100644 --- a/documentation/tools/host_packages_scripts/opensuse_essential.sh +++ b/documentation/tools/host_packages_scripts/opensuse_essential.sh @@ -1,2 +1,2 @@ -sudo zypper install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd +sudo zypper --non-interactive install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd sudo pip3 install GitPython