From patchwork Mon Jan 27 18:37:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Schulz X-Patchwork-Id: 56170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77D27C0218A for ; Mon, 27 Jan 2025 18:37:32 +0000 (UTC) Received: from smtp-42ac.mail.infomaniak.ch (smtp-42ac.mail.infomaniak.ch [84.16.66.172]) by mx.groups.io with SMTP id smtpd.web11.848.1738003043872412558 for ; Mon, 27 Jan 2025 10:37:24 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: 0leil.net, ip: 84.16.66.172, mailfrom: foss+yocto@0leil.net) Received: from smtp-3-0000.mail.infomaniak.ch (smtp-3-0000.mail.infomaniak.ch [10.4.36.107]) by smtp-3-3000.mail.infomaniak.ch (Postfix) with ESMTPS id 4YhcbQ2ZrGztXZ; Mon, 27 Jan 2025 19:37:22 +0100 (CET) Received: from unknown by smtp-3-0000.mail.infomaniak.ch (Postfix) with ESMTPA id 4YhcbP6GBJz7p5; Mon, 27 Jan 2025 19:37:21 +0100 (CET) From: Quentin Schulz Date: Mon, 27 Jan 2025 19:37:04 +0100 Subject: [PATCH v3 1/2] docs: use literalinclude for system requirements MIME-Version: 1.0 Message-Id: <20250127-instructions-shell-container-v3-1-9373d3f156f5@cherry.de> References: <20250127-instructions-shell-container-v3-0-9373d3f156f5@cherry.de> In-Reply-To: <20250127-instructions-shell-container-v3-0-9373d3f156f5@cherry.de> To: Quentin Schulz , docs@lists.yoctoproject.org X-Mailer: b4 0.14.2 X-Infomaniak-Routing: alpha List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 27 Jan 2025 18:37:32 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/6200 From: Quentin Schulz The YAML variables for the host dependencies are updated by hand and actually only used inside code blocks. Let's migrate all instructions into separate shell scripts that are then literalinclude'd into the Sphinx documentation. This allows a few things: - ability to run shellcheck on the scripts if we ever want to - manually calling the appropriate script from a supported distro to build stuff (distro or bitbake/yocto stuff) - use this script to create containers to do CI of documentation on different distros, to make sure our instructions are all up to date, Signed-off-by: Quentin Schulz --- documentation/poky.yaml.in | 234 --------------------- documentation/ref-manual/system-requirements.rst | 82 +++++--- .../tools/host_packages_scripts/almalinux_docs.sh | 1 + .../host_packages_scripts/almalinux_docs_pdf.sh | 1 + .../host_packages_scripts/almalinux_essential.sh | 5 + .../tools/host_packages_scripts/fedora_docs.sh | 1 + .../tools/host_packages_scripts/fedora_docs_pdf.sh | 1 + .../host_packages_scripts/fedora_essential.sh | 1 + .../tools/host_packages_scripts/opensuse_docs.sh | 1 + .../host_packages_scripts/opensuse_docs_pdf.sh | 1 + .../host_packages_scripts/opensuse_essential.sh | 2 + .../tools/host_packages_scripts/pip3_docs.sh | 1 + .../tools/host_packages_scripts/ubuntu_docs.sh | 1 + .../tools/host_packages_scripts/ubuntu_docs_pdf.sh | 1 + .../host_packages_scripts/ubuntu_essential.sh | 1 + 15 files changed, 65 insertions(+), 269 deletions(-) diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in index d045ff596e6d7d7d46be373078e8bb8b38483267..c93b7806642b89fe78343513f0196c8b9318fc68 100644 --- a/documentation/poky.yaml.in +++ b/documentation/poky.yaml.in @@ -24,237 +24,3 @@ MIN_DISK_SPACE : "90" MIN_DISK_SPACE_RM_WORK : "40" # RAM (Gbytes) needed to generate qemux86-64 core-image-sato on Ubuntu 22.04 (x86-64) on a 4 core system MIN_RAM : "8" - -# -# Dependencies -# - -# Shared between distros -PIP3_HOST_PACKAGES_DOC: sphinx sphinx_rtd_theme pyyaml - -UBUNTU_DEBIAN_HOST_PACKAGES_ESSENTIAL: >- - build-essential - chrpath - cpio - debianutils - diffstat - file - gawk - gcc - git - iputils-ping - libacl1 - liblz4-tool - locales - python3 - python3-git - python3-jinja2 - python3-pexpect - python3-pip - python3-subunit - socat - texinfo - unzip - wget - xz-utils - zstd - -UBUNTU_DEBIAN_HOST_PACKAGES_DOC: >- - git - librsvg2-bin - locales - make - python3-saneyaml - python3-sphinx-rtd-theme - sphinx - -UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF: >- - fonts-freefont-otf - latexmk - tex-gyre - texlive-fonts-extra - texlive-fonts-recommended - texlive-lang-all - texlive-latex-extra - texlive-latex-recommended - texlive-xetex - -FEDORA_HOST_PACKAGES_ESSENTIAL: >- - bzip2 - ccache - chrpath - cpio - cpp - diffstat - diffutils - file - findutils - gawk - gcc - gcc-c++ - git - glibc-devel - glibc-langpack-en - gzip - hostname - libacl - lz4 - make - patch - perl - perl-Data-Dumper - perl-File-Compare - perl-File-Copy - perl-FindBin - perl-Text-ParseWords - perl-Thread-Queue - perl-bignum - perl-locale - python - python3 - python3-GitPython - python3-jinja2 - python3-pexpect - python3-pip - rpcgen - socat - tar - texinfo - unzip - wget - which - xz - zstd - -FEDORA_HOST_PACKAGES_DOC: >- - git - glibc-locale-source - librsvg2-tools - make - python3-pip - which - -FEDORA_HOST_PACKAGES_DOC_PDF: >- - 'texlive-collection-lang*' - latexmk - texlive-collection-fontsextra - texlive-collection-fontsrecommended - texlive-collection-latex - texlive-collection-latexextra - texlive-collection-latexrecommended - texlive-collection-xetex - texlive-fncychap - texlive-gnu-freefont - texlive-tex-gyre - texlive-xetex - -OPENSUSE_HOST_PACKAGES_ESSENTIAL: >- - bzip2 - chrpath - diffstat - gcc - gcc-c++ - git - gzip - hostname - libacl1 - lz4 - make - makeinfo - patch - python - python-curses - python-xml - python3 - python3-Jinja2 - python3-curses - python3-pexpect - python3-pip - rpcgen - socat - tar - wget - which - xz - zstd - -OPENSUSE_PIP3_HOST_PACKAGES_ESSENTIAL: GitPython - -OPENSUSE_HOST_PACKAGES_DOC: >- - git - glibc-i18ndata - make - python3-pip - rsvg-convert - which - -OPENSUSE_HOST_PACKAGES_DOC_PDF: >- - 'texlive-collection-lang*' - texlive-collection-fontsextra - texlive-collection-fontsrecommended - texlive-collection-latex - texlive-collection-latexextra - texlive-collection-latexrecommended - texlive-collection-xetex - texlive-fncychap - texlive-gnu-freefont - texlive-latexmk - texlive-tex-gyre - texlive-xetex - -ALMALINUX_HOST_PACKAGES_ESSENTIAL: >- - bzip2 - ccache - chrpath - cpio - cpp - diffstat - diffutils - gawk - gcc - gcc-c++ - git - glibc-devel - glibc-langpack-en - gzip - libacl - lz4 - make - patch - perl - perl-Data-Dumper - perl-Text-ParseWords - perl-Thread-Queue - python3 - python3-GitPython - python3-jinja2 - python3-pexpect - python3-pip - rpcgen - socat - tar - texinfo - unzip - wget - which - xz - zstd - -ALMALINUX_HOST_PACKAGES_DOC: >- - git - glibc-locale-source - librsvg2-tools - make - python3-pip - which - -ALMALINUX_HOST_PACKAGES_DOC_PDF: >- - latexmk - texlive-collection-fontsrecommended - texlive-collection-latex - texlive-collection-latexrecommended - texlive-collection-xetex - texlive-fncychap - texlive-gnu-freefont - texlive-tex-gyre - texlive-xetex diff --git a/documentation/ref-manual/system-requirements.rst b/documentation/ref-manual/system-requirements.rst index b087d374d23a71104dc9c050e96e4a2c20f5abb0..93b95a7fb251c4a0192c2d29a1dbf365eb69871e 100644 --- a/documentation/ref-manual/system-requirements.rst +++ b/documentation/ref-manual/system-requirements.rst @@ -158,9 +158,10 @@ Ubuntu and Debian ----------------- Here are the packages needed to build an image on a headless system -with a supported Ubuntu or Debian Linux distribution:: +with a supported Ubuntu or Debian Linux distribution: - $ sudo apt install &UBUNTU_DEBIAN_HOST_PACKAGES_ESSENTIAL; +.. literalinclude:: ../tools/host_packages_scripts/ubuntu_essential.sh + :language: shell You also need to ensure you have the ``en_US.UTF-8`` locale enabled:: @@ -189,64 +190,71 @@ If this is not the case, you can reconfigure the ``locales`` package to add it $ sudo apt build-dep qemu $ sudo apt remove oss4-dev -Here are the packages needed to build Project documentation manuals:: +Here are the packages needed to build Project documentation manuals: - $ sudo apt install &UBUNTU_DEBIAN_HOST_PACKAGES_DOC; +.. literalinclude:: ../tools/host_packages_scripts/ubuntu_docs.sh + :language: shell In addition to the previous packages, here are the packages needed to build the -documentation in PDF format:: +documentation in PDF format: - $ sudo apt install &UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF; +.. literalinclude:: ../tools/host_packages_scripts/ubuntu_docs_pdf.sh + :language: shell Fedora Packages --------------- Here are the packages needed to build an image on a headless system -with a supported Fedora Linux distribution:: +with a supported Fedora Linux distribution: - $ sudo dnf install &FEDORA_HOST_PACKAGES_ESSENTIAL; +.. literalinclude:: ../tools/host_packages_scripts/fedora_essential.sh + :language: shell -Here are the packages needed to build Project documentation manuals:: +Here are the packages needed to build Project documentation manuals: - $ sudo dnf install &FEDORA_HOST_PACKAGES_DOC; - $ sudo pip3 install &PIP3_HOST_PACKAGES_DOC; +.. literalinclude:: ../tools/host_packages_scripts/fedora_docs.sh + :language: shell + +.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh + :language: shell In addition to the previous packages, here are the packages needed to build the -documentation in PDF format:: +documentation in PDF format: - $ sudo dnf install &FEDORA_HOST_PACKAGES_DOC_PDF; +.. literalinclude:: ../tools/host_packages_scripts/fedora_docs_pdf.sh + :language: shell openSUSE Packages ----------------- Here are the packages needed to build an image on a headless system -with a supported openSUSE distribution:: +with a supported openSUSE distribution: - $ sudo zypper install &OPENSUSE_HOST_PACKAGES_ESSENTIAL; - $ sudo pip3 install &OPENSUSE_PIP3_HOST_PACKAGES_ESSENTIAL; +.. literalinclude:: ../tools/host_packages_scripts/opensuse_essential.sh + :language: shell -Here are the packages needed to build Project documentation manuals:: +Here are the packages needed to build Project documentation manuals: - $ sudo zypper install &OPENSUSE_HOST_PACKAGES_DOC; - $ sudo pip3 install &PIP3_HOST_PACKAGES_DOC; +.. literalinclude:: ../tools/host_packages_scripts/opensuse_docs.sh + :language: shell + +.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh + :language: shell In addition to the previous packages, here are the packages needed to build the -documentation in PDF format:: - - $ sudo zypper install &OPENSUSE_HOST_PACKAGES_DOC_PDF; +documentation in PDF format: +.. literalinclude:: ../tools/host_packages_scripts/opensuse_docs_pdf.sh + :language: shell AlmaLinux Packages ------------------ Here are the packages needed to build an image on a headless system -with a supported AlmaLinux distribution:: +with a supported AlmaLinux distribution: - $ sudo dnf install -y epel-release - $ sudo yum install dnf-plugins-core - $ sudo dnf config-manager --set-enabled crb - $ sudo dnf makecache - $ sudo dnf install &ALMALINUX_HOST_PACKAGES_ESSENTIAL; +.. literalinclude:: ../tools/host_packages_scripts/almalinux_essential.sh + :language: shell .. note:: @@ -261,15 +269,13 @@ with a supported AlmaLinux distribution:: - The ``makecache`` command consumes additional Metadata from ``epel-release``. -Here are the packages needed to build Project documentation manuals:: +Here are the packages needed to build Project documentation manuals: - $ sudo dnf install &ALMALINUX_HOST_PACKAGES_DOC; - $ sudo pip3 install &PIP3_HOST_PACKAGES_DOC; +.. literalinclude:: ../tools/host_packages_scripts/almalinux_docs.sh + :language: shell -In addition to the previous packages, here are the packages needed to build the -documentation in PDF format:: - - $ sudo dnf install &ALMALINUX_HOST_PACKAGES_DOC_PDF; +.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh + :language: shell .. warning:: @@ -278,6 +284,12 @@ documentation in PDF format:: ``texlive-collection-latexextra``, so you may run into issues. These may be installed using `tlmgr `_. +In addition to the previous packages, here are the packages needed to build the +documentation in PDF format: + +.. literalinclude:: ../tools/host_packages_scripts/almalinux_docs_pdf.sh + :language: shell + .. _system-requirements-buildtools: Required Git, tar, Python, make and gcc Versions diff --git a/documentation/tools/host_packages_scripts/almalinux_docs.sh b/documentation/tools/host_packages_scripts/almalinux_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..8188d529a1e9b8034e5b14ed4e9e633e7ae0600b --- /dev/null +++ b/documentation/tools/host_packages_scripts/almalinux_docs.sh @@ -0,0 +1 @@ +sudo dnf install git glibc-locale-source librsvg2-tools make python3-pip which diff --git a/documentation/tools/host_packages_scripts/almalinux_docs_pdf.sh b/documentation/tools/host_packages_scripts/almalinux_docs_pdf.sh new file mode 100644 index 0000000000000000000000000000000000000000..8341eb8c252d50b8a5cd3ff09ff013b1d49249bf --- /dev/null +++ b/documentation/tools/host_packages_scripts/almalinux_docs_pdf.sh @@ -0,0 +1 @@ +sudo dnf install latexmk texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-tex-gyre texlive-xetex diff --git a/documentation/tools/host_packages_scripts/almalinux_essential.sh b/documentation/tools/host_packages_scripts/almalinux_essential.sh new file mode 100644 index 0000000000000000000000000000000000000000..16f25fa1f29c84d512df97988f527d1ea834bea7 --- /dev/null +++ b/documentation/tools/host_packages_scripts/almalinux_essential.sh @@ -0,0 +1,5 @@ +sudo dnf install -y epel-release +sudo yum install dnf-plugins-core +sudo dnf config-manager --set-enabled crb +sudo dnf makecache +sudo dnf install bzip2 ccache chrpath cpio cpp diffstat diffutils gawk gcc gcc-c++ git glibc-devel glibc-langpack-en gzip libacl lz4 make patch perl perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3 python3-GitPython python3-jinja2 python3-pexpect python3-pip rpcgen socat tar texinfo unzip wget which xz zstd diff --git a/documentation/tools/host_packages_scripts/fedora_docs.sh b/documentation/tools/host_packages_scripts/fedora_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..8188d529a1e9b8034e5b14ed4e9e633e7ae0600b --- /dev/null +++ b/documentation/tools/host_packages_scripts/fedora_docs.sh @@ -0,0 +1 @@ +sudo dnf install git glibc-locale-source librsvg2-tools make python3-pip which diff --git a/documentation/tools/host_packages_scripts/fedora_docs_pdf.sh b/documentation/tools/host_packages_scripts/fedora_docs_pdf.sh new file mode 100644 index 0000000000000000000000000000000000000000..816f1c36ec35e046782d341bb70dff1834bfd8a1 --- /dev/null +++ b/documentation/tools/host_packages_scripts/fedora_docs_pdf.sh @@ -0,0 +1 @@ +sudo dnf install 'texlive-collection-lang*' latexmk texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-tex-gyre texlive-xetex diff --git a/documentation/tools/host_packages_scripts/fedora_essential.sh b/documentation/tools/host_packages_scripts/fedora_essential.sh new file mode 100644 index 0000000000000000000000000000000000000000..4841853e1e224683b563c8d6112446f0633a081b --- /dev/null +++ b/documentation/tools/host_packages_scripts/fedora_essential.sh @@ -0,0 +1 @@ +sudo dnf install bzip2 ccache chrpath cpio cpp diffstat diffutils file findutils gawk gcc gcc-c++ git glibc-devel glibc-langpack-en gzip hostname libacl lz4 make patch perl perl-Data-Dumper perl-File-Compare perl-File-Copy perl-FindBin perl-Text-ParseWords perl-Thread-Queue perl-bignum perl-locale python python3 python3-GitPython python3-jinja2 python3-pexpect python3-pip rpcgen socat tar texinfo unzip wget which xz zstd diff --git a/documentation/tools/host_packages_scripts/opensuse_docs.sh b/documentation/tools/host_packages_scripts/opensuse_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..7d36eb0f99668168dc4fed44d71181338bdc59a3 --- /dev/null +++ b/documentation/tools/host_packages_scripts/opensuse_docs.sh @@ -0,0 +1 @@ +sudo zypper install git glibc-i18ndata make python3-pip rsvg-convert which diff --git a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh new file mode 100644 index 0000000000000000000000000000000000000000..ee9f07886ce1c0eac8e1d1118cda8ddc496f0139 --- /dev/null +++ b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh @@ -0,0 +1 @@ +sudo zypper install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex diff --git a/documentation/tools/host_packages_scripts/opensuse_essential.sh b/documentation/tools/host_packages_scripts/opensuse_essential.sh new file mode 100644 index 0000000000000000000000000000000000000000..a784f4a5dc05f3734177f08ed97663a04d20a978 --- /dev/null +++ b/documentation/tools/host_packages_scripts/opensuse_essential.sh @@ -0,0 +1,2 @@ +sudo zypper install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd +sudo pip3 install GitPython diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..fd6ad98053fb6aa1ff55669191dc99578c830455 --- /dev/null +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh @@ -0,0 +1 @@ +sudo pip3 install sphinx sphinx_rtd_theme pyyaml diff --git a/documentation/tools/host_packages_scripts/ubuntu_docs.sh b/documentation/tools/host_packages_scripts/ubuntu_docs.sh new file mode 100644 index 0000000000000000000000000000000000000000..d2a1832e279057519321a1fb7820efd60d886564 --- /dev/null +++ b/documentation/tools/host_packages_scripts/ubuntu_docs.sh @@ -0,0 +1 @@ +sudo apt install git librsvg2-bin locales make python3-saneyaml python3-saneyaml python3-sphinx-rtd-theme sphinx diff --git a/documentation/tools/host_packages_scripts/ubuntu_docs_pdf.sh b/documentation/tools/host_packages_scripts/ubuntu_docs_pdf.sh new file mode 100644 index 0000000000000000000000000000000000000000..d6310bd21deae81df32d0cb843aab4a5622ac4fa --- /dev/null +++ b/documentation/tools/host_packages_scripts/ubuntu_docs_pdf.sh @@ -0,0 +1 @@ +sudo apt install fonts-freefont-otf latexmk tex-gyre texlive-fonts-extra texlive-fonts-recommended texlive-lang-all texlive-latex-extra texlive-latex-recommended texlive-xetex diff --git a/documentation/tools/host_packages_scripts/ubuntu_essential.sh b/documentation/tools/host_packages_scripts/ubuntu_essential.sh new file mode 100644 index 0000000000000000000000000000000000000000..f3388c88c93d885e3b035651e434c669654ee241 --- /dev/null +++ b/documentation/tools/host_packages_scripts/ubuntu_essential.sh @@ -0,0 +1 @@ +sudo apt-get install build-essential chrpath cpio debianutils diffstat file gawk gcc git iputils-ping libacl1 liblz4-tool locales python3 python3-git python3-jinja2 python3-pexpect python3-pip python3-subunit socat texinfo unzip wget xz-utils zstd From patchwork Mon Jan 27 18:37:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Quentin Schulz X-Patchwork-Id: 56171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78653C02191 for ; Mon, 27 Jan 2025 18:37:32 +0000 (UTC) Received: from smtp-42a8.mail.infomaniak.ch (smtp-42a8.mail.infomaniak.ch [84.16.66.168]) by mx.groups.io with SMTP id smtpd.web11.849.1738003044576999054 for ; Mon, 27 Jan 2025 10:37:25 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: 0leil.net, ip: 84.16.66.168, mailfrom: foss+yocto@0leil.net) Received: from smtp-3-0000.mail.infomaniak.ch (smtp-3-0000.mail.infomaniak.ch [10.4.36.107]) by smtp-3-3000.mail.infomaniak.ch (Postfix) with ESMTPS id 4YhcbQ54nzztXc; Mon, 27 Jan 2025 19:37:22 +0100 (CET) Received: from unknown by smtp-3-0000.mail.infomaniak.ch (Postfix) with ESMTPA id 4YhcbQ2Tvbz6yy; Mon, 27 Jan 2025 19:37:22 +0100 (CET) From: Quentin Schulz Date: Mon, 27 Jan 2025 19:37:05 +0100 Subject: [PATCH v3 2/2] tools: add script for building documentation inside containers MIME-Version: 1.0 Message-Id: <20250127-instructions-shell-container-v3-2-9373d3f156f5@cherry.de> References: <20250127-instructions-shell-container-v3-0-9373d3f156f5@cherry.de> In-Reply-To: <20250127-instructions-shell-container-v3-0-9373d3f156f5@cherry.de> To: Quentin Schulz , docs@lists.yoctoproject.org X-Mailer: b4 0.14.2 X-Infomaniak-Routing: alpha List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 27 Jan 2025 18:37:32 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/6201 From: Quentin Schulz This adds a script for building a container and building the documentation within that new container image. The openSUSE instructions now require a --non-interactive flag otherwise they fail to run. Sadly there doesn't seem to be a way to have this in an environment variable à-la DEBIAN_FRONTEND=noninteractive, so we simply do a sed on the scripts to add --non-interactive to the zypper commands to avoid having those in the instructions provided to our users. Somehow tzdata package in Ubuntu doesn't respect DEBIAN_FRONTEND=noninteractive hence why the timezone needs to be set by hand. Signed-off-by: Quentin Schulz --- documentation/tools/Containerfile.almalinux | 1 + documentation/tools/Containerfile.apt | 26 +++++ documentation/tools/Containerfile.debian | 1 + documentation/tools/Containerfile.dnf | 25 ++++ documentation/tools/Containerfile.fedora | 1 + documentation/tools/Containerfile.ubuntu | 1 + documentation/tools/Containerfile.zypper | 32 +++++ documentation/tools/build-docs-container | 175 ++++++++++++++++++++++++++++ 8 files changed, 262 insertions(+) diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux new file mode 120000 index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 --- /dev/null +++ b/documentation/tools/Containerfile.almalinux @@ -0,0 +1 @@ +Containerfile.dnf \ No newline at end of file diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt new file mode 100644 index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517 --- /dev/null +++ b/documentation/tools/Containerfile.apt @@ -0,0 +1,26 @@ +ARG ARG_FROM=debian:12 # default value to avoid warning +FROM $ARG_FROM + +ARG DOCS=ubuntu_docs.sh +ARG DOCS_PDF=ubuntu_docs_pdf.sh + +ENV DEBIAN_FRONTEND=noninteractive +ARG TZ=Europe/Vienna + +# relative to the location of the dockerfile +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh + +RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \ + && apt-get update \ + && apt-get install -y sudo \ + && yes | /temp/host_packages_docs.sh \ + && yes | /temp/host_packages_docs_pdf.sh \ + && apt-get --yes autoremove \ + && apt-get clean \ + && rm -rf /temp + +RUN git config --global --add safe.directory /docs + +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] +CMD ["publish"] diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian new file mode 120000 index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a --- /dev/null +++ b/documentation/tools/Containerfile.debian @@ -0,0 +1 @@ +Containerfile.apt \ No newline at end of file diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf new file mode 100644 index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886 --- /dev/null +++ b/documentation/tools/Containerfile.dnf @@ -0,0 +1,25 @@ +ARG ARG_FROM=fedora:40 # default value to avoid warning +FROM $ARG_FROM + +ARG DOCS=fedora_docs.sh +ARG DOCS_PDF=fedora_docs_pdf.sh +ARG PIP3=pip3_docs.sh + +# relative to the location of the dockerfile +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh + +RUN dnf update -y \ + && dnf install -y sudo \ + && yes | /temp/host_packages_docs.sh \ + && yes | /temp/host_packages_docs_pdf.sh \ + && yes | /temp/pip3_docs.sh \ + && dnf autoremove -y \ + && dnf clean all -y \ + && rm -rf /temp + +RUN git config --global --add safe.directory /docs + +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] +CMD ["publish"] diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora new file mode 120000 index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74 --- /dev/null +++ b/documentation/tools/Containerfile.fedora @@ -0,0 +1 @@ +Containerfile.dnf \ No newline at end of file diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu new file mode 120000 index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a --- /dev/null +++ b/documentation/tools/Containerfile.ubuntu @@ -0,0 +1 @@ +Containerfile.apt \ No newline at end of file diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper new file mode 100644 index 0000000000000000000000000000000000000000..f27ad1b476999f4f2cebb47b89025b1c9f89cec1 --- /dev/null +++ b/documentation/tools/Containerfile.zypper @@ -0,0 +1,32 @@ +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning +FROM $ARG_FROM + +ARG DOCS=opensuse_docs.sh +ARG DOCS_PDF=opensuse_docs_pdf.sh +ARG PIP3=pip3_docs.sh + +# relative to the location of the dockerfile +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh + +# Zypper doesn't have environment variables to specify whether to run in +# non-interactive mode like Debian does with DEBIAN_FRONTEND and piping yes to +# the scripts doesn't need to be enough as well, so let's force all zypper calls +# to be non-interactive by adding the appropriate flag in the scripts. +RUN for script in /temp/*.sh; do \ + sed -i 's/zypper/zypper --non-interactive/' $script; \ + done + +RUN zypper update -y \ + && zypper install -y sudo \ + && yes | /temp/host_packages_docs.sh \ + && yes | /temp/host_packages_docs_pdf.sh \ + && yes | /temp/pip3_docs.sh \ + && zypper clean --all \ + && rm -rf /temp + +RUN git config --global --add safe.directory /docs + +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"] +CMD ["publish"] diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container new file mode 100755 index 0000000000000000000000000000000000000000..6b4d4254342bd00a1c1ba6758075bd1720334888 --- /dev/null +++ b/documentation/tools/build-docs-container @@ -0,0 +1,175 @@ +#!/usr/bin/env bash +# -*- vim: set expandtab tabstop=2 shiftwidth=2: +# +# Build a container ready to build the documentation be reading the dependencies +# listed in shell scripts in documentation/tools/host_packages_scripts, and +# start a documentation build in this container. +# +# Usage: +# +# ./documentation/tools/build-docs-container [] +# +# e.g.: +# +# ./documentation/tools/build-docs-container ubuntu:24.04 html +# +# Will build the docs in an Ubuntu 24.04 container in html. +# +# The container engine can be selected by exporting CONTAINERCMD in the +# environment. The default is docker, but podman can also be used. + +set -eu -o pipefail + +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) +CONTAINERCMD=${CONTAINERCMD:-docker} +DOCS_DIR="$SCRIPT_DIR/../.." +SH_DIR="$SCRIPT_DIR/host_packages_scripts" + +function usage() +{ + echo "$0 -- script to build documentation from within a container + +$0 OCI_IMAGE [make arguments...] + + OCI_IMAGE is an image:tag of an OCI image hosted on hub.docker.com. It is one + of: + - debian:12 + - fedora:38 + - fedora:39 + - fedora:40 + - leap:15.4 + - leap:15.5 + - ubuntu:22.04 + - ubuntu:24.04 + + [make arguments] is one or more argument to pass to the make command of + documentation/Makefile, see that file for what's supported. This is typically + intended to be used to provide specific make targets. + Default: publish +" +} + +main () +{ + if [ "$#" -lt 1 ]; then + usage + exit 1 + fi + + local image="$1" + shift + + OCI=$(which "$CONTAINERCMD") + + # docker build doesn't accept 2 colons, so "sanitize" the name + local sanitized_dockername + sanitized_dockername=$(echo "$image" | tr ':.' '-') + + local version + version=$(echo "$image" | awk -F: '{print $NF}') + + case $image in + # Missing latexmk texlive-gnu-freefont packages at the very least + # "almalinux:8"*|\ + # "almalinux:9"*) + # containerfile=Containerfile.almalinux + # docs=almalinux_docs.sh + # docs_pdf=almalinux_docs_pdf.sh + # pip3=pip3_docs.sh + # ;; + # Missing python3-saneyaml + # "debian:11"*|\ + "debian:12"*) + containerfile=Containerfile.debian + docs=ubuntu_docs.sh + docs_pdf=ubuntu_docs_pdf.sh + ;; + "fedora:38"*|\ + "fedora:39"*|\ + "fedora:40"*) + containerfile=Containerfile.fedora + docs=fedora_docs.sh + docs_pdf=fedora_docs_pdf.sh + pip3=pip3_docs.sh + ;; + "leap:15.4"*|\ + "leap:15.5"*) + # Seems like issue with permissions package, c.f. + # + # Updating /etc/sysconfig/security ... + # /dev/zero: chown: Permission denied + # /dev/null: chown: Permission denied + # /dev/full: chown: Permission denied + # ERROR: not all operations were successful. + # Checking permissions and ownerships - using the permissions files + # /etc/permissions + # /etc/permissions.easy + # /etc/permissions.local + # setting / to root:root 0755. (wrong permissions 0555) + # setting /dev/zero to root:root 0666. (wrong owner/group 65534:65534) + # setting /dev/null to root:root 0666. (wrong owner/group 65534:65534) + # setting /dev/full to root:root 0666. (wrong owner/group 65534:65534) + # warning: %post(permissions-20240826-150600.10.12.1.x86_64) scriptlet failed, exit status 1 + # + # "leap:15.6"*) + image=opensuse/leap:$version + containerfile=Containerfile.zypper + docs=opensuse_docs.sh + docs_pdf=opensuse_docs_pdf.sh + pip3=pip3_docs.sh + ;; + # Missing python3-saneyaml + # "ubuntu:18.04"*|\ + # "ubuntu:20.04"*|\ + # Cannot fetch packages anymore + # "ubuntu:23.04"*|\ + "ubuntu:22.04"*|\ + "ubuntu:24.04"*) + containerfile=Containerfile.ubuntu + docs=ubuntu_docs.sh + docs_pdf=ubuntu_docs_pdf.sh + ;; + *) + echo "$image not supported!" + usage + exit 1 + ;; + esac + + $OCI build \ + --tag "yocto-docs-$sanitized_dockername:latest" \ + --build-arg ARG_FROM="docker.io/$image" \ + --build-arg DOCS="$docs" \ + --build-arg DOCS_PDF="$docs_pdf" \ + --build-arg PIP3="${pip3:-}" \ + --file "$SCRIPT_DIR/$containerfile" \ + "$SH_DIR/" + + local -a args_run=( + --rm + --interactive + --tty + --volume="$DOCS_DIR:/docs:rw" + --workdir=/docs + --security-opt label=disable + ) + + if [ "$OCI" = "docker" ]; then + args_run+=( + --user="$(id -u)":"$(id -g)" + ) + elif [ "$OCI" = "podman" ]; then + # we need net access to fetch bitbake terms + args_run+=( + --cap-add=NET_RAW + --userns=keep-id + ) + fi + + $OCI run \ + "${args_run[@]}" \ + "yocto-docs-$sanitized_dockername" \ + "$@" +} + +main "$@"