| Message ID | 20251202-fix-make-multi-target-v2-1-7eea1d64cf6b@cherry.de |
|---|---|
| State | Under Review |
| Headers | show |
| Series | [v2] convert SVGs to PDF and PNG using sphinxcontrib.rsvgconverter plugin | expand |
Hi Antonin, On 12/2/25 6:06 PM, Quentin Schulz wrote: > From: Quentin Schulz <quentin.schulz@cherry.de> > > The sphinxcontrib.rsvgconverter plugin allows to generate a PDF or PNG > from an SVG. > This is what we already do manually via Make targets but it isn't good > enough. > > Sphinx generates a cache upon first parsing and stores it in a doctrees > directory. Sphinx claims that it can be shared between all builders[1]. > For glob patterns in image or figure directives, Sphinx will look for > all possible matches in the source tree and store those in a > "candidates" map for each file, along with the associated mimetype. When > building, Sphinx will then look in this map to try to find a match in > the current builder's supported_image_types. If none is found, the build > will fail. > > The latexpdf (using the LaTeXBuilder) target does not support SVGs by > default[2]. We used Make to generate PDFs from them before generating > the doc PDF though (see PDFs variable and %.pdf in Makefile) and that > type is supported by default[2]. > > The epub (via the Epub3Builder) target does support SVGs by default[3] > but we disabled their support in commit ff3876ca4910 ("conf.py: use PNG > first in EPUB output"). We used Make to generate PNGs from them before > generating the doc epub though (see PNGs variable and %.png in Makefile) > and that type is supported (c.f. Epub3Builder.supported_image_types in > our conf.py). > > The issue is that this is done transparently from Sphinx. When we > generate the PDFs or PNGs variants of the SVGs, we put them in-tree > directly along their source file. Then, when caching, Sphinx will find > both the source file and the appropriate variant. However, the cache > isn't updated if there are new files in the tree and the source rST > files aren't modified. So, the cache will not have its map updated and > we won't be able to find the new variant when building for a > non-SVG-compatible builder. Take the following scenario: > > - start from a clean source file (fresh clone or git clean -ffdx) > - build the html target (which supports SVGs by default[4]) > - sphinx will find all the files in the source tree matching the glob > pattern in ".. image:: test.*", in our case only an SVG since the PDFs > and PNGs are only generated for the latexpdf and epub targets > respectively. The cache will only store the path to the SVG file > because it is the only source file that matches the glob, > - attempt to build the epub target (which doesn't support SVGs, only > PNGs) > - Sphinx checks for the file to include for '.. image:: test.*' and > finds an SVG in the cache map and then check the list of supported > image types for the Epub3Builder and find that it doesn't support > that. It cannot find anything satisfying the dependency and thus fails > to build. > > This scenario can easily be reproduced by running the `make all` command > since the html builder will be used first, then epub and finally > latexpdf. > > The `make publish` target works by chance, because the epub builder is > built first and will cache SVG + PNG for each glob, then the latexpdf > builder is built and supports PNGs so that's fine and then html, which > supports SVG as well. > > To fix this issue, we could simply always generate PDFs and PNGs of all > SVGs in the source tree, but this isn't ideal. > > Instead, let's use an ImageConverter from a Sphinx plugin. This allows > to map a plugin as being able to generate a file of type Y from a file > of type X. When Sphinx wants to build an image, it'll try to find the > image with the type the current builder supports in the cache. If it > cannot, it's going to try to find an ImageConverter plugin that is able > to convert one of the image types in cache with one of the image types > the current builder supports. Then Sphinx will call this plugin to > generate the file and put it into the build directory (not in the > source!). > > This allows to simplify the Makefile as well and is a much cleaner > approach. > > The epub target is removed as the catch-all target contains the same > instructions as the epub target we remove in this commit. > > sphinxcontrib.rsvgconverter is a plugin from > sphinxcontrib-svg2pdfconverter python module. SVGs to PNGs is only > supported since 2.0.0. > > [1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-d > [2] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.latex.LaTeXBuilder.supported_image_types > [3] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.epub3.Epub3Builder.supported_image_types > [4] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.html.StandaloneHTMLBuilder > > Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de> > --- > This depends on > https://lore.kernel.org/openembedded-core/20251202-svg2pdf-v1-0-bf4bd322e528@cherry.de/ > in OE-Core for still being able to build the docs with the > buildtools-docs-tarball SDK. This has merged as far as I could tell, c.f. https://git.openembedded.org/openembedded-core/commit/meta/recipes-core/meta/buildtools-docs-tarball.bb?id=056686eaf1a3a27f5e83537e5a1b7c837bc16fb9 so this can now be freely reviewed and/or merged, thanks! Cheers, Quentin
Hi, On Mon Dec 15, 2025 at 11:58 AM CET, Quentin Schulz via lists.yoctoproject.org wrote: > Hi Antonin, > > On 12/2/25 6:06 PM, Quentin Schulz wrote: >> From: Quentin Schulz <quentin.schulz@cherry.de> >> >> The sphinxcontrib.rsvgconverter plugin allows to generate a PDF or PNG >> from an SVG. >> This is what we already do manually via Make targets but it isn't good >> enough. >> >> Sphinx generates a cache upon first parsing and stores it in a doctrees >> directory. Sphinx claims that it can be shared between all builders[1]. >> For glob patterns in image or figure directives, Sphinx will look for >> all possible matches in the source tree and store those in a >> "candidates" map for each file, along with the associated mimetype. When >> building, Sphinx will then look in this map to try to find a match in >> the current builder's supported_image_types. If none is found, the build >> will fail. >> >> The latexpdf (using the LaTeXBuilder) target does not support SVGs by >> default[2]. We used Make to generate PDFs from them before generating >> the doc PDF though (see PDFs variable and %.pdf in Makefile) and that >> type is supported by default[2]. >> >> The epub (via the Epub3Builder) target does support SVGs by default[3] >> but we disabled their support in commit ff3876ca4910 ("conf.py: use PNG >> first in EPUB output"). We used Make to generate PNGs from them before >> generating the doc epub though (see PNGs variable and %.png in Makefile) >> and that type is supported (c.f. Epub3Builder.supported_image_types in >> our conf.py). >> >> The issue is that this is done transparently from Sphinx. When we >> generate the PDFs or PNGs variants of the SVGs, we put them in-tree >> directly along their source file. Then, when caching, Sphinx will find >> both the source file and the appropriate variant. However, the cache >> isn't updated if there are new files in the tree and the source rST >> files aren't modified. So, the cache will not have its map updated and >> we won't be able to find the new variant when building for a >> non-SVG-compatible builder. Take the following scenario: >> >> - start from a clean source file (fresh clone or git clean -ffdx) >> - build the html target (which supports SVGs by default[4]) >> - sphinx will find all the files in the source tree matching the glob >> pattern in ".. image:: test.*", in our case only an SVG since the PDFs >> and PNGs are only generated for the latexpdf and epub targets >> respectively. The cache will only store the path to the SVG file >> because it is the only source file that matches the glob, >> - attempt to build the epub target (which doesn't support SVGs, only >> PNGs) >> - Sphinx checks for the file to include for '.. image:: test.*' and >> finds an SVG in the cache map and then check the list of supported >> image types for the Epub3Builder and find that it doesn't support >> that. It cannot find anything satisfying the dependency and thus fails >> to build. >> >> This scenario can easily be reproduced by running the `make all` command >> since the html builder will be used first, then epub and finally >> latexpdf. >> >> The `make publish` target works by chance, because the epub builder is >> built first and will cache SVG + PNG for each glob, then the latexpdf >> builder is built and supports PNGs so that's fine and then html, which >> supports SVG as well. >> >> To fix this issue, we could simply always generate PDFs and PNGs of all >> SVGs in the source tree, but this isn't ideal. >> >> Instead, let's use an ImageConverter from a Sphinx plugin. This allows >> to map a plugin as being able to generate a file of type Y from a file >> of type X. When Sphinx wants to build an image, it'll try to find the >> image with the type the current builder supports in the cache. If it >> cannot, it's going to try to find an ImageConverter plugin that is able >> to convert one of the image types in cache with one of the image types >> the current builder supports. Then Sphinx will call this plugin to >> generate the file and put it into the build directory (not in the >> source!). >> >> This allows to simplify the Makefile as well and is a much cleaner >> approach. >> >> The epub target is removed as the catch-all target contains the same >> instructions as the epub target we remove in this commit. >> >> sphinxcontrib.rsvgconverter is a plugin from >> sphinxcontrib-svg2pdfconverter python module. SVGs to PNGs is only >> supported since 2.0.0. >> >> [1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-d >> [2] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.latex.LaTeXBuilder.supported_image_types >> [3] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.epub3.Epub3Builder.supported_image_types >> [4] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.html.StandaloneHTMLBuilder >> >> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de> >> --- >> This depends on >> https://lore.kernel.org/openembedded-core/20251202-svg2pdf-v1-0-bf4bd322e528@cherry.de/ >> in OE-Core for still being able to build the docs with the >> buildtools-docs-tarball SDK. > > This has merged as far as I could tell, c.f. > https://git.openembedded.org/openembedded-core/commit/meta/recipes-core/meta/buildtools-docs-tarball.bb?id=056686eaf1a3a27f5e83537e5a1b7c837bc16fb9 > so this can now be freely reviewed and/or merged, thanks! Sadly, I can't merge this until [1] is updated. I was waiting for 5.3 to release to update this URL. The closest milestone that will include your new recipe is 6.0M1. This will be available in https://downloads.yoctoproject.org/releases/yocto/milestones/. So we'll have to delay this until this milestone, unless we manually hijack a custom built tarball before. [1]: https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/run-docs-build?id=0c0c7b41dea1b4f2da06bdf4cf4f7771ca33bcf7#n29 Antonin
Hi Antonin, On 12/16/25 10:09 AM, Antonin Godard wrote: > Hi, > > On Mon Dec 15, 2025 at 11:58 AM CET, Quentin Schulz via lists.yoctoproject.org wrote: >> Hi Antonin, >> >> On 12/2/25 6:06 PM, Quentin Schulz wrote: >>> From: Quentin Schulz <quentin.schulz@cherry.de> >>> >>> The sphinxcontrib.rsvgconverter plugin allows to generate a PDF or PNG >>> from an SVG. >>> This is what we already do manually via Make targets but it isn't good >>> enough. >>> >>> Sphinx generates a cache upon first parsing and stores it in a doctrees >>> directory. Sphinx claims that it can be shared between all builders[1]. >>> For glob patterns in image or figure directives, Sphinx will look for >>> all possible matches in the source tree and store those in a >>> "candidates" map for each file, along with the associated mimetype. When >>> building, Sphinx will then look in this map to try to find a match in >>> the current builder's supported_image_types. If none is found, the build >>> will fail. >>> >>> The latexpdf (using the LaTeXBuilder) target does not support SVGs by >>> default[2]. We used Make to generate PDFs from them before generating >>> the doc PDF though (see PDFs variable and %.pdf in Makefile) and that >>> type is supported by default[2]. >>> >>> The epub (via the Epub3Builder) target does support SVGs by default[3] >>> but we disabled their support in commit ff3876ca4910 ("conf.py: use PNG >>> first in EPUB output"). We used Make to generate PNGs from them before >>> generating the doc epub though (see PNGs variable and %.png in Makefile) >>> and that type is supported (c.f. Epub3Builder.supported_image_types in >>> our conf.py). >>> >>> The issue is that this is done transparently from Sphinx. When we >>> generate the PDFs or PNGs variants of the SVGs, we put them in-tree >>> directly along their source file. Then, when caching, Sphinx will find >>> both the source file and the appropriate variant. However, the cache >>> isn't updated if there are new files in the tree and the source rST >>> files aren't modified. So, the cache will not have its map updated and >>> we won't be able to find the new variant when building for a >>> non-SVG-compatible builder. Take the following scenario: >>> >>> - start from a clean source file (fresh clone or git clean -ffdx) >>> - build the html target (which supports SVGs by default[4]) >>> - sphinx will find all the files in the source tree matching the glob >>> pattern in ".. image:: test.*", in our case only an SVG since the PDFs >>> and PNGs are only generated for the latexpdf and epub targets >>> respectively. The cache will only store the path to the SVG file >>> because it is the only source file that matches the glob, >>> - attempt to build the epub target (which doesn't support SVGs, only >>> PNGs) >>> - Sphinx checks for the file to include for '.. image:: test.*' and >>> finds an SVG in the cache map and then check the list of supported >>> image types for the Epub3Builder and find that it doesn't support >>> that. It cannot find anything satisfying the dependency and thus fails >>> to build. >>> >>> This scenario can easily be reproduced by running the `make all` command >>> since the html builder will be used first, then epub and finally >>> latexpdf. >>> >>> The `make publish` target works by chance, because the epub builder is >>> built first and will cache SVG + PNG for each glob, then the latexpdf >>> builder is built and supports PNGs so that's fine and then html, which >>> supports SVG as well. >>> >>> To fix this issue, we could simply always generate PDFs and PNGs of all >>> SVGs in the source tree, but this isn't ideal. >>> >>> Instead, let's use an ImageConverter from a Sphinx plugin. This allows >>> to map a plugin as being able to generate a file of type Y from a file >>> of type X. When Sphinx wants to build an image, it'll try to find the >>> image with the type the current builder supports in the cache. If it >>> cannot, it's going to try to find an ImageConverter plugin that is able >>> to convert one of the image types in cache with one of the image types >>> the current builder supports. Then Sphinx will call this plugin to >>> generate the file and put it into the build directory (not in the >>> source!). >>> >>> This allows to simplify the Makefile as well and is a much cleaner >>> approach. >>> >>> The epub target is removed as the catch-all target contains the same >>> instructions as the epub target we remove in this commit. >>> >>> sphinxcontrib.rsvgconverter is a plugin from >>> sphinxcontrib-svg2pdfconverter python module. SVGs to PNGs is only >>> supported since 2.0.0. >>> >>> [1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-d >>> [2] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.latex.LaTeXBuilder.supported_image_types >>> [3] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.epub3.Epub3Builder.supported_image_types >>> [4] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.html.StandaloneHTMLBuilder >>> >>> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de> >>> --- >>> This depends on >>> https://lore.kernel.org/openembedded-core/20251202-svg2pdf-v1-0-bf4bd322e528@cherry.de/ >>> in OE-Core for still being able to build the docs with the >>> buildtools-docs-tarball SDK. >> >> This has merged as far as I could tell, c.f. >> https://git.openembedded.org/openembedded-core/commit/meta/recipes-core/meta/buildtools-docs-tarball.bb?id=056686eaf1a3a27f5e83537e5a1b7c837bc16fb9 >> so this can now be freely reviewed and/or merged, thanks! > > Sadly, I can't merge this until [1] is updated. I was waiting for 5.3 to release > to update this URL. The closest milestone that will include your new recipe is > 6.0M1. This will be available in https://downloads.yoctoproject.org/releases/yocto/milestones/. > Ah shoot, I had completely forgotten about the autobuilder. Sorry for the ping! > So we'll have to delay this until this milestone, unless we manually hijack a > custom built tarball before. > It's been broken for a few releases, one more isn't going to hurt anyone :) Cheers, Quentin
diff --git a/documentation/Makefile b/documentation/Makefile index bade78fe8..897db2b5a 100644 --- a/documentation/Makefile +++ b/documentation/Makefile @@ -10,11 +10,8 @@ VALEOPTS ?= --no-wrap --glob '!migration-guides/release-notes-*.rst' SOURCEDIR = . VALEDOCS ?= $(SOURCEDIR) SPHINXLINTDOCS ?= $(SOURCEDIR) -IMAGEDIRS = */svg BUILDDIR = _build DESTDIR = final -SVG2PNG = rsvg-convert -SVG2PDF = rsvg-convert ifeq ($(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi),0) $(error "The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed") @@ -24,7 +21,7 @@ endif help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) -.PHONY: all checks help Makefile clean stylecheck publish epub latexpdf +.PHONY: all checks help Makefile clean stylecheck publish latexpdf publish: Makefile checks epub latexpdf html singlehtml rm -rf $(BUILDDIR)/$(DESTDIR)/ @@ -35,22 +32,8 @@ publish: Makefile checks epub latexpdf html singlehtml cp $(BUILDDIR)/singlehtml/index.html $(BUILDDIR)/$(DESTDIR)/singleindex.html sed -i -e 's@index.html#@singleindex.html#@g' $(BUILDDIR)/$(DESTDIR)/singleindex.html -# Build a list of SVG files to convert to PDFs -PDFs := $(foreach dir, $(IMAGEDIRS), $(patsubst %.svg,%.pdf,$(wildcard $(SOURCEDIR)/$(dir)/*.svg))) - -# Build a list of SVG files to convert to PNGs -PNGs := $(foreach dir, $(IMAGEDIRS), $(patsubst %.svg,%.png,$(wildcard $(SOURCEDIR)/$(dir)/*.svg))) - -# Pattern rule for converting SVG to PDF -%.pdf : %.svg - $(SVG2PDF) --format=Pdf --output=$@ $< - -# Pattern rule for converting SVG to PNG -%.png : %.svg - $(SVG2PNG) --format=Png --output=$@ $< - clean: - @rm -rf $(BUILDDIR) $(PNGs) $(PDFs) poky.yaml sphinx-static/switchers.js releases.rst + @rm -rf $(BUILDDIR) poky.yaml sphinx-static/switchers.js releases.rst checks: $(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR) @@ -62,14 +45,10 @@ stylecheck: sphinx-lint: sphinx-lint $(SPHINXLINTDOCS) -epub: $(PNGs) - $(SOURCEDIR)/set_versions.py - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) - # Note: we need to pass buf_size here (which is also configurable from # texmf.cnf), to avoid following error: # Unable to read an entire line---bufsize=200000. Please increase buf_size in texmf.cnf. -latexpdf: $(PDFs) +latexpdf: $(SOURCEDIR)/set_versions.py buf_size=10000000 $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/documentation/Pipfile b/documentation/Pipfile index 7ee1d2290..1e186abb3 100644 --- a/documentation/Pipfile +++ b/documentation/Pipfile @@ -9,6 +9,8 @@ verify_ssl = true sphinx = "*" sphinx-rtd-theme = "*" pyyaml = "*" +# SVG to PNG only supported since 2.0.0 +sphinxcontrib-svg2pdfconverter = ">=2.0.0" [requires] python_version = "3" diff --git a/documentation/conf.py b/documentation/conf.py index c07b6c419..9ca8a5415 100644 --- a/documentation/conf.py +++ b/documentation/conf.py @@ -66,6 +66,7 @@ extensions = [ 'sphinx.ext.autosectionlabel', 'sphinx.ext.extlinks', 'sphinx.ext.intersphinx', + 'sphinxcontrib.rsvgconverter', 'yocto-vars' ] autosectionlabel_prefix_document = True diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh index fd6ad9805..4f92c2a98 100644 --- a/documentation/tools/host_packages_scripts/pip3_docs.sh +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh @@ -1 +1 @@ -sudo pip3 install sphinx sphinx_rtd_theme pyyaml +sudo pip3 install sphinx sphinx_rtd_theme pyyaml 'sphinxcontrib-svg2pdfconverter>=2.0.0'