diff mbox series

[v3] convert SVGs to PDF and PNG using sphinxcontrib.rsvgconverter plugin

Message ID 20260211-fix-make-multi-target-v3-1-1168a6f1525f@cherry.de
State Changes Requested
Headers show
Series [v3] convert SVGs to PDF and PNG using sphinxcontrib.rsvgconverter plugin | expand

Commit Message

Quentin Schulz Feb. 11, 2026, 3:25 p.m. UTC 
From: Quentin Schulz <quentin.schulz@cherry.de>

The sphinxcontrib.rsvgconverter plugin allows to generate a PDF or PNG
from an SVG.
This is what we already do manually via Make targets but it isn't good
enough.

Sphinx generates a cache upon first parsing and stores it in a doctrees
directory. Sphinx claims that it can be shared between all builders[1].
For glob patterns in image or figure directives, Sphinx will look for
all possible matches in the source tree and store those in a
"candidates" map for each file, along with the associated mimetype. When
building, Sphinx will then look in this map to try to find a match in
the current builder's supported_image_types. If none is found, the build
will fail.

The latexpdf (using the LaTeXBuilder) target does not support SVGs by
default[2]. We used Make to generate PDFs from them before generating
the doc PDF though (see PDFs variable and %.pdf in Makefile) and that
type is supported by default[2].

The epub (via the Epub3Builder) target does support SVGs by default[3]
but we disabled their support in commit ff3876ca4910 ("conf.py: use PNG
first in EPUB output"). We used Make to generate PNGs from them before
generating the doc epub though (see PNGs variable and %.png in Makefile)
and that type is supported (c.f. Epub3Builder.supported_image_types in
our conf.py).

The issue is that this is done transparently from Sphinx. When we
generate the PDFs or PNGs variants of the SVGs, we put them in-tree
directly along their source file. Then, when caching, Sphinx will find
both the source file and the appropriate variant. However, the cache
isn't updated if there are new files in the tree and the source rST
files aren't modified. So, the cache will not have its map updated and
we won't be able to find the new variant when building for a
non-SVG-compatible builder. Take the following scenario:

- start from a clean source file (fresh clone or git clean -ffdx)
- build the html target (which supports SVGs by default[4])
- sphinx will find all the files in the source tree matching the glob
  pattern in ".. image:: test.*", in our case only an SVG since the PDFs
  and PNGs are only generated for the latexpdf and epub targets
  respectively. The cache will only store the path to the SVG file
  because it is the only source file that matches the glob,
- attempt to build the epub target (which doesn't support SVGs, only
  PNGs)
- Sphinx checks for the file to include for '.. image:: test.*' and
  finds an SVG in the cache map and then check the list of supported
  image types for the Epub3Builder and find that it doesn't support
  that. It cannot find anything satisfying the dependency and thus fails
  to build.

This scenario can easily be reproduced by running the `make all` command
since the html builder will be used first, then epub and finally
latexpdf.

The `make publish` target works by chance, because the epub builder is
built first and will cache SVG + PNG for each glob, then the latexpdf
builder is built and supports PNGs so that's fine and then html, which
supports SVG as well.

To fix this issue, we could simply always generate PDFs and PNGs of all
SVGs in the source tree, but this isn't ideal.

Instead, let's use an ImageConverter from a Sphinx plugin. This allows
to map a plugin as being able to generate a file of type Y from a file
of type X. When Sphinx wants to build an image, it'll try to find the
image with the type the current builder supports in the cache. If it
cannot, it's going to try to find an ImageConverter plugin that is able
to convert one of the image types in cache with one of the image types
the current builder supports. Then Sphinx will call this plugin to
generate the file and put it into the build directory (not in the
source!).

This allows to simplify the Makefile as well and is a much cleaner
approach.

The epub target is removed as the catch-all target contains the same
instructions as the epub target we remove in this commit.

sphinxcontrib.rsvgconverter is a plugin from
sphinxcontrib-svg2pdfconverter python module. SVGs to PNGs is only
supported since 2.0.0.

[1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-d
[2] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.latex.LaTeXBuilder.supported_image_types
[3] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.epub3.Epub3Builder.supported_image_types
[4] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.html.StandaloneHTMLBuilder

Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
Changes in v3:
- rebased on top of master, rebuilt with `make all` no build error to be
  seen,
- Link to v2: https://patch.msgid.link/20251202-fix-make-multi-target-v2-1-7eea1d64cf6b@cherry.de

Changes in v2:
- switched to upstream sphinxcontrib-svg2pdfconverter since the
  necessary MR got merged and a new release made,
- removed IMAGEDIRS Make variable as it is now unused,
- added sphinxcontrib-svg2pdfconverter to Pipfile (not tested) and pip
  installation script,
- Link to v1: https://patch.msgid.link/20251030-fix-make-multi-target-v1-0-213616ed1f0a@cherry.de
---
 documentation/Makefile                             | 27 +++-------------------
 documentation/Pipfile                              |  2 ++
 documentation/conf.py                              |  1 +
 .../tools/host_packages_scripts/pip3_docs.sh       |  2 +-
 4 files changed, 7 insertions(+), 25 deletions(-)


---
base-commit: 095981c08b9d63905472df5d1d60c07af96f0250
change-id: 20251029-fix-make-multi-target-d2de9d4ff7ec

Best regards,
--  
Quentin Schulz <quentin.schulz@cherry.de>

Comments

Antonin Godard Feb. 12, 2026, 8:50 a.m. UTC | #1 
Hi,

On Wed Feb 11, 2026 at 4:25 PM CET, Quentin Schulz via lists.yoctoproject.org wrote:
[...]
> diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
> index 907ecec55..3e4045e7e 100644
> --- a/documentation/tools/host_packages_scripts/pip3_docs.sh
> +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
> @@ -1 +1 @@
> -sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
> +sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton 'sphinxcontrib-svg2pdfconverter>=2.0.0'

pip3_docs.sh is not used for ubuntu and debian, meaning we need to find the
equivalent package to add to
documentation/tools/host_packages_scripts/ubuntu_docs.sh, or write a custom
pip3_docs.sh file for ubuntu/debian.

Or drop the ubuntu_docs.sh file and use pip3_docs.sh for ubuntu/debian, which
would make this kind of mistakes unlikely to happen. What do you think?

Antonin
Quentin Schulz Feb. 12, 2026, 9:39 a.m. UTC | #2 
Hi Antonin,

On 2/12/26 9:50 AM, Antonin Godard wrote:
> Hi,
> 
> On Wed Feb 11, 2026 at 4:25 PM CET, Quentin Schulz via lists.yoctoproject.org wrote:
> [...]
>> diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
>> index 907ecec55..3e4045e7e 100644
>> --- a/documentation/tools/host_packages_scripts/pip3_docs.sh
>> +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
>> @@ -1 +1 @@
>> -sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
>> +sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton 'sphinxcontrib-svg2pdfconverter>=2.0.0'
> 
> pip3_docs.sh is not used for ubuntu and debian, meaning we need to find the

Woopsies.

> equivalent package to add to
> documentation/tools/host_packages_scripts/ubuntu_docs.sh, or write a custom

There's none as far as I could tell and even if there was one, unlikely 
to be 2.0.0 for all Debian/Ubuntu versions as they are generally pretty 
conservative with upgrades and this was a recent one.

> pip3_docs.sh file for ubuntu/debian.
> 
> Or drop the ubuntu_docs.sh file and use pip3_docs.sh for ubuntu/debian, which
> would make this kind of mistakes unlikely to happen. What do you think?
> 

I don't mind either way. Makes it easier to maintain if all use the same 
packages from pip is my opinion.

What would you prefer?

Cheers,
Quentin
Antonin Godard Feb. 12, 2026, 9:49 a.m. UTC | #3 
On Thu Feb 12, 2026 at 10:39 AM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 2/12/26 9:50 AM, Antonin Godard wrote:
>> Hi,
>> 
>> On Wed Feb 11, 2026 at 4:25 PM CET, Quentin Schulz via lists.yoctoproject.org wrote:
>> [...]
>>> diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>> index 907ecec55..3e4045e7e 100644
>>> --- a/documentation/tools/host_packages_scripts/pip3_docs.sh
>>> +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>> @@ -1 +1 @@
>>> -sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
>>> +sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton 'sphinxcontrib-svg2pdfconverter>=2.0.0'
>> 
>> pip3_docs.sh is not used for ubuntu and debian, meaning we need to find the
>
> Woopsies.
>
>> equivalent package to add to
>> documentation/tools/host_packages_scripts/ubuntu_docs.sh, or write a custom
>
> There's none as far as I could tell and even if there was one, unlikely 
> to be 2.0.0 for all Debian/Ubuntu versions as they are generally pretty 
> conservative with upgrades and this was a recent one.
>
>> pip3_docs.sh file for ubuntu/debian.
>> 
>> Or drop the ubuntu_docs.sh file and use pip3_docs.sh for ubuntu/debian, which
>> would make this kind of mistakes unlikely to happen. What do you think?
>> 
>
> I don't mind either way. Makes it easier to maintain if all use the same 
> packages from pip is my opinion.
>
> What would you prefer?

I prefer having pip3_docs.sh shared between all. Do you mind deleting
ubuntu_docs.sh and replacing it with pip3_docs.sh in a separate patch?

Thanks!
Antonin
Quentin Schulz Feb. 12, 2026, 10:57 a.m. UTC | #4 
Hi Antonin,

On 2/12/26 10:49 AM, Antonin Godard wrote:
> On Thu Feb 12, 2026 at 10:39 AM CET, Quentin Schulz wrote:
>> Hi Antonin,
>>
>> On 2/12/26 9:50 AM, Antonin Godard wrote:
>>> Hi,
>>>
>>> On Wed Feb 11, 2026 at 4:25 PM CET, Quentin Schulz via lists.yoctoproject.org wrote:
>>> [...]
>>>> diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>>> index 907ecec55..3e4045e7e 100644
>>>> --- a/documentation/tools/host_packages_scripts/pip3_docs.sh
>>>> +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>>> @@ -1 +1 @@
>>>> -sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
>>>> +sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton 'sphinxcontrib-svg2pdfconverter>=2.0.0'
>>>
>>> pip3_docs.sh is not used for ubuntu and debian, meaning we need to find the
>>
>> Woopsies.
>>
>>> equivalent package to add to
>>> documentation/tools/host_packages_scripts/ubuntu_docs.sh, or write a custom
>>
>> There's none as far as I could tell and even if there was one, unlikely
>> to be 2.0.0 for all Debian/Ubuntu versions as they are generally pretty
>> conservative with upgrades and this was a recent one.
>>
>>> pip3_docs.sh file for ubuntu/debian.
>>>
>>> Or drop the ubuntu_docs.sh file and use pip3_docs.sh for ubuntu/debian, which
>>> would make this kind of mistakes unlikely to happen. What do you think?
>>>
>>
>> I don't mind either way. Makes it easier to maintain if all use the same
>> packages from pip is my opinion.
>>
>> What would you prefer?
> 
> I prefer having pip3_docs.sh shared between all. Do you mind deleting
> ubuntu_docs.sh and replacing it with pip3_docs.sh in a separate patch?
> 

Working on it, and I'll check that the docs can be built with all 
containers this time. Good catch :)

Cheers,
Quentin
diff mbox series

Patch

diff --git a/documentation/Makefile b/documentation/Makefile
index e144a50b4..897db2b5a 100644
--- a/documentation/Makefile
+++ b/documentation/Makefile
@@ -10,11 +10,8 @@  VALEOPTS       ?= --no-wrap --glob '!migration-guides/release-notes-*.rst'
 SOURCEDIR      = .
 VALEDOCS       ?= $(SOURCEDIR)
 SPHINXLINTDOCS ?= $(SOURCEDIR)
-IMAGEDIRS      = */svg
 BUILDDIR       = _build
 DESTDIR        = final
-SVG2PNG        = rsvg-convert
-SVG2PDF        = rsvg-convert
 
 ifeq ($(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi),0)
 $(error "The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed")
@@ -24,7 +21,7 @@  endif
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-.PHONY: all checks help Makefile clean stylecheck publish epub latexpdf
+.PHONY: all checks help Makefile clean stylecheck publish latexpdf
 
 publish: Makefile checks epub latexpdf html singlehtml
 	rm -rf $(BUILDDIR)/$(DESTDIR)/
@@ -35,22 +32,8 @@  publish: Makefile checks epub latexpdf html singlehtml
 	cp $(BUILDDIR)/singlehtml/index.html $(BUILDDIR)/$(DESTDIR)/singleindex.html
 	sed -i -e 's@index.html#@singleindex.html#@g' $(BUILDDIR)/$(DESTDIR)/singleindex.html
 
-# Build a list of SVG files to convert to PDFs
-PDFs := $(foreach dir, $(IMAGEDIRS), $(patsubst %.svg,%.pdf,$(wildcard $(SOURCEDIR)/$(dir)/*.svg)))
-
-# Build a list of SVG files to convert to PNGs
-PNGs := $(foreach dir, $(IMAGEDIRS), $(patsubst %.svg,%.png,$(wildcard $(SOURCEDIR)/$(dir)/*.svg)))
-
-# Pattern rule for converting SVG to PDF
-%.pdf : %.svg
-	$(SVG2PDF) --format=pdf --output=$@ $<
-
-# Pattern rule for converting SVG to PNG
-%.png : %.svg
-	$(SVG2PNG) --format=png --output=$@ $<
-
 clean:
-	@rm -rf $(BUILDDIR) $(PNGs) $(PDFs) poky.yaml sphinx-static/switchers.js releases.rst
+	@rm -rf $(BUILDDIR) poky.yaml sphinx-static/switchers.js releases.rst
 
 checks:
 	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
@@ -62,14 +45,10 @@  stylecheck:
 sphinx-lint:
 	sphinx-lint $(SPHINXLINTDOCS)
 
-epub: $(PNGs)
-	$(SOURCEDIR)/set_versions.py
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
-
 # Note: we need to pass buf_size here (which is also configurable from
 # texmf.cnf), to avoid following error:
 #   Unable to read an entire line---bufsize=200000. Please increase buf_size in texmf.cnf.
-latexpdf: $(PDFs)
+latexpdf:
 	$(SOURCEDIR)/set_versions.py
 	buf_size=10000000 $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
diff --git a/documentation/Pipfile b/documentation/Pipfile
index 1fa2df31e..67fce078d 100644
--- a/documentation/Pipfile
+++ b/documentation/Pipfile
@@ -10,6 +10,8 @@  sphinx = "*"
 sphinx-rtd-theme = "*"
 pyyaml = "*"
 sphinx-copybutton = "*"
+# SVG to PNG only supported since 2.0.0
+sphinxcontrib-svg2pdfconverter = ">=2.0.0"
 
 [requires]
 python_version = "3"
diff --git a/documentation/conf.py b/documentation/conf.py
index 3d04f9057..5a69977cf 100644
--- a/documentation/conf.py
+++ b/documentation/conf.py
@@ -67,6 +67,7 @@  extensions = [
     'sphinx.ext.extlinks',
     'sphinx.ext.intersphinx',
     'sphinx_copybutton',
+    'sphinxcontrib.rsvgconverter',
     'yocto-vars'
 ]
 autosectionlabel_prefix_document = True
diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
index 907ecec55..3e4045e7e 100644
--- a/documentation/tools/host_packages_scripts/pip3_docs.sh
+++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
@@ -1 +1 @@ 
-sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
+sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton 'sphinxcontrib-svg2pdfconverter>=2.0.0'