| Message ID | 20251222-concurrent-safety-v1-5-e3d86e44cd38@bootlin.com |
|---|---|
| State | New |
| Headers | show |
| Series | Parallel docs build improvements | expand |
Hi Antonin, On 12/22/25 1:27 PM, Antonin Godard via lists.yoctoproject.org wrote: > The set_versions.py script modified files in the source directory. To > improve concurrent docs builds, wrap the set_versions.py with flock, > with a 30 seconds timeout. > I don't think that's enough. I can have sphinx-build of build A use poky.yaml, switchers.js, releases.rst while set_versions.py from build B (over)write the files, ending with corrupted or partial files. I think we'd need a lock on set_versions.py + sphinx-build run inside a subshell, something like the fourth example in the manpage. (; flock -w 30 9 || exit 1; # ... commands executed under lock ...; ) 9>/var/lock/mylockfile Another option (haven't looked up if it's possible) might be to migrate set_versions.py into a sphinx plugin such that the plugin is keeping the files in DRAM, or writing them to the BUILDDIR directly for consumption. > Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> > --- > documentation/.gitignore | 1 + > documentation/Makefile | 8 ++++---- > 2 files changed, 5 insertions(+), 4 deletions(-) > > diff --git a/documentation/.gitignore b/documentation/.gitignore > index b23d598054..a5a5efc19c 100644 > --- a/documentation/.gitignore > +++ b/documentation/.gitignore > @@ -9,3 +9,4 @@ releases.rst > */svg/*.pdf > styles/* > !styles/config > +set-versions.lock > diff --git a/documentation/Makefile b/documentation/Makefile > index 5f84a93e32..fe49d74cc6 100644 > --- a/documentation/Makefile > +++ b/documentation/Makefile > @@ -50,7 +50,7 @@ PNGs := $(foreach dir, $(IMAGEDIRS), $(patsubst %.svg,%.png,$(wildcard $(SOURCED > $(SVG2PNG) --format=png --output=$@ $< > > clean: > - @rm -rf $(BUILDDIR) $(PNGs) $(PDFs) poky.yaml sphinx-static/switchers.js releases.rst > + @rm -rf $(BUILDDIR) $(PNGs) $(PDFs) poky.yaml sphinx-static/switchers.js releases.rst set-versions.lock > That would allow us to run make clean; make html in parallel and bypass the flock entirely. If the lock is released, it's fine to still have the file no? If it's stuck for some reason (someone Ctrl+C the build and flock doesn't release the lock?), we can always say you need to remove the lock by hand? Cheers, Quentin
diff --git a/documentation/.gitignore b/documentation/.gitignore index b23d598054..a5a5efc19c 100644 --- a/documentation/.gitignore +++ b/documentation/.gitignore @@ -9,3 +9,4 @@ releases.rst */svg/*.pdf styles/* !styles/config +set-versions.lock diff --git a/documentation/Makefile b/documentation/Makefile index 5f84a93e32..fe49d74cc6 100644 --- a/documentation/Makefile +++ b/documentation/Makefile @@ -50,7 +50,7 @@ PNGs := $(foreach dir, $(IMAGEDIRS), $(patsubst %.svg,%.png,$(wildcard $(SOURCED $(SVG2PNG) --format=png --output=$@ $< clean: - @rm -rf $(BUILDDIR) $(PNGs) $(PDFs) poky.yaml sphinx-static/switchers.js releases.rst + @rm -rf $(BUILDDIR) $(PNGs) $(PDFs) poky.yaml sphinx-static/switchers.js releases.rst set-versions.lock checks: $(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR) @@ -63,14 +63,14 @@ sphinx-lint: sphinx-lint $(SPHINXLINTDOCS) epub: $(PNGs) - $(SOURCEDIR)/set_versions.py + flock -w 30 "$(SOURCEDIR)"/set-versions.lock -c $(SOURCEDIR)/set_versions.py @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) # Note: we need to pass buf_size here (which is also configurable from # texmf.cnf), to avoid following error: # Unable to read an entire line---bufsize=200000. Please increase buf_size in texmf.cnf. latexpdf: $(PDFs) - $(SOURCEDIR)/set_versions.py + flock -w 30 "$(SOURCEDIR)"/set-versions.lock -c $(SOURCEDIR)/set_versions.py buf_size=10000000 $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) all: html epub latexpdf @@ -78,5 +78,5 @@ all: html epub latexpdf # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: - $(SOURCEDIR)/set_versions.py + flock -w 30 "$(SOURCEDIR)"/set-versions.lock -c $(SOURCEDIR)/set_versions.py @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
The set_versions.py script modified files in the source directory. To improve concurrent docs builds, wrap the set_versions.py with flock, with a 30 seconds timeout. Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> --- documentation/.gitignore | 1 + documentation/Makefile | 8 ++++---- 2 files changed, 5 insertions(+), 4 deletions(-)