| Message ID | 20260924192929.907566-1-adrian.freihofer@siemens.com |
|---|---|
| Headers | show |
| Series | dev-manual/devtool: ide-sdk doc rewrite and Mermaid diagrams | expand |
On Thu, 2026-09-24 at 21:29 +0200, AdrianF wrote: > From: Adrian Freihofer <adrian.freihofer@siemens.com> > > Update the "devtool ide-sdk" section to reflect its current behavior, > and fix outdated non-patch local file handling docs (5.1 change). > > Add the sphinxcontrib-mermaid Sphinx extension, add workflow/ > architecture diagrams for "devtool ide-sdk", and convert the existing > devtool add/modify/upgrade flow chart PNGs to Mermaid too. > > Diagrams render client-side via a CDN-hosted mermaid.js, so the build > server needs no new dependency. Hi Adrian, Expecting clients to fetch the CDN-hosted mermaid.js breaks the ability to render the diagrams offline for local reading. We also generate PDF and ePub versions of the docs - would this change require CDN access to build those? We may be better carrying a minified copy of mermaid.js and setting mermaid_use_local [1]. [1]: https://sphinxcontrib-mermaid-demo.readthedocs.io/en/latest/#mermaid-use-local Best regards,
On Sat, 2026-09-26 at 14:21 +0100, Paul Barker via lists.yoctoproject.org wrote: > On Thu, 2026-09-24 at 21:29 +0200, AdrianF wrote: > > From: Adrian Freihofer <adrian.freihofer@siemens.com> > > > > Update the "devtool ide-sdk" section to reflect its current > > behavior, > > and fix outdated non-patch local file handling docs (5.1 change). > > > > Add the sphinxcontrib-mermaid Sphinx extension, add workflow/ > > architecture diagrams for "devtool ide-sdk", and convert the > > existing > > devtool add/modify/upgrade flow chart PNGs to Mermaid too. > > > > Diagrams render client-side via a CDN-hosted mermaid.js, so the > > build > > server needs no new dependency. > > Hi Adrian, > > Expecting clients to fetch the CDN-hosted mermaid.js breaks the > ability > to render the diagrams offline for local reading. We also generate > PDF > and ePub versions of the docs - would this change require CDN access > to > build those? > > We may be better carrying a minified copy of mermaid.js and setting > mermaid_use_local [1]. > > [1]: > https://sphinxcontrib-mermaid-demo.readthedocs.io/en/latest/#mermaid-use-local Hi Paul That might be a good idea for the HTML case, but I was unsure how it would work for PDF or EPUB outputs. It’s not possible to embed the dynamic Mermaid rendering engine directly into a PDF or EPUB file. For PDF and EPUB cases, the rendering needs to take place at Sphinx build-time, which requires a JavaScript runtime on our build machines. I had a quick chat with Antonin about this, and he mentioned that an infrastructure change like this would need some evaluation. We can actually support both formats seamlessly using the sphinxcontrib-mermaid extension along with mermaid-cli (mmdc): * For HTML builds: Sphinx serves client-side JavaScript as planned, rendering diagrams in the reader's browser without extra build dependencies. * For PDF / EPUB builds: Sphinx automatically uses mermaid-cli (@mermaid-js/mermaid-cli) behind the scenes during build-time to rasterize/convert Mermaid blocks into static SVG or PNG images, embedding them directly into the output document. On the infrastructure side, this requires adding Node.js and running npm install -g @mermaid-js/mermaid-cli on the documentation build runners/CI environment. Would this be feasible? Best regards, Adrian > > Best regards, > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#10641): > https://lists.yoctoproject.org/g/docs/message/10641 > Mute This Topic: https://lists.yoctoproject.org/mt/121415542/4454582 > Group Owner: docs+owner@lists.yoctoproject.org > Unsubscribe: > https://lists.yoctoproject.org/g/docs/unsub [adrian.freihofer@gmail.com > ] > -=-=-=-=-=-=-=-=-=-=-=-
On 9/26/26 3:45 PM, Adrian Freihofer via lists.yoctoproject.org wrote: > On Sat, 2026-09-26 at 14:21 +0100, Paul Barker via > lists.yoctoproject.org wrote: >> On Thu, 2026-09-24 at 21:29 +0200, AdrianF wrote: >>> From: Adrian Freihofer <adrian.freihofer@siemens.com> >>> >>> Update the "devtool ide-sdk" section to reflect its current >>> behavior, >>> and fix outdated non-patch local file handling docs (5.1 change). >>> >>> Add the sphinxcontrib-mermaid Sphinx extension, add workflow/ >>> architecture diagrams for "devtool ide-sdk", and convert the >>> existing >>> devtool add/modify/upgrade flow chart PNGs to Mermaid too. >>> >>> Diagrams render client-side via a CDN-hosted mermaid.js, so the >>> build >>> server needs no new dependency. >> >> Hi Adrian, >> >> Expecting clients to fetch the CDN-hosted mermaid.js breaks the >> ability >> to render the diagrams offline for local reading. We also generate >> PDF >> and ePub versions of the docs - would this change require CDN access >> to >> build those? >> >> We may be better carrying a minified copy of mermaid.js and setting >> mermaid_use_local [1]. >> >> [1]: >> https://sphinxcontrib-mermaid-demo.readthedocs.io/en/latest/#mermaid-use-local > > Hi Paul > > That might be a good idea for the HTML case, but I was unsure how it > would work for PDF or EPUB outputs. It’s not possible to embed the > dynamic Mermaid rendering engine directly into a PDF or EPUB file. > > For PDF and EPUB cases, the rendering needs to take place at Sphinx > build-time, which requires a JavaScript runtime on our build machines. > I had a quick chat with Antonin about this, and he mentioned that an > infrastructure change like this would need some evaluation. > > We can actually support both formats seamlessly using the > sphinxcontrib-mermaid extension along with mermaid-cli (mmdc): > > * For HTML builds: Sphinx serves client-side JavaScript as planned, > rendering diagrams in the reader's browser without extra build > dependencies. Do we really want to require the user to run JavaScript to render "images"? I've very quickly checked our docs with NoScript enabled, the search doesn't work (but it tells you why) and the drop-down menu to select the version of the docs is gone as well but that isn't something essential I believe. Since it's possible to render images at build time, I think it'd make more sense to render them also for HTML, and it also gives us consistency in behavior with PDF/ePUB (and we would detect earlier if rendering is broken in the HTML variant, the one that I assume most people consume). > * For PDF / EPUB builds: Sphinx automatically uses mermaid-cli > (@mermaid-js/mermaid-cli) behind the scenes during build-time to > rasterize/convert Mermaid blocks into static SVG or PNG images, > embedding them directly into the output document. > > On the infrastructure side, this requires adding Node.js and running > npm install -g @mermaid-js/mermaid-cli on the documentation build > runners/CI environment. Would this be feasible? > I think this means we need a recipe for mermaid-cli in OE-Core and have it installed in the buildtools-docs. This means we now have a critical package in OE-Core that uses NodeJS and the npm fetcher. We've recently re-enabled the NPM fetcher but it's been a liability before so not sure if the project would like to depend on that for building the docs. Maybe there are other alternatives (Python would likely be fine as we already depend on it for Sphinx and we already tell the user to install some packages via pip; for the rest, it would be *very* beneficial to have packages in *all* distros we support). Cheers, Quentin
On Wed Sep 30, 2026 at 5:35 PM CEST, Quentin Schulz wrote: > > > On 9/26/26 3:45 PM, Adrian Freihofer via lists.yoctoproject.org wrote: >> On Sat, 2026-09-26 at 14:21 +0100, Paul Barker via >> lists.yoctoproject.org wrote: >>> On Thu, 2026-09-24 at 21:29 +0200, AdrianF wrote: >>>> From: Adrian Freihofer <adrian.freihofer@siemens.com> >>>> >>>> Update the "devtool ide-sdk" section to reflect its current >>>> behavior, >>>> and fix outdated non-patch local file handling docs (5.1 change). >>>> >>>> Add the sphinxcontrib-mermaid Sphinx extension, add workflow/ >>>> architecture diagrams for "devtool ide-sdk", and convert the >>>> existing >>>> devtool add/modify/upgrade flow chart PNGs to Mermaid too. >>>> >>>> Diagrams render client-side via a CDN-hosted mermaid.js, so the >>>> build >>>> server needs no new dependency. >>> >>> Hi Adrian, >>> >>> Expecting clients to fetch the CDN-hosted mermaid.js breaks the >>> ability >>> to render the diagrams offline for local reading. We also generate >>> PDF >>> and ePub versions of the docs - would this change require CDN access >>> to >>> build those? >>> >>> We may be better carrying a minified copy of mermaid.js and setting >>> mermaid_use_local [1]. >>> >>> [1]: >>> https://sphinxcontrib-mermaid-demo.readthedocs.io/en/latest/#mermaid-use-local >> >> Hi Paul >> >> That might be a good idea for the HTML case, but I was unsure how it >> would work for PDF or EPUB outputs. It’s not possible to embed the >> dynamic Mermaid rendering engine directly into a PDF or EPUB file. >> >> For PDF and EPUB cases, the rendering needs to take place at Sphinx >> build-time, which requires a JavaScript runtime on our build machines. >> I had a quick chat with Antonin about this, and he mentioned that an >> infrastructure change like this would need some evaluation. >> >> We can actually support both formats seamlessly using the >> sphinxcontrib-mermaid extension along with mermaid-cli (mmdc): >> >> * For HTML builds: Sphinx serves client-side JavaScript as planned, >> rendering diagrams in the reader's browser without extra build >> dependencies. > > Do we really want to require the user to run JavaScript to render "images"? > > I've very quickly checked our docs with NoScript enabled, the search > doesn't work (but it tells you why) and the drop-down menu to select the > version of the docs is gone as well but that isn't something essential I > believe. > > Since it's possible to render images at build time, I think it'd make > more sense to render them also for HTML, and it also gives us > consistency in behavior with PDF/ePUB (and we would detect earlier if > rendering is broken in the HTML variant, the one that I assume most > people consume). I agree, it would probably be more consistent to convert all mermaid diagrams to SVGs at build-time and just render them like other SVGs in the documentation. >> * For PDF / EPUB builds: Sphinx automatically uses mermaid-cli >> (@mermaid-js/mermaid-cli) behind the scenes during build-time to >> rasterize/convert Mermaid blocks into static SVG or PNG images, >> embedding them directly into the output document. >> >> On the infrastructure side, this requires adding Node.js and running >> npm install -g @mermaid-js/mermaid-cli on the documentation build >> runners/CI environment. Would this be feasible? >> > I think this means we need a recipe for mermaid-cli in OE-Core and have > it installed in the buildtools-docs. This means we now have a critical > package in OE-Core that uses NodeJS and the npm fetcher. We've recently > re-enabled the NPM fetcher but it's been a liability before so not sure > if the project would like to depend on that for building the docs. Maybe > there are other alternatives (Python would likely be fine as we already > depend on it for Sphinx and we already tell the user to install some > packages via pip; for the rest, it would be *very* beneficial to have > packages in *all* distros we support). I agree, I doubt OE-Core would accept it. I'm sorry Adrian, this hasn't gone quite as I hoped it would! Well, I looked at the diagrams anyway and they look good. So, maybe you can still convert them manually to SVGs for now, and include the mermaid code as a comment in the SVG itself? Not as fancy as converting them dynamically, but at least your work isn't lost and this series can make progress. If possible, please try to follow our standards.md diagram guidelines[1]. At least the fonts, and possibly colors. Hopefully this works for you :) [1]: https://git.yoctoproject.org/yocto-docs/tree/documentation/standards.md#n173 Thanks, Antonin
From: Adrian Freihofer <adrian.freihofer@siemens.com> Update the "devtool ide-sdk" section to reflect its current behavior, and fix outdated non-patch local file handling docs (5.1 change). Add the sphinxcontrib-mermaid Sphinx extension, add workflow/ architecture diagrams for "devtool ide-sdk", and convert the existing devtool add/modify/upgrade flow chart PNGs to Mermaid too. Diagrams render client-side via a CDN-hosted mermaid.js, so the build server needs no new dependency. Adrian Freihofer (5): dev-manual/devtool: fix non-patch local file handling docs dev-manual/devtool: update devtool ide-sdk section for recent features conf.py: support mermaid plugin dev-manual/devtool: add Mermaid diagrams for the ide-sdk workflow and architecture dev-manual/devtool: convert add/modify/upgrade flow diagrams to Mermaid documentation/Pipfile | 1 + documentation/conf.py | 6 + documentation/dev-manual/devtool.rst | 1026 +++++++++++------ .../dev-manual/figures/devtool-add-flow.mmd | 73 ++ .../dev-manual/figures/devtool-add-flow.png | Bin 181699 -> 0 bytes .../figures/devtool-ide-sdk-architecture.mmd | 53 + .../devtool-ide-sdk-nfs-architecture.mmd | 58 + .../figures/devtool-ide-sdk-workflow.mmd | 43 + .../figures/devtool-modify-flow.mmd | 77 ++ .../figures/devtool-modify-flow.png | Bin 171676 -> 0 bytes .../figures/devtool-upgrade-flow.mmd | 52 + .../figures/devtool-upgrade-flow.png | Bin 138917 -> 0 bytes .../sphinx-static/theme_overrides.css | 17 + 13 files changed, 1046 insertions(+), 360 deletions(-) create mode 100644 documentation/dev-manual/figures/devtool-add-flow.mmd delete mode 100644 documentation/dev-manual/figures/devtool-add-flow.png create mode 100644 documentation/dev-manual/figures/devtool-ide-sdk-architecture.mmd create mode 100644 documentation/dev-manual/figures/devtool-ide-sdk-nfs-architecture.mmd create mode 100644 documentation/dev-manual/figures/devtool-ide-sdk-workflow.mmd create mode 100644 documentation/dev-manual/figures/devtool-modify-flow.mmd delete mode 100644 documentation/dev-manual/figures/devtool-modify-flow.png create mode 100644 documentation/dev-manual/figures/devtool-upgrade-flow.mmd delete mode 100644 documentation/dev-manual/figures/devtool-upgrade-flow.png