mbox series

[v2,0/5] dev-manual/devtool: ide-sdk doc rewrite and Mermaid diagrams

Message ID 20260924192929.907566-1-adrian.freihofer@siemens.com
Headers show
Series dev-manual/devtool: ide-sdk doc rewrite and Mermaid diagrams | expand

Message

Freihofer, Adrian Sept. 24, 2026, 7:29 p.m. UTC
From: Adrian Freihofer <adrian.freihofer@siemens.com>

Update the "devtool ide-sdk" section to reflect its current behavior,
and fix outdated non-patch local file handling docs (5.1 change).

Add the sphinxcontrib-mermaid Sphinx extension, add workflow/
architecture diagrams for "devtool ide-sdk", and convert the existing
devtool add/modify/upgrade flow chart PNGs to Mermaid too.

Diagrams render client-side via a CDN-hosted mermaid.js, so the build
server needs no new dependency.

Adrian Freihofer (5):
  dev-manual/devtool: fix non-patch local file handling docs
  dev-manual/devtool: update devtool ide-sdk section for recent features
  conf.py: support mermaid plugin
  dev-manual/devtool: add Mermaid diagrams for the ide-sdk workflow and
    architecture
  dev-manual/devtool: convert add/modify/upgrade flow diagrams to
    Mermaid

 documentation/Pipfile                         |    1 +
 documentation/conf.py                         |    6 +
 documentation/dev-manual/devtool.rst          | 1026 +++++++++++------
 .../dev-manual/figures/devtool-add-flow.mmd   |   73 ++
 .../dev-manual/figures/devtool-add-flow.png   |  Bin 181699 -> 0 bytes
 .../figures/devtool-ide-sdk-architecture.mmd  |   53 +
 .../devtool-ide-sdk-nfs-architecture.mmd      |   58 +
 .../figures/devtool-ide-sdk-workflow.mmd      |   43 +
 .../figures/devtool-modify-flow.mmd           |   77 ++
 .../figures/devtool-modify-flow.png           |  Bin 171676 -> 0 bytes
 .../figures/devtool-upgrade-flow.mmd          |   52 +
 .../figures/devtool-upgrade-flow.png          |  Bin 138917 -> 0 bytes
 .../sphinx-static/theme_overrides.css         |   17 +
 13 files changed, 1046 insertions(+), 360 deletions(-)
 create mode 100644 documentation/dev-manual/figures/devtool-add-flow.mmd
 delete mode 100644 documentation/dev-manual/figures/devtool-add-flow.png
 create mode 100644 documentation/dev-manual/figures/devtool-ide-sdk-architecture.mmd
 create mode 100644 documentation/dev-manual/figures/devtool-ide-sdk-nfs-architecture.mmd
 create mode 100644 documentation/dev-manual/figures/devtool-ide-sdk-workflow.mmd
 create mode 100644 documentation/dev-manual/figures/devtool-modify-flow.mmd
 delete mode 100644 documentation/dev-manual/figures/devtool-modify-flow.png
 create mode 100644 documentation/dev-manual/figures/devtool-upgrade-flow.mmd
 delete mode 100644 documentation/dev-manual/figures/devtool-upgrade-flow.png

Comments

Paul Barker Sept. 26, 2026, 1:21 p.m. UTC | #1
On Thu, 2026-09-24 at 21:29 +0200, AdrianF wrote:
> From: Adrian Freihofer <adrian.freihofer@siemens.com>
> 
> Update the "devtool ide-sdk" section to reflect its current behavior,
> and fix outdated non-patch local file handling docs (5.1 change).
> 
> Add the sphinxcontrib-mermaid Sphinx extension, add workflow/
> architecture diagrams for "devtool ide-sdk", and convert the existing
> devtool add/modify/upgrade flow chart PNGs to Mermaid too.
> 
> Diagrams render client-side via a CDN-hosted mermaid.js, so the build
> server needs no new dependency.

Hi Adrian,

Expecting clients to fetch the CDN-hosted mermaid.js breaks the ability
to render the diagrams offline for local reading. We also generate PDF
and ePub versions of the docs - would this change require CDN access to
build those?

We may be better carrying a minified copy of mermaid.js and setting
mermaid_use_local [1].

[1]: https://sphinxcontrib-mermaid-demo.readthedocs.io/en/latest/#mermaid-use-local

Best regards,
adrian.freihofer@gmail.com Sept. 26, 2026, 1:45 p.m. UTC | #2
On Sat, 2026-09-26 at 14:21 +0100, Paul Barker via
lists.yoctoproject.org wrote:
> On Thu, 2026-09-24 at 21:29 +0200, AdrianF wrote:
> > From: Adrian Freihofer <adrian.freihofer@siemens.com>
> > 
> > Update the "devtool ide-sdk" section to reflect its current
> > behavior,
> > and fix outdated non-patch local file handling docs (5.1 change).
> > 
> > Add the sphinxcontrib-mermaid Sphinx extension, add workflow/
> > architecture diagrams for "devtool ide-sdk", and convert the
> > existing
> > devtool add/modify/upgrade flow chart PNGs to Mermaid too.
> > 
> > Diagrams render client-side via a CDN-hosted mermaid.js, so the
> > build
> > server needs no new dependency.
> 
> Hi Adrian,
> 
> Expecting clients to fetch the CDN-hosted mermaid.js breaks the
> ability
> to render the diagrams offline for local reading. We also generate
> PDF
> and ePub versions of the docs - would this change require CDN access
> to
> build those?
> 
> We may be better carrying a minified copy of mermaid.js and setting
> mermaid_use_local [1].
> 
> [1]:
> https://sphinxcontrib-mermaid-demo.readthedocs.io/en/latest/#mermaid-use-local

Hi Paul

That might be a good idea for the HTML case, but I was unsure how it
would work for PDF or EPUB outputs. It’s not possible to embed the
dynamic Mermaid rendering engine directly into a PDF or EPUB file.

For PDF and EPUB cases, the rendering needs to take place at Sphinx
build-time, which requires a JavaScript runtime on our build machines.
I had a quick chat with Antonin about this, and he mentioned that an
infrastructure change like this would need some evaluation.

We can actually support both formats seamlessly using the
sphinxcontrib-mermaid extension along with mermaid-cli (mmdc):

 * For HTML builds: Sphinx serves client-side JavaScript as planned,
   rendering diagrams in the reader's browser without extra build
   dependencies.
 * For PDF / EPUB builds: Sphinx automatically uses mermaid-cli
   (@mermaid-js/mermaid-cli) behind the scenes during build-time to
   rasterize/convert Mermaid blocks into static SVG or PNG images,
   embedding them directly into the output document.

On the infrastructure side, this requires adding Node.js and running
npm install -g @mermaid-js/mermaid-cli on the documentation build
runners/CI environment. Would this be feasible?

Best regards,
Adrian



> 
> Best regards,
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#10641):
> https://lists.yoctoproject.org/g/docs/message/10641
> Mute This Topic: https://lists.yoctoproject.org/mt/121415542/4454582
> Group Owner: docs+owner@lists.yoctoproject.org
> Unsubscribe:
> https://lists.yoctoproject.org/g/docs/unsub [adrian.freihofer@gmail.com
> ]
> -=-=-=-=-=-=-=-=-=-=-=-
Quentin Schulz Sept. 30, 2026, 3:35 p.m. UTC | #3
On 9/26/26 3:45 PM, Adrian Freihofer via lists.yoctoproject.org wrote:
> On Sat, 2026-09-26 at 14:21 +0100, Paul Barker via
> lists.yoctoproject.org wrote:
>> On Thu, 2026-09-24 at 21:29 +0200, AdrianF wrote:
>>> From: Adrian Freihofer <adrian.freihofer@siemens.com>
>>>
>>> Update the "devtool ide-sdk" section to reflect its current
>>> behavior,
>>> and fix outdated non-patch local file handling docs (5.1 change).
>>>
>>> Add the sphinxcontrib-mermaid Sphinx extension, add workflow/
>>> architecture diagrams for "devtool ide-sdk", and convert the
>>> existing
>>> devtool add/modify/upgrade flow chart PNGs to Mermaid too.
>>>
>>> Diagrams render client-side via a CDN-hosted mermaid.js, so the
>>> build
>>> server needs no new dependency.
>>
>> Hi Adrian,
>>
>> Expecting clients to fetch the CDN-hosted mermaid.js breaks the
>> ability
>> to render the diagrams offline for local reading. We also generate
>> PDF
>> and ePub versions of the docs - would this change require CDN access
>> to
>> build those?
>>
>> We may be better carrying a minified copy of mermaid.js and setting
>> mermaid_use_local [1].
>>
>> [1]:
>> https://sphinxcontrib-mermaid-demo.readthedocs.io/en/latest/#mermaid-use-local
> 
> Hi Paul
> 
> That might be a good idea for the HTML case, but I was unsure how it
> would work for PDF or EPUB outputs. It’s not possible to embed the
> dynamic Mermaid rendering engine directly into a PDF or EPUB file.
> 
> For PDF and EPUB cases, the rendering needs to take place at Sphinx
> build-time, which requires a JavaScript runtime on our build machines.
> I had a quick chat with Antonin about this, and he mentioned that an
> infrastructure change like this would need some evaluation.
> 
> We can actually support both formats seamlessly using the
> sphinxcontrib-mermaid extension along with mermaid-cli (mmdc):
> 
>   * For HTML builds: Sphinx serves client-side JavaScript as planned,
>     rendering diagrams in the reader's browser without extra build
>     dependencies.

Do we really want to require the user to run JavaScript to render "images"?

I've very quickly checked our docs with NoScript enabled, the search 
doesn't work (but it tells you why) and the drop-down menu to select the 
version of the docs is gone as well but that isn't something essential I 
believe.

Since it's possible to render images at build time, I think it'd make 
more sense to render them also for HTML, and it also gives us 
consistency in behavior with PDF/ePUB (and we would detect earlier if 
rendering is broken in the HTML variant, the one that I assume most 
people consume).

>   * For PDF / EPUB builds: Sphinx automatically uses mermaid-cli
>     (@mermaid-js/mermaid-cli) behind the scenes during build-time to
>     rasterize/convert Mermaid blocks into static SVG or PNG images,
>     embedding them directly into the output document.
> 
> On the infrastructure side, this requires adding Node.js and running
> npm install -g @mermaid-js/mermaid-cli on the documentation build
> runners/CI environment. Would this be feasible?
> 
I think this means we need a recipe for mermaid-cli in OE-Core and have 
it installed in the buildtools-docs. This means we now have a critical 
package in OE-Core that uses NodeJS and the npm fetcher. We've recently 
re-enabled the NPM fetcher but it's been a liability before so not sure 
if the project would like to depend on that for building the docs. Maybe 
there are other alternatives (Python would likely be fine as we already 
depend on it for Sphinx and we already tell the user to install some 
packages via pip; for the rest, it would be *very* beneficial to have 
packages in *all* distros we support).

Cheers,
Quentin
Antonin Godard Oct. 15, 2026, 10:22 a.m. UTC | #4
On Wed Sep 30, 2026 at 5:35 PM CEST, Quentin Schulz wrote:
>
>
> On 9/26/26 3:45 PM, Adrian Freihofer via lists.yoctoproject.org wrote:
>> On Sat, 2026-09-26 at 14:21 +0100, Paul Barker via
>> lists.yoctoproject.org wrote:
>>> On Thu, 2026-09-24 at 21:29 +0200, AdrianF wrote:
>>>> From: Adrian Freihofer <adrian.freihofer@siemens.com>
>>>>
>>>> Update the "devtool ide-sdk" section to reflect its current
>>>> behavior,
>>>> and fix outdated non-patch local file handling docs (5.1 change).
>>>>
>>>> Add the sphinxcontrib-mermaid Sphinx extension, add workflow/
>>>> architecture diagrams for "devtool ide-sdk", and convert the
>>>> existing
>>>> devtool add/modify/upgrade flow chart PNGs to Mermaid too.
>>>>
>>>> Diagrams render client-side via a CDN-hosted mermaid.js, so the
>>>> build
>>>> server needs no new dependency.
>>>
>>> Hi Adrian,
>>>
>>> Expecting clients to fetch the CDN-hosted mermaid.js breaks the
>>> ability
>>> to render the diagrams offline for local reading. We also generate
>>> PDF
>>> and ePub versions of the docs - would this change require CDN access
>>> to
>>> build those?
>>>
>>> We may be better carrying a minified copy of mermaid.js and setting
>>> mermaid_use_local [1].
>>>
>>> [1]:
>>> https://sphinxcontrib-mermaid-demo.readthedocs.io/en/latest/#mermaid-use-local
>> 
>> Hi Paul
>> 
>> That might be a good idea for the HTML case, but I was unsure how it
>> would work for PDF or EPUB outputs. It’s not possible to embed the
>> dynamic Mermaid rendering engine directly into a PDF or EPUB file.
>> 
>> For PDF and EPUB cases, the rendering needs to take place at Sphinx
>> build-time, which requires a JavaScript runtime on our build machines.
>> I had a quick chat with Antonin about this, and he mentioned that an
>> infrastructure change like this would need some evaluation.
>> 
>> We can actually support both formats seamlessly using the
>> sphinxcontrib-mermaid extension along with mermaid-cli (mmdc):
>> 
>>   * For HTML builds: Sphinx serves client-side JavaScript as planned,
>>     rendering diagrams in the reader's browser without extra build
>>     dependencies.
>
> Do we really want to require the user to run JavaScript to render "images"?
>
> I've very quickly checked our docs with NoScript enabled, the search 
> doesn't work (but it tells you why) and the drop-down menu to select the 
> version of the docs is gone as well but that isn't something essential I 
> believe.
>
> Since it's possible to render images at build time, I think it'd make 
> more sense to render them also for HTML, and it also gives us 
> consistency in behavior with PDF/ePUB (and we would detect earlier if 
> rendering is broken in the HTML variant, the one that I assume most 
> people consume).

I agree, it would probably be more consistent to convert all mermaid diagrams to
SVGs at build-time and just render them like other SVGs in the documentation.

>>   * For PDF / EPUB builds: Sphinx automatically uses mermaid-cli
>>     (@mermaid-js/mermaid-cli) behind the scenes during build-time to
>>     rasterize/convert Mermaid blocks into static SVG or PNG images,
>>     embedding them directly into the output document.
>> 
>> On the infrastructure side, this requires adding Node.js and running
>> npm install -g @mermaid-js/mermaid-cli on the documentation build
>> runners/CI environment. Would this be feasible?
>> 
> I think this means we need a recipe for mermaid-cli in OE-Core and have 
> it installed in the buildtools-docs. This means we now have a critical 
> package in OE-Core that uses NodeJS and the npm fetcher. We've recently 
> re-enabled the NPM fetcher but it's been a liability before so not sure 
> if the project would like to depend on that for building the docs. Maybe 
> there are other alternatives (Python would likely be fine as we already 
> depend on it for Sphinx and we already tell the user to install some 
> packages via pip; for the rest, it would be *very* beneficial to have 
> packages in *all* distros we support).

I agree, I doubt OE-Core would accept it.

I'm sorry Adrian, this hasn't gone quite as I hoped it would!

Well, I looked at the diagrams anyway and they look good. So, maybe you can
still convert them manually to SVGs for now, and include the mermaid code as a
comment in the SVG itself? Not as fancy as converting them dynamically, but at
least your work isn't lost and this series can make progress.

If possible, please try to follow our standards.md diagram guidelines[1]. At
least the fonts, and possibly colors.

Hopefully this works for you :)

[1]: https://git.yoctoproject.org/yocto-docs/tree/documentation/standards.md#n173

Thanks,
Antonin