[v3,5/5] tools: Add check-confusables pre-commit hook

From: Niko Mauno <niko.mauno@vaisala.com>

Add a check-confusables script, in the same fashion as
check-glossaries, that scans the documentation .rst sources for
non-ASCII "confusable" characters (curly quotes, en/em dashes,
non-breaking and zero-width spaces, etc.) and reports each occurrence
with its location and suggested ASCII replacement, exiting non-zero if
any are found. This guards against the class of breakage fixed in the
preceding commit, e.g. curly quotes causing recipe ParseErrors.

Legitimate non-ASCII such as box-drawing characters used in directory
trees, accented letters in contributor names and CJK characters are
intentionally left untouched. No-break spaces are likewise tolerated
on lines containing box-drawing characters, since the tree command
emits them as indentation in directory listings.

Wire it up both as a local pre-commit hook, which checks the changed
files, and in the Makefile "checks" target, which scans the whole
tree, alongside check-glossaries.

Suggested-by: Quentin Schulz <quentin.schulz@cherry.de>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
---
 .pre-commit-config.yaml               |   5 ++
 documentation/Makefile                |   1 +
 documentation/tools/check-confusables | 115 ++++++++++++++++++++++++++
 3 files changed, 121 insertions(+)
 create mode 100755 documentation/tools/check-confusables

On Tue, 2026-07-28 at 12:07 +0300, Niko Mauno via lists.yoctoproject.org
wrote:
> From: Niko Mauno <niko.mauno@vaisala.com>
> 
> Add a check-confusables script, in the same fashion as
> check-glossaries, that scans the documentation .rst sources for
> non-ASCII "confusable" characters (curly quotes, en/em dashes,
> non-breaking and zero-width spaces, etc.) and reports each occurrence
> with its location and suggested ASCII replacement, exiting non-zero if
> any are found. This guards against the class of breakage fixed in the
> preceding commit, e.g. curly quotes causing recipe ParseErrors.
> 
> Legitimate non-ASCII such as box-drawing characters used in directory
> trees, accented letters in contributor names and CJK characters are
> intentionally left untouched. No-break spaces are likewise tolerated
> on lines containing box-drawing characters, since the tree command
> emits them as indentation in directory listings.
> 
> Wire it up both as a local pre-commit hook, which checks the changed
> files, and in the Makefile "checks" target, which scans the whole
> tree, alongside check-glossaries.
> 
> Suggested-by: Quentin Schulz <quentin.schulz@cherry.de>
> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>

Hi Niko,

Are there any existing tools we can use instead of implementing this
ourselves? Perhaps https://pypi.org/project/confusables/ ?

Some comments below if we do want to merge our own implementation of
this...

> ---
>  .pre-commit-config.yaml               |   5 ++
>  documentation/Makefile                |   1 +
>  documentation/tools/check-confusables | 115 ++++++++++++++++++++++++++
>  3 files changed, 121 insertions(+)
>  create mode 100755 documentation/tools/check-confusables
> 
> diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
> index f2b73a481..876546f9a 100644
> --- a/.pre-commit-config.yaml
> +++ b/.pre-commit-config.yaml
> @@ -6,3 +6,8 @@ repos:
>          entry: ./documentation/tools/check-glossaries
>          language: python
>          pass_filenames: false
> +      - id: check-confusables
> +        name: Check for non-ASCII confusable characters
> +        entry: ./documentation/tools/check-confusables
> +        language: python
> +        files: \.rst$
> diff --git a/documentation/Makefile b/documentation/Makefile
> index fe0574537..87a6f8a8b 100644
> --- a/documentation/Makefile
> +++ b/documentation/Makefile
> @@ -37,6 +37,7 @@ clean:
>  
>  checks:
>  	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
> +	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
>  
>  stylecheck:
>  	vale sync
> diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables
> new file mode 100755
> index 000000000..d9f27b337
> --- /dev/null
> +++ b/documentation/tools/check-confusables
> @@ -0,0 +1,115 @@
> +#!/usr/bin/env python3
> +

This needs a copyright and license header.

> +import argparse
> +import sys
> +
> +from pathlib import Path
> +
> +
> +def parse_arguments() -> argparse.Namespace:
> +    parser = argparse.ArgumentParser(
> +        description="Check documentation sources for non-ASCII typographic "
> +                    "characters that should be plain ASCII")
> +
> +    parser.add_argument("files",
> +                        nargs="*",
> +                        type=Path,
> +                        help="Specific files to check; if none are given, "
> +                             "all *.rst files under --docs-dir are scanned")
> +
> +    parser.add_argument("-d", "--docs-dir",
> +                        type=Path,
> +                        default=Path(__file__).resolve().parent.parent,
> +                        help="Path to documentation/ directory in yocto-docs")
> +
> +    return parser.parse_args()
> +
> +
> +# Map of "confusable" characters that are frequently introduced by editors,
> +# word processors or copy-pasting, to their plain ASCII replacement. These
> +# look almost identical to regular ASCII but break tooling, e.g. a curly
> +# quote in a recipe example causes:
> +#
> +#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = “foo”'
> +#
> +# Only these characters are flagged; legitimate non-ASCII such as box-drawing
> +# characters used in directory trees, accented letters in contributor names
> +# and CJK characters are intentionally left alone.
> +confusables = {
> +    "‘": "'",      # LEFT SINGLE QUOTATION MARK
> +    "’": "'",      # RIGHT SINGLE QUOTATION MARK
> +    "“": '"',      # LEFT DOUBLE QUOTATION MARK
> +    "”": '"',      # RIGHT DOUBLE QUOTATION MARK
> +    "′": "'",      # PRIME
> +    "″": '"',      # DOUBLE PRIME
> +    "–": "-",      # EN DASH
> +    "—": "--",     # EM DASH
> +    "‐": "-",      # HYPHEN
> +    "‑": "-",      # NON-BREAKING HYPHEN
> +    "−": "-",      # MINUS SIGN
> +    " ": " ",      # NO-BREAK SPACE
> +    " ": " ",      # NARROW NO-BREAK SPACE
> +    "​": "",       # ZERO WIDTH SPACE
> +    "": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
> +    "­": "",       # SOFT HYPHEN
> +}

We should not directly use confusing unicode characters in this array,
we shoud use '\u' escape sequences.

I worry that the above is only a very small subset of the confusable
characters, and this list will expand over time.

> +
> +NO_BREAK_SPACE = " "

As above, better to use a '\u' escape sequence here.

> +
> +
> +def is_box_drawing(char: str) -> bool:
> +    # Box Drawing Unicode block (U+2500..U+257F), used for the directory
> +    # trees rendered in the manuals.
> +    return "─" <= char <= "╿"

... and probably here as well.

> +
> +
> +def check_file(path: Path, display: str) -> bool:
> +    found = False
> +
> +    with open(path, "r", encoding="utf-8") as f:
> +        for lineno, line in enumerate(f, start=1):
> +            # The tree(1) command indents its directory listings with
> +            # no-break spaces; such listings are embedded verbatim in the
> +            # manuals. A no-break space is therefore tolerated on any line
> +            # that also contains box-drawing characters (i.e. inside a
> +            # rendered directory tree), but still flagged elsewhere.
> +            in_tree = any(is_box_drawing(c) for c in line)
> +            for col, char in enumerate(line, start=1):
> +                if char not in confusables:
> +                    continue
> +                if char == NO_BREAK_SPACE and in_tree:
> +                    continue
> +                replacement = confusables[char]
> +                hint = f"'{replacement}'" if replacement else "(remove)"
> +                print(f"WARNING: {display}:{lineno}:{col}: non-ASCII "
> +                      f"character U+{ord(char):04X} should be "
> +                      f"replaced with {hint}")
> +                found = True
> +
> +    return found
> +
> +
> +def main():
> +
> +    args = parse_arguments()
> +
> +    # When invoked with explicit files (e.g. by pre-commit, which passes the
> +    # staged filenames) only those are checked; otherwise the whole tree of
> +    # *.rst files under --docs-dir is scanned (e.g. by "make checks").
> +    if args.files:
> +        targets = [(path, str(path)) for path in args.files]
> +    else:
> +        docs_dir = Path(args.docs_dir)
> +        targets = [(path, str(path.relative_to(docs_dir)))
> +                   for path in sorted(docs_dir.rglob("*.rst"))]
> +
> +    exit_code = 0
> +    for path, display in targets:
> +        if check_file(path, display):
> +            exit_code = 1
> +
> +    sys.exit(exit_code)
> +
> +
> +if __name__ == "__main__":
> +    main()

Best regards,