| Message ID | 20260728072328.11926-6-niko.mauno@vaisala.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | documentation: minor fixes, plus a confusables check | expand |
Hi, On Tue Jul 28, 2026 at 9:23 AM CEST, Niko Mauno via lists.yoctoproject.org wrote: > From: Niko Mauno <niko.mauno@vaisala.com> > > Add a check-confusables script, in the same fashion as > check-glossaries, that scans the documentation .rst sources for > non-ASCII "confusable" characters (curly quotes, en/em dashes, > horizontal ellipsis, non-breaking and zero-width spaces, etc.) and > reports each occurrence with its location and suggested ASCII > replacement, exiting non-zero if any are found. This guards against > the class of breakage fixed in the preceding commit, e.g. curly quotes > causing recipe ParseErrors. > > Legitimate non-ASCII such as box-drawing characters used in directory > trees, accented letters in contributor names and CJK characters are > intentionally left untouched. > > Wire it up both as a local pre-commit hook and in the Makefile "checks" > target, alongside check-glossaries. > > Suggested-by: Quentin Schulz <quentin.schulz@cherry.de> > Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> > Signed-off-by: Niko Mauno <niko.mauno@vaisala.com> > --- > .pre-commit-config.yaml | 5 ++ > documentation/Makefile | 1 + > documentation/tools/check-confusables | 77 +++++++++++++++++++++++++++ > 3 files changed, 83 insertions(+) > create mode 100755 documentation/tools/check-confusables > > diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml > index f2b73a481..d6008b609 100644 > --- a/.pre-commit-config.yaml > +++ b/.pre-commit-config.yaml > @@ -6,3 +6,8 @@ repos: > entry: ./documentation/tools/check-glossaries > language: python > pass_filenames: false > + - id: check-confusables > + name: Check for non-ASCII confusable characters > + entry: ./documentation/tools/check-confusables > + language: python > + pass_filenames: false I think we should pass filenames and check only the modified files, no need to checking the entire tree each time? > diff --git a/documentation/Makefile b/documentation/Makefile > index fe0574537..87a6f8a8b 100644 > --- a/documentation/Makefile > +++ b/documentation/Makefile > @@ -37,6 +37,7 @@ clean: > > checks: > $(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR) > + $(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR) > > stylecheck: > vale sync > diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables > new file mode 100755 > index 000000000..f79ee046c > --- /dev/null > +++ b/documentation/tools/check-confusables > @@ -0,0 +1,77 @@ > +#!/usr/bin/env python3 > + > +import argparse > +import sys > + > +from pathlib import Path > + > + > +def parse_arguments() -> argparse.Namespace: > + parser = argparse.ArgumentParser( > + description="Check documentation sources for non-ASCII typographic " > + "characters that should be plain ASCII") > + > + parser.add_argument("-d", "--docs-dir", > + type=Path, > + default=Path(__file__).resolve().parent.parent, > + help="Path to documentation/ directory in yocto-docs") > + > + return parser.parse_args() > + > + > +# Map of "confusable" characters that are frequently introduced by editors, > +# word processors or copy-pasting, to their plain ASCII replacement. These > +# look almost identical to regular ASCII but break tooling, e.g. a curly > +# quote in a recipe example causes: > +# > +# ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = "foo"' I don't see the curly quote here > +# > +# Only these characters are flagged; legitimate non-ASCII such as box-drawing > +# characters used in directory trees, accented letters in contributor names > +# and CJK characters are intentionally left alone. > +confusables = { > + "‘": "'", # LEFT SINGLE QUOTATION MARK > + "’": "'", # RIGHT SINGLE QUOTATION MARK > + "“": '"', # LEFT DOUBLE QUOTATION MARK > + "”": '"', # RIGHT DOUBLE QUOTATION MARK > + "′": "'", # PRIME > + "″": '"', # DOUBLE PRIME > + "–": "-", # EN DASH > + "—": "--", # EM DASH > + "‐": "-", # HYPHEN > + "‑": "-", # NON-BREAKING HYPHEN > + "−": "-", # MINUS SIGN > + "…": "...", # HORIZONTAL ELLIPSIS > + " ": " ", # NO-BREAK SPACE Actually, since the `tree` command uses no-break spaces, I think it's OK to keep them otherwise we'd have to convert each tree output, which also might be confusing. > + " ": " ", # NARROW NO-BREAK SPACE > + "": "", # ZERO WIDTH SPACE > + "": "", # ZERO WIDTH NO-BREAK SPACE / BOM > + "": "", # SOFT HYPHEN > +} > + > + > +def main(): > + > + args = parse_arguments() > + exit_code = 0 > + > + for rst_path in sorted(Path(args.docs_dir).rglob("*.rst")): > + rel = rst_path.relative_to(args.docs_dir) > + > + with open(rst_path, "r", encoding="utf-8") as f: > + for lineno, line in enumerate(f, start=1): > + for col, char in enumerate(line, start=1): > + if char in confusables: > + replacement = confusables[char] > + hint = (f"'{replacement}'" if replacement > + else "(remove)") > + print(f"WARNING: {rel}:{lineno}:{col}: non-ASCII " > + f"character U+{ord(char):04X} should be " > + f"replaced with {hint}") > + exit_code = 1 > + > + sys.exit(exit_code) > + > + > +if __name__ == "__main__": > + main() Antonin
On 7/28/26 10:53 AM, Antonin Godard wrote: > Hi, > > On Tue Jul 28, 2026 at 9:23 AM CEST, Niko Mauno via lists.yoctoproject.org wrote: >> From: Niko Mauno <niko.mauno@vaisala.com> >> >> Add a check-confusables script, in the same fashion as >> check-glossaries, that scans the documentation .rst sources for >> non-ASCII "confusable" characters (curly quotes, en/em dashes, >> horizontal ellipsis, non-breaking and zero-width spaces, etc.) and >> reports each occurrence with its location and suggested ASCII >> replacement, exiting non-zero if any are found. This guards against >> the class of breakage fixed in the preceding commit, e.g. curly quotes >> causing recipe ParseErrors. >> >> Legitimate non-ASCII such as box-drawing characters used in directory >> trees, accented letters in contributor names and CJK characters are >> intentionally left untouched. >> >> Wire it up both as a local pre-commit hook and in the Makefile "checks" >> target, alongside check-glossaries. >> >> Suggested-by: Quentin Schulz <quentin.schulz@cherry.de> >> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> >> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com> >> --- >> .pre-commit-config.yaml | 5 ++ >> documentation/Makefile | 1 + >> documentation/tools/check-confusables | 77 +++++++++++++++++++++++++++ >> 3 files changed, 83 insertions(+) >> create mode 100755 documentation/tools/check-confusables >> >> diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml >> index f2b73a481..d6008b609 100644 >> --- a/.pre-commit-config.yaml >> +++ b/.pre-commit-config.yaml >> @@ -6,3 +6,8 @@ repos: >> entry: ./documentation/tools/check-glossaries >> language: python >> pass_filenames: false >> + - id: check-confusables >> + name: Check for non-ASCII confusable characters >> + entry: ./documentation/tools/check-confusables >> + language: python >> + pass_filenames: false > > I think we should pass filenames and check only the modified files, no need to > checking the entire tree each time? > >> diff --git a/documentation/Makefile b/documentation/Makefile >> index fe0574537..87a6f8a8b 100644 >> --- a/documentation/Makefile >> +++ b/documentation/Makefile >> @@ -37,6 +37,7 @@ clean: >> >> checks: >> $(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR) >> + $(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR) >> >> stylecheck: >> vale sync >> diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables >> new file mode 100755 >> index 000000000..f79ee046c >> --- /dev/null >> +++ b/documentation/tools/check-confusables >> @@ -0,0 +1,77 @@ >> +#!/usr/bin/env python3 >> + >> +import argparse >> +import sys >> + >> +from pathlib import Path >> + >> + >> +def parse_arguments() -> argparse.Namespace: >> + parser = argparse.ArgumentParser( >> + description="Check documentation sources for non-ASCII typographic " >> + "characters that should be plain ASCII") >> + >> + parser.add_argument("-d", "--docs-dir", >> + type=Path, >> + default=Path(__file__).resolve().parent.parent, >> + help="Path to documentation/ directory in yocto-docs") >> + >> + return parser.parse_args() >> + >> + >> +# Map of "confusable" characters that are frequently introduced by editors, >> +# word processors or copy-pasting, to their plain ASCII replacement. These >> +# look almost identical to regular ASCII but break tooling, e.g. a curly >> +# quote in a recipe example causes: >> +# >> +# ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = "foo"' > > I don't see the curly quote here > >> +# >> +# Only these characters are flagged; legitimate non-ASCII such as box-drawing >> +# characters used in directory trees, accented letters in contributor names >> +# and CJK characters are intentionally left alone. >> +confusables = { >> + "‘": "'", # LEFT SINGLE QUOTATION MARK >> + "’": "'", # RIGHT SINGLE QUOTATION MARK >> + "“": '"', # LEFT DOUBLE QUOTATION MARK >> + "”": '"', # RIGHT DOUBLE QUOTATION MARK >> + "′": "'", # PRIME >> + "″": '"', # DOUBLE PRIME >> + "–": "-", # EN DASH >> + "—": "--", # EM DASH >> + "‐": "-", # HYPHEN >> + "‑": "-", # NON-BREAKING HYPHEN >> + "−": "-", # MINUS SIGN >> + "…": "...", # HORIZONTAL ELLIPSIS >> + " ": " ", # NO-BREAK SPACE > > Actually, since the `tree` command uses no-break spaces, I think it's OK to keep > them otherwise we'd have to convert each tree output, which also might be > confusing. > >> + " ": " ", # NARROW NO-BREAK SPACE >> + "": "", # ZERO WIDTH SPACE >> + "": "", # ZERO WIDTH NO-BREAK SPACE / BOM >> + "": "", # SOFT HYPHEN >> +} >> + >> + >> +def main(): >> + >> + args = parse_arguments() >> + exit_code = 0 >> + >> + for rst_path in sorted(Path(args.docs_dir).rglob("*.rst")): >> + rel = rst_path.relative_to(args.docs_dir) >> + >> + with open(rst_path, "r", encoding="utf-8") as f: >> + for lineno, line in enumerate(f, start=1): >> + for col, char in enumerate(line, start=1): >> + if char in confusables: >> + replacement = confusables[char] >> + hint = (f"'{replacement}'" if replacement >> + else "(remove)") >> + print(f"WARNING: {rel}:{lineno}:{col}: non-ASCII " >> + f"character U+{ord(char):04X} should be " >> + f"replaced with {hint}") >> + exit_code = 1 >> + >> + sys.exit(exit_code) >> + >> + >> +if __name__ == "__main__": >> + main() > > > Antonin Thank You Antonin, I have now submitted v3 which should address the observations You pointed out above, plus adds Quentin's Reviewed-by line in the bottom three commits which remained unchanged since v1, and also leaves out conversion of "horizontal ellipsis" characters which seemed to be used for denoting git hash truncation and therefore seemed to add just noise to the fix commit. -Niko
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index f2b73a481..d6008b609 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -6,3 +6,8 @@ repos: entry: ./documentation/tools/check-glossaries language: python pass_filenames: false + - id: check-confusables + name: Check for non-ASCII confusable characters + entry: ./documentation/tools/check-confusables + language: python + pass_filenames: false diff --git a/documentation/Makefile b/documentation/Makefile index fe0574537..87a6f8a8b 100644 --- a/documentation/Makefile +++ b/documentation/Makefile @@ -37,6 +37,7 @@ clean: checks: $(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR) + $(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR) stylecheck: vale sync diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables new file mode 100755 index 000000000..f79ee046c --- /dev/null +++ b/documentation/tools/check-confusables @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 + +import argparse +import sys + +from pathlib import Path + + +def parse_arguments() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Check documentation sources for non-ASCII typographic " + "characters that should be plain ASCII") + + parser.add_argument("-d", "--docs-dir", + type=Path, + default=Path(__file__).resolve().parent.parent, + help="Path to documentation/ directory in yocto-docs") + + return parser.parse_args() + + +# Map of "confusable" characters that are frequently introduced by editors, +# word processors or copy-pasting, to their plain ASCII replacement. These +# look almost identical to regular ASCII but break tooling, e.g. a curly +# quote in a recipe example causes: +# +# ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = "foo"' +# +# Only these characters are flagged; legitimate non-ASCII such as box-drawing +# characters used in directory trees, accented letters in contributor names +# and CJK characters are intentionally left alone. +confusables = { + "‘": "'", # LEFT SINGLE QUOTATION MARK + "’": "'", # RIGHT SINGLE QUOTATION MARK + "“": '"', # LEFT DOUBLE QUOTATION MARK + "”": '"', # RIGHT DOUBLE QUOTATION MARK + "′": "'", # PRIME + "″": '"', # DOUBLE PRIME + "–": "-", # EN DASH + "—": "--", # EM DASH + "‐": "-", # HYPHEN + "‑": "-", # NON-BREAKING HYPHEN + "−": "-", # MINUS SIGN + "…": "...", # HORIZONTAL ELLIPSIS + " ": " ", # NO-BREAK SPACE + " ": " ", # NARROW NO-BREAK SPACE + "": "", # ZERO WIDTH SPACE + "": "", # ZERO WIDTH NO-BREAK SPACE / BOM + "": "", # SOFT HYPHEN +} + + +def main(): + + args = parse_arguments() + exit_code = 0 + + for rst_path in sorted(Path(args.docs_dir).rglob("*.rst")): + rel = rst_path.relative_to(args.docs_dir) + + with open(rst_path, "r", encoding="utf-8") as f: + for lineno, line in enumerate(f, start=1): + for col, char in enumerate(line, start=1): + if char in confusables: + replacement = confusables[char] + hint = (f"'{replacement}'" if replacement + else "(remove)") + print(f"WARNING: {rel}:{lineno}:{col}: non-ASCII " + f"character U+{ord(char):04X} should be " + f"replaced with {hint}") + exit_code = 1 + + sys.exit(exit_code) + + +if __name__ == "__main__": + main()