| Message ID | 20260804060205.179715-1-daniel.turull@ericsson.com |
|---|---|
| Headers | show |
| Series | devtool: upgrade: improve changelog extraction for git-log-style ChangeLogs | expand |
On Tue, 2026-08-04 at 08:02 +0200, Daniel Turull via lists.openembedded.org wrote: > From: Daniel Turull <daniel.turull@ericsson.com> > > bb.process.run() always decodes command output as UTF-8, which raises > UnicodeDecodeError for changelogs containing non-UTF-8 bytes (e.g. > Latin-1 author names). Add _git_show_file(), which redirects > `git show` to a temp file and reads it back with errors='replace' to > tolerate that, and use it for the existing per-version release notes > lookup in _extract_changelog(), which had the same crash exposure. > > AI-Generated: Kiro with Claude Sonnet 5 > Signed-off-by: Daniel Turull <daniel.turull@ericsson.com> > --- > scripts/lib/devtool/upgrade.py | 16 ++++++++++++++-- > 1 file changed, 14 insertions(+), 2 deletions(-) > > diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py > index 495a5b8217..6883c99cce 100644 > --- a/scripts/lib/devtool/upgrade.py > +++ b/scripts/lib/devtool/upgrade.py > @@ -589,6 +589,18 @@ def _resolve_rst_includes(content, srctree): > return ''.join(result) > > > +def _git_show_file(srctree, ref, fname): > + """Return the content of fname at ref. Changelogs occasionally contain > + non-UTF-8 bytes (e.g. Latin-1 author names), which bb.process.run() > + can't handle since it always decodes command output as UTF-8; > + redirecting to a temp file and reading it back leniently avoids that > + restriction.""" > + with tempfile.NamedTemporaryFile(prefix='devtool-changelog') as tmpf: > + _run('git show %s > %s' % (shlex.quote('%s:%s' % (ref, fname)), shlex.quote(tmpf.name)), > srctree) The alternative to use a tmp file was to modify _run to allow other encodings than utf-8, but it will be a bigger refactor. Or not use _run and call directly the raw subprocess calls. I can change it to either option or keep the current patch. > + with open(tmpf.name, 'r', errors='replace') as f: > + return f.read() > + > + > Daniel
From: Daniel Turull <daniel.turull@ericsson.com> devtool upgrade's changelog extraction misattributed unrelated commits when upgrading recipes like nghttp2, whose ChangeLog is the literal output of `git log`, regenerated wholesale on every release. Changes from v1: - Header-skipping in the structural diff now relies on the blank line git log always inserts before the commit message, instead of enumerating header prefixes with a regex. - The UTF-8 decode fix is split into its own patch and applied to both git-show call sites instead of just one. - A single-use helper that only sequenced two operations with no reuse benefit has been inlined into its caller. - The git-diff fallback is now guarded against UnicodeDecodeError too, so a bad byte in the diff hunk itself doesn't crash extraction. Daniel Turull (3): devtool: upgrade: ignore changelogs from 3rd party devtool: upgrade: read changelog blobs via a temp file devtool: upgrade: diff git-log-style changelogs by commit hash scripts/lib/devtool/upgrade.py | 90 +++++++++++++++++++++++++++++----- 1 file changed, 79 insertions(+), 11 deletions(-)