mbox series

[AUH,v2,0/9] upgrade_helper: scarthgap compatibility, stable updates and changelog extraction

Message ID 20260424114603.2444938-1-daniel.turull@ericsson.com
Headers show
Series upgrade_helper: scarthgap compatibility, stable updates and changelog extraction | expand

Message

Daniel Turull April 24, 2026, 11:45 a.m. UTC 
From: Daniel Turull <daniel.turull@ericsson.com>

This patch series allow to run upgrade-helper both in master and scarthgap.

This allows to be able to automatically generate patches for updating patched or
minor versions, which can help on the security work fixing vulnerabilities.

It adds also the option to extract the changelog with different strategies

It can also prebuild the recipe to be able to compare before and after the update,
so it could be connected with ABI changes with meta-binaryaudit [1],
so if there are incompatible changes the upgrade fails.

Example with OE-core scarthgap, there are 156 potential updates

vte: no suitable version available, skipping
 ########### The list of recipes to be upgraded #############
 , libubootenv, 0.3.5, 0.3.7, Stefano Babic <sbabic@denx.de>, 1e3511ed77f794ee5decc0974d54c8e5af26f64c
 , bind, 9.18.44, 9.18.48, Unassigned <unassigned@yoctoproject.org>, N/A
 , dhcpcd, 10.0.6, 10.0.10, Unassigned <unassigned@yoctoproject.org>, N/A
 , libpcap, 1.10.4, 1.10.6, Anuj Mittal <anuj.mittal@intel.com>, N/A
 , ppp, 2.5.0, 2.5.2, Hongxu Jia <hongxu.jia@windriver.com>, N/A
 , socat, 1.8.0.0, 1.8.0.3, Hongxu Jia <hongxu.jia@windriver.com>, N/A
 , base-passwd, 3.6.3, 3.6.8, Anuj Mittal <anuj.mittal@intel.com>, N/A
 , ifupdown, 0.8.41, 0.8.45, Anuj Mittal <anuj.mittal@intel.com>, N/A
 , libxcrypt-compat, 4.4.36, 4.4.38, Khem Raj <raj.khem@gmail.com>, N/A
 , libxcrypt, 4.4.36, 4.4.38, Khem Raj <raj.khem@gmail.com>, N/A
 , ovmf, edk2-stable202402, edk2-stable202602, Unassigned <unassigned@yoctoproject.org>, b7a715f7c03c45c6b4575bf88596bfd79658b8ce
 , util-linux-libuuid, 2.39.3, 2.39.4, Chen Qi <Qi.Chen@windriver.com>, N/A
 , util-linux, 2.39.3, 2.39.4, Chen Qi <Qi.Chen@windriver.com>, N/A
...
 , numactl, 2.0.18, 2.0.19, Richard Purdie <richard.purdie@linuxfoundation.org>, 3bc85e37d5a30da6790cb7e8bb488bb8f679170f
 , taglib, 2.0.1, 2.0.2, Anuj Mittal <anuj.mittal@intel.com>, N/A
 , utfcpp, 4.0.5, 4.0.9, Anuj Mittal <anuj.mittal@intel.com>, 63d64de49fd6b829f7c8694df5ab2ee625cb7134
 , xxhash, 0.8.2, 0.8.3, Alexander Kanavin <alex.kanavin@gmail.com>, e626a72bc2321cd320e953a0ccf1584cad60f363

[...]
 ############################################################
 Building gcc runtimes ...
  building gcc runtime for qemux86-64
  building gcc runtime for qemuarm_musl
 ATTEMPT PACKAGE GROUP 1/156
[...]

Tested with on openembedded-core with latest scarthgap version (2026-04-07)
../auto-upgrade-helper/upgrade-helper.py all --layer-dir ../openembedded-core --changelog  --stable --prebuild

    TOTAL: attempted=135 succeeded=81(60.00%) failed=54(40.00%)

Examples of the patches in my contrib branch [2]

[1] https://github.com/Nordix/meta-binaryaudit
[2] https://git.openembedded.org/openembedded-core-contrib/log/?h=dturull/scarthgap-auh

Changes in v2:
- Split scarthgap compat into two commits: compatibility [1/9] and
  fetch error handling [2/9]
- Fix iterator exhaustion: materialise executor.map() results into a
  list so a single fetch error no longer silently truncates the scan [2/9]
- Split --stable into its own commit (was squashed into changelog) [5/9]
- Fix --minor help text and docstring to accurately describe behavior
  (any same-major upgrade); drop version length equality constraint [6/9]
- Fix --prebuild to pass space-separated targets to bitbake instead
  of comma-joined group name [7/9]
- Fix --resume to match individual recipe names against the skip set
  instead of the comma-joined group name [8/9]
- Add scan_workers rate limiting to avoid overwhelming upstream servers
  during version checks (configurable via scan_workers in config) [9/9]
- Reorder commits so each feature is independently functional:
  scarthgap compat, layer-dir, changelog, stable, minor, prebuild,
  resume, scan_workers

Daniel Turull (9):
  upgrade-helper.py: Add compatibility with Yocto scarthgap
  upgrade-helper.py: Handle fetch errors gracefully during version scan
  upgrade-helper.py: allow to set the git directory from layer-dir
  upgrade-helper.py: add changelog flag
  upgrade-helper.py: Add stable option for patch-only upgrades
  upgrade-helper.py: Add minor option for minor version upgrades
  upgrade-helper.py: Add prebuild option
  upgrade-helper.py: Add resume option to skip already attempted recipes
  upgrade-helper.py: Limit number of requests to remote servers

 modules/changelog.py     | 256 +++++++++++++++++++++++++++++++++++++++
 modules/steps.py         |  22 +++-
 modules/utils/bitbake.py |   8 +-
 modules/utils/version.py | 175 ++++++++++++++++++++++++++
 upgrade-helper.py        | 152 ++++++++++++++++++++++-
 5 files changed, 609 insertions(+), 4 deletions(-)
 create mode 100644 modules/changelog.py
 create mode 100644 modules/utils/version.py