diff mbox series

[AUH,v2,8/9] upgrade-helper.py: Add resume option to skip already attempted recipes

Message ID 20260424114603.2444938-9-daniel.turull@ericsson.com
State New
Headers show
Series upgrade_helper: scarthgap compatibility, stable updates and changelog extraction | expand

Commit Message

Daniel Turull April 24, 2026, 11:46 a.m. UTC
From: Daniel Turull <daniel.turull@ericsson.com>

Add --resume option that reads the most recent run's log and skips
recipes that were already attempted (either upgraded or failed pre-build).

This allows restarting an interrupted run without re-processing recipes
that have already been handled, saving significant time on large runs
(e.g. 12h for 135 recipes).

The skip set is built by parsing 'Upgrading to' and 'Pre-build FAILED'
lines from the previous run's upgrade-helper.log.

Assisted-by: Claude, Anthropic
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 upgrade-helper.py | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

Comments

Alexander Kanavin April 27, 2026, 10:48 a.m. UTC | #1
On Fri, 24 Apr 2026 at 13:46, <daniel.turull@ericsson.com> wrote:
> The skip set is built by parsing 'Upgrading to' and 'Pre-build FAILED'
> lines from the previous run's upgrade-helper.log.

I wonder if this could be less brittle? For instance the skip set
could be built by checking if the upgrade output is symlinked in
failed/ or succeeded/. Currently those symlinks are created after
everything is finished, but it could be done directly after each
particular update.

Alex
Daniel Turull April 27, 2026, 1:21 p.m. UTC | #2
> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: Monday, 27 April 2026 12:49
> To: Daniel Turull <daniel.turull@ericsson.com>
> Cc: yocto-patches@lists.yoctoproject.org; paul@pbarker.dev;
> ross.burton@arm.com; yoann.congal@smile.fr
> Subject: Re: [AUH][PATCH v2 8/9] upgrade-helper.py: Add resume option to
> skip already attempted recipes
> 
> On Fri, 24 Apr 2026 at 13:46, <daniel.turull@ericsson.com> wrote:
> > The skip set is built by parsing 'Upgrading to' and 'Pre-build FAILED'
> > lines from the previous run's upgrade-helper.log.
> 
> I wonder if this could be less brittle? For instance the skip set could be built
> by checking if the upgrade output is symlinked in failed/ or succeeded/.
> Currently those symlinks are created after everything is finished, but it could
> be done directly after each particular update.

Good idea. I'll redo the patch as you suggested in v3.
diff mbox series

Patch

diff --git a/upgrade-helper.py b/upgrade-helper.py
index 11a97d2..58fd660 100755
--- a/upgrade-helper.py
+++ b/upgrade-helper.py
@@ -108,6 +108,9 @@  def parse_cmdline():
     parser.add_argument("--prebuild", action="store_true", default=False,
                         help="build each recipe at its current version before attempting upgrade")
 
+    parser.add_argument("--resume", action="store_true", default=False,
+                        help="skip recipes already attempted in the last run")
+
     parser.add_argument("-d", "--debug-level", type=int, default=4, choices=range(1, 6),
                         help="set the debug level: CRITICAL=1, ERROR=2, WARNING=3, INFO=4, DEBUG=5")
     parser.add_argument("-e", "--send-emails", action="store_true", default=False,
@@ -515,9 +518,33 @@  class Updater(object):
         succeeded_pkggroups_ctx = []
         failed_pkggroups_ctx = []
         attempted_pkggroups = 0
+
+        # --resume: skip recipes already attempted in the most recent run
+        resume_skip = set()
+        if self.args.resume:
+            import glob as _glob
+            prev_runs = sorted(_glob.glob(os.path.join(self.uh_dir, "20*")))
+            # skip the current run dir (last entry) if it exists
+            prev_runs = [r for r in prev_runs if r != self.uh_work_dir]
+            if prev_runs:
+                prev_log = os.path.join(prev_runs[-1], "upgrade-helper.log")
+                if os.path.isfile(prev_log):
+                    with open(prev_log) as f:
+                        for line in f:
+                            m = re.match(r'\s+(\S+): (?:Upgrading to|Pre-build FAILED)', line)
+                            if m:
+                                resume_skip.add(m.group(1))
+                    I(" --resume: skipping %d recipes from %s" %
+                      (len(resume_skip), prev_runs[-1]))
         for g in pkggroups_ctx:
             attempted_pkggroups += 1
             pkggroup_name = g["name"]
+
+            if any(p['PN'] in resume_skip for p in g['pkgs']):
+                I(" SKIP PACKAGE GROUP %d/%d (already attempted): %s" %
+                  (attempted_pkggroups, total_pkggroups, pkggroup_name))
+                continue
+
             I(" ATTEMPT PACKAGE GROUP %d/%d" % (attempted_pkggroups, total_pkggroups))
 
             # Build current version first to verify it builds and seed ABI reference