diff mbox series

[AUH,v2,9/9] upgrade-helper.py: Limit number of requests to remote servers

Message ID 20260424114603.2444938-10-daniel.turull@ericsson.com
State New
Headers show
Series upgrade_helper: scarthgap compatibility, stable updates and changelog extraction | expand

Commit Message

Daniel Turull April 24, 2026, 11:46 a.m. UTC 
From: Daniel Turull <daniel.turull@ericsson.com>

In systems with lots of CPUs, we overflood some remote servers
and then we get timeouts because of rate control.

Be more conservative. Defaults to 8 parallel workers instead of 64.
Configurable via scan_workers=N in upgrade-helper.conf

For example, ftp.gnu.org was timing out

Assisted-by: Claude, Anthropic
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 upgrade-helper.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

Comments

Alexander Kanavin April 27, 2026, 10:35 a.m. UTC | #1 
On Fri, 24 Apr 2026 at 13:46, <daniel.turull@ericsson.com> wrote:
> +                pkggroups_iter = oe.recipeutils.get_recipe_upgrade_status(layer_recipes)

Any such rate limiting needs to be handled inside this function by
passing in a parameter, and certainly not by hacking
oe.utils.cpu_count. Also, I'm not seeing the issue locally (256 cores,
64 worker threads in 'devtool check-upgrade-status'), are you
redirected to some gnu mirror which does it?

I understand that you want to automate scarthgap updates, and such
tweaks can't be easily backported in oe-core, but code shouldn't just
'fix issues', it also needs to be maintainable in the long term.

Alex
Daniel Turull April 27, 2026, 12:31 p.m. UTC | #2 
Hi,

I was experimenting to run it with an early version if scarthgap and GNU_MIRROR was not corrected.

-GNU_MIRROR = "https://ftp.gnu.org/gnu"
+GNU_MIRROR = "https://ftpmirror.gnu.org/gnu"

We can probably drop it, just wanted to be a bit more gentle against upstream servers. I'll remove it from v3

Daniel

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: Monday, 27 April 2026 12:35
> To: Daniel Turull <daniel.turull@ericsson.com>
> Cc: yocto-patches@lists.yoctoproject.org; paul@pbarker.dev;
> ross.burton@arm.com; yoann.congal@smile.fr
> Subject: Re: [AUH][PATCH v2 9/9] upgrade-helper.py: Limit number of
> requests to remote servers
> 
> On Fri, 24 Apr 2026 at 13:46, <daniel.turull@ericsson.com> wrote:
> > +                pkggroups_iter =
> > + oe.recipeutils.get_recipe_upgrade_status(layer_recipes)
> 
> Any such rate limiting needs to be handled inside this function by passing in a
> parameter, and certainly not by hacking oe.utils.cpu_count. Also, I'm not
> seeing the issue locally (256 cores,
> 64 worker threads in 'devtool check-upgrade-status'), are you redirected to
> some gnu mirror which does it?
> 
> I understand that you want to automate scarthgap updates, and such tweaks
> can't be easily backported in oe-core, but code shouldn't just 'fix issues', it
> also needs to be maintainable in the long term.
> 
> Alex
diff mbox series

Patch

diff --git a/upgrade-helper.py b/upgrade-helper.py
index 58fd660..11b8d58 100755
--- a/upgrade-helper.py
+++ b/upgrade-helper.py
@@ -808,9 +808,17 @@  class UniverseUpdater(Updater):
             return pkg_to_upgrade
 
         upgrade_pkggroups = []
+        scan_workers = int(settings.get("scan_workers", "8"))
 
         for layer_name, layer_recipes in self.recipes:
-            pkggroups_iter = oe.recipeutils.get_recipe_upgrade_status(layer_recipes)
+            # Limit parallel version checks to avoid overwhelming upstream
+            # servers (oe-core defaults to cpu_count which can be 64+).
+            _orig_cpu_count = oe.utils.cpu_count
+            oe.utils.cpu_count = lambda at_least=1, at_most=64: min(max(at_least, scan_workers), at_most)
+            try:
+                pkggroups_iter = oe.recipeutils.get_recipe_upgrade_status(layer_recipes)
+            finally:
+                oe.utils.cpu_count = _orig_cpu_count
 
             # Collect results, logging any fetch error that terminates
             # the iterator early (oe-core raises inside executor.map,