diff mbox series

[3/4] recipeutils/get_recipe_upgrade_status: group recipes when they need to be upgraded together

Message ID 20240717182216.1661015-3-alex.kanavin@gmail.com
State New
Headers show
Series [1/4] lib/oe/recipeutils: return a dict in get_recipe_upgrade_status() instead of a tuple | expand

Commit Message

Alexander Kanavin July 17, 2024, 6:22 p.m. UTC
From: Alexander Kanavin <alex@linutronix.de>

This will allow 'lockstep upgrades' of such recipes, improving success
rates in automated version updating process.

devtool check-upgrade-status now prints:

These recipes need to be upgraded together {
glib-2.0                  2.80.2          2.80.4          Anuj Mittal <anuj.mittal@intel.com>
glib-2.0-initial          2.80.2          2.80.4          Anuj Mittal <anuj.mittal@intel.com>
}
These recipes need to be upgraded together {
util-linux                2.39.3          2.40.2          Chen Qi <Qi.Chen@windriver.com>
util-linux-libuuid        2.39.3          2.40.2          Chen Qi <Qi.Chen@windriver.com>
}
These recipes need to be upgraded together {
cmake                     3.29.3          3.30.0          Unassigned <unassigned@yoctoproject.org>
cmake-native              3.29.3          3.30.0          Unassigned <unassigned@yoctoproject.org>
}

etc.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/oe/recipeutils.py                 | 23 +++++++++++++++++-
 meta/lib/oeqa/selftest/cases/distrodata.py |  6 ++---
 scripts/lib/devtool/upgrade.py             | 28 +++++++++++++++-------
 3 files changed, 44 insertions(+), 13 deletions(-)

Comments

patchtest@automation.yoctoproject.org July 17, 2024, 6:38 p.m. UTC | #1
Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/3-4-recipeutils-get_recipe_upgrade_status-group-recipes-when-they-need-to-be-upgraded-together.patch

FAIL: test max line length: Patch line too long (current length 217, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)
FAIL: test shortlog length: Edit shortlog so that it is 90 characters or less (currently 91 characters) (test_mbox.TestMbox.test_shortlog_length)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!
diff mbox series

Patch

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 7586332fe0c..56be75dc9c9 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -1154,6 +1154,7 @@  def get_recipe_upgrade_status(recipes=None):
         if not recipes:
             recipes = tinfoil.all_recipe_files(variants=False)
 
+        recipeincludes = {}
         for fn in recipes:
             try:
                 if fn.startswith("/"):
@@ -1178,11 +1179,13 @@  def get_recipe_upgrade_status(recipes=None):
 
             data_copy_list.append(data_copy)
 
+            recipeincludes[data.getVar('FILE')] = {'bbincluded':data.getVar('BBINCLUDED').split(),'pn':data.getVar('PN')}
+
     from concurrent.futures import ProcessPoolExecutor
     with ProcessPoolExecutor(max_workers=utils.cpu_count()) as executor:
         pkgs_list = executor.map(_get_recipe_upgrade_status, data_copy_list)
 
-    return pkgs_list
+    return _group_recipes(pkgs_list, _get_common_include_recipes(recipeincludes))
 
 def get_common_include_recipes():
     with bb.tinfoil.Tinfoil() as tinfoil:
@@ -1220,3 +1223,21 @@  def _get_common_include_recipes(recipeincludes_all):
                 recipes_with_shared_includes.append(recipeset)
 
         return recipes_with_shared_includes
+
+def _group_recipes(recipes, groups):
+    recipedict = {}
+    for r in recipes:
+        recipedict[r['pn']] = r
+
+    recipegroups = []
+    for g in groups:
+        recipeset = []
+        for r in g:
+            if r in recipedict.keys():
+                recipeset.append(recipedict[r])
+                del recipedict[r]
+        recipegroups.append(recipeset)
+
+    for r in recipedict.values():
+        recipegroups.append([r])
+    return recipegroups
diff --git a/meta/lib/oeqa/selftest/cases/distrodata.py b/meta/lib/oeqa/selftest/cases/distrodata.py
index bc561605220..bd375523649 100644
--- a/meta/lib/oeqa/selftest/cases/distrodata.py
+++ b/meta/lib/oeqa/selftest/cases/distrodata.py
@@ -20,10 +20,10 @@  class Distrodata(OESelftestTestCase):
         feature = 'LICENSE_FLAGS_ACCEPTED += " commercial"\n'
         self.write_config(feature)
 
-        pkgs = oe.recipeutils.get_recipe_upgrade_status()
+        pkggroups = oe.recipeutils.get_recipe_upgrade_status()
 
-        regressed_failures = [pkg['pn'] for pkg in pkgs if pkg['status'] == 'UNKNOWN_BROKEN']
-        regressed_successes = [pkg['pn'] for pkg in pkgs if pkg['status'] == 'KNOWN_BROKEN']
+        regressed_failures = [pkg['pn'] for pkgs in pkggroups for pkg in pkgs if pkg['status'] == 'UNKNOWN_BROKEN']
+        regressed_successes = [pkg['pn'] for pkgs in pkggroups for pkg in pkgs if pkg['status'] == 'KNOWN_BROKEN']
         msg = ""
         if len(regressed_failures) > 0:
             msg = msg + """
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 10b4f8b5ee5..4c268af3a7c 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -654,18 +654,28 @@  def latest_version(args, config, basepath, workspace):
     return 0
 
 def check_upgrade_status(args, config, basepath, workspace):
+    def _print_status(recipe):
+        print("{:25} {:15} {:15} {} {} {}".format(   recipe['pn'],
+                                                               recipe['cur_ver'],
+                                                               recipe['status'] if recipe['status'] != 'UPDATE' else (recipe['next_ver'] if not recipe['next_ver'].endswith("new-commits-available") else "new commits"),
+                                                               recipe['maintainer'],
+                                                               recipe['revision'] if recipe['revision'] != 'N/A' else "",
+                                                               "cannot be updated due to: %s" %(recipe['no_upgrade_reason']) if recipe['no_upgrade_reason'] else ""))
     if not args.recipe:
         logger.info("Checking the upstream status for all recipes may take a few minutes")
     results = oe.recipeutils.get_recipe_upgrade_status(args.recipe)
-    for result in results:
-        # pn, update_status, current, latest, maintainer, latest_commit, no_update_reason
-        if args.all or result['status'] != 'MATCH':
-            print("{:25} {:15} {:15} {} {} {}".format(   result['pn'],
-                                                               result['cur_ver'],
-                                                               result['status'] if result['status'] != 'UPDATE' else (result['next_ver'] if not result['next_ver'].endswith("new-commits-available") else "new commits"),
-                                                               result['maintainer'],
-                                                               result['revision'] if result['revision'] != 'N/A' else "",
-                                                               "cannot be updated due to: %s" %(result['no_upgrade_reason']) if result['no_upgrade_reason'] else ""))
+    for recipegroup in results:
+        upgrades = [r for r in recipegroup if r['status'] != 'MATCH']
+        currents = [r for r in recipegroup if r['status'] == 'MATCH']
+        if len(upgrades) > 1:
+            print("These recipes need to be upgraded together {")
+        for r in upgrades:
+            _print_status(r)
+        if len(upgrades) > 1:
+            print("}")
+        for r in currents:
+            if args.all:
+                _print_status(r)
 
 def register_commands(subparsers, context):
     """Register devtool subcommands from this plugin"""