| Message ID | 20240717182216.1661015-2-alex.kanavin@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/4] lib/oe/recipeutils: return a dict in get_recipe_upgrade_status() instead of a tuple | expand |
On Wed, 2024-07-17 at 20:22 +0200, Alexander Kanavin via lists.openembedded.org wrote: > From: Alexander Kanavin <alex@linutronix.de> > > This functionality is needed for 'lockstep version upgrades' where several > recipes need to be upgraded at the same time to produce a buildable > outcome. > > The function itself obtains BBINCLUDED for each recipe and then massages > the data until it takes the form of a list of sets: > > [{'cmake','cmake-native'}, > {'qemu','qemu-native','qemu-system-native'}, > ... ] > > There's also a selftest that checks for the above. > > Unfortunately this won't detect mutually exclusive recipes like mesa and mesa-gl > as they're chosen with PREFERRED_PROVIDER and can't be enabled in the same build > at the same time. ('devtool upgrade' will also accept just one of them but not the other) This is partly why I was suggesting the internal list of files that bitbake has instead of BBINCLUDED, even if it comes from the same data. Even if a recipe is skipped (as conflicting providers are), I think the cache entry in bitbake is still needed to know if/when to reparse the recipe. Your patch is good and I'm happy to merge as is, I just wanted to mention that it might be possible to catch the mesa issue. The challenge is that even if it were identifiable, the code still probably can't know how to actually enable it for the upgrade/testing :(. Cheers, Richard
On Mon, 22 Jul 2024 at 16:44, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > Your patch is good and I'm happy to merge as is, I just wanted to > mention that it might be possible to catch the mesa issue. Thanks, I just pushed the corresponding changes to AUH. It's possible next AUH run won't be perfect, we have to wait and see (I did test passing/failing lockstep upggrades with glib, but AUH is notorious for running into corner cases over the complete recipe set). But you should be getting perfect qemu upgrade patches from now on ;) > The challenge is that even if it were identifiable, the code still > probably can't know how to actually enable it for the upgrade/testing > :(. Yes. We could insert that data into the recipe to be used by AUH/devtool, but that's even more work, and all for a single known recipe in core (mesa). For mesa I'm tempted to go back to the 'simple file rename' proposal that's already implemented; in reality no one tests mesa-gl either when they submit mesa version updates. And we have to keep telling people to include mesa-gl, because it's only natural to overlook that. Alex
On Tue, 2024-07-23 at 09:15 +0200, Alexander Kanavin wrote: > On Mon, 22 Jul 2024 at 16:44, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > Your patch is good and I'm happy to merge as is, I just wanted to > > mention that it might be possible to catch the mesa issue. > > Thanks, I just pushed the corresponding changes to AUH. It's possible > next AUH run won't be perfect, we have to wait and see (I did test > passing/failing lockstep upggrades with glib, but AUH is notorious > for > running into corner cases over the complete recipe set). But you > should be getting perfect qemu upgrade patches from now on ;) > > > The challenge is that even if it were identifiable, the code still > > probably can't know how to actually enable it for the > > upgrade/testing > > :(. > > Yes. We could insert that data into the recipe to be used by > AUH/devtool, but that's even more work, and all for a single known > recipe in core (mesa). For mesa I'm tempted to go back to the 'simple > file rename' proposal that's already implemented; in reality no one > tests mesa-gl either when they submit mesa version updates. And we > have to keep telling people to include mesa-gl, because it's only > natural to overlook that. FWIW, PV doesn't have to come from the filename. We could set PV from the inc file and rename the files so PV isn't there... Cheers, Richard
On Tue, 23 Jul 2024 at 09:21, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > FWIW, PV doesn't have to come from the filename. We could set PV from > the inc file and rename the files so PV isn't there... That's fine with me too. Let's do that then. Alex
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py index f9d7dfe253a..7586332fe0c 100644 --- a/meta/lib/oe/recipeutils.py +++ b/meta/lib/oe/recipeutils.py @@ -1183,3 +1183,40 @@ def get_recipe_upgrade_status(recipes=None): pkgs_list = executor.map(_get_recipe_upgrade_status, data_copy_list) return pkgs_list + +def get_common_include_recipes(): + with bb.tinfoil.Tinfoil() as tinfoil: + tinfoil.prepare(config_only=False) + + recipes = tinfoil.all_recipe_files(variants=False) + + recipeincludes = {} + for fn in recipes: + data = tinfoil.parse_recipe_file(fn) + recipeincludes[fn] = {'bbincluded':data.getVar('BBINCLUDED').split(),'pn':data.getVar('PN')} + return _get_common_include_recipes(recipeincludes) + +def _get_common_include_recipes(recipeincludes_all): + recipeincludes = {} + for fn,data in recipeincludes_all.items(): + bbincluded_filtered = [i for i in data['bbincluded'] if os.path.dirname(i) == os.path.dirname(fn) and i != fn] + if bbincluded_filtered: + recipeincludes[data['pn']] = bbincluded_filtered + + recipeincludes_inverted = {} + for k,v in recipeincludes.items(): + for i in v: + recipeincludes_inverted.setdefault(i,set()).add(k) + + recipeincludes_inverted_filtered = {k:v for k,v in recipeincludes_inverted.items() if len(v) > 1} + + recipes_with_shared_includes = list() + for v in recipeincludes_inverted_filtered.values(): + recipeset = v + for v1 in recipeincludes_inverted_filtered.values(): + if recipeset.intersection(v1): + recipeset.update(v1) + if recipeset not in recipes_with_shared_includes: + recipes_with_shared_includes.append(recipeset) + + return recipes_with_shared_includes diff --git a/meta/lib/oeqa/selftest/cases/distrodata.py b/meta/lib/oeqa/selftest/cases/distrodata.py index b60913dbca4..bc561605220 100644 --- a/meta/lib/oeqa/selftest/cases/distrodata.py +++ b/meta/lib/oeqa/selftest/cases/distrodata.py @@ -115,3 +115,15 @@ The list of oe-core recipes with maintainers is empty. This may indicate that th self.fail(""" Unable to find recipes for the following entries in maintainers.inc: """ + "\n".join(['%s' % i for i in missing_recipes])) + + def test_common_include_recipes(self): + """ + Summary: Test that obtaining recipes that share includes between them returns a sane result + Expected: At least cmake and qemu entries are present in the output + Product: oe-core + Author: Alexander Kanavin <alex.kanavin@gmail.com> + """ + recipes = oe.recipeutils.get_common_include_recipes() + + self.assertIn({'qemu-system-native', 'qemu', 'qemu-native'}, recipes) + self.assertIn({'cmake-native', 'cmake'}, recipes)