Message ID | 20250729144906.1665908-1-JPEWhacker@gmail.com |
---|---|
State | New |
Headers | show |
Series | [bitbake-devel] cooker: Add layer-specific ignore for dangling appends | expand |
On Tue, 29 Jul 2025 at 16:49, Joshua Watt via lists.openembedded.org <JPEWhacker=gmail.com@lists.openembedded.org> wrote: > Our setup is that we have a "disto" layer that we need to maintain > against several version of Yocto at once. In this disto layer, we make > policy choices that sometimes are tied to a specific version of a recipe > found in a specific version of Yocto using a bbappend; and often that > version is only in one or two Yocto versions, which means that the > bbappend will be dangling in the others. This is where the weak point of the argument is. Yocto does not make concessions to layers that attempt to be compatible across yocto releases, patches that help such layers had been previously rejected, and this would create a precedent to the contrary. It's always possible to make a branch. Alex
On Tue, Jul 29, 2025 at 8:57 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > On Tue, 29 Jul 2025 at 16:49, Joshua Watt via lists.openembedded.org > <JPEWhacker=gmail.com@lists.openembedded.org> wrote: > > Our setup is that we have a "disto" layer that we need to maintain > > against several version of Yocto at once. In this disto layer, we make > > policy choices that sometimes are tied to a specific version of a recipe > > found in a specific version of Yocto using a bbappend; and often that > > version is only in one or two Yocto versions, which means that the > > bbappend will be dangling in the others. > > This is where the weak point of the argument is. Yocto does not make > concessions to layers that attempt to be compatible across yocto > releases, patches that help such layers had been previously rejected, > and this would create a precedent to the contrary. > > It's always possible to make a branch. > Branching isn't an option for us; we are unwilling to add the maintenance overhead of dealing with that. > > Alex >
On Tue, 2025-07-29 at 08:47 -0600, Joshua Watt via lists.openembedded.org wrote: > Implements a new layer-specific variable called > BB_DANGLINGAPPENDS_IGNORE. This variable is similar to the removed > BB_DANGLINGAPPENDS_WARNONLY, except that it is scoped to a specific > layer, and it also causes appends to be completely ignored instead of > warning about them. > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > BACKGROUND AND RATIONAL: > > I was fully on board with removing BB_DANGLINGAPPENDS_WARNONLY, since it > was pretty bad that it operated on all layers if any layer set the > variable. However, I did mention at the time that it was going to cause > us some trouble, but not wanting to hold up a good change, I indicated > that I would attempt a fix for it later. Later is now :) > > After much poking around in our setup and deliberation, the conclusion > that I came to is that we do actually need _something_ along the lines > of being able to ignore dangling appends, at least in a few specific > layers. > > Our setup is that we have a "disto" layer that we need to maintain > against several version of Yocto at once. In this disto layer, we make > policy choices that sometimes are tied to a specific version of a recipe > found in a specific version of Yocto using a bbappend; and often that > version is only in one or two Yocto versions, which means that the > bbappend will be dangling in the others. > > dynamic-layers doesn't work here because we are keying off the Yocto > version, not the existence of some layer (unless there is something I'm > unaware of). > > BBMASK is also a less than satisfactory solution (IMHO). Part of the > reason we have structured our distro layer in the way we have is to > reduce the cognitive overhead required to maintain our dozens of > products that more or less need to behave similarly. Keeping all the > distro stuff together makes the quite easy to reason about, but having > then again to track (probably quite complicated, and possibly > per-product) BBMASKs erodes that maintainability. > > We did go through and eliminate as many dangling appends with other > methods as possible, but at the end of the day, we ended up with some > for which we didn't really see a viable (and maintainable) solution. > > I'm open to suggestions if this change is entirely reprehensible, but > our primary goal is to keep the maintenance of our cross-yocto distro > layer simple. > > I also decided to simplify this to just ignore dangling BBAPPENDS, since > I suspect anyone setting this is simply going to ignore the warnings > anyway, but that can be changed if there's a strong option to do it > differently. > > bitbake/lib/bb/cooker.py | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py > index 1810bcc604..349e27d538 100644 > --- a/bitbake/lib/bb/cooker.py > +++ b/bitbake/lib/bb/cooker.py > @@ -953,7 +953,11 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si > appends_without_recipes[mc] = [] > for _, appendfn in self.collections[mc].bbappends: > if not appendfn in applied_appends: > - appends_without_recipes[mc].append(appendfn) > + layername = self.collections[mc].calc_bbfile_priority(appendfn)[2] > + ignore = self.databuilder.mcdata[mc].getVar("BB_DANGLINGAPPENDS_IGNORE_%s" % layername, > + False) or "no" > + if ignore.lower() not in ("1", "yes", "true"): At least use bb.utils.to_boolean() so this behaves the same as any other true/false value we have. > + appends_without_recipes[mc].append(appendfn) I'm torn on this. On the one hand I do want to support our user base, on the other this is adding back some API we decided we really didn't want. I appreciate it is different in that it is layer specific and it is silent rather than warning, but I'm not convinced either of those things sell me on the idea that we want to encourage people to be doing this. We did consider making this layer specific and decided against that at the time it was removed. The choice here is between defined and explicit configurations and "mashing a load of stuff together" and hoping it works. We directed people to BBMASK as that is much more explicit. I suspect with some distro version overrides, BBMASK can be made to work but it will be more effort that just ignoring any mismatches. Cheers, Richard
On 7/29/25 07:59, Joshua Watt via lists.openembedded.org wrote: > > > On Tue, Jul 29, 2025 at 8:57 AM Alexander Kanavin > <alex.kanavin@gmail.com <mailto:alex.kanavin@gmail.com>> wrote: > > On Tue, 29 Jul 2025 at 16:49, Joshua Watt via lists.openembedded.org > <http://lists.openembedded.org> > <JPEWhacker=gmail.com@lists.openembedded.org > <mailto:gmail.com@lists.openembedded.org>> wrote: > > Our setup is that we have a "disto" layer that we need to maintain > > against several version of Yocto at once. In this disto layer, we > make > > policy choices that sometimes are tied to a specific version of a > recipe > > found in a specific version of Yocto using a bbappend; and often that > > version is only in one or two Yocto versions, which means that the > > bbappend will be dangling in the others. > > This is where the weak point of the argument is. Yocto does not make > concessions to layers that attempt to be compatible across yocto > releases, patches that help such layers had been previously rejected, > and this would create a precedent to the contrary. > > It's always possible to make a branch. > > > Branching isn't an option for us; we are unwilling to add the > maintenance overhead of dealing with that. I have resorted to some dirty tricks to solve problems like this in the past with great success. If you move the incompatible bbappends into their own '$branch' directories in your distro layer, then you can add lines like this to your `layer.conf`: BBFILES += "${@ "${LAYERDIR}/$branch/recipes-*/*/*.bbappend" if d.getVar("DISTRO_CODENAME") == "$branch" else "" }" I just tested that this trick does appear to work. But as no one has jumped in to offer this solution before now, I suspect that this actually might be a bad idea for... reasons? Cheers, Zach
On Tue Jul 29, 2025 at 7:59 AM PDT, Joshua Watt via lists.openembedded.org wrote: > On Tue, Jul 29, 2025 at 8:57 AM Alexander Kanavin <alex.kanavin@gmail.com> > wrote: > >> On Tue, 29 Jul 2025 at 16:49, Joshua Watt via lists.openembedded.org >> <JPEWhacker=gmail.com@lists.openembedded.org> wrote: >> > >> > Our setup is that we have a "disto" layer that we need to maintain against >> > several version of Yocto at once. In this disto layer, we make policy >> > choices that sometimes are tied to a specific version of a recipe found in >> > a specific version of Yocto using a bbappend; and often that version is >> > only in one or two Yocto versions, which means that the bbappend will be >> > dangling in the others. >> >> This is where the weak point of the argument is. Yocto does not make >> concessions to layers that attempt to be compatible across yocto releases, >> patches that help such layers had been previously rejected, and this would >> create a precedent to the contrary. Yes, but perhaps some consideration is merited? This is a perennial problem for OEMs with long lived products. >> It's always possible to make a branch. While this is logically correct, it is also an unreasonable response. The argument is being made with sincerity, clarity, and consistency. I think it there is a lot to be gained by taking more time to understand it. > Branching isn't an option for us; we are unwilling to add the maintenance > overhead of dealing with that. You are not alone in this respect. Perhaps we need to find a better way to explain this to the YP community? I believe adequate consideration of the problem will improve more than just the plight of OEMs with long lived products. ..Ch:W..
On Sat, 9 Aug 2025 at 21:59, Chuck Wolber <chuck@wolber.net> wrote: > > Branching isn't an option for us; we are unwilling to add the maintenance > > overhead of dealing with that. > > You are not alone in this respect. Perhaps we need to find a better way to > explain this to the YP community? I believe adequate consideration of the > problem will improve more than just the plight of OEMs with long lived > products. I am totally listening, but you (and Joshua) aren't actually saying much. You do need to explain with sincerity and clarity and full details what is this 'maintenance overhead' that is so burdensome that branching becomes impossible. Pretty much every community layer manages this, so why not your product layer? Alex
> I am totally listening, but you (and Joshua) aren't actually saying much. > > You do need to explain with sincerity and clarity and full details > what is this 'maintenance overhead' that is so burdensome that > branching becomes impossible. Pretty much every community layer > manages this, so why not your product layer? > > Ok. THE BACKGROUND: We don't produce just one product, we produce many products simultaneously (north of 60), and we do so at scale. One of the things we highly value is the efficiency with which we can develop so many products at the same time. Developer efficiency is one of the primary reasons that we chose to use Yocto in the first place; while the first product we did had a bit of a steep learning curve, it was very clear that we could efficiently scale it out much better than competitors. Looking back, I think the project has actually exceeded my expectations for just how well we've been able to scale. Highly valuing efficient with our development has a few side effects: 1) We do (more or less) trunk development on all our products at once. This means there is a single branch where we do all development work on all our products simultaneously, regardless of what version of upstream Yocto they are using, which BSP they are using etc. This not only means its easy for developers to find all products, but our CI also tests any change made against all products to make sure it doesn't break anything. We aren't willing to give up this development style and trade away the efficiencies this provides, which is why we can't simply branch our internal repos. 2) We like all our products to behave consistently, at least in userspace if not sooner. This means that as much as possible we configure our distro to act the same, regardless of Yocto release, BSP, etc. Similarly, if the BSP allows it, we like to use the same boot flow across all products, etc. Consistency allows our userspace developers to be more efficient because they know that the code they write should work the same regardless of which platform it's running on, the development and debugging experience is pretty much the same across all products, etc. It (mostly) eliminates the need for a developer to have to learn 60+ different ways of doing development. However, we still need to customize the behavior of many things (think distro policy settings, etc.), so we need a way to make sure we get as similar behavior in these customization as possible, even across multiple Yocto releases. 3) We don't diverge from master of upstream OE (or any other layer). We have internal mirrors of all the layers we use, and we have a branch for each Yocto version in those mirrors that we maintain. I will apply changes to these internal mirror branches for a given release (effectively forking it), but I *never* fork master (e.g. master always exactly matches upstream master). Any change that is made to a release branch in the mirror I require to exist on the master branch first (this is occasionally unpopular). The reason for this is so that as we move to newer releases, we aren't carrying along patches that are not upstreamed. Another way to think about this is that I'll more or less backport anything to a release branch, as long as the behavior matches master. I wouldn't say this is entirely ideal, but it does IMHO strike the proper balance between allowing us to do trunk development and not diverging from upstream in the long run. THE PROBLEM: Hopefully, that sets the stage for *why* we do things the way that we do, and now we can look a concrete example of why we'd prefer to allow dangling bbappends. Here is the list of bbappend we use in our disto layer for wpa-supplicant: wpa-supplicant_2.%.bbappend wpa-supplicant_2.10.bbappend wpa-supplicant_2.11.bbappend wpa-supplicant_2.5.bbappend wpa-supplicant_2.6.bbappend wpa-supplicant_2.7.bbappend wpa-supplicant_2.9.bbappend wpa-supplicant-CHIP.bbappend The primary purpose here is to configure wpa-supplicant based on our product requirements, and remember that we want consistent behavior of wpa-supplicant across all products. As such, we put as much of the configuration as we can into wpa-supplication_2.%.bbappend so that it applies to all wpa-supplicants. However, it's simply not possible to use the same configuration across all wpa-supplicant versions; new options are added, old one removed etc. so some configuration changes can only be applied to specific version, or some versions need a patch to behave like the latest version, etc. Finally, some BSPs choose to provide their own wpa-supplicant recipe, and those have to be configured also, which is what the wpa-supplicant-CHIP.bbappend stands for. From the perspective of dangling bbappends, this is awful. However, from the perspective of our developers, this is pretty efficient. Our developers can go to a single directory and see *all* the distro changes for all versions of wpa-supplicant. If multiple versions need to share a config file (this happens frequently), they can easily do that since they all live in the same directory. It's much easier for our developers to make correct changes, and understand the implication of changes made in this directory. Overall, we've been happy to trade off dangling bbappends for this. A POSSIBLE SOLUTION: For completeness, since it has appeared that upstream is unwilling to accommodate this, we have tried to work around this. The approach is to make our conf/layer.conf look like this: BBFILES += " \ ${LAYERDIR}/recipes-*/*/*.bb \ ${LAYERDIR}/recipes-*/*/*.bbappend \ ${LAYERDIR}/yocto-${LAYERSERIES_CORENAMES}/recipes-*/*/*.bbappend \ " Effectively placing the Yocto version specific bbappends into their own directory. However, this alone is insufficient because we also have to account for dangling bbappends in layers that might not be present in a given product (e.g. because it doesn't use a specific BSP). This effectively means using dynamic layers is forced in order to prevent dangling bbappends, however it gets worse when you have dynamic layers that have version specific bbappends, because then we end up with something like this: BBFILES_DYNAMIC += " \ raspberrypi:${LAYERDIR}/dynamic-layers/raspberrypi/*/*/*.bb \ raspberrypi:${LAYERDIR}/dynamic-layers/raspberrypi/*/*/*.bbappend \ raspberrypi:${LAYERDIR}/dynamic-layers/yocto-${LAYERSERIES_CORENAMES}/raspberrypi/*/*/*.bbappend \ " This approach (or other similar to it) are certainly better than branching since we can still do trunk development, but what this effectively doing is placing each bbappend in it's own directory, so instead of looking in a single place to understand how we are configuring wpa-supplicant, we now must look in no less than 8 places, and probably more since we'd have to look in all the places a bbappend _might_ be instead of all the places that _actually_ have one. Also, doing this makes it much more difficult to share a common config file across multiple versions, since the bbappends are all in separate directories. This might be what we'll do if upstream refuses to allow the dangling bbappends.... but we'd still just prefer to have the dangling bbappends as the lesser of two evils. FINAL THOUGHTS: Looking behind the curtain a little, my struggle with this is that there is some pressure to fork master (and break rule #3) so that we can continue to have the developer efficiency we are accustom to, so I'm probably going to have some hard conversions about where we go from here. That's not really upstreams problem, but I wanted to shed some light on the ramifications of these types of changes. Hopefully that accurately outlines our struggle here and why we would like this feature. Thanks Joshua Watt
On Sat, 2025-08-09 at 12:59 -0700, Chuck Wolber via lists.openembedded.org wrote: > On Tue Jul 29, 2025 at 7:59 AM PDT, Joshua Watt via lists.openembedded.org > wrote: > > On Tue, Jul 29, 2025 at 8:57 AM Alexander Kanavin <alex.kanavin@gmail.com> > > wrote: > > > > > On Tue, 29 Jul 2025 at 16:49, Joshua Watt via lists.openembedded.org > > > <JPEWhacker=gmail.com@lists.openembedded.org> wrote: > > > > > > > > Our setup is that we have a "disto" layer that we need to maintain against > > > > several version of Yocto at once. In this disto layer, we make policy > > > > choices that sometimes are tied to a specific version of a recipe found in > > > > a specific version of Yocto using a bbappend; and often that version is > > > > only in one or two Yocto versions, which means that the bbappend will be > > > > dangling in the others. > > > > > > This is where the weak point of the argument is. Yocto does not make > > > concessions to layers that attempt to be compatible across yocto releases, > > > patches that help such layers had been previously rejected, and this would > > > create a precedent to the contrary. > > Yes, but perhaps some consideration is merited? This is a perennial problem for > OEMs with long lived products. > > > > > It's always possible to make a branch. > > While this is logically correct, it is also an unreasonable response. > > The argument is being made with sincerity, clarity, and consistency. I think it > there is a lot to be gained by taking more time to understand it. > > > > Branching isn't an option for us; we are unwilling to add the maintenance > > overhead of dealing with that. > > You are not alone in this respect. Perhaps we need to find a better way to > explain this to the YP community? I believe adequate consideration of the > problem will improve more than just the plight of OEMs with long lived > products. We took a decision to drop dangling bbappend support. We did that for very specific reasons after discussion at the time. It wasn't a universally popular decision but the issues that they caused and the improvements dropping them allowed seemed attractive. The decision was made and acted upon. Now, we're being asked to effectively reverse that decision (with a slightly different implementation we could have switched to at the time but chose not to). It is clear that the usecase this is being asked for is not something we've officially supported, ever, and that has been really clear. The precedent here is problematic if I do take this, since whether we officially say so or not, we're effectively unofficially supporting that use case. We have two YP platinum members saying "we really want this". Ironically we also have the wider YP membership telling me to say "no" more often and more forcefully. To me, this isn't an issue of discussing the pros/cons as personally, I'm well aware of the trade offs on both sides of this. Rather than the technical merits, the main concern for me is that I do want to encourage project adoption and use and to push back against this will hurt that and also potentially hurt some of our key developers standing within their companies. As such, I don't feel like I may have a lot of choice but to take this, but perhaps not for the reasons people expect. Cheers, Richard
On Mon, 11 Aug 2025 at 17:49, Joshua Watt <jpewhacker@gmail.com> wrote:
> Hopefully that accurately outlines our struggle here and why we would like this feature.
Thanks for the detailed write-up Joshua. Given what RP said in this
thread after this email appeared, I'd like to abstain from further
commenting.
Alex
On Fri, 15 Aug 2025 at 12:33, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > Hopefully that accurately outlines our struggle here and why we would like this feature. > > Thanks for the detailed write-up Joshua. Given what RP said in this > thread after this email appeared, I'd like to abstain from further > commenting. Famous last words :) After I sent it an idea popped into my head: why not have two layers: - one layer that exists for the sole purpose of hosting bbappends, and yes it would have branches for each of the respective yocto release. You'd have to devise a process for patch flow that ensures bbappend tweaks get applied on every affected branch. There's also a benefit in having less file clutter, as any branch would contain only files that are directly relevant and used, and one wouldn't have to figure out which bbappend goes into which product. - another layer that contains everything else, including distro definitions, and product-related recipes, and has only a single trunk branch like now. Would that be still unworkable and cause too much overhead? I can't imagine that bbappends get a lot of churn and are changed often. Alex
> -----Original Message----- > From: bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org> On Behalf Of Alexander Kanavin via lists.openembedded.org > Sent: den 15 augusti 2025 13:10 > To: alex.kanavin@gmail.com > Cc: Joshua Watt <jpewhacker@gmail.com>; Chuck Wolber <chuck@wolber.net>; bitbake-devel@lists.openembedded.org > Subject: Re: [bitbake-devel][PATCH] cooker: Add layer-specific ignore for dangling appends > > On Fri, 15 Aug 2025 at 12:33, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > > Hopefully that accurately outlines our struggle here and why we would like this feature. > > > > Thanks for the detailed write-up Joshua. Given what RP said in this > > thread after this email appeared, I'd like to abstain from further > > commenting. > > Famous last words :) > > After I sent it an idea popped into my head: > > why not have two layers: > > - one layer that exists for the sole purpose of hosting bbappends, and > yes it would have branches for each of the respective yocto release. > You'd have to devise a process for patch flow that ensures bbappend > tweaks get applied on every affected branch. There's also a benefit in > having less file clutter, as any branch would contain only files that > are directly relevant and used, and one wouldn't have to figure out > which bbappend goes into which product. > > - another layer that contains everything else, including distro > definitions, and product-related recipes, and has only a single trunk > branch like now. > > Would that be still unworkable and cause too much overhead? I can't > imagine that bbappends get a lot of churn and are changed often. > > Alex That is basically what we do. We have our normal distro/application layers. Their master branches work with the latest Poky release (currently Walnascar 5.2.1, but soon 5.2.2). Then to build with some other version of Poky (in our case only with master of Poky), we have a separate layer where we do all adaptations. This layer typically has a long list of BBMASKs to drop the versioned bbappends that are in our application layers, either because they are no longer needed, or updated versions of the bbappends are in the adaptation layer. When it is time to upgrade to a new version of Poky, all the changes in the adaptation layer are applied to the real layers, and the adaptation layer is cleaned out, starting a new cycle. //Peter
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index 1810bcc604..349e27d538 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -953,7 +953,11 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si appends_without_recipes[mc] = [] for _, appendfn in self.collections[mc].bbappends: if not appendfn in applied_appends: - appends_without_recipes[mc].append(appendfn) + layername = self.collections[mc].calc_bbfile_priority(appendfn)[2] + ignore = self.databuilder.mcdata[mc].getVar("BB_DANGLINGAPPENDS_IGNORE_%s" % layername, + False) or "no" + if ignore.lower() not in ("1", "yes", "true"): + appends_without_recipes[mc].append(appendfn) msgs = [] for mc in sorted(appends_without_recipes.keys()):
Implements a new layer-specific variable called BB_DANGLINGAPPENDS_IGNORE. This variable is similar to the removed BB_DANGLINGAPPENDS_WARNONLY, except that it is scoped to a specific layer, and it also causes appends to be completely ignored instead of warning about them. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- BACKGROUND AND RATIONAL: I was fully on board with removing BB_DANGLINGAPPENDS_WARNONLY, since it was pretty bad that it operated on all layers if any layer set the variable. However, I did mention at the time that it was going to cause us some trouble, but not wanting to hold up a good change, I indicated that I would attempt a fix for it later. Later is now :) After much poking around in our setup and deliberation, the conclusion that I came to is that we do actually need _something_ along the lines of being able to ignore dangling appends, at least in a few specific layers. Our setup is that we have a "disto" layer that we need to maintain against several version of Yocto at once. In this disto layer, we make policy choices that sometimes are tied to a specific version of a recipe found in a specific version of Yocto using a bbappend; and often that version is only in one or two Yocto versions, which means that the bbappend will be dangling in the others. dynamic-layers doesn't work here because we are keying off the Yocto version, not the existence of some layer (unless there is something I'm unaware of). BBMASK is also a less than satisfactory solution (IMHO). Part of the reason we have structured our distro layer in the way we have is to reduce the cognitive overhead required to maintain our dozens of products that more or less need to behave similarly. Keeping all the distro stuff together makes the quite easy to reason about, but having then again to track (probably quite complicated, and possibly per-product) BBMASKs erodes that maintainability. We did go through and eliminate as many dangling appends with other methods as possible, but at the end of the day, we ended up with some for which we didn't really see a viable (and maintainable) solution. I'm open to suggestions if this change is entirely reprehensible, but our primary goal is to keep the maintenance of our cross-yocto distro layer simple. I also decided to simplify this to just ignore dangling BBAPPENDS, since I suspect anyone setting this is simply going to ignore the warnings anyway, but that can be changed if there's a strong option to do it differently. bitbake/lib/bb/cooker.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)