| Message ID | 20250729144906.1665908-1-JPEWhacker@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [bitbake-devel] cooker: Add layer-specific ignore for dangling appends | expand |
On Tue, 29 Jul 2025 at 16:49, Joshua Watt via lists.openembedded.org <JPEWhacker=gmail.com@lists.openembedded.org> wrote: > Our setup is that we have a "disto" layer that we need to maintain > against several version of Yocto at once. In this disto layer, we make > policy choices that sometimes are tied to a specific version of a recipe > found in a specific version of Yocto using a bbappend; and often that > version is only in one or two Yocto versions, which means that the > bbappend will be dangling in the others. This is where the weak point of the argument is. Yocto does not make concessions to layers that attempt to be compatible across yocto releases, patches that help such layers had been previously rejected, and this would create a precedent to the contrary. It's always possible to make a branch. Alex
On Tue, Jul 29, 2025 at 8:57 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > On Tue, 29 Jul 2025 at 16:49, Joshua Watt via lists.openembedded.org > <JPEWhacker=gmail.com@lists.openembedded.org> wrote: > > Our setup is that we have a "disto" layer that we need to maintain > > against several version of Yocto at once. In this disto layer, we make > > policy choices that sometimes are tied to a specific version of a recipe > > found in a specific version of Yocto using a bbappend; and often that > > version is only in one or two Yocto versions, which means that the > > bbappend will be dangling in the others. > > This is where the weak point of the argument is. Yocto does not make > concessions to layers that attempt to be compatible across yocto > releases, patches that help such layers had been previously rejected, > and this would create a precedent to the contrary. > > It's always possible to make a branch. > Branching isn't an option for us; we are unwilling to add the maintenance overhead of dealing with that. > > Alex >
On Tue, 2025-07-29 at 08:47 -0600, Joshua Watt via lists.openembedded.org wrote: > Implements a new layer-specific variable called > BB_DANGLINGAPPENDS_IGNORE. This variable is similar to the removed > BB_DANGLINGAPPENDS_WARNONLY, except that it is scoped to a specific > layer, and it also causes appends to be completely ignored instead of > warning about them. > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > BACKGROUND AND RATIONAL: > > I was fully on board with removing BB_DANGLINGAPPENDS_WARNONLY, since it > was pretty bad that it operated on all layers if any layer set the > variable. However, I did mention at the time that it was going to cause > us some trouble, but not wanting to hold up a good change, I indicated > that I would attempt a fix for it later. Later is now :) > > After much poking around in our setup and deliberation, the conclusion > that I came to is that we do actually need _something_ along the lines > of being able to ignore dangling appends, at least in a few specific > layers. > > Our setup is that we have a "disto" layer that we need to maintain > against several version of Yocto at once. In this disto layer, we make > policy choices that sometimes are tied to a specific version of a recipe > found in a specific version of Yocto using a bbappend; and often that > version is only in one or two Yocto versions, which means that the > bbappend will be dangling in the others. > > dynamic-layers doesn't work here because we are keying off the Yocto > version, not the existence of some layer (unless there is something I'm > unaware of). > > BBMASK is also a less than satisfactory solution (IMHO). Part of the > reason we have structured our distro layer in the way we have is to > reduce the cognitive overhead required to maintain our dozens of > products that more or less need to behave similarly. Keeping all the > distro stuff together makes the quite easy to reason about, but having > then again to track (probably quite complicated, and possibly > per-product) BBMASKs erodes that maintainability. > > We did go through and eliminate as many dangling appends with other > methods as possible, but at the end of the day, we ended up with some > for which we didn't really see a viable (and maintainable) solution. > > I'm open to suggestions if this change is entirely reprehensible, but > our primary goal is to keep the maintenance of our cross-yocto distro > layer simple. > > I also decided to simplify this to just ignore dangling BBAPPENDS, since > I suspect anyone setting this is simply going to ignore the warnings > anyway, but that can be changed if there's a strong option to do it > differently. > > bitbake/lib/bb/cooker.py | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py > index 1810bcc604..349e27d538 100644 > --- a/bitbake/lib/bb/cooker.py > +++ b/bitbake/lib/bb/cooker.py > @@ -953,7 +953,11 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si > appends_without_recipes[mc] = [] > for _, appendfn in self.collections[mc].bbappends: > if not appendfn in applied_appends: > - appends_without_recipes[mc].append(appendfn) > + layername = self.collections[mc].calc_bbfile_priority(appendfn)[2] > + ignore = self.databuilder.mcdata[mc].getVar("BB_DANGLINGAPPENDS_IGNORE_%s" % layername, > + False) or "no" > + if ignore.lower() not in ("1", "yes", "true"): At least use bb.utils.to_boolean() so this behaves the same as any other true/false value we have. > + appends_without_recipes[mc].append(appendfn) I'm torn on this. On the one hand I do want to support our user base, on the other this is adding back some API we decided we really didn't want. I appreciate it is different in that it is layer specific and it is silent rather than warning, but I'm not convinced either of those things sell me on the idea that we want to encourage people to be doing this. We did consider making this layer specific and decided against that at the time it was removed. The choice here is between defined and explicit configurations and "mashing a load of stuff together" and hoping it works. We directed people to BBMASK as that is much more explicit. I suspect with some distro version overrides, BBMASK can be made to work but it will be more effort that just ignoring any mismatches. Cheers, Richard
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index 1810bcc604..349e27d538 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -953,7 +953,11 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si appends_without_recipes[mc] = [] for _, appendfn in self.collections[mc].bbappends: if not appendfn in applied_appends: - appends_without_recipes[mc].append(appendfn) + layername = self.collections[mc].calc_bbfile_priority(appendfn)[2] + ignore = self.databuilder.mcdata[mc].getVar("BB_DANGLINGAPPENDS_IGNORE_%s" % layername, + False) or "no" + if ignore.lower() not in ("1", "yes", "true"): + appends_without_recipes[mc].append(appendfn) msgs = [] for mc in sorted(appends_without_recipes.keys()):
Implements a new layer-specific variable called BB_DANGLINGAPPENDS_IGNORE. This variable is similar to the removed BB_DANGLINGAPPENDS_WARNONLY, except that it is scoped to a specific layer, and it also causes appends to be completely ignored instead of warning about them. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- BACKGROUND AND RATIONAL: I was fully on board with removing BB_DANGLINGAPPENDS_WARNONLY, since it was pretty bad that it operated on all layers if any layer set the variable. However, I did mention at the time that it was going to cause us some trouble, but not wanting to hold up a good change, I indicated that I would attempt a fix for it later. Later is now :) After much poking around in our setup and deliberation, the conclusion that I came to is that we do actually need _something_ along the lines of being able to ignore dangling appends, at least in a few specific layers. Our setup is that we have a "disto" layer that we need to maintain against several version of Yocto at once. In this disto layer, we make policy choices that sometimes are tied to a specific version of a recipe found in a specific version of Yocto using a bbappend; and often that version is only in one or two Yocto versions, which means that the bbappend will be dangling in the others. dynamic-layers doesn't work here because we are keying off the Yocto version, not the existence of some layer (unless there is something I'm unaware of). BBMASK is also a less than satisfactory solution (IMHO). Part of the reason we have structured our distro layer in the way we have is to reduce the cognitive overhead required to maintain our dozens of products that more or less need to behave similarly. Keeping all the distro stuff together makes the quite easy to reason about, but having then again to track (probably quite complicated, and possibly per-product) BBMASKs erodes that maintainability. We did go through and eliminate as many dangling appends with other methods as possible, but at the end of the day, we ended up with some for which we didn't really see a viable (and maintainable) solution. I'm open to suggestions if this change is entirely reprehensible, but our primary goal is to keep the maintenance of our cross-yocto distro layer simple. I also decided to simplify this to just ignore dangling BBAPPENDS, since I suspect anyone setting this is simply going to ignore the warnings anyway, but that can be changed if there's a strong option to do it differently. bitbake/lib/bb/cooker.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)