| Message ID | 20220101112911.1866500-1-alex@linutronix.de |
|---|---|
| State | New |
| Headers | show |
| Series | providers.py: do not skip recipes providing 'virtual/x' in world builds | expand |
On Sat, 2022-01-01 at 12:29 +0100, Alexander Kanavin wrote: > This is very old code pre-dating 2005, and the reasons are probably lost in > history. They're not, I remember this. > Doing this skipping means that anything providing 'virtual/x' is excluded > from populating packages to deploy/ in world builds, and thus, from reproducubility > and packaging tests. > > This issue has been largely masked until 'package_ipk/deb/rpm: Drop recursive do_build task dependencies' > as previously recipes without virtual/x provides would pull in the dependent rpms instead of the world target > doing that directly. > > I have noticed the issue by observing that go-runtime is no longer reported as > non-reproducible when it should be. The principle is that virtual/XXX should only be built if something actually depends upon them as they're usually used to denote things which can be provided from multiple sources. I'd argue that the code is correct and wonder whether: a) go-runtime should be providing a virtual b) that OE-Core should have something depending on go-runtime for test purposes Cheers, Richard > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > bitbake/lib/bb/providers.py | 4 ---- > 1 file changed, 4 deletions(-) > > diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py > index 8c1c31a5c5..3c4462fbdd 100644 > --- a/bitbake/lib/bb/providers.py > +++ b/bitbake/lib/bb/providers.py > @@ -429,10 +429,6 @@ def buildWorldTargetList(dataCache, task=None): > terminal = False > > for p in dataCache.pn_provides[pn]: > - if p.startswith('virtual/'): > - logger.debug2("World build skipping %s due to %s provider starting with virtual/", f, p) > - terminal = False > - break > for pf in dataCache.providers[p]: > if dataCache.pkg_fn[pf] != pn: > logger.debug2("World build skipping %s due to both us and %s providing %s", f, pf, p) > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#13209): https://lists.openembedded.org/g/bitbake-devel/message/13209 > Mute This Topic: https://lists.openembedded.org/mt/88071309/1686473 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [richard.purdie@linuxfoundation.org] > -=-=-=-=-=-=-=-=-=-=-=- >
This seems like something that could be moved to oe-core rather than being in bitbake proper, set EXCLUDE_FROM_WORLD’s default value if providing virtual/, in which case this functionality could be easily disabled if needed as well. -- Christopher “kergoth” Larson chris_larson@mentor.com, chris.larson@siemens.com, kergoth@gmail.com Founder - BitBake, OpenEmbedded, OpenZaurus Senior Software Engineer, Mentor Graphics, a Siemens Business On Jan 3, 2022, 3:38 PM -0700, Richard Purdie <richard.purdie@linuxfoundation.org>, wrote: > On Sat, 2022-01-01 at 12:29 +0100, Alexander Kanavin wrote: > > This is very old code pre-dating 2005, and the reasons are probably lost in > > history. > > They're not, I remember this. > > > Doing this skipping means that anything providing 'virtual/x' is excluded > > from populating packages to deploy/ in world builds, and thus, from reproducubility > > and packaging tests. > > > > This issue has been largely masked until 'package_ipk/deb/rpm: Drop recursive do_build task dependencies' > > as previously recipes without virtual/x provides would pull in the dependent rpms instead of the world target > > doing that directly. > > > > I have noticed the issue by observing that go-runtime is no longer reported as > > non-reproducible when it should be. > > The principle is that virtual/XXX should only be built if something actually > depends upon them as they're usually used to denote things which can be provided > from multiple sources. I'd argue that the code is correct and wonder whether: > > a) go-runtime should be providing a virtual > b) that OE-Core should have something depending on go-runtime for test purposes > > Cheers, > > Richard > > > > > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > > --- > > bitbake/lib/bb/providers.py | 4 ---- > > 1 file changed, 4 deletions(-) > > > > diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py > > index 8c1c31a5c5..3c4462fbdd 100644 > > --- a/bitbake/lib/bb/providers.py > > +++ b/bitbake/lib/bb/providers.py > > @@ -429,10 +429,6 @@ def buildWorldTargetList(dataCache, task=None): > > terminal = False > > > > for p in dataCache.pn_provides[pn]: > > - if p.startswith('virtual/'): > > - logger.debug2("World build skipping %s due to %s provider starting with virtual/", f, p) > > - terminal = False > > - break > > for pf in dataCache.providers[p]: > > if dataCache.pkg_fn[pf] != pn: > > logger.debug2("World build skipping %s due to both us and %s providing %s", f, pf, p) > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#13212): https://lists.openembedded.org/g/bitbake-devel/message/13212 > Mute This Topic: https://lists.openembedded.org/mt/88071309/3617123 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [kergoth@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Mon, 3 Jan 2022 at 23:38, Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > The principle is that virtual/XXX should only be built if something > actually > depends upon them as they're usually used to denote things which can be > provided > from multiple sources. I'm not sure I follow. The 'world' set is aiming to have the fullest amount of target items in it. If there are items which have the same 'virtual/x' provides in their recipes, why should the recipes be all excluded from the set? They're not going to clash in package deploy, or in sysroots (since sysroots became recipe-specific). Note that it's not 'virtual/x' being excluded, but the parent recipes themselves. > I'd argue that the code is correct and wonder whether: > > a) go-runtime should be providing a virtual > b) that OE-Core should have something depending on go-runtime for test > purposes > go-runtime is how I spotted the issue, but there are actually a fair bit more items with the same problem, some of them major. DEBUG: collating packages for "world" DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/libsdl2/libsdl2_2.0.18.bb due to virtual/libsdl2 provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/make/make_4.3.bb due to virtual/make provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-extended/libtirpc/libtirpc_1.3.2.bb due to virtual/librpc provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/go/go-runtime_1.17.5.bb due to virtual/x86_64-poky-linux-go-runtime provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/libxcrypt/libxcrypt_4.4.26.bb due to virtual/crypt provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/xorg-xserver/ xserver-xorg_21.1.1.bb due to virtual/xserver-xf86 provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.10.bb due to virtual/kernel provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-sato/rxvt-unicode/ rxvt-unicode_9.30.bb due to virtual/x-terminal-emulator provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/gettext/gettext_0.21.bb due to virtual/libintl provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/glibc/glibc-locale_2.34.bb due to virtual/libc-locale provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-multimedia/libomxil/ libomxil_0.9.3.bb due to virtual/libomxil provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/gcc/libgcc-initial_11.2.bb as task do_build doesn't exist DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/glibc/glibc_2.34.bb due to virtual/libc provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/perf/perf.bb due to virtual/perf provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-bsp/u-boot/u-boot_2021.10.bb due to virtual/bootloader provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/mesa/mesa_21.3.1.bb due to virtual/libgl provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.15.bb due to virtual/kernel provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/dpkg/dpkg_1.21.1.bb due to virtual/update-alternatives provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/xorg-lib/libx11_1.7.3.1.bb due to virtual/libx11 provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.14.bb due to virtual/kernel provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/opkg-utils/ opkg-utils_0.5.0.bb due to virtual/update-alternatives provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/gcc/gcc-runtime_11.2.bb due to virtual/x86_64-poky-linux-compilerlibs provider starting with virtual/ DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/pseudo/pseudo_git.bb due to virtual/fakeroot provider starting with virtual/ Alex
To follow this up, if there are indeed multiple providers, they are
excluded even earlier in base.bbclass:
=====
provs = (d.getVar("PROVIDES") or "").split()
multiwhitelist = (d.getVar("MULTI_PROVIDER_WHITELIST") or
"").split()
for p in provs:
if p.startswith("virtual/") and p not in multiwhitelist:
profprov = d.getVar("PREFERRED_PROVIDER_" + p)
if profprov and pn != profprov:
raise bb.parse.SkipRecipe("PREFERRED_PROVIDER_%s
set to %s, not %s" % (p, profprov, pn))
=====
So the bitbake code only excludes the sole remaining option from world,
which makes even less sense.
Alex
On Tue, 4 Jan 2022 at 08:43, Alexander Kanavin via lists.openembedded.org
<alex.kanavin=gmail.com@lists.openembedded.org> wrote:
> On Mon, 3 Jan 2022 at 23:38, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> The principle is that virtual/XXX should only be built if something
>> actually
>> depends upon them as they're usually used to denote things which can be
>> provided
>> from multiple sources.
>
>
> I'm not sure I follow. The 'world' set is aiming to have the fullest
> amount of target items in it. If there are items which have the same
> 'virtual/x' provides in their recipes, why should the recipes be all
> excluded from the set? They're not going to clash in package deploy, or in
> sysroots (since sysroots became recipe-specific). Note that it's not
> 'virtual/x' being excluded, but the parent recipes themselves.
>
>
>> I'd argue that the code is correct and wonder whether:
>>
>> a) go-runtime should be providing a virtual
>> b) that OE-Core should have something depending on go-runtime for test
>> purposes
>>
>
> go-runtime is how I spotted the issue, but there are actually a fair bit
> more items with the same problem, some of them major.
>
> DEBUG: collating packages for "world"
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-graphics/libsdl2/
> libsdl2_2.0.18.bb due to virtual/libsdl2 provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-devtools/make/make_4.3.bb due to
> virtual/make provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-extended/libtirpc/
> libtirpc_1.3.2.bb due to virtual/librpc provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-devtools/go/go-runtime_1.17.5.bb
> due to virtual/x86_64-poky-linux-go-runtime provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-core/libxcrypt/
> libxcrypt_4.4.26.bb due to virtual/crypt provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-graphics/xorg-xserver/
> xserver-xorg_21.1.1.bb due to virtual/xserver-xf86 provider starting with
> virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.10.bb
> due to virtual/kernel provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-sato/rxvt-unicode/
> rxvt-unicode_9.30.bb due to virtual/x-terminal-emulator provider starting
> with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-core/gettext/gettext_0.21.bb due
> to virtual/libintl provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-core/glibc/glibc-locale_2.34.bb
> due to virtual/libc-locale provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-multimedia/libomxil/
> libomxil_0.9.3.bb due to virtual/libomxil provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-devtools/gcc/
> libgcc-initial_11.2.bb as task do_build doesn't exist
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-core/glibc/glibc_2.34.bb due to
> virtual/libc provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-kernel/perf/perf.bb due to
> virtual/perf provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-bsp/u-boot/u-boot_2021.10.bb due
> to virtual/bootloader provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-graphics/mesa/mesa_21.3.1.bb due
> to virtual/libgl provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.15.bb
> due to virtual/kernel provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-devtools/dpkg/dpkg_1.21.1.bb due
> to virtual/update-alternatives provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-graphics/xorg-lib/
> libx11_1.7.3.1.bb due to virtual/libx11 provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.14.bb
> due to virtual/kernel provider starting with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-devtools/opkg-utils/
> opkg-utils_0.5.0.bb due to virtual/update-alternatives provider starting
> with virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-devtools/gcc/gcc-runtime_11.2.bb
> due to virtual/x86_64-poky-linux-compilerlibs provider starting with
> virtual/
> DEBUG: World build skipping
> /home/alex/development/poky/meta/recipes-devtools/pseudo/pseudo_git.bb
> due to virtual/fakeroot provider starting with virtual/
>
> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13215):
> https://lists.openembedded.org/g/bitbake-devel/message/13215
> Mute This Topic: https://lists.openembedded.org/mt/88071309/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
On Tue, 4 Jan 2022 at 10:29, Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > > Building musl on a glibc configured system is pointless. Similarly, > building > clang when you're targeting things with gcc is pointless too. > As mentioned, musl and clang and systemd are all excluded earlier by skipping them via base.bbclass. So this code only ends up excluding the sole remaining options (preferred alternatives set via PREFERRED_PROVIDER), which are both valid and useful in world builds. > > DEBUG: collating packages for "world" > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-graphics/libsdl2/ > libsdl2_2.0.18.bb due to virtual/libsdl2 provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-devtools/make/make_4.3.bb due to > virtual/make provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-extended/libtirpc/ > libtirpc_1.3.2.bb due to virtual/librpc provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-devtools/go/go-runtime_1.17.5.bb > due to virtual/x86_64-poky-linux-go-runtime provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-core/libxcrypt/ > libxcrypt_4.4.26.bb due to virtual/crypt provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-graphics/xorg-xserver/ > xserver-xorg_21.1.1.bb due to virtual/xserver-xf86 provider starting with > virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.10.bb > due to virtual/kernel provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-sato/rxvt-unicode/ > rxvt-unicode_9.30.bb due to virtual/x-terminal-emulator provider starting > with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-core/gettext/gettext_0.21.bb due > to virtual/libintl provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-core/glibc/glibc-locale_2.34.bb > due to virtual/libc-locale provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-multimedia/libomxil/ > libomxil_0.9.3.bb due to virtual/libomxil provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-devtools/gcc/ > libgcc-initial_11.2.bb as task do_build doesn't exist > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-core/glibc/glibc_2.34.bb due to > virtual/libc provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-kernel/perf/perf.bb due to > virtual/perf provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-bsp/u-boot/u-boot_2021.10.bb due > to virtual/bootloader provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-graphics/mesa/mesa_21.3.1.bb due > to virtual/libgl provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.15.bb > due to virtual/kernel provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-devtools/dpkg/dpkg_1.21.1.bb due > to virtual/update-alternatives provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-graphics/xorg-lib/ > libx11_1.7.3.1.bb due to virtual/libx11 provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.14.bb > due to virtual/kernel provider starting with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-devtools/opkg-utils/ > opkg-utils_0.5.0.bb due to virtual/update-alternatives provider starting > with virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-devtools/gcc/gcc-runtime_11.2.bb > due to virtual/x86_64-poky-linux-compilerlibs provider starting with > virtual/ > > DEBUG: World build skipping > /home/alex/development/poky/meta/recipes-devtools/pseudo/pseudo_git.bb > due to virtual/fakeroot provider starting with virtual/ > > Most of these would be have things depending on them and therefore would > get > built anyway. If they don't, they're probably not "useful" in the world > build > context so I still think the code is actually doing what was intended? > The issue here is not a build for purposes of populating another sysroot (without packaging for target), the issue is that there's no guarantee anything here would end up in package deploy and thus be tested for reproducibility. How should we solve this? I'd still want to drop the world skipping (given that it does not actually exclude non-useful alternatives). I would reluctantly agree to '-k universe' too in the repro test. Alex
On Tue, 4 Jan 2022 at 10:46, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > Most of these would be have things depending on them and therefore would > get > >> built anyway. If they don't, they're probably not "useful" in the world >> build >> context so I still think the code is actually doing what was intended? >> > > The issue here is not a build for purposes of populating another sysroot > (without packaging for target), the issue is that there's no guarantee > anything here would end up in package deploy and thus be tested for > reproducibility. How should we solve this? I'd still want to drop the world > skipping (given that it does not actually exclude non-useful alternatives). > I would reluctantly agree to '-k universe' too in the repro test. > As a specific example, go-runtime is already part of the world builds because it's needed by go-helloworld - but only up to do_populate_sysroot. do_package and later simply do not run due to being excluded from world. What to do that doesn't require maintaining a list of such items manually? Alex
diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py index 8c1c31a5c5..3c4462fbdd 100644 --- a/bitbake/lib/bb/providers.py +++ b/bitbake/lib/bb/providers.py @@ -429,10 +429,6 @@ def buildWorldTargetList(dataCache, task=None): terminal = False for p in dataCache.pn_provides[pn]: - if p.startswith('virtual/'): - logger.debug2("World build skipping %s due to %s provider starting with virtual/", f, p) - terminal = False - break for pf in dataCache.providers[p]: if dataCache.pkg_fn[pf] != pn: logger.debug2("World build skipping %s due to both us and %s providing %s", f, pf, p)
This is very old code pre-dating 2005, and the reasons are probably lost in history. Doing this skipping means that anything providing 'virtual/x' is excluded from populating packages to deploy/ in world builds, and thus, from reproducubility and packaging tests. This issue has been largely masked until 'package_ipk/deb/rpm: Drop recursive do_build task dependencies' as previously recipes without virtual/x provides would pull in the dependent rpms instead of the world target doing that directly. I have noticed the issue by observing that go-runtime is no longer reported as non-reproducible when it should be. Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- bitbake/lib/bb/providers.py | 4 ---- 1 file changed, 4 deletions(-)