| Message ID | 20240626140500.1857-1-sreejith.ravi087@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | package.py: Add Requires.private field in process_pkgconfig | expand |
I believe this causes the following reproducibility failures: https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4998/steps/13/logs/stdio https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240627-3tyeozgn/packages/diff-html/ On 26/06/2024 14:05:00+0000, Sreejith Ravi wrote: > Currently, the process_pkgconfig() only handles the "Requires" key > field from the .pc file and ignores the "Requires.private" field while > generating the dev dependency chain. This results in a broken dependency > list and requires the installation of recommended packages to resolve > the build dependencies when using the dev IPKs. This increases the > hard disk space usage and download time and installs many unnecessary > packages as part of the recommendations. > > This patch ensures that the "Requires.private" field is also considered > when creating the dependency list for the dev IPKs. With this, the dev > IPK will have the proper dependencies listed, eliminating the need to > install recommended packages to resolve the build time dependencies. > > Example: usr/lib/pkgconfig/libical.pc > ---------- > Libs: -L${libdir} -lical -licalss -licalvcal > Libs.private: -lpthread > Requires.private: icu-i18n > ---------- > Depends field generated for libical-dev > Depends: glib-2.0-dev, libical (= 3.0.7-r0) > ------------ > > When trying to resolve the build time dependency with libical > package using “-dev” ipk generated, it will throw the below error. > ----------- > Package icu-i18n was not found in the pkg-config search path. > Perhaps you should add the directory containing `icu-i18n.pc' > to the PKG_CONFIG_PATH environment variable > Package 'icu-i18n', required by 'libical', not found > ----------- > > This patch will fix the broken dependency list. > > ------- > libical-dev depends field generated with this patch > Depends: glib-2.0-dev, icu-dev, libical (= 3.0.7-r0) > ------- > > Other examples of packages generated with broken dev dependency. > > libflac-dev : https://packages.debian.org/sid/libflac-dev > Without patch: > Depends: flac (= 1.3.3-r0), libflac, libflac++ > with patch: > Depends: flac (= 1.3.3-r0), libflac, libflac++, libogg-dev > > libglib2.0-dev : https://packages.debian.org/buster/libglib2.0-dev > without patch: > Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev > with patch: > Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev, > util-linux-dev, zlib-dev > > Signed-off-by: Sreejith Ravi <sreejith.ravi087@gmail.com> > --- > meta/lib/oe/package.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py > index d1303f32bf..e6b46a0dc5 100644 > --- a/meta/lib/oe/package.py > +++ b/meta/lib/oe/package.py > @@ -1853,7 +1853,7 @@ def process_pkgconfig(pkgfiles, d): > if m: > hdr = m.group(1) > exp = pd.expand(m.group(2)) > - if hdr == 'Requires': > + if hdr == 'Requires' or hdr == 'Requires.private': > pkgconfig_needed[pkg] += exp.replace(',', ' ').split() > continue > m = var_re.match(l) > -- > 2.34.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#201166): https://lists.openembedded.org/g/openembedded-core/message/201166 > Mute This Topic: https://lists.openembedded.org/mt/106889869/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Thu, 2024-06-27 at 11:29 +0200, Alexandre Belloni via lists.openembedded.org wrote: > I believe this causes the following reproducibility failures: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4998/steps/13/logs/stdio > > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240627-3tyeozgn/packages/diff-html/ > This is exposing a problem with the way we're handling python library functions and their effect or lack of effect on the task hashes. If you do something like: bitbake libdrm -S none then look at bitbake-dumpsig tmp/stamps/core2-64-poky-linux/libdrm/2.4.120.do_package.sigdata.* you'll see: List of dependencies for variable oe.package.process_pkgconfig is ['PACKAGES', 'PKGDEST', 'SHLIBSDIRS', 'SHLIBSWORKDIR'] List of dependencies for variable package_do_pkgconfig is ['oe.package.process_pkgconfig'] Variable package_do_pkgconfig value is oe.package.process_pkgconfig(pkgfiles, d) Variable oe.package.process_pkgconfig value is i.e. it knows the function is being used but isn't caching its value. That was by design but it does mean when you change funcitonality, you have to manually tweak the tashhash which I'm not sure is a great idea when put like this. I'm pondering the best way to improve/fix this. Cheers, Richard
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index d1303f32bf..e6b46a0dc5 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -1853,7 +1853,7 @@ def process_pkgconfig(pkgfiles, d): if m: hdr = m.group(1) exp = pd.expand(m.group(2)) - if hdr == 'Requires': + if hdr == 'Requires' or hdr == 'Requires.private': pkgconfig_needed[pkg] += exp.replace(',', ' ').split() continue m = var_re.match(l)
Currently, the process_pkgconfig() only handles the "Requires" key field from the .pc file and ignores the "Requires.private" field while generating the dev dependency chain. This results in a broken dependency list and requires the installation of recommended packages to resolve the build dependencies when using the dev IPKs. This increases the hard disk space usage and download time and installs many unnecessary packages as part of the recommendations. This patch ensures that the "Requires.private" field is also considered when creating the dependency list for the dev IPKs. With this, the dev IPK will have the proper dependencies listed, eliminating the need to install recommended packages to resolve the build time dependencies. Example: usr/lib/pkgconfig/libical.pc ---------- Libs: -L${libdir} -lical -licalss -licalvcal Libs.private: -lpthread Requires.private: icu-i18n ---------- Depends field generated for libical-dev Depends: glib-2.0-dev, libical (= 3.0.7-r0) ------------ When trying to resolve the build time dependency with libical package using “-dev” ipk generated, it will throw the below error. ----------- Package icu-i18n was not found in the pkg-config search path. Perhaps you should add the directory containing `icu-i18n.pc' to the PKG_CONFIG_PATH environment variable Package 'icu-i18n', required by 'libical', not found ----------- This patch will fix the broken dependency list. ------- libical-dev depends field generated with this patch Depends: glib-2.0-dev, icu-dev, libical (= 3.0.7-r0) ------- Other examples of packages generated with broken dev dependency. libflac-dev : https://packages.debian.org/sid/libflac-dev Without patch: Depends: flac (= 1.3.3-r0), libflac, libflac++ with patch: Depends: flac (= 1.3.3-r0), libflac, libflac++, libogg-dev libglib2.0-dev : https://packages.debian.org/buster/libglib2.0-dev without patch: Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev with patch: Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev, util-linux-dev, zlib-dev Signed-off-by: Sreejith Ravi <sreejith.ravi087@gmail.com> --- meta/lib/oe/package.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)