| Message ID | 20230324083620.2028587-1-xiangyu.chen@eng.windriver.com |
|---|---|
| State | Accepted, archived |
| Commit | a73e269d3e591a10bb397b94b82e3fb960112d33 |
| Headers | show |
| Series | [v2] package: moving field data process before variable process in process_pkgconfig | expand |
On Fri, 2023-03-24 at 16:36 +0800, Xiangyu Chen wrote: > From: Xiangyu Chen <xiangyu.chen@windriver.com> > > Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's > basic package config(.pc file) as below: > > prefix=/usr > exec_prefix=${prefix} > > ...... > > Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125, > absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125, > absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125 > ...... > > Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the > absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the > process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py > try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking > and finally bitbake stop building and reporting an error as below: > > "Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake." > > This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field > data as variable. > > Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com> > --- > > Changes: > V1 -> V2: changing the package.py instead of bitbake's data_smart.py > v1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525 This looks like a much better fix than the workaround in v1, thanks! Cheers, Richard
On Fri, 2023-03-24 at 16:36 +0800, Xiangyu Chen wrote: > From: Xiangyu Chen <xiangyu.chen@windriver.com> > > Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's > basic package config(.pc file) as below: > > prefix=/usr > exec_prefix=${prefix} > > ...... > > Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125, > absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125, > absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125 > ...... > > Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the > absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the > process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py > try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking > and finally bitbake stop building and reporting an error as below: > > "Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake." > > This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field > data as variable. > > Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com> > --- > > Changes: > V1 -> V2: changing the package.py instead of bitbake's data_smart.py > v1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525 > > --- > meta/lib/oe/package.py | 12 ++++++------ > 1 file changed, 6 insertions(+), 6 deletions(-) > > diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py > index c9eb75d852..7a6b31957a 100644 > --- a/meta/lib/oe/package.py > +++ b/meta/lib/oe/package.py > @@ -1823,18 +1823,18 @@ def process_pkgconfig(pkgfiles, d): > with open(file, 'r') as f: > lines = f.readlines() > for l in lines: > - m = var_re.match(l) > - if m: > - name = m.group(1) > - val = m.group(2) > - pd.setVar(name, pd.expand(val)) > - continue > m = field_re.match(l) > if m: > hdr = m.group(1) > exp = pd.expand(m.group(2)) > if hdr == 'Requires': > pkgconfig_needed[pkg] += exp.replace(',', ' ').split() > + continue > + m = var_re.match(l) > + if m: > + name = m.group(1) > + val = m.group(2) > + pd.setVar(name, pd.expand(val)) > > for pkg in packages.split(): > pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist") It is worth noting this fix actually resolved a number of latent bugs too. This breakage in reproducibility was as a result of this change: http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230325-0k1j3_4w/packages/diff-html/ which was because the output changed but the sstate hashes didn't. That is because package.py code isn't part of the sstate hashes (currently). This isn't a fault of this patch. I've a patch in progress to bump various versions to make sure this doesn't break anything further and it does raise a question on whether we should checksum the python functions and track that due to the adverse impact an issue like this can have. The issue here was missing dependencies, which were missing from -dev packages when there was a version constraint present. I wanted to highlight why fixing things at the root cause is good! If we'd hacked this in bitbake, the other issues would have remained unseen. Cheers, Richard
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py index c9eb75d852..7a6b31957a 100644 --- a/meta/lib/oe/package.py +++ b/meta/lib/oe/package.py @@ -1823,18 +1823,18 @@ def process_pkgconfig(pkgfiles, d): with open(file, 'r') as f: lines = f.readlines() for l in lines: - m = var_re.match(l) - if m: - name = m.group(1) - val = m.group(2) - pd.setVar(name, pd.expand(val)) - continue m = field_re.match(l) if m: hdr = m.group(1) exp = pd.expand(m.group(2)) if hdr == 'Requires': pkgconfig_needed[pkg] += exp.replace(',', ' ').split() + continue + m = var_re.match(l) + if m: + name = m.group(1) + val = m.group(2) + pd.setVar(name, pd.expand(val)) for pkg in packages.split(): pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")