| Message ID | 20220513055923.170822-1-sundeep.kokkonda@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [V4] meta: rust - Bug fix for target definitions returning 'NoneType' for arm | expand |
On Thu, 2022-05-12 at 22:59 -0700, Sundeep KOKKONDA wrote: > [Yocto bug #14742] > The build shows below error while building for arm machines. > Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' > Detailed error info : > > Steps to reproduce: > 1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"' > 2. bitbake core-image-minimal -cpopulate_sdk > > Complete Error: > ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated: > The stack trace of python calls that resulted in this exception/failure was: > File: 'exec_func_python() autogenerated', lineno: 2, function: <module> > 0001: > *** 0002:do_rust_gen_targets(d) > 0003: > File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets > 0027: > 0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}" > 0029:python do_rust_gen_targets () { > 0030: wd = d.getVar('WORKDIR') + '/targets/' > *** 0031: rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH')) > 0032: rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH')) > 0033: rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH')) > 0034:} > 0035: > File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target > 0326: # build tspec > 0327: tspec = {} > 0328: tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi) > 0329: tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi) > *** 0330: tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi)) > 0331: tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi) > 0332: tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi) > 0333: tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi) > 0334: tspec['arch'] = arch_to_rust_target_arch(rust_arch) > Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' > > Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType. > (a) Locals at rust_gen_target for arm:: > tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'> > tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'> > DEBUG: Python function do_rust_gen_targets finished > (b) Locals at rust_gen_target for aarch64:: > tspec['data-layout'] = aarch64-unknown-linux-gnu, Type of tspec['data-layout'] = <class 'str'> > tspec['max-atomic-width'] = 128, Type of tspec['max-atomic-width'] = <class 'int'> > > Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue. > > The 'rust_gen_targets' Task added with its dependent variable list. > > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@gmail.com> > --- > meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++------- > 1 file changed, 9 insertions(+), 7 deletions(-) > > diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc > index 310aecef22..984fe9099e 100644 > --- a/meta/recipes-devtools/rust/rust-common.inc > +++ b/meta/recipes-devtools/rust/rust-common.inc > @@ -119,13 +119,13 @@ def llvm_features(d): > > > ## arm-unknown-linux-gnueabihf > -DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" > -LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}" > -TARGET_ENDIAN[arm-eabi] = "little" > -TARGET_POINTER_WIDTH[arm-eabi] = "32" > -TARGET_C_INT_WIDTH[arm-eabi] = "32" > -MAX_ATOMIC_WIDTH[arm-eabi] = "64" > -FEATURES[arm-eabi] = "+v6,+vfp2" > +DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" > +LLVM_TARGET[arm] = "${RUST_TARGET_SYS}" > +TARGET_ENDIAN[arm] = "little" > +TARGET_POINTER_WIDTH[arm] = "32" > +TARGET_C_INT_WIDTH[arm] = "32" > +MAX_ATOMIC_WIDTH[arm] = "64" > +FEATURES[arm] = "+v6,+vfp2" > > ## armv7-unknown-linux-gnueabihf > DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" > @@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""): > with open(wd + sys + '.json', 'w') as f: > json.dump(tspec, f, indent=4) > > +do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES" > + > python do_rust_gen_targets () { > wd = d.getVar('WORKDIR') + '/targets/' > build_arch = d.getVar('BUILD_ARCH') We need to split this into two. One patch to fix the determinism issue, the other to fix the arm bug. The "good" news is that this patch now fails consistently on the autobuilder, for example: https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3522 sstatetests.SStateTests.test_sstate_allarch_samesigs is failing which you can rerun with: oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs Now it fails reliably, we need to work out why the arm change breaks that. If you split the patches into two we can likely merge the reproducibility one, then onwards with the arm issue. In case it isn't clear, the failing signatures are for adwaita-icon- theme which depends on librsvg-native which depends on rust-native. What this means is that the signature of rust-native is changing when MACHINE is changed in that test. This shouldn't happen. We already knew that from our other discussion but now we can prove it. It would be nice if a test showed us the native hash was changing and that is the other issue we discussed but one step at a time! To be clear, rust-native (and it's signature) should be independent of MACHINE so we need to fix that. This means that currently rust-native varies and rebuilds when you change MACHINE (after the dependency fix in this patch). Cheers, Richard
Hello, On 14-05-2022 12:46, richard.purdie@linuxfoundation.org wrote: > On Thu, 2022-05-12 at 22:59 -0700, Sundeep KOKKONDA wrote: >> [Yocto bug #14742] >> The build shows below error while building for arm machines. >> Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' >> Detailed error info : >> >> Steps to reproduce: >> 1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"' >> 2. bitbake core-image-minimal -cpopulate_sdk >> >> Complete Error: >> ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated: >> The stack trace of python calls that resulted in this exception/failure was: >> File: 'exec_func_python() autogenerated', lineno: 2, function: <module> >> 0001: >> *** 0002:do_rust_gen_targets(d) >> 0003: >> File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets >> 0027: >> 0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}" >> 0029:python do_rust_gen_targets () { >> 0030: wd = d.getVar('WORKDIR') + '/targets/' >> *** 0031: rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH')) >> 0032: rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH')) >> 0033: rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH')) >> 0034:} >> 0035: >> File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target >> 0326: # build tspec >> 0327: tspec = {} >> 0328: tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi) >> 0329: tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi) >> *** 0330: tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi)) >> 0331: tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi) >> 0332: tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi) >> 0333: tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi) >> 0334: tspec['arch'] = arch_to_rust_target_arch(rust_arch) >> Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' >> >> Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType. >> (a) Locals at rust_gen_target for arm:: >> tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'> >> tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'> >> DEBUG: Python function do_rust_gen_targets finished >> (b) Locals at rust_gen_target for aarch64:: >> tspec['data-layout'] = aarch64-unknown-linux-gnu, Type of tspec['data-layout'] = <class 'str'> >> tspec['max-atomic-width'] = 128, Type of tspec['max-atomic-width'] = <class 'int'> >> >> Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue. >> >> The 'rust_gen_targets' Task added with its dependent variable list. >> >> Signed-off-by: Sundeep KOKKONDA<sundeep.kokkonda@gmail.com> >> --- >> meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++------- >> 1 file changed, 9 insertions(+), 7 deletions(-) >> >> diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc >> index 310aecef22..984fe9099e 100644 >> --- a/meta/recipes-devtools/rust/rust-common.inc >> +++ b/meta/recipes-devtools/rust/rust-common.inc >> @@ -119,13 +119,13 @@ def llvm_features(d): >> >> >> ## arm-unknown-linux-gnueabihf >> -DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" >> -LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}" >> -TARGET_ENDIAN[arm-eabi] = "little" >> -TARGET_POINTER_WIDTH[arm-eabi] = "32" >> -TARGET_C_INT_WIDTH[arm-eabi] = "32" >> -MAX_ATOMIC_WIDTH[arm-eabi] = "64" >> -FEATURES[arm-eabi] = "+v6,+vfp2" >> +DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" >> +LLVM_TARGET[arm] = "${RUST_TARGET_SYS}" >> +TARGET_ENDIAN[arm] = "little" >> +TARGET_POINTER_WIDTH[arm] = "32" >> +TARGET_C_INT_WIDTH[arm] = "32" >> +MAX_ATOMIC_WIDTH[arm] = "64" >> +FEATURES[arm] = "+v6,+vfp2" >> >> ## armv7-unknown-linux-gnueabihf >> DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" >> @@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""): >> with open(wd + sys + '.json', 'w') as f: >> json.dump(tspec, f, indent=4) >> >> +do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES" >> + >> python do_rust_gen_targets () { >> wd = d.getVar('WORKDIR') + '/targets/' >> build_arch = d.getVar('BUILD_ARCH') > We need to split this into two. One patch to fix the determinism issue, > the other to fix the arm bug. > > The "good" news is that this patch now fails consistently on the > autobuilder, for example: > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3522 > > sstatetests.SStateTests.test_sstate_allarch_samesigs is failing which > you can rerun with: > > oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs > > Now it fails reliably, we need to work out why the arm change breaks > that. > > If you split the patches into two we can likely merge the > reproducibility one, then onwards with the arm issue. > > In case it isn't clear, the failing signatures are for adwaita-icon- > theme which depends on librsvg-native which depends on rust-native. > What this means is that the signature of rust-native is changing when > MACHINE is changed in that test. This shouldn't happen. We already knew > that from our other discussion but now we can prove it. It would be > nice if a test showed us the native hash was changing and that is the > other issue we discussed but one step at a time! > > To be clear, rust-native (and it's signature) should be independent of > MACHINE so we need to fix that. This means that currently rust-native > varies and rebuilds when you change MACHINE (after the dependency fix > in this patch). Hello Richard, I analyzed the test failure and it is because of the newly added dependency on variable LLVM_TARGET. In think the patch split will not help here. We need to understand the reason for the introduced change in first place from 'arm' to 'arm-eabi' & why the rust_gen_target is not added dependency on these variables. Hello Alex, We need a few inputs from you regarding a 'arm' bug which is in discussion in this patch. The changes in file /meta/recipes-devtools/rust/rust-common.inc/ with commit ID '/d6b563710e6cc0857843433d85023d47f9f2037d/' are causing the build error with 'arm' architecture (the /tspec /variables returning '/NoneType/') when '/TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}/" added in /local.conf/ (See the detailed build error at beginning of this email). And to fix that error the changes for '/arm/' architecture are reverted (#arm-unknown-linux-gnueabihf variables reverted to '/arm/' from '/arm-eabi/') and the build error is solved. But, because no Task is depending on these variables, /bitbake /is unable to detect these changes while rebuilding (In other words, the Sstate-cache sigdata is not changed). So, the rust_gen_target task dependency to these variables are added and then the Sstate sigdata is updated with rebuild. However, the dependency (on variable LLVM_TARGET) is causing the failure in /oe-selftest test_sstate_allarch_samesigs/ testcase. Can you let us know is it the right thing to add these variable dependencies for rust_gen_target task? And, to fix the 'arm NoneType' error is it the correct fix to revert the '/arm-eabi/' to '/arm/'? Let me know if anything is not clear. Below is the dependency list taken from /bitbake-diffsigs/ from the /test_sstate_allarch_samesigs/ test sigdata for x86_64 & qemuarm machines. The change in sigdata caused by differed values for variable TUNE_FEATURES{callconvention-hard}. ================= ... List of dependencies for variable LLVM_TARGET[aarch64] is ['RUST_TARGET_SYS'] List of dependencies for variable LLVM_TARGET[arm] is ['RUST_TARGET_SYS'] ... List of dependencies for variable RUST_TARGET_SYS is ['rust_base_triple'] ... List of dependencies for variable rust_base_triple is ['determine_libc', 'target_is_armv7'] ... List of dependencies for variable determine_libc is ['BUILD_SYS', 'HOST_SYS', 'RUST_LIBC', 'RUST_LIBC:class-native'] # The determine_libc is using the TUNE_FEATURES List of dependencies for variable target_is_armv7 is [] ... TUNE_FEATURES{callconvention-hard} = *Unset *#in x86_64 TUNE_FEATURES{callconvention-hard} = *Set *#in qemuarm ... ==================== Thanks, Sundeep K. > > Cheers, > > Richard >
Hello Sundeep, this looks like something that requires a bit of time to set up, reproduce and understand, and unfortunately I do not have the time this week or the next. If you need my assistance you have to wait. Alex On Thu, 19 May 2022 at 13:12, Sundeep KOKKONDA <sundeep.kokkonda@gmail.com> wrote: > > Hello, > On 14-05-2022 12:46, richard.purdie@linuxfoundation.org wrote: > > On Thu, 2022-05-12 at 22:59 -0700, Sundeep KOKKONDA wrote: > > [Yocto bug #14742] > The build shows below error while building for arm machines. > Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' > Detailed error info : > > Steps to reproduce: > 1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"' > 2. bitbake core-image-minimal -cpopulate_sdk > > Complete Error: > ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated: > The stack trace of python calls that resulted in this exception/failure was: > File: 'exec_func_python() autogenerated', lineno: 2, function: <module> > 0001: > *** 0002:do_rust_gen_targets(d) > 0003: > File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets > 0027: > 0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}" > 0029:python do_rust_gen_targets () { > 0030: wd = d.getVar('WORKDIR') + '/targets/' > *** 0031: rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH')) > 0032: rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH')) > 0033: rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH')) > 0034:} > 0035: > File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target > 0326: # build tspec > 0327: tspec = {} > 0328: tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi) > 0329: tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi) > *** 0330: tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi)) > 0331: tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi) > 0332: tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi) > 0333: tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi) > 0334: tspec['arch'] = arch_to_rust_target_arch(rust_arch) > Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' > > Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType. > (a) Locals at rust_gen_target for arm:: > tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'> > tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'> > DEBUG: Python function do_rust_gen_targets finished > (b) Locals at rust_gen_target for aarch64:: > tspec['data-layout'] = aarch64-unknown-linux-gnu, Type of tspec['data-layout'] = <class 'str'> > tspec['max-atomic-width'] = 128, Type of tspec['max-atomic-width'] = <class 'int'> > > Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue. > > The 'rust_gen_targets' Task added with its dependent variable list. > > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@gmail.com> > --- > meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++------- > 1 file changed, 9 insertions(+), 7 deletions(-) > > diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc > index 310aecef22..984fe9099e 100644 > --- a/meta/recipes-devtools/rust/rust-common.inc > +++ b/meta/recipes-devtools/rust/rust-common.inc > @@ -119,13 +119,13 @@ def llvm_features(d): > > > ## arm-unknown-linux-gnueabihf > -DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" > -LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}" > -TARGET_ENDIAN[arm-eabi] = "little" > -TARGET_POINTER_WIDTH[arm-eabi] = "32" > -TARGET_C_INT_WIDTH[arm-eabi] = "32" > -MAX_ATOMIC_WIDTH[arm-eabi] = "64" > -FEATURES[arm-eabi] = "+v6,+vfp2" > +DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" > +LLVM_TARGET[arm] = "${RUST_TARGET_SYS}" > +TARGET_ENDIAN[arm] = "little" > +TARGET_POINTER_WIDTH[arm] = "32" > +TARGET_C_INT_WIDTH[arm] = "32" > +MAX_ATOMIC_WIDTH[arm] = "64" > +FEATURES[arm] = "+v6,+vfp2" > > ## armv7-unknown-linux-gnueabihf > DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" > @@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""): > with open(wd + sys + '.json', 'w') as f: > json.dump(tspec, f, indent=4) > > +do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES" > + > python do_rust_gen_targets () { > wd = d.getVar('WORKDIR') + '/targets/' > build_arch = d.getVar('BUILD_ARCH') > > We need to split this into two. One patch to fix the determinism issue, > the other to fix the arm bug. > > The "good" news is that this patch now fails consistently on the > autobuilder, for example: > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3522 > > sstatetests.SStateTests.test_sstate_allarch_samesigs is failing which > you can rerun with: > > oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs > > Now it fails reliably, we need to work out why the arm change breaks > that. > > If you split the patches into two we can likely merge the > reproducibility one, then onwards with the arm issue. > > In case it isn't clear, the failing signatures are for adwaita-icon- > theme which depends on librsvg-native which depends on rust-native. > What this means is that the signature of rust-native is changing when > MACHINE is changed in that test. This shouldn't happen. We already knew > that from our other discussion but now we can prove it. It would be > nice if a test showed us the native hash was changing and that is the > other issue we discussed but one step at a time! > > To be clear, rust-native (and it's signature) should be independent of > MACHINE so we need to fix that. This means that currently rust-native > varies and rebuilds when you change MACHINE (after the dependency fix > in this patch). > > Hello Richard, > > I analyzed the test failure and it is because of the newly added dependency on variable LLVM_TARGET. In think the patch split will not help here. We need to understand the reason for the introduced change in first place from 'arm' to 'arm-eabi' & why the rust_gen_target is not added dependency on these variables. > > Hello Alex, > > We need a few inputs from you regarding a 'arm' bug which is in discussion in this patch. > > The changes in file meta/recipes-devtools/rust/rust-common.inc with commit ID 'd6b563710e6cc0857843433d85023d47f9f2037d' are causing the build error with 'arm' architecture (the tspec variables returning 'NoneType') when 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}" added in local.conf (See the detailed build error at beginning of this email). > > And to fix that error the changes for 'arm' architecture are reverted (#arm-unknown-linux-gnueabihf variables reverted to 'arm' from 'arm-eabi') and the build error is solved. But, because no Task is depending on these variables, bitbake is unable to detect these changes while rebuilding (In other words, the Sstate-cache sigdata is not changed). So, the rust_gen_target task dependency to these variables are added and then the Sstate sigdata is updated with rebuild. However, the dependency (on variable LLVM_TARGET) is causing the failure in oe-selftest test_sstate_allarch_samesigs testcase. > > Can you let us know is it the right thing to add these variable dependencies for rust_gen_target task? And, to fix the 'arm NoneType' error is it the correct fix to revert the 'arm-eabi' to 'arm'? > > Let me know if anything is not clear. > > Below is the dependency list taken from bitbake-diffsigs from the test_sstate_allarch_samesigs test sigdata for x86_64 & qemuarm machines. The change in sigdata caused by differed values for variable TUNE_FEATURES{callconvention-hard}. > ================= > ... > List of dependencies for variable LLVM_TARGET[aarch64] is ['RUST_TARGET_SYS'] > List of dependencies for variable LLVM_TARGET[arm] is ['RUST_TARGET_SYS'] > ... > List of dependencies for variable RUST_TARGET_SYS is ['rust_base_triple'] > ... > List of dependencies for variable rust_base_triple is ['determine_libc', 'target_is_armv7'] > ... > List of dependencies for variable determine_libc is ['BUILD_SYS', 'HOST_SYS', 'RUST_LIBC', 'RUST_LIBC:class-native'] # The determine_libc is using the TUNE_FEATURES > List of dependencies for variable target_is_armv7 is [] > ... > TUNE_FEATURES{callconvention-hard} = Unset #in x86_64 > TUNE_FEATURES{callconvention-hard} = Set #in qemuarm > ... > ==================== > > > Thanks, > Sundeep K. > > > > Cheers, > > Richard > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#165881): https://lists.openembedded.org/g/openembedded-core/message/165881 > Mute This Topic: https://lists.openembedded.org/mt/91074788/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Let me jump in and explain a bit about what I think we need to do here
to move forward.
With the patch in this series applied, "oe-selftest -r
sstatetests.SStateTests.test_sstate_allarch_samesigs" fails. Looking at
the test, it sets two different MACHINE values, "qemuarm" and "qemux86-
64". Lets see if we can get more information about what is going on.
With MACHINE = "qemux86-64", I run:
bitbake adwaita-icon-theme rust-native cargo-native -S none
(since adwaita-icon-theme is what is changing when it shouldn't and we
suspect a problem with rust-native and cargo-native).
This generates a locked-sigs.inc, lets move that to a different
location:
mv locked-sigs.inc locked-sigs2.inc
Now with MACHINE = "qemuarm", I run:
bitbake adwaita-icon-theme rust-native cargo-native -S none
and we get another locked-sigs.inc.
Comparing locked-sigs.inc with locked-sigs2.inc is revealing. I like:
meld locked-sigs.inc locked-sigs2.inc
but you could use diff too or whatever way you prefer to view
differences.
adwaita-icon-theme is changing, there are a load of changes from
SIGGEN_LOCKEDSIGS_t-core2-64 to SIGGEN_LOCKEDSIGS_t-cortexa15t2hf-neon
which isn't surprising. You can see binutils-cross-arm becoming
binutils-cross-x86_64 which again, isn't surprising. cargo-native
changes which is a worry. rust-native also changes and is the likely
cause of our issues, it is furthest down the dependency stack. The
tasks which change are:
rust-native:do_build
rust-native:do_compile
rust-native:do_install
rust-native:do_populate_sysroot
rust-native:do_rust_gen_targets
and since the patch is changing do_rust_gen_targets, that is likely our
problem.
Now we've proven that the signature of that task changes between these
two different configs, we need to investigate what the issue is there
and bitbake-diffsigs can likely help with that. The idea is that the
native recipes should be independent of the target machine choice so
that is the issue which now needs to be fixed.
I did notice that if you compare qemux86-64 with qemuarm64, you don't
see this issue, it only happens with qemuarm. This probably means that
some 32 vs 64 bit issue is happening in rust-native.
Note I did all of this without actually building anything, just some
recipe parses to dump the signatures.
Does that help move things forward with the issue?
Cheers,
Richard
On Fri, 2022-05-20 at 10:56 +0100, Richard Purdie via lists.openembedded.org wrote: > Let me jump in and explain a bit about what I think we need to do here > to move forward. > > With the patch in this series applied, "oe-selftest -r > sstatetests.SStateTests.test_sstate_allarch_samesigs" fails. Looking at > the test, it sets two different MACHINE values, "qemuarm" and "qemux86- > 64". Lets see if we can get more information about what is going on. > > With MACHINE = "qemux86-64", I run: > > bitbake adwaita-icon-theme rust-native cargo-native -S none > > (since adwaita-icon-theme is what is changing when it shouldn't and we > suspect a problem with rust-native and cargo-native). > > This generates a locked-sigs.inc, lets move that to a different > location: > > mv locked-sigs.inc locked-sigs2.inc > > Now with MACHINE = "qemuarm", I run: > > bitbake adwaita-icon-theme rust-native cargo-native -S none > > and we get another locked-sigs.inc. > > Comparing locked-sigs.inc with locked-sigs2.inc is revealing. I like: > > meld locked-sigs.inc locked-sigs2.inc > > but you could use diff too or whatever way you prefer to view > differences. > > adwaita-icon-theme is changing, there are a load of changes from > SIGGEN_LOCKEDSIGS_t-core2-64 to SIGGEN_LOCKEDSIGS_t-cortexa15t2hf-neon > which isn't surprising. You can see binutils-cross-arm becoming > binutils-cross-x86_64 which again, isn't surprising. cargo-native > changes which is a worry. rust-native also changes and is the likely > cause of our issues, it is furthest down the dependency stack. The > tasks which change are: > > rust-native:do_build > rust-native:do_compile > rust-native:do_install > rust-native:do_populate_sysroot > rust-native:do_rust_gen_targets > > and since the patch is changing do_rust_gen_targets, that is likely our > problem. > > Now we've proven that the signature of that task changes between these > two different configs, we need to investigate what the issue is there > and bitbake-diffsigs can likely help with that. The idea is that the > native recipes should be independent of the target machine choice so > that is the issue which now needs to be fixed. > > I did notice that if you compare qemux86-64 with qemuarm64, you don't > see this issue, it only happens with qemuarm. This probably means that > some 32 vs 64 bit issue is happening in rust-native. > > Note I did all of this without actually building anything, just some > recipe parses to dump the signatures. > > Does that help move things forward with the issue? Since I now had the environment, I help but complete this (taking the hashes from the difference between the locked-sigs diff): $ bitbake-diffsigs tmp/stamps/x86_64-linux/rust-native/1.60.0-r0.do_rust_gen_targets.sigdata.63* tmp/stamps/x86_64-linux/rust-native/1.60.0-r0.do_rust_gen_targets.sigdata.34* NOTE: Starting bitbake server... basehash changed from 860b8f11b10182dc5b2737f62cdb697477f714adb63eeb4d4b932d67cac8eec2 to 9379e8b9df9696e8056fec7d1534661f34dda073f6d816e241b09a2dff76ae2d Variable rust_base_triple value changed: @@ -36,4 +36,4 @@ # In some cases uname and the toolchain differ on their idea of the arch name -TUNE_FEATURES{callconvention-hard} = Set +TUNE_FEATURES{callconvention-hard} = Unset The question now becomes how to fix that since I doubt rust-native really does depend on that. Cheers, Richard
On Fri, 2022-05-20 at 12:04 +0100, Richard Purdie via lists.openembedded.org wrote: > On Fri, 2022-05-20 at 10:56 +0100, Richard Purdie via > lists.openembedded.org wrote: > > Let me jump in and explain a bit about what I think we need to do here > > to move forward. > > > > With the patch in this series applied, "oe-selftest -r > > sstatetests.SStateTests.test_sstate_allarch_samesigs" fails. Looking at > > the test, it sets two different MACHINE values, "qemuarm" and "qemux86- > > 64". Lets see if we can get more information about what is going on. > > > > With MACHINE = "qemux86-64", I run: > > > > bitbake adwaita-icon-theme rust-native cargo-native -S none > > > > (since adwaita-icon-theme is what is changing when it shouldn't and we > > suspect a problem with rust-native and cargo-native). > > > > This generates a locked-sigs.inc, lets move that to a different > > location: > > > > mv locked-sigs.inc locked-sigs2.inc > > > > Now with MACHINE = "qemuarm", I run: > > > > bitbake adwaita-icon-theme rust-native cargo-native -S none > > > > and we get another locked-sigs.inc. > > > > Comparing locked-sigs.inc with locked-sigs2.inc is revealing. I like: > > > > meld locked-sigs.inc locked-sigs2.inc > > > > but you could use diff too or whatever way you prefer to view > > differences. > > > > adwaita-icon-theme is changing, there are a load of changes from > > SIGGEN_LOCKEDSIGS_t-core2-64 to SIGGEN_LOCKEDSIGS_t-cortexa15t2hf-neon > > which isn't surprising. You can see binutils-cross-arm becoming > > binutils-cross-x86_64 which again, isn't surprising. cargo-native > > changes which is a worry. rust-native also changes and is the likely > > cause of our issues, it is furthest down the dependency stack. The > > tasks which change are: > > > > rust-native:do_build > > rust-native:do_compile > > rust-native:do_install > > rust-native:do_populate_sysroot > > rust-native:do_rust_gen_targets > > > > and since the patch is changing do_rust_gen_targets, that is likely our > > problem. > > > > Now we've proven that the signature of that task changes between these > > two different configs, we need to investigate what the issue is there > > and bitbake-diffsigs can likely help with that. The idea is that the > > native recipes should be independent of the target machine choice so > > that is the issue which now needs to be fixed. > > > > I did notice that if you compare qemux86-64 with qemuarm64, you don't > > see this issue, it only happens with qemuarm. This probably means that > > some 32 vs 64 bit issue is happening in rust-native. > > > > Note I did all of this without actually building anything, just some > > recipe parses to dump the signatures. > > > > Does that help move things forward with the issue? > > Since I now had the environment, I help but complete this (taking the > hashes from the difference between the locked-sigs diff): > > $ bitbake-diffsigs tmp/stamps/x86_64-linux/rust-native/1.60.0-r0.do_rust_gen_targets.sigdata.63* tmp/stamps/x86_64-linux/rust-native/1.60.0-r0.do_rust_gen_targets.sigdata.34* > NOTE: Starting bitbake server... > basehash changed from 860b8f11b10182dc5b2737f62cdb697477f714adb63eeb4d4b932d67cac8eec2 to 9379e8b9df9696e8056fec7d1534661f34dda073f6d816e241b09a2dff76ae2d > Variable rust_base_triple value changed: > @@ -36,4 +36,4 @@ > > > # In some cases uname and the toolchain differ on their idea of the arch name > -TUNE_FEATURES{callconvention-hard} = Set > +TUNE_FEATURES{callconvention-hard} = Unset > > The question now becomes how to fix that since I doubt rust-native > really does depend on that. The fix is probably this: diff --git a/meta/classes/rust-common.bbclass b/meta/classes/rust-common.bbclass index 02a538258af..cb811ac5da7 100644 --- a/meta/classes/rust-common.bbclass +++ b/meta/classes/rust-common.bbclass @@ -117,8 +117,11 @@ RUST_BUILD_ARCH = "${@oe.rust.arch_to_rust_arch(d.getVar('BUILD_ARCH'))}" # its likely best to not use the triple suffix due to potential confusion. RUST_BUILD_SYS = "${@rust_base_triple(d, 'BUILD')}" +RUST_BUILD_SYS[vardepvalue] = "${RUST_BUILD_SYS}" RUST_HOST_SYS = "${@rust_base_triple(d, 'HOST')}" +RUST_HOST_SYS[vardepvalue] = "${RUST_HOST_SYS}" RUST_TARGET_SYS = "${@rust_base_triple(d, 'TARGET')}" +RUST_TARGET_SYS[vardepvalue] = "${RUST_TARGET_SYS}" # wrappers to get around the fact that Rust needs a single # binary but Yocto's compiler and linker commands have which makes the sstate signatures depend on the determined values rather than the ingredients in them (which is usually what we want but not here). Cheers, Richard
Hello Richard, The patch you sent in rust-common.bbclass fixing the ' oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs' test issue but the regression test shows that the 'sstatetests.SStateTests.test_sstate_32_64_same_hash' is getting failed. So, I think this change is not correct. With this patch, the diffsigs shows 'rust-native' sigdata is changed based on the changed value for RUST_TARGET_SYS variable (in x86_64 :: x86_64-unknown-linux-gnu & in i686 :: i686-unknown-linux-gnu). I tried several attempts to fix it but nothing worked out yet. @Alex. Did you get sometime to look into this 'arm' issue introduced with your commit? Thanks, Sundeep K.
On Tue, 2022-05-24 at 03:21 -0700, Sundeep KOKKONDA wrote: > Hello Richard, > > The patch you sent in rust-common.bbclass fixing the 'oe-selftest -r > sstatetests.SStateTests.test_sstate_allarch_samesigs' test issue but > the regression test shows that the > 'sstatetests.SStateTests.test_sstate_32_64_same_hash' is getting > failed. So, I think this change is not correct. With this patch, the > diffsigs shows 'rust-native' sigdata is changed based on the changed > value for RUST_TARGET_SYS variable (in x86_64 :: x86_64-unknown- > linux-gnu & in i686 :: i686-unknown-linux-gnu). I tried several > attempts to fix it but nothing worked out yet. You should have seen a further couple of patches which fixed other issues and with current master, the issues should be resolved? Cheers, Richard
On 24-05-2022 16:34, richard.purdie@linuxfoundation.org wrote: > On Tue, 2022-05-24 at 03:21 -0700, Sundeep KOKKONDA wrote: >> Hello Richard, >> >> The patch you sent in rust-common.bbclass fixing the 'oe-selftest -r >> sstatetests.SStateTests.test_sstate_allarch_samesigs' test issue but >> the regression test shows that the >> 'sstatetests.SStateTests.test_sstate_32_64_same_hash' is getting >> failed. So, I think this change is not correct. With this patch, the >> diffsigs shows 'rust-native' sigdata is changed based on the changed >> value for RUST_TARGET_SYS variable (in x86_64 :: x86_64-unknown- >> linux-gnu & in i686 :: i686-unknown-linux-gnu). I tried several >> attempts to fix it but nothing worked out yet. > You should have seen a further couple of patches which fixed other > issues and with current master, the issues should be resolved? I used old sources. Thanks for info. Issue is fixed. Thanks, Sundeep K. > > Cheers, > > Richard > >
diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc index 310aecef22..984fe9099e 100644 --- a/meta/recipes-devtools/rust/rust-common.inc +++ b/meta/recipes-devtools/rust/rust-common.inc @@ -119,13 +119,13 @@ def llvm_features(d): ## arm-unknown-linux-gnueabihf -DATA_LAYOUT[arm-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" -LLVM_TARGET[arm-eabi] = "${RUST_TARGET_SYS}" -TARGET_ENDIAN[arm-eabi] = "little" -TARGET_POINTER_WIDTH[arm-eabi] = "32" -TARGET_C_INT_WIDTH[arm-eabi] = "32" -MAX_ATOMIC_WIDTH[arm-eabi] = "64" -FEATURES[arm-eabi] = "+v6,+vfp2" +DATA_LAYOUT[arm] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" +LLVM_TARGET[arm] = "${RUST_TARGET_SYS}" +TARGET_ENDIAN[arm] = "little" +TARGET_POINTER_WIDTH[arm] = "32" +TARGET_C_INT_WIDTH[arm] = "32" +MAX_ATOMIC_WIDTH[arm] = "64" +FEATURES[arm] = "+v6,+vfp2" ## armv7-unknown-linux-gnueabihf DATA_LAYOUT[armv7-eabi] = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64" @@ -360,6 +360,8 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""): with open(wd + sys + '.json', 'w') as f: json.dump(tspec, f, indent=4) +do_rust_gen_targets[vardeps] += "DATA_LAYOUT LLVM_TARGET TARGET_ENDIAN TARGET_POINTER_WIDTH TARGET_C_INT_WIDTH MAX_ATOMIC_WIDTH FEATURES" + python do_rust_gen_targets () { wd = d.getVar('WORKDIR') + '/targets/' build_arch = d.getVar('BUILD_ARCH')
[Yocto bug #14742] The build shows below error while building for arm machines. Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' Detailed error info : Steps to reproduce: 1. Set MACHINE ?= "qemuarm" in local.conf & add 'TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"' 2. bitbake core-image-minimal -cpopulate_sdk Complete Error: ERROR: rust-cross-canadian-arm-1.59.0-r0 do_rust_gen_targets: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_rust_gen_targets(d) 0003: File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-cross-canadian-common.inc', lineno: 31, function: do_rust_gen_targets 0027: 0028:LLVM_TARGET[x86_64] = "${RUST_HOST_SYS}" 0029:python do_rust_gen_targets () { 0030: wd = d.getVar('WORKDIR') + '/targets/' *** 0031: rust_gen_target(d, 'TARGET', wd, d.getVar('TARGET_LLVM_FEATURES') or "", d.getVar('TARGET_LLVM_CPU'), d.getVar('TARGET_ARCH')) 0032: rust_gen_target(d, 'HOST', wd, "", "generic", d.getVar('HOST_ARCH')) 0033: rust_gen_target(d, 'BUILD', wd, "", "generic", d.getVar('BUILD_ARCH')) 0034:} 0035: File: '/ala-lpggp31/skokkonda/yocto/poky/meta/recipes-devtools/rust/rust-common.inc', lineno: 330, function: rust_gen_target 0326: # build tspec 0327: tspec = {} 0328: tspec['llvm-target'] = d.getVarFlag('LLVM_TARGET', arch_abi) 0329: tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi) *** 0330: tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi)) 0331: tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi) 0332: tspec['target-c-int-width'] = d.getVarFlag('TARGET_C_INT_WIDTH', arch_abi) 0333: tspec['target-endian'] = d.getVarFlag('TARGET_ENDIAN', arch_abi) 0334: tspec['arch'] = arch_to_rust_target_arch(rust_arch) Exception: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' Below are the local variables from rust_gen_target function for arm and aarch64 targets. Refer below, the tspec varibles for 'arm' generated with NoneType. (a) Locals at rust_gen_target for arm:: tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'> tspec['data-layout'] = None, Type of tspec['data-layout'] = <class 'NoneType'> DEBUG: Python function do_rust_gen_targets finished (b) Locals at rust_gen_target for aarch64:: tspec['data-layout'] = aarch64-unknown-linux-gnu, Type of tspec['data-layout'] = <class 'str'> tspec['max-atomic-width'] = 128, Type of tspec['max-atomic-width'] = <class 'int'> Reason for changing arm-eabi to arm: The earlier changes introduced this bug, so reverting the change 'arm-eabi' to 'arm' fixed the issue. The 'rust_gen_targets' Task added with its dependent variable list. Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@gmail.com> --- meta/recipes-devtools/rust/rust-common.inc | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)