| Message ID | 20240816155419.213703-1-Yash.Shinde@windriver.com |
|---|---|
| State | Accepted, archived |
| Commit | 81bbd6db47f45628ff4be400e1f2fa5b09ccd0bb |
| Headers | show |
| Series | [1/2] rust: Oe-selftest changes for rust v1.78 | expand |
On 2024-08-16 11:54 a.m., Yash.Shinde@windriver.com wrote: > From: Yash Shinde<Yash.Shinde@windriver.com> > > Add some of the upsupported/failing tests to the exclude list > and ignore the failing unit tests. > > Signed-off-by: Yash Shinde<Yash.Shinde@windriver.com> > --- > meta/lib/oeqa/selftest/cases/rust.py | 34 +++++++++- > .../rust/files/rust-oe-selftest.patch | 64 ++++++++++++++++++- > 2 files changed, 95 insertions(+), 3 deletions(-) > > diff --git a/meta/lib/oeqa/selftest/cases/rust.py b/meta/lib/oeqa/selftest/cases/rust.py > index 62ca762cd0..aa5814eccf 100644 > --- a/meta/lib/oeqa/selftest/cases/rust.py > +++ b/meta/lib/oeqa/selftest/cases/rust.py > @@ -205,7 +205,39 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): > 'library/std/src/sync/rwlock/tests.rs', > 'library/std/src/thread/tests.rs', > 'library/test/src/stats/tests.rs', > - 'tidyselftest' > + 'tidyselftest', > + 'compiler/rustc_span', > + 'compiler/rustc_expand', > + 'tests/ui/array-slice-vec', > + 'tests/ui/associated-consts', > + 'tests/ui/associated-types', > + 'tests/ui/allocator', > + 'tests/ui/abi/extern', > + 'tests/codegen/non-terminate/infinite-loop-1.rs', > + 'tests/codegen/coercions.rs', > + 'tests/codegen/const_scalar_pair.rs', > + 'tests/codegen/cold-call-declare-and-call.rs', > + 'tests/codegen/issues', > + 'tests/codegen/bool-cmp.rs', > + 'tests/codegen/ascii-char.rs', > + 'tests/codegen/frame-pointer.rs', > + 'tests/codegen/common_prim_int_ptr.rs', > + 'tests/codegen/default-requires-uwtable.rs', > + 'tests/codegen/debug-accessibility/public-enum.rs', > + 'tests/codegen/meta-filecheck/filecheck-flags.rs', > + 'tests/codegen/float_math.rs', > + 'tests/codegen/debug-accessibility/tuple-fields.rs', > + 'tests/codegen/consts.rs', > + 'tests/codegen/enum/enum-bounds-check.rs', > + 'tests/codegen/debug-alignment.rs', > + 'tests/codegen/export-no-mangle.rs', > + 'tests/codegen/fastcall-inreg.rs', > + 'tests/codegen/debuginfo-inline-callsite-location.rs', > + 'tests/codegen/enum/enum-debug-tagged.rs', > + 'tests/codegen/adjustments.rs', > + 'tests/codegen/macos/x86_64-macosx-deployment-target.rs', > + 'tests/codegen/frame-pointer.rs', > + 'tests/codegen/meta-filecheck/check-prefix.rs' > ] Hi Yash, There was a problem with a SDK test failing in 1.78 due to a linkage problem for rustc in the SDK. You replied to the email saying: "This also resolves the "bitbake core-image-sato -c do_testsdk" error for cargo." but I don't see a patch or change that would resolve that problem. Can you explain briefly what the problem was and what you did to fix it for me and anyone who might be interested, please? It seems that we now have just over 170 tests that we're skipping! Ugh! Skipping tests is okay as a temporary work-around but once we have reached feature freeze next week on Aug 26, please start looking at why these tests are failing and try to fix the root cause so the tests can be re-enable. $ rg -A 175 "exclude_list = " meta/lib/oeqa/selftest/cases/rust.py Where should you start, you might ask? Let's look at this list... Oh, it's not sorted alphabetically as we agreed it should be! Please fix that as a single commit now or as part of the next Rust update if that makes more sense for you given that the feature freeze deadline is less than a week away. BTW, you should be able to do this sorting within your editor (emacs can do it at least! ;-) ) but let's do it in the shell: # cut on the ' delimiter: $ rg -A 172 "exclude_list = " meta/lib/oeqa/selftest/cases/rust.py | tail -172 | cut -d"'" -f2 | sort > /tmp/after The 172 was determined manually. Here's why you want the list sorted: ❯ uniq -c /tmp/after | grep -v " 1 " 2 compiler/rustc_interface/src/tests.rs 2 library/std/src/thread/tests.rs 2 library/test/src/stats/tests.rs 2 tests/codegen/frame-pointer.rs 2 tests/rustdoc-json 2 tests/rustdoc-ui/doc-test-rustdoc-feature.rs 2 tests/rustdoc-ui/no-run-flag.rs so there are "only" 165 tests being skipped: ❯ uniq -c /tmp/after | wc -l 165 I'm most concerned about the compiler and library/std tests but we do need to work on understanding and hopefully fixing all the skipped tests. Please take a look at say up to 5 tests, hopefully related, and work on fixing them as a group, rather than trying to fix the entire list in one work item and one commit. In future commits, please include a summary of the total number of tests that are being skipped and the total number that are being run so we can have a numerical indicator to track the rust quality. Oh, you had exact statistics in this poky commit: commit c496b1eaf4b72ca979d47c37e73e44cd145fed1c Author: Yash Shinde <Yash.Shinde@windriver.com> Date: Sat Jul 29 03:00:21 2023 ... * The tests that are passed and skipped are as follows- Target PASS SKIPPED ARM 15507 428 ARM64 15535 400 MIPS64 15479 456 X86 15528 407 X86-64 15643 292 so it looks like things have improved. Yay! Do you know offhand if upstream Rust passes all tests for x86-64 and arm64 ? I know that running these tests takes quite a long time so perhaps you can run the full testsuite for all targets once per YP release, on the final Rust update so that the results are in a commit log. Given that I've changed requirements, late in this release, if that's a problem then just send the update to 1.80.x in without the full set of all target statistics. Hopefully during the M4 bug fix cycle, you can fix some of the failing tests and provide statistics for all targets, including riscv64. Please let me know what you think. Thanks, ../Randy
On 21-08-2024 07:13, Randy MacLeod wrote: > On 2024-08-16 11:54 a.m., Yash.Shinde@windriver.com wrote: >> From: Yash Shinde<Yash.Shinde@windriver.com> >> >> Add some of the upsupported/failing tests to the exclude list >> and ignore the failing unit tests. >> >> Signed-off-by: Yash Shinde<Yash.Shinde@windriver.com> >> --- >> meta/lib/oeqa/selftest/cases/rust.py | 34 +++++++++- >> .../rust/files/rust-oe-selftest.patch | 64 ++++++++++++++++++- >> 2 files changed, 95 insertions(+), 3 deletions(-) >> >> diff --git a/meta/lib/oeqa/selftest/cases/rust.py b/meta/lib/oeqa/selftest/cases/rust.py >> index 62ca762cd0..aa5814eccf 100644 >> --- a/meta/lib/oeqa/selftest/cases/rust.py >> +++ b/meta/lib/oeqa/selftest/cases/rust.py >> @@ -205,7 +205,39 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): >> 'library/std/src/sync/rwlock/tests.rs', >> 'library/std/src/thread/tests.rs', >> 'library/test/src/stats/tests.rs', >> - 'tidyselftest' >> + 'tidyselftest', >> + 'compiler/rustc_span', >> + 'compiler/rustc_expand', >> + 'tests/ui/array-slice-vec', >> + 'tests/ui/associated-consts', >> + 'tests/ui/associated-types', >> + 'tests/ui/allocator', >> + 'tests/ui/abi/extern', >> + 'tests/codegen/non-terminate/infinite-loop-1.rs', >> + 'tests/codegen/coercions.rs', >> + 'tests/codegen/const_scalar_pair.rs', >> + 'tests/codegen/cold-call-declare-and-call.rs', >> + 'tests/codegen/issues', >> + 'tests/codegen/bool-cmp.rs', >> + 'tests/codegen/ascii-char.rs', >> + 'tests/codegen/frame-pointer.rs', >> + 'tests/codegen/common_prim_int_ptr.rs', >> + 'tests/codegen/default-requires-uwtable.rs', >> + 'tests/codegen/debug-accessibility/public-enum.rs', >> + 'tests/codegen/meta-filecheck/filecheck-flags.rs', >> + 'tests/codegen/float_math.rs', >> + 'tests/codegen/debug-accessibility/tuple-fields.rs', >> + 'tests/codegen/consts.rs', >> + 'tests/codegen/enum/enum-bounds-check.rs', >> + 'tests/codegen/debug-alignment.rs', >> + 'tests/codegen/export-no-mangle.rs', >> + 'tests/codegen/fastcall-inreg.rs', >> + 'tests/codegen/debuginfo-inline-callsite-location.rs', >> + 'tests/codegen/enum/enum-debug-tagged.rs', >> + 'tests/codegen/adjustments.rs', >> + 'tests/codegen/macos/x86_64-macosx-deployment-target.rs', >> + 'tests/codegen/frame-pointer.rs', >> + 'tests/codegen/meta-filecheck/check-prefix.rs' >> ] > > Hi Yash, > > > There was a problem with a SDK test failing in 1.78 due to a linkage > problem for rustc in the SDK. > You replied to the email saying: > "This also resolves the "bitbake core-image-sato -c do_testsdk" > error for cargo." > but I don't see a patch or change that would resolve that problem. Can > you explain briefly what > the problem was and what you did to fix it for me and anyone who might > be interested, please? > The problem was related to the changes in cargo with the following patch: repro-issue-fix-with-cc-crate-hashmap.patch The patch was implemented to fix the rust reproducibility issue. I had successfully implemented and tested the patch to upgrade rust to v1.78 version. However, the issue was observed after integrating the rust repro issue fix. The issue seemed to be somehow related to the checksum and json related files. After encountering the cargo SDK test error, I spent quite a lot of time to debug the error from the logs. It was observed that the issue was related to the cargo changes in "vendor/cc-1.0.79/.cargo-checksum.json" file from repro issue fix patch file. Hence, I implemented the version upgrade to v1.78 from scratch and fixed the issue. > > It seems that we now have just over 170 tests that we're skipping! Ugh! > > Skipping tests is okay as a temporary work-around but once we have > reached feature freeze next week on Aug 26, > please start looking at why these tests are failing and try to fix the > root cause so the tests can be re-enable. > > $ rg -A 175 "exclude_list = " meta/lib/oeqa/selftest/cases/rust.py > > > Where should you start, you might ask? Let's look at this list... > > Oh, it's not sorted alphabetically as we agreed it should be! > Please fix that as a single commit now or as part of the next Rust > update if that makes more sense for you > given that the feature freeze deadline is less than a week away. > > > BTW, you should be able to do this sorting within your editor (emacs > can do it at least! ;-) ) but let's do it in the shell: > > # cut on the ' delimiter: > $ rg -A 172 "exclude_list = " meta/lib/oeqa/selftest/cases/rust.py | > tail -172 | cut -d"'" -f2 | sort > /tmp/after > > The 172 was determined manually. > > Here's why you want the list sorted: > > ❯ uniq -c /tmp/after | grep -v " 1 " > 2 compiler/rustc_interface/src/tests.rs > 2 library/std/src/thread/tests.rs > 2 library/test/src/stats/tests.rs > 2 tests/codegen/frame-pointer.rs > 2 tests/rustdoc-json > 2 tests/rustdoc-ui/doc-test-rustdoc-feature.rs > 2 tests/rustdoc-ui/no-run-flag.rs > > so there are "only" 165 tests being skipped: > > ❯ uniq -c /tmp/after | wc -l > 165 > I'm most concerned about the compiler and library/std tests but > we do need to work on understanding and hopefully fixing all the > skipped tests. > > Please take a look at say up to 5 tests, hopefully related, and work > on fixing them as a group, > rather than trying to fix the entire list in one work item and one commit. > We have already started to work on clearing the exclude list based on rust v1.77.2. We have removed around 50 tests from the exclude list and are being successfully tested(including a few from compiler and library/std tests). Currently, the testing is being performed with v1.78 and will be submitted upstream if there are no issues. There are plans to improve this process further for v1.80 by removing more tests from the exclude list. > > In future commits, please include a summary of the total number of > tests that are being skipped > and the total number that are being run so we can have a numerical > indicator to track the rust > quality. > > > Oh, you had exact statistics in this poky commit: > > commit c496b1eaf4b72ca979d47c37e73e44cd145fed1c > Author: Yash Shinde <Yash.Shinde@windriver.com> > Date: Sat Jul 29 03:00:21 2023 > > ... > > * The tests that are passed and skipped are as follows- > > Target PASS SKIPPED > > ARM 15507 428 > ARM64 15535 400 > MIPS64 15479 456 > X86 15528 407 > X86-64 15643 292 > > so it looks like things have improved. Yay! > > Do you know offhand if upstream Rust passes all tests for x86-64 and > arm64 ? > > > I know that running these tests takes quite a long time so perhaps you > can run > the full testsuite for all targets once per YP release, on the final > Rust update so > that the results are in a commit log. Given that I've changed > requirements, late in this release, > if that's a problem then just send the update to 1.80.x in without the > full set of all target statistics. > Hopefully during the M4 bug fix cycle, you can fix some of the failing > tests and provide statistics > for all targets, including riscv64. > > > Please let me know what you think. > Thanks Randy. This seems to be a great idea to measure the overall rust quality. We will provide the summary of test results for set of all target statistics along with the upgrades. Mips, PPC and Riscv are tier-2 and tier-3 targets for which even rust upstream doesn't do official tests. So, they may or may not work. Many of the tests in the exclude lists come from mips and ppc failures. We are investigating more about the test failures that occurs only for tier-2 and tier-3 targets. We will prepare a report on the same and share it with you after the analysis. Please refer tier 2 and tier 3 section from the following rust document: https://doc.rust-lang.org/nightly/rustc/platform-support.html Regards, Yash > Thanks, > > ../Randy > > >
diff --git a/meta/lib/oeqa/selftest/cases/rust.py b/meta/lib/oeqa/selftest/cases/rust.py index 62ca762cd0..aa5814eccf 100644 --- a/meta/lib/oeqa/selftest/cases/rust.py +++ b/meta/lib/oeqa/selftest/cases/rust.py @@ -205,7 +205,39 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): 'library/std/src/sync/rwlock/tests.rs', 'library/std/src/thread/tests.rs', 'library/test/src/stats/tests.rs', - 'tidyselftest' + 'tidyselftest', + 'compiler/rustc_span', + 'compiler/rustc_expand', + 'tests/ui/array-slice-vec', + 'tests/ui/associated-consts', + 'tests/ui/associated-types', + 'tests/ui/allocator', + 'tests/ui/abi/extern', + 'tests/codegen/non-terminate/infinite-loop-1.rs', + 'tests/codegen/coercions.rs', + 'tests/codegen/const_scalar_pair.rs', + 'tests/codegen/cold-call-declare-and-call.rs', + 'tests/codegen/issues', + 'tests/codegen/bool-cmp.rs', + 'tests/codegen/ascii-char.rs', + 'tests/codegen/frame-pointer.rs', + 'tests/codegen/common_prim_int_ptr.rs', + 'tests/codegen/default-requires-uwtable.rs', + 'tests/codegen/debug-accessibility/public-enum.rs', + 'tests/codegen/meta-filecheck/filecheck-flags.rs', + 'tests/codegen/float_math.rs', + 'tests/codegen/debug-accessibility/tuple-fields.rs', + 'tests/codegen/consts.rs', + 'tests/codegen/enum/enum-bounds-check.rs', + 'tests/codegen/debug-alignment.rs', + 'tests/codegen/export-no-mangle.rs', + 'tests/codegen/fastcall-inreg.rs', + 'tests/codegen/debuginfo-inline-callsite-location.rs', + 'tests/codegen/enum/enum-debug-tagged.rs', + 'tests/codegen/adjustments.rs', + 'tests/codegen/macos/x86_64-macosx-deployment-target.rs', + 'tests/codegen/frame-pointer.rs', + 'tests/codegen/meta-filecheck/check-prefix.rs' ] exclude_fail_tests = " ".join([" --exclude " + item for item in exclude_list]) diff --git a/meta/recipes-devtools/rust/files/rust-oe-selftest.patch b/meta/recipes-devtools/rust/files/rust-oe-selftest.patch index 1d287335d4..a69c1e5237 100644 --- a/meta/recipes-devtools/rust/files/rust-oe-selftest.patch +++ b/meta/recipes-devtools/rust/files/rust-oe-selftest.patch @@ -223,9 +223,9 @@ index 1a9d3d3f12f..0a9cfc48806 100644 fn test_rw_arc_no_poison_rr() { let arc = Arc::new(RwLock::new(1)); let arc2 = arc.clone(); -@@ -82,6 +85,7 @@ fn test_rw_arc_no_poison_rr() { - assert_eq!(*lock, 1); +@@ -127,6 +127,7 @@ } + #[test] +#[ignore] fn test_rw_arc_no_poison_rw() { @@ -330,3 +330,63 @@ diff --git a/library/alloc/src/slice/tests.rs b/library/alloc/src/slice/tests.rs #[cfg_attr(target_os = "emscripten", ignore)] // no threads #[cfg_attr(not(panic = "unwind"), ignore = "test requires unwinding support")] fn panic_safe() { +diff --git a/library/test/src/tests.rs b/library/test/src/tests.rs +--- a/library/test/src/tests.rs ++++ b/library/test/src/tests.rs +@@ -424,6 +424,7 @@ + } + + #[test] ++#[ignore] + fn test_time_options_threshold() { + let unit = TimeThreshold::new(Duration::from_millis(50), Duration::from_millis(100)); + let integration = TimeThreshold::new(Duration::from_millis(500), Duration::from_millis(1000)); +diff --git a/library/std/src/sync/mutex/tests.rs b/library/std/src/sync/mutex/tests.rs +--- a/library/std/src/sync/mutex/tests.rs ++++ b/library/std/src/sync/mutex/tests.rs +@@ -193,6 +193,7 @@ + } + + #[test] ++#[ignore] + fn test_mutex_arc_poison_mapped() { + let arc = Arc::new(Mutex::new(1)); + assert!(!arc.is_poisoned()); +diff --git a/library/std/src/sync/mutex/tests.rs b/library/std/src/sync/mutex/tests.rs +--- a/library/std/src/sync/mutex/tests.rs ++++ b/library/std/src/sync/mutex/tests.rs +@@ -272,6 +272,7 @@ + } + + #[test] ++#[ignore] + fn panic_while_mapping_unlocked_poison() { + let lock = Mutex::new(()); + +diff --git a/ +--- a/library/std/src/sync/rwlock/tests.rs ++++ b/library/std/src/sync/rwlock/tests.rs +@@ -60,6 +60,7 @@ + } + + #[test] ++#[ignore] + fn test_rw_arc_poison_mapped_w_r() { + let arc = Arc::new(RwLock::new(1)); + let arc2 = arc.clone(); +@@ -88,6 +89,7 @@ + } + + #[test] ++#[ignore] + fn test_rw_arc_poison_mapped_w_w() { + let arc = Arc::new(RwLock::new(1)); + let arc2 = arc.clone(); +@@ -438,6 +440,7 @@ + } + + #[test] ++#[ignore] + fn panic_while_mapping_write_unlocked_poison() { + let lock = RwLock::new(()); +