| Message ID | 20221225215921.12868-1-alex.kiernan@gmail.com |
|---|---|
| Headers | show |
| Series | Add rust runtime tests | expand |
On Sun, 2022-12-25 at 21:59 +0000, Alex Kiernan wrote: > This series adds runtime tests for rust and cargo, merges (most of) the > include files into the base recipes and fixes target cargo builds. > > Also add SDK_TOOLCHAIN_LANGS for selection of target toolchains as part > of packagegroup-core-sdk. > > Changes in v4: > - Fixup `__stack_chk_fail_local` in musl-x86 > - Add crossbeam atomics fixes for ppc/mips > > Changes in v3: > - Switch to packagegroup-core-sdk driven from TARGET_TOOLCHAIN_LANGS > (which defaults to SDK_TOOLCHAIN_LANGS) > > Changes in v2: > - Drop rust.inc/rust.bb merge, not actually made use of in the change > set > - Reorder so cargo test goes after build fixes > > Alex Kiernan (11): > rust-llvm: Merge .inc into .bb > rust-llvm: Update LLVM_VERSION to match embedded version > packagegroup-rust-sdk-target: Add Rust SDK target packagegroup > packagegroup-core-sdk: Add SDK toolchain language selection support > cargo: Merge .inc into .bb > cargo: Extend DEBUG_PREFIX_MAP to cover vendor > rust: Merge .inc into .bb > rust: Move musl-x86 fix for `__stack_chk_fail_local` to rust-source > cargo: Override crossbeam-utils with 0.8.14 > packagegroup-rust-sdk-target: Add cargo > oeqa/runtime/rust: Add cargo test Looks much better thanks. There was one remaining issue which showed up in testing with a reproducibility issue: https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/2119/steps/12/logs/stdio We did already have an open bug for a rust reproducibility issue and we excluded it from the tests until we could get to the bottom of it, I suspect cargo may have the same issue. It frustrates me a bit as it is the one reproducibility exclusion for core. Ideally we would fix it but excluding it from the tests and noting it in the bug may be an option if it is the same issue. It may be simpler to reproduce and fix the issue with cargo too. Cc: Sundeep who as been looking at it. Cheers, Richard
On Mon, Dec 26, 2022 at 6:33 PM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Sun, 2022-12-25 at 21:59 +0000, Alex Kiernan wrote: > > This series adds runtime tests for rust and cargo, merges (most of) the > > include files into the base recipes and fixes target cargo builds. > > > > Also add SDK_TOOLCHAIN_LANGS for selection of target toolchains as part > > of packagegroup-core-sdk. > > > > Changes in v4: > > - Fixup `__stack_chk_fail_local` in musl-x86 > > - Add crossbeam atomics fixes for ppc/mips > > > > Changes in v3: > > - Switch to packagegroup-core-sdk driven from TARGET_TOOLCHAIN_LANGS > > (which defaults to SDK_TOOLCHAIN_LANGS) > > > > Changes in v2: > > - Drop rust.inc/rust.bb merge, not actually made use of in the change > > set > > - Reorder so cargo test goes after build fixes > > > > Alex Kiernan (11): > > rust-llvm: Merge .inc into .bb > > rust-llvm: Update LLVM_VERSION to match embedded version > > packagegroup-rust-sdk-target: Add Rust SDK target packagegroup > > packagegroup-core-sdk: Add SDK toolchain language selection support > > cargo: Merge .inc into .bb > > cargo: Extend DEBUG_PREFIX_MAP to cover vendor > > rust: Merge .inc into .bb > > rust: Move musl-x86 fix for `__stack_chk_fail_local` to rust-source > > cargo: Override crossbeam-utils with 0.8.14 > > packagegroup-rust-sdk-target: Add cargo > > oeqa/runtime/rust: Add cargo test > > Looks much better thanks. There was one remaining issue which showed up > in testing with a reproducibility issue: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/2119/steps/12/logs/stdio > > We did already have an open bug for a rust reproducibility issue and we > excluded it from the tests until we could get to the bottom of it, I > suspect cargo may have the same issue. It frustrates me a bit as it is > the one reproducibility exclusion for core. > > Ideally we would fix it but excluding it from the tests and noting it > in the bug may be an option if it is the same issue. It may be simpler > to reproduce and fix the issue with cargo too. > > Cc: Sundeep who as been looking at it. > Just reading the ticket and the upstream ones... the upstream ticket is closed as remap path fixes it (which it looks like it does to me), there are build paths embedded in the build directory which don't get caught by that, but they're in thing like bootstrap which isn't packaged (which may or may not be an issue, but I'm suspecting not); it kinda feels like we're missing something. Can I get hold of the cargo packages generated here to have a poke at them?
On Tue, 2022-12-27 at 18:09 +0000, Alex Kiernan wrote: > On Mon, Dec 26, 2022 at 6:33 PM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Sun, 2022-12-25 at 21:59 +0000, Alex Kiernan wrote: > > > This series adds runtime tests for rust and cargo, merges (most of) the > > > include files into the base recipes and fixes target cargo builds. > > > > > > Also add SDK_TOOLCHAIN_LANGS for selection of target toolchains as part > > > of packagegroup-core-sdk. > > > > > > Changes in v4: > > > - Fixup `__stack_chk_fail_local` in musl-x86 > > > - Add crossbeam atomics fixes for ppc/mips > > > > > > Changes in v3: > > > - Switch to packagegroup-core-sdk driven from TARGET_TOOLCHAIN_LANGS > > > (which defaults to SDK_TOOLCHAIN_LANGS) > > > > > > Changes in v2: > > > - Drop rust.inc/rust.bb merge, not actually made use of in the change > > > set > > > - Reorder so cargo test goes after build fixes > > > > > > Alex Kiernan (11): > > > rust-llvm: Merge .inc into .bb > > > rust-llvm: Update LLVM_VERSION to match embedded version > > > packagegroup-rust-sdk-target: Add Rust SDK target packagegroup > > > packagegroup-core-sdk: Add SDK toolchain language selection support > > > cargo: Merge .inc into .bb > > > cargo: Extend DEBUG_PREFIX_MAP to cover vendor > > > rust: Merge .inc into .bb > > > rust: Move musl-x86 fix for `__stack_chk_fail_local` to rust-source > > > cargo: Override crossbeam-utils with 0.8.14 > > > packagegroup-rust-sdk-target: Add cargo > > > oeqa/runtime/rust: Add cargo test > > > > Looks much better thanks. There was one remaining issue which showed up > > in testing with a reproducibility issue: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/2119/steps/12/logs/stdio > > > > We did already have an open bug for a rust reproducibility issue and we > > excluded it from the tests until we could get to the bottom of it, I > > suspect cargo may have the same issue. It frustrates me a bit as it is > > the one reproducibility exclusion for core. > > > > Ideally we would fix it but excluding it from the tests and noting it > > in the bug may be an option if it is the same issue. It may be simpler > > to reproduce and fix the issue with cargo too. > > > > Cc: Sundeep who as been looking at it. > > > > Just reading the ticket and the upstream ones... the upstream ticket > is closed as remap path fixes it (which it looks like it does to me), > there are build paths embedded in the build directory which don't get > caught by that, but they're in thing like bootstrap which isn't > packaged (which may or may not be an issue, but I'm suspecting not); > it kinda feels like we're missing something. > > Can I get hold of the cargo packages generated here to have a poke at them? Yes. The key line in the log is: 022-12-26 11:56:30,555 - oe-selftest - INFO - Non-reproducible packages will be copied to /srv/autobuilder/autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk i.e. https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk where you'll find both the package files and diffoscope output: https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk/packages/diff-html/ which seems to be saying the symbol table changed size which is unusual. I didn't look very deeply. Cheers, Richard
On Wed, Dec 28, 2022 at 12:01 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Tue, 2022-12-27 at 18:09 +0000, Alex Kiernan wrote: > > On Mon, Dec 26, 2022 at 6:33 PM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > On Sun, 2022-12-25 at 21:59 +0000, Alex Kiernan wrote: > > > > This series adds runtime tests for rust and cargo, merges (most of) the > > > > include files into the base recipes and fixes target cargo builds. > > > > > > > > Also add SDK_TOOLCHAIN_LANGS for selection of target toolchains as part > > > > of packagegroup-core-sdk. > > > > > > > > Changes in v4: > > > > - Fixup `__stack_chk_fail_local` in musl-x86 > > > > - Add crossbeam atomics fixes for ppc/mips > > > > > > > > Changes in v3: > > > > - Switch to packagegroup-core-sdk driven from TARGET_TOOLCHAIN_LANGS > > > > (which defaults to SDK_TOOLCHAIN_LANGS) > > > > > > > > Changes in v2: > > > > - Drop rust.inc/rust.bb merge, not actually made use of in the change > > > > set > > > > - Reorder so cargo test goes after build fixes > > > > > > > > Alex Kiernan (11): > > > > rust-llvm: Merge .inc into .bb > > > > rust-llvm: Update LLVM_VERSION to match embedded version > > > > packagegroup-rust-sdk-target: Add Rust SDK target packagegroup > > > > packagegroup-core-sdk: Add SDK toolchain language selection support > > > > cargo: Merge .inc into .bb > > > > cargo: Extend DEBUG_PREFIX_MAP to cover vendor > > > > rust: Merge .inc into .bb > > > > rust: Move musl-x86 fix for `__stack_chk_fail_local` to rust-source > > > > cargo: Override crossbeam-utils with 0.8.14 > > > > packagegroup-rust-sdk-target: Add cargo > > > > oeqa/runtime/rust: Add cargo test > > > > > > Looks much better thanks. There was one remaining issue which showed up > > > in testing with a reproducibility issue: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/2119/steps/12/logs/stdio > > > > > > We did already have an open bug for a rust reproducibility issue and we > > > excluded it from the tests until we could get to the bottom of it, I > > > suspect cargo may have the same issue. It frustrates me a bit as it is > > > the one reproducibility exclusion for core. > > > > > > Ideally we would fix it but excluding it from the tests and noting it > > > in the bug may be an option if it is the same issue. It may be simpler > > > to reproduce and fix the issue with cargo too. > > > > > > Cc: Sundeep who as been looking at it. > > > > > > > Just reading the ticket and the upstream ones... the upstream ticket > > is closed as remap path fixes it (which it looks like it does to me), > > there are build paths embedded in the build directory which don't get > > caught by that, but they're in thing like bootstrap which isn't > > packaged (which may or may not be an issue, but I'm suspecting not); > > it kinda feels like we're missing something. > > > > Can I get hold of the cargo packages generated here to have a poke at them? > > Yes. The key line in the log is: > > 022-12-26 11:56:30,555 - oe-selftest - INFO - Non-reproducible packages will be copied to /srv/autobuilder/autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk > > i.e. > > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk > > where you'll find both the package files and diffoscope output: > > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk/packages/diff-html/ > > which seems to be saying the symbol table changed size which is > unusual. I didn't look very deeply. > Upstream has this ticket which, given how I did the patching for cargo, feels like the right kind of thing (rust has something similar out of the box): https://github.com/rust-lang/rust/issues/98185 I've switched it around so that it uses local patching rather than cargo patching, which if it is this, seems like it might resolve it. -- Alex Kiernan
On Wed, 2022-12-28 at 14:51 +0000, Alex Kiernan wrote: > On Wed, Dec 28, 2022 at 12:01 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Tue, 2022-12-27 at 18:09 +0000, Alex Kiernan wrote: > > > On Mon, Dec 26, 2022 at 6:33 PM Richard Purdie > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > On Sun, 2022-12-25 at 21:59 +0000, Alex Kiernan wrote: > > > > > This series adds runtime tests for rust and cargo, merges (most of) the > > > > > include files into the base recipes and fixes target cargo builds. > > > > > > > > > > Also add SDK_TOOLCHAIN_LANGS for selection of target toolchains as part > > > > > of packagegroup-core-sdk. > > > > > > > > > > Changes in v4: > > > > > - Fixup `__stack_chk_fail_local` in musl-x86 > > > > > - Add crossbeam atomics fixes for ppc/mips > > > > > > > > > > Changes in v3: > > > > > - Switch to packagegroup-core-sdk driven from TARGET_TOOLCHAIN_LANGS > > > > > (which defaults to SDK_TOOLCHAIN_LANGS) > > > > > > > > > > Changes in v2: > > > > > - Drop rust.inc/rust.bb merge, not actually made use of in the change > > > > > set > > > > > - Reorder so cargo test goes after build fixes > > > > > > > > > > Alex Kiernan (11): > > > > > rust-llvm: Merge .inc into .bb > > > > > rust-llvm: Update LLVM_VERSION to match embedded version > > > > > packagegroup-rust-sdk-target: Add Rust SDK target packagegroup > > > > > packagegroup-core-sdk: Add SDK toolchain language selection support > > > > > cargo: Merge .inc into .bb > > > > > cargo: Extend DEBUG_PREFIX_MAP to cover vendor > > > > > rust: Merge .inc into .bb > > > > > rust: Move musl-x86 fix for `__stack_chk_fail_local` to rust-source > > > > > cargo: Override crossbeam-utils with 0.8.14 > > > > > packagegroup-rust-sdk-target: Add cargo > > > > > oeqa/runtime/rust: Add cargo test > > > > > > > > Looks much better thanks. There was one remaining issue which showed up > > > > in testing with a reproducibility issue: > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/2119/steps/12/logs/stdio > > > > > > > > We did already have an open bug for a rust reproducibility issue and we > > > > excluded it from the tests until we could get to the bottom of it, I > > > > suspect cargo may have the same issue. It frustrates me a bit as it is > > > > the one reproducibility exclusion for core. > > > > > > > > Ideally we would fix it but excluding it from the tests and noting it > > > > in the bug may be an option if it is the same issue. It may be simpler > > > > to reproduce and fix the issue with cargo too. > > > > > > > > Cc: Sundeep who as been looking at it. > > > > > > > > > > Just reading the ticket and the upstream ones... the upstream ticket > > > is closed as remap path fixes it (which it looks like it does to me), > > > there are build paths embedded in the build directory which don't get > > > caught by that, but they're in thing like bootstrap which isn't > > > packaged (which may or may not be an issue, but I'm suspecting not); > > > it kinda feels like we're missing something. > > > > > > Can I get hold of the cargo packages generated here to have a poke at them? > > > > Yes. The key line in the log is: > > > > 022-12-26 11:56:30,555 - oe-selftest - INFO - Non-reproducible packages will be copied to /srv/autobuilder/autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk > > > > i.e. > > > > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk > > > > where you'll find both the package files and diffoscope output: > > > > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk/packages/diff-html/ > > > > which seems to be saying the symbol table changed size which is > > unusual. I didn't look very deeply. > > > > Upstream has this ticket which, given how I did the patching for > cargo, feels like the right kind of thing (rust has something similar > out of the box): > > https://github.com/rust-lang/rust/issues/98185 > > I've switched it around so that it uses local patching rather than > cargo patching, which if it is this, seems like it might resolve it. It was a different issue, thanks for tracking that down and fixing it! The tests all passed so I merged the v5 series. Thanks for working through things, I think this makes our rust situation much more robust and should help future proof it :) Cheers, Richard
On Thu, Dec 29, 2022 at 12:53 PM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Wed, 2022-12-28 at 14:51 +0000, Alex Kiernan wrote: > > On Wed, Dec 28, 2022 at 12:01 AM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > On Tue, 2022-12-27 at 18:09 +0000, Alex Kiernan wrote: > > > > On Mon, Dec 26, 2022 at 6:33 PM Richard Purdie > > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > > > On Sun, 2022-12-25 at 21:59 +0000, Alex Kiernan wrote: > > > > > > This series adds runtime tests for rust and cargo, merges (most of) the > > > > > > include files into the base recipes and fixes target cargo builds. > > > > > > > > > > > > Also add SDK_TOOLCHAIN_LANGS for selection of target toolchains as part > > > > > > of packagegroup-core-sdk. > > > > > > > > > > > > Changes in v4: > > > > > > - Fixup `__stack_chk_fail_local` in musl-x86 > > > > > > - Add crossbeam atomics fixes for ppc/mips > > > > > > > > > > > > Changes in v3: > > > > > > - Switch to packagegroup-core-sdk driven from TARGET_TOOLCHAIN_LANGS > > > > > > (which defaults to SDK_TOOLCHAIN_LANGS) > > > > > > > > > > > > Changes in v2: > > > > > > - Drop rust.inc/rust.bb merge, not actually made use of in the change > > > > > > set > > > > > > - Reorder so cargo test goes after build fixes > > > > > > > > > > > > Alex Kiernan (11): > > > > > > rust-llvm: Merge .inc into .bb > > > > > > rust-llvm: Update LLVM_VERSION to match embedded version > > > > > > packagegroup-rust-sdk-target: Add Rust SDK target packagegroup > > > > > > packagegroup-core-sdk: Add SDK toolchain language selection support > > > > > > cargo: Merge .inc into .bb > > > > > > cargo: Extend DEBUG_PREFIX_MAP to cover vendor > > > > > > rust: Merge .inc into .bb > > > > > > rust: Move musl-x86 fix for `__stack_chk_fail_local` to rust-source > > > > > > cargo: Override crossbeam-utils with 0.8.14 > > > > > > packagegroup-rust-sdk-target: Add cargo > > > > > > oeqa/runtime/rust: Add cargo test > > > > > > > > > > Looks much better thanks. There was one remaining issue which showed up > > > > > in testing with a reproducibility issue: > > > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/2119/steps/12/logs/stdio > > > > > > > > > > We did already have an open bug for a rust reproducibility issue and we > > > > > excluded it from the tests until we could get to the bottom of it, I > > > > > suspect cargo may have the same issue. It frustrates me a bit as it is > > > > > the one reproducibility exclusion for core. > > > > > > > > > > Ideally we would fix it but excluding it from the tests and noting it > > > > > in the bug may be an option if it is the same issue. It may be simpler > > > > > to reproduce and fix the issue with cargo too. > > > > > > > > > > Cc: Sundeep who as been looking at it. > > > > > > > > > > > > > Just reading the ticket and the upstream ones... the upstream ticket > > > > is closed as remap path fixes it (which it looks like it does to me), > > > > there are build paths embedded in the build directory which don't get > > > > caught by that, but they're in thing like bootstrap which isn't > > > > packaged (which may or may not be an issue, but I'm suspecting not); > > > > it kinda feels like we're missing something. > > > > > > > > Can I get hold of the cargo packages generated here to have a poke at them? > > > > > > Yes. The key line in the log is: > > > > > > 022-12-26 11:56:30,555 - oe-selftest - INFO - Non-reproducible packages will be copied to /srv/autobuilder/autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk > > > > > > i.e. > > > > > > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk > > > > > > where you'll find both the package files and diffoscope output: > > > > > > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20221226-0vk5efsk/packages/diff-html/ > > > > > > which seems to be saying the symbol table changed size which is > > > unusual. I didn't look very deeply. > > > > > > > Upstream has this ticket which, given how I did the patching for > > cargo, feels like the right kind of thing (rust has something similar > > out of the box): > > > > https://github.com/rust-lang/rust/issues/98185 > > > > I've switched it around so that it uses local patching rather than > > cargo patching, which if it is this, seems like it might resolve it. > > It was a different issue, thanks for tracking that down and fixing it! > I'm not so sure - there's a .cargo/config.toml in the vendored rustc tarball which explicitly patches one of the vendored sources (it feels odd, but hey...), so I think we'll trip over this issue (assuming that's what the problem is). > The tests all passed so I merged the v5 series. Thanks for working > through things, I think this makes our rust situation much more robust > and should help future proof it :) > :)