| Message ID | 20260826155417.2774608-1-alhe@linux.microsoft.com |
|---|---|
| State | Under Review |
| Headers | show |
| Series | [v3] rust: Avoid passing host-specific information to crates Strict Version Hash | expand |
Hi Alejandro, Thanks for chasing this one down. Small notes on v3: The commit is tagged [YOCTO #15554], but that bug is "reproducibility failures with 'rustdoc' on rust 1.75" (path-length dependent), RESOLVED FIXED since 2024-08-01 via codegen-units=1 / lto=off. What you describe here - artifacts differing by build host arch, causing cross-architecture sstate matches - looks like: [YOCTO #16376] AB-INT: build system architecture reproducibility issue https://bugzilla.yoctoproject.org/show_bug.cgi?id=16376 which is still open. Could you retag it in v4, unless you had a reason to point at 15554 that I'm missing? --- One pointer that may or may not be relevant: the 1.97.1 upgrade (65271a5d3b) carries an LLVM backport: 0043-LoopUnroll-Iterate-exit-fold-candidates-in-determini.patch added for a repro issue seen with rust 1.97.0. I haven't looked at whether it interacts with the SVH path at all - just flagging it in case it's useful context. Thanks, Sunil Dora
On Wed, 2026-08-26 at 11:25 -0700, Dora, Sunil Kumar via lists.openembedded.org wrote: > Hi Alejandro, > > Thanks for chasing this one down. Small notes on v3: > > The commit is tagged [YOCTO #15554], but that bug is "reproducibility > failures with 'rustdoc' on rust 1.75" (path-length dependent), RESOLVED FIXED > since 2024-08-01 via codegen-units=1 / lto=off. > > What you describe here - artifacts differing by build host arch, causing > cross-architecture sstate matches - looks like: > > [YOCTO #16376] AB-INT: build system architecture reproducibility issue > https://bugzilla.yoctoproject.org/show_bug.cgi?id=16376 > > which is still open. Could you retag it in v4, unless you had a reason to > point at 15554 that I'm missing? > --- > One pointer that may or may not be relevant: the 1.97.1 upgrade (65271a5d3b) > carries an LLVM backport: > > 0043-LoopUnroll-Iterate-exit-fold-candidates-in-determini.patch > > added for a repro issue seen with rust 1.97.0. I haven't looked at whether it > interacts with the SVH path at all - just flagging it in case it's useful context. I put v3 in for testing on the autobuilder and it looks promising, there are some rust test suite failures though: https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4472 https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/4577 and probably more to follow. We may need to exclude those tests? I also triggered the reproducibility test targets: https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/3 https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/3 https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/3 those builds are ongoing. Cheers, Richard
On 8/26/2026 1:17 PM, Richard Purdie via lists.openembedded.org wrote: > On Wed, 2026-08-26 at 11:25 -0700, Dora, Sunil Kumar via lists.openembedded.org wrote: >> Hi Alejandro, >> >> Thanks for chasing this one down. Small notes on v3: >> >> The commit is tagged [YOCTO #15554], but that bug is "reproducibility >> failures with 'rustdoc' on rust 1.75" (path-length dependent), RESOLVED FIXED >> since 2024-08-01 via codegen-units=1 / lto=off. >> >> What you describe here - artifacts differing by build host arch, causing >> cross-architecture sstate matches - looks like: >> >> [YOCTO #16376] AB-INT: build system architecture reproducibility issue >> https://bugzilla.yoctoproject.org/show_bug.cgi?id=16376 >> >> which is still open. Could you retag it in v4, unless you had a reason to >> point at 15554 that I'm missing? >> --- >> One pointer that may or may not be relevant: the 1.97.1 upgrade (65271a5d3b) >> carries an LLVM backport: >> >> 0043-LoopUnroll-Iterate-exit-fold-candidates-in-determini.patch >> >> added for a repro issue seen with rust 1.97.0. I haven't looked at whether it >> interacts with the SVH path at all - just flagging it in case it's useful context. Hey Richard, Sorry for the incorrect tag. I'll look into these failures as well and send a v4 Alejandro > I put v3 in for testing on the autobuilder and it looks promising, there are some rust test suite failures though: > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4472 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/4577 > > and probably more to follow. We may need to exclude those tests? > > I also triggered the reproducibility test targets: > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/3 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/3 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/3 > > those builds are ongoing. > > Cheers, > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#244412): https://lists.openembedded.org/g/openembedded-core/message/244412 > Mute This Topic: https://lists.openembedded.org/mt/120939810/4354175 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alhe@linux.microsoft.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Wed, 2026-08-26 at 14:08 -0600, Alejandro Hernandez wrote: > > On 8/26/2026 1:17 PM, Richard Purdie via lists.openembedded.org > wrote: > > On Wed, 2026-08-26 at 11:25 -0700, Dora, Sunil Kumar via > > lists.openembedded.org wrote: > > > Hi Alejandro, > > > > > > Thanks for chasing this one down. Small notes on v3: > > > > > > The commit is tagged [YOCTO #15554], but that bug is > > > "reproducibility > > > failures with 'rustdoc' on rust 1.75" (path-length dependent), > > > RESOLVED FIXED > > > since 2024-08-01 via codegen-units=1 / lto=off. > > > > > > What you describe here - artifacts differing by build host arch, > > > causing > > > cross-architecture sstate matches - looks like: > > > > > > [YOCTO #16376] AB-INT: build system architecture > > > reproducibility issue > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=16376 > > > > > > which is still open. Could you retag it in v4, unless you had a > > > reason to > > > point at 15554 that I'm missing? > > > --- > > > One pointer that may or may not be relevant: the 1.97.1 upgrade > > > (65271a5d3b) > > > carries an LLVM backport: > > > > > > 0043-LoopUnroll-Iterate-exit-fold-candidates-in- > > > determini.patch > > > > > > added for a repro issue seen with rust 1.97.0. I haven't looked > > > at whether it > > > interacts with the SVH path at all - just flagging it in case > > > it's useful context. > > Hey Richard, > > > Sorry for the incorrect tag. No problem, that is at least easily fixed! > I'll look into these failures as well and send a v4 Thanks! I appreciate the work on this as it should improve things a lot. > > I put v3 in for testing on the autobuilder and it looks promising, > > there are some rust test suite failures though: > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4472 > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/4577 > > > > and probably more to follow. We may need to exclude those tests? > > > > I also triggered the reproducibility test targets: > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/121/builds/3 > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/119/builds/3 > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/3 > > > > those builds are ongoing. One of these completed: https://autobuilder.yoctoproject.org/valkyrie/#/builders/120/builds/3 which leads to: https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260826-plagtpam/packages/diff-html/ There looks to be a locale issue, some python config tweaks, curl- config binconfig needing a tweak and a smaller rust issue left. Definitely progress, much fewer recipes showing issues and fewer binary ones and more text ones left... Cheers, Richard
diff --git a/meta/recipes-devtools/rust/files/0003-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch b/meta/recipes-devtools/rust/files/0003-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch new file mode 100644 index 0000000000..f9d171df75 --- /dev/null +++ b/meta/recipes-devtools/rust/files/0003-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch @@ -0,0 +1,35 @@ +rust: Avoid passing host-dependent fingerprint to build artifacts + +To fix reproducibility issues, pass a fixed value instead of passing a +host-dependent fingerprint to the rust build artifacts via the Strict +Version Hash (SVH) + +Upstream-Status: Inappropriate [OE-specific reproducibility policy] +Signed-off-by: Alejandro Hernandez <alhe@linux.microsoft.com> +--- +--- a/compiler/rustc_span/src/def_id.rs ++++ b/compiler/rustc_span/src/def_id.rs +@@ -163,7 +163,7 @@ + crate_name: Symbol, + is_exe: bool, + mut metadata: Vec<String>, +- cfg_version: &'static str, ++ _cfg_version: &'static str, + ) -> StableCrateId { + let mut hasher = StableHasher::new(); + // We must hash the string text of the crate name, not the id, as the id is not stable +@@ -195,11 +195,9 @@ + // + // RUSTC_FORCE_RUSTC_VERSION is used to inject rustc version information + // during testing. +- if let Some(val) = std::env::var_os("RUSTC_FORCE_RUSTC_VERSION") { +- hasher.write(val.to_string_lossy().into_owned().as_bytes()) +- } else { +- hasher.write(cfg_version.as_bytes()) +- } ++ // OE reproducible builds use a fixed value so host-varying bootstrap ++ // fingerprints do not perturb StableCrateId. ++ hasher.write(b"oe-stable-crate-id-no-cfg-version"); + + StableCrateId(hasher.finish()) + } diff --git a/meta/recipes-devtools/rust/rust-source.inc b/meta/recipes-devtools/rust/rust-source.inc index 68f3009a7b..2bff69de3d 100644 --- a/meta/recipes-devtools/rust/rust-source.inc +++ b/meta/recipes-devtools/rust/rust-source.inc @@ -6,6 +6,7 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n file://repro-issue-fix-with-cc-crate-hashmap.patch;patchdir=${RUSTSRC} \ file://0001-src-core-build_steps-tool.rs-switch-off-lto-for-rust.patch;patchdir=${RUSTSRC} \ file://0002-Fix-rust-build-failure-with-unstable-options.patch;patchdir=${RUSTSRC} \ + file://0003-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch;patchdir=${RUSTSRC} \ " SRC_URI[rust.sha256sum] = "0ed06fdaffd4722a7702e0b4eebfafc897ab8f513e8e1b247cdd7e5c6df6ded2"
rustc computes each crate's Strict Version Hash (SVH) using inputs that include the *stage0/stage1 bootstrap compiler* fingerprint, which in turn depends on the build host arch. This eventually may cause sstate matches across architectures for artifacts that are actually different, causing autobuilder intermitent reproducibility issues. To avoid this, pass a fixed value instead of host-specific bits to the specified hash. [YOCTO #15554] Assisted-by: AI - OpenAI Signed-off-by: Alejandro Hernandez <alhe@linux.microsoft.com> --- ...ide-cfg-version-from-stable-crate-id.patch | 35 +++++++++++++++++++ meta/recipes-devtools/rust/rust-source.inc | 1 + 2 files changed, 36 insertions(+) create mode 100644 meta/recipes-devtools/rust/files/0003-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch