diff mbox series

[v4] rust: Avoid passing host-specific information to crates

Message ID 20260908155315.2468903-1-alhe@linux.microsoft.com
State New
Headers show
Series [v4] rust: Avoid passing host-specific information to crates | expand

Commit Message

Alejandro Hernandez Sept. 8, 2026, 3:53 p.m. UTC
There were several build contamination issues found on our rust builds:

SVH - rustc computes each crate's Strict Version Hash (SVH) using inputs that
    include the *stage0/stage1 bootstrap compiler* fingerprint, which in turn
    depends on the build host arch.

    This eventually may cause sstate matches across architectures for artifacts
    that are actually different, causing autobuilder intermitent reproducibility
    issues.

    To avoid this, pass a fixed value instead of host-specific bits to the
    specified hash.

Cmetadata - Cargo hashes the `host:` line of `rustc -vV` into the metadata of
    host units (build scripts and proc-macros), so target libraries that depend
    on a crate carrying a build script inherited the build host triple through
    -Cmetadata and, from there, each crate's StableCrateId.

Unordered data - DocLinkResMap is an UnordMap wrapping an FxHashMap, so the
    doc-link table was serialised into crate metadata in hash iteration order,
    which is not stable across hosts. Switch it to FxIndexMap, which is already
    Encodable, Decodable and HashStable.

    HygieneEncodeContext::encode consumed `latest_ctxts` and `latest_expns`
    directly from an `FxHashSet` without sorting. Payload blobs are serialized
    into the crate metadata buffer during this loop in hash iteration order.
    Sort both before encoding so payloads are written into metadata in
    deterministic order.

With all three addressed, Rust metadata and compiled libraries are reproducible
across mixed-architecture build hosts (x86_64 vs aarch64).

[YOCTO #16376]

Assisted-by: AI - OpenAI
Signed-off-by: Alejandro Hernandez <alhe@linux.microsoft.com>
---
 ...ide-cfg-version-from-stable-crate-id.patch | 36 +++++++++++
 ...-host-triple-from-unit-metadata-hash.patch | 55 +++++++++++++++++
 ...oc-link-metadata-order-deterministic.patch | 36 +++++++++++
 ...hygiene-encoding-order-deterministic.patch | 60 +++++++++++++++++++
 meta/recipes-devtools/rust/rust-source.inc    |  4 ++
 5 files changed, 191 insertions(+)
 create mode 100644 meta/recipes-devtools/rust/files/0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch
 create mode 100644 meta/recipes-devtools/rust/files/0007-cargo-omit-host-triple-from-unit-metadata-hash.patch
 create mode 100644 meta/recipes-devtools/rust/files/0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch
 create mode 100644 meta/recipes-devtools/rust/files/0009-rustc-span-make-hygiene-encoding-order-deterministic.patch

Comments

patchtest@automation.yoctoproject.org Sept. 8, 2026, 4:01 p.m. UTC | #1
Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/v4-rust-Avoid-passing-host-specific-information-to-crates.patch

FAIL: test Signed-off-by presence: A patch file has been added without a Signed-off-by tag: '0009-rustc-span-make-hygiene-encoding-order-deterministic.patch' (test_patch.TestPatch.test_signed_off_by_presence)

PASS: test CVE tag format (test_patch.TestPatch.test_cve_tag_format)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test Upstream-Status presence (test_patch.TestPatch.test_upstream_status_presence_format)
PASS: test auh changelog truncation notice (test_mbox.TestMbox.test_auh_changelog_truncation_notice)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test bugzilla entry format (test_mbox.TestMbox.test_bugzilla_entry_format)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest pylint: No python related patches, skipping test (test_python_pylint.PyLint.pretest_pylint)
SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping test (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!
diff mbox series

Patch

diff --git a/meta/recipes-devtools/rust/files/0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch b/meta/recipes-devtools/rust/files/0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch
new file mode 100644
index 00000000000..a7d03d9c363
--- /dev/null
+++ b/meta/recipes-devtools/rust/files/0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch
@@ -0,0 +1,36 @@ 
+rust: Avoid passing host-dependent fingerprint to build artifacts
+
+To fix reproducibility issues, pass a fixed value instead of passing a
+host-dependent fingerprint to the rust build artifacts via the Strict
+Version Hash (SVH)
+
+Upstream-Status: Inappropriate [OE-specific]
+Assisted-by: AI - OpenAI
+Signed-off-by: Alejandro Hernandez <alhe@linux.microsoft.com>
+---
+--- a/compiler/rustc_span/src/def_id.rs
++++ b/compiler/rustc_span/src/def_id.rs
+@@ -163,7 +163,7 @@
+         crate_name: Symbol,
+         is_exe: bool,
+         mut metadata: Vec<String>,
+-        cfg_version: &'static str,
++        _cfg_version: &'static str,
+     ) -> StableCrateId {
+         let mut hasher = StableHasher::new();
+         // We must hash the string text of the crate name, not the id, as the id is not stable
+@@ -195,11 +195,9 @@
+         //
+         // RUSTC_FORCE_RUSTC_VERSION is used to inject rustc version information
+         // during testing.
+-        if let Some(val) = std::env::var_os("RUSTC_FORCE_RUSTC_VERSION") {
+-            hasher.write(val.to_string_lossy().into_owned().as_bytes())
+-        } else {
+-            hasher.write(cfg_version.as_bytes())
+-        }
++        // OE reproducible builds use a fixed value so host-varying bootstrap
++        // fingerprints do not perturb StableCrateId.
++        hasher.write(b"oe-stable-crate-id-no-cfg-version");
+ 
+         StableCrateId(hasher.finish())
+     }
diff --git a/meta/recipes-devtools/rust/files/0007-cargo-omit-host-triple-from-unit-metadata-hash.patch b/meta/recipes-devtools/rust/files/0007-cargo-omit-host-triple-from-unit-metadata-hash.patch
new file mode 100644
index 00000000000..ca842858c10
--- /dev/null
+++ b/meta/recipes-devtools/rust/files/0007-cargo-omit-host-triple-from-unit-metadata-hash.patch
@@ -0,0 +1,55 @@ 
+cargo: omit the build host triple from the unit metadata hash
+
+Cargo mixes the `host:` line of `rustc -vV` into the metadata hash of host
+units (build scripts and proc-macros), and every unit additionally hashes the
+metadata of its dependencies. Target libraries that depend on a crate carrying
+a build script therefore inherit the build host triple, which ends up in
+`-Cmetadata` and consequently in each crate's StableCrateId.
+
+The result is that libraries built for the same target are not reproducible
+across build hosts of different architectures: only `core` (which has no
+build-script dependency) keeps a stable crate id, while `alloc`, `std` and
+everything downstream change.
+
+Stop hashing the host triple so the metadata of a unit depends on the target
+being built rather than on the machine performing the build.
+
+Upstream-Status: Inappropriate [OE-specific]
+
+Assisted-by: AI - OpenAI
+Signed-off-by: Alejandro Hernandez <alhe@linux.microsoft.com>
+
+---
+--- a/src/tools/cargo/src/cargo/core/compiler/build_runner/compilation_files.rs
++++ b/src/tools/cargo/src/cargo/core/compiler/build_runner/compilation_files.rs
+@@ -877,7 +877,7 @@
+ }
+ 
+ /// Hash the version of rustc being used during the build process.
+-fn hash_rustc_version(bcx: &BuildContext<'_, '_>, hasher: &mut StableHasher, unit: &Unit) {
++fn hash_rustc_version(bcx: &BuildContext<'_, '_>, hasher: &mut StableHasher, _unit: &Unit) {
+     let vers = &bcx.rustc().version;
+     if vers.pre.is_empty() || bcx.gctx.cli_unstable().separate_nightlies {
+         // For stable, keep the artifacts separate. This helps if someone is
+@@ -886,7 +886,7 @@
+         // omitted since rustc should produce the same output for each target
+         // regardless of the host.
+         for line in bcx.rustc().verbose_version.lines() {
+-            if unit.kind.is_host() || !line.starts_with("host: ") {
++            if !line.starts_with("host: ") {
+                 line.hash(hasher);
+             }
+         }
+@@ -899,12 +899,6 @@
+     // This assumes that the first segment is the important bit ("nightly",
+     // "beta", "dev", etc.). Skip other parts like the `.3` in `-beta.3`.
+     vers.pre.split('.').next().hash(hasher);
+-    // Keep "host" since some people switch hosts to implicitly change
+-    // targets, (like gnu vs musl or gnu vs msvc). In the future, we may want
+-    // to consider hashing `unit.kind.short_name()` instead.
+-    if unit.kind.is_host() {
+-        bcx.rustc().host.hash(hasher);
+-    }
+     // None of the other lines are important. Currently they are:
+     // binary: rustc  <-- or "rustdoc"
+     // commit-hash: 38114ff16e7856f98b2b4be7ab4cd29b38bed59a
diff --git a/meta/recipes-devtools/rust/files/0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch b/meta/recipes-devtools/rust/files/0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch
new file mode 100644
index 00000000000..46fb2ed9712
--- /dev/null
+++ b/meta/recipes-devtools/rust/files/0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch
@@ -0,0 +1,36 @@ 
+rustc_hir: serialise doc-link resolutions in a deterministic order
+
+DocLinkResMap is an UnordMap, which wraps an FxHashMap and derives its
+Encodable implementation, so crate metadata records the doc-link table in hash
+iteration order. That order is not stable across build hosts, leaving a few
+hundred bytes of the .rustc section, and the DefIndex values that follow it,
+different for otherwise identical builds.
+
+Use an insertion-ordered FxIndexMap instead. Entries are added while walking
+the AST, so insertion order is deterministic, and the consumers of this map
+only ever look entries up by key.
+
+Upstream-Status: Inappropriate [OE-specific]
+Assisted-by: AI - OpenAI
+Signed-off-by: Alejandro Hernandez <alhe@linux.microsoft.com>
+---
+--- a/compiler/rustc_hir/src/def.rs
++++ b/compiler/rustc_hir/src/def.rs
+@@ -4,7 +4,7 @@
+ 
+ use rustc_ast as ast;
+ use rustc_ast::NodeId;
+-use rustc_data_structures::unord::UnordMap;
++use rustc_data_structures::fx::FxIndexMap;
+ use rustc_error_messages::{DiagArgValue, IntoDiagArg};
+ use rustc_macros::{Decodable, Encodable, StableHash};
+ use rustc_span::Symbol;
+@@ -969,4 +969,7 @@
+     ElidedAnchor { start: NodeId, end: NodeId },
+ }
+ 
+-pub type DocLinkResMap = UnordMap<(Symbol, Namespace), Option<Res<NodeId>>>;
++// Serialise doc-link resolutions in insertion order: UnordMap wraps an
++// FxHashMap, whose iteration order varies with the build host and leaves
++// crate metadata unreproducible across builders.
++pub type DocLinkResMap = FxIndexMap<(Symbol, Namespace), Option<Res<NodeId>>>;
diff --git a/meta/recipes-devtools/rust/files/0009-rustc-span-make-hygiene-encoding-order-deterministic.patch b/meta/recipes-devtools/rust/files/0009-rustc-span-make-hygiene-encoding-order-deterministic.patch
new file mode 100644
index 00000000000..93b0fcd68e1
--- /dev/null
+++ b/meta/recipes-devtools/rust/files/0009-rustc-span-make-hygiene-encoding-order-deterministic.patch
@@ -0,0 +1,60 @@ 
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: Alejandro Hernandez <alhe@linux.microsoft.com>
+Date: Mon, 7 Sep 2026 00:00:00 +0000
+Subject: [PATCH] rustc_span: Make hygiene metadata encoding order deterministic
+
+In `HygieneEncodeContext::encode`, `latest_ctxts` (`FxHashSet<SyntaxContext>`)
+and `latest_expns` (`FxHashSet<ExpnId>`) are consumed by iterating directly
+over the hash set.
+
+While the table entries storing offsets for syntax contexts and expansions
+are indexed by ID, `encode_ctxt` and `encode_expn` serialize the actual
+`SyntaxContextData` and `ExpnData` payload blobs directly into the crate
+metadata buffer (`self.opaque`) during this loop.
+
+Because `FxHashSet` iteration order depends on hash values that differ
+between build host architectures, the byte stream of hygiene payloads
+in metadata was non-deterministic across x86_64 and aarch64 build hosts.
+
+Sort `latest_ctxts` and `latest_expns` prior to encoding so hygiene
+payloads are written into metadata in deterministic ascending ID order.
+
+Upstream-Status: Submitted [https://github.com/rust-lang/rust/pull/...]
+[YOCTO #15554]
+
+AI-Generated: Uses GitHub Copilot
+
+--- a/compiler/rustc_span/src/hygiene.rs
++++ b/compiler/rustc_span/src/hygiene.rs
+@@ -1314,24 +1326,26 @@
+             // Consume the current round of syntax contexts.
+             // Drop the lock() temporary early.
+-            // It's fine to iterate over a HashMap, because the serialization of the table
+-            // that we insert data into doesn't depend on insertion order.
+             #[allow(rustc::potential_query_instability)]
+-            let latest_ctxts = { mem::take(&mut *self.latest_ctxts.lock()) }.into_iter();
++            let mut latest_ctxts: Vec<_> = { mem::take(&mut *self.latest_ctxts.lock()) }.into_iter().collect();
++            latest_ctxts.sort_by_key(|ctxt| ctxt.0);
+             let all_ctxt_data: Vec<_> = HygieneData::with(|data| {
+                 latest_ctxts
++                    .into_iter()
+                     .map(|ctxt| (ctxt, data.syntax_context_data[ctxt.0 as usize].key()))
+                     .collect()
+             });
+             for (ctxt, ctxt_key) in all_ctxt_data {
+                 if self.serialized_ctxts.lock().insert(ctxt) {
+                     encode_ctxt(encoder, ctxt.0, &ctxt_key);
+                 }
+             }
+ 
+             // Same as above, but for expansions instead of syntax contexts.
+             #[allow(rustc::potential_query_instability)]
+-            let latest_expns = { mem::take(&mut *self.latest_expns.lock()) }.into_iter();
++            let mut latest_expns: Vec<_> = { mem::take(&mut *self.latest_expns.lock()) }.into_iter().collect();
++            latest_expns.sort_by_key(|expn| (expn.krate, expn.local_id));
+             let all_expn_data: Vec<_> = HygieneData::with(|data| {
+                 latest_expns
++                    .into_iter()
+                     .map(|expn| (expn, data.expn_data(expn).clone(), data.expn_hash(expn)))
+                     .collect()
+             });
diff --git a/meta/recipes-devtools/rust/rust-source.inc b/meta/recipes-devtools/rust/rust-source.inc
index 019e2585e28..1d192b1c716 100644
--- a/meta/recipes-devtools/rust/rust-source.inc
+++ b/meta/recipes-devtools/rust/rust-source.inc
@@ -9,6 +9,10 @@  SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n
             file://0003-bootstrap-skip-StdarchVerify-when-remote-testing.patch;patchdir=${RUSTSRC} \
             file://0004-Backport-commits-from-rust-Fix-selftest-llvm23.patch;patchdir=${RUSTSRC} \
             file://0005-rustc_codegen_llvm-Do-not-pass-amx-tf32-to-LLVM-23.patch;patchdir=${RUSTSRC} \
+            file://0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch;patchdir=${RUSTSRC} \
+            file://0007-cargo-omit-host-triple-from-unit-metadata-hash.patch;patchdir=${RUSTSRC} \
+            file://0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch;patchdir=${RUSTSRC} \
+            file://0009-rustc-span-make-hygiene-encoding-order-deterministic.patch;patchdir=${RUSTSRC} \
 "
 SRC_URI[rust.sha256sum] = "be1816e7f6c40abb90245ad6e024bed2a7e88d7dda4561e4d5470207df616b9f"