| Message ID | 2e75c6169cac05273b3edde52ea820506ce9d3fe.camel@linuxfoundation.org |
|---|---|
| State | New |
| Headers | show |
| Series | Rust reproducibility issue summary | expand |
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information: --- Testing patch /home/patchtest/share/mboxes/Rust-reproducibility-issue-summary.patch FAIL: test Signed-off-by presence: Mbox is missing Signed-off-by. Add it manually or with "git commit --amend -s" (test_mbox.TestMbox.test_signed_off_by_presence) FAIL: test shortlog format: Commit shortlog (first line of commit message) should follow the format "<target>: <summary>" (test_mbox.TestMbox.test_shortlog_format) PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint) PASS: test author valid (test_mbox.TestMbox.test_author_valid) PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence) PASS: test max line length (test_metadata.TestMetadata.test_max_line_length) PASS: test mbox format (test_mbox.TestMbox.test_mbox_format) PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade) PASS: test pylint (test_python_pylint.PyLint.test_pylint) PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length) PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list) SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files) SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore) SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format) SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence) SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format) SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format) SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned) SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence) SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence) SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head) SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files) SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence) --- Please address the issues identified and submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
On Fri, 31 May 2024 at 15:28, Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > I've been asked what the reproducibility issue is and what data we > have. I ran a couple of test builds on the current autobuilder and the > test cluster. I added in the diff at the end of this email, just to > limit the test to what we're interested in. > > The test on the new test cluster passed: > > https://valkyrie.yoctoproject.org/#/builders/37/builds/36 > > However the build on the current autobuilder did not: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876 > > and the different binaries generated are here: > > http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/ > > We need to get to the bottom of why these are different and the test is > sometimes failing. Any help appreciated! > > It is unclear what we need to do to reproduce the issue on demand. One possible lead is in the original attempt to fix this; https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7 It is not yet clear if disabling PGO makes things reproducible, as the same commit also prevented the non reproducible binary (rustdoc) from being installed (with the demangler line). PGO as the culprit should be either confirmed, or ruled out first. If PGO indeed helps then the next step is figure out how to disable (or tweak) it in a way that doesn't cause issues with chromium: https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 Alex
On 01-Jun-24 15:57, Alexander Kanavin wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Fri, 31 May 2024 at 15:28, Richard Purdie via > lists.openembedded.org > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: >> I've been asked what the reproducibility issue is and what data we >> have. I ran a couple of test builds on the current autobuilder and the >> test cluster. I added in the diff at the end of this email, just to >> limit the test to what we're interested in. >> >> The test on the new test cluster passed: >> >> https://valkyrie.yoctoproject.org/#/builders/37/builds/36 >> >> However the build on the current autobuilder did not: >> >> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876 >> >> and the different binaries generated are here: >> >> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/ >> >> We need to get to the bottom of why these are different and the test is >> sometimes failing. Any help appreciated! >> >> It is unclear what we need to do to reproduce the issue on demand. > One possible lead is in the original attempt to fix this; > https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7 > > It is not yet clear if disabling PGO makes things reproducible, as the > same commit also prevented the non reproducible binary (rustdoc) from > being installed (with the demangler line). > > PGO as the culprit should be either confirmed, or ruled out first. > > If PGO indeed helps then the next step is figure out how to disable > (or tweak) it in a way that doesn't cause issues with chromium: > > https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 Hello Alex, Initially we did enabled PGO & rust-demangler options for rustdoc reproducibility. https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/5 I did not get any info (from doc or community) on how and why optimation is affecting rustdoc binary. https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/6 However, later we learned that the '/rustdoc/' is reproducible without PGO also, and so removed for chromium project. And after removing PGO also we've not seen this rustdoc repro issue on autobuilder. The issue is not reproduced on my end to confirm the PGO is root cause or any recent changes on rust recipe causing the issue? /Sundeep > > Alex
On 03-Jun-24 9:53, Sundeep KOKKONDA via lists.openembedded.org wrote: > > > On 01-Jun-24 15:57, Alexander Kanavin wrote: >> CAUTION: This email comes from a non Wind River email account! >> Do not click links or open attachments unless you recognize the sender and know the content is safe. >> >> On Fri, 31 May 2024 at 15:28, Richard Purdie via >> lists.openembedded.org >> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: >>> I've been asked what the reproducibility issue is and what data we >>> have. I ran a couple of test builds on the current autobuilder and the >>> test cluster. I added in the diff at the end of this email, just to >>> limit the test to what we're interested in. >>> >>> The test on the new test cluster passed: >>> >>> https://valkyrie.yoctoproject.org/#/builders/37/builds/36 >>> >>> However the build on the current autobuilder did not: >>> >>> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876 >>> >>> and the different binaries generated are here: >>> >>> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/ >>> >>> We need to get to the bottom of why these are different and the test is >>> sometimes failing. Any help appreciated! >>> >>> It is unclear what we need to do to reproduce the issue on demand. >> One possible lead is in the original attempt to fix this; >> https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7 >> >> It is not yet clear if disabling PGO makes things reproducible, as the >> same commit also prevented the non reproducible binary (rustdoc) from >> being installed (with the demangler line). >> >> PGO as the culprit should be either confirmed, or ruled out first. >> >> If PGO indeed helps then the next step is figure out how to disable >> (or tweak) it in a way that doesn't cause issues with chromium: >> >> https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 > Hello Alex, > > Initially we did enabled PGO & rust-demangler options for rustdoc > reproducibility. > https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/5 > > I did not get any info (from doc or community) on how and why > optimation is affecting rustdoc binary. > https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/6 > > However, later we learned that the '/rustdoc/' is reproducible without > PGO also, and so removed for chromium project. And after removing PGO > also we've not seen this rustdoc repro issue on autobuilder. > > The issue is not reproduced on my end to confirm the PGO is root cause > or any recent changes on rust recipe causing the issue? Further analysis on autobuilder binaries shows that the deviations are similar to that of earlier /rustdoc /issue. The deviations are in |.llvm.<hash>| suffixes. As discussed in issue <https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027>, may be we should check by disabling PGO (by default it is enabled and as in comment <in https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/3>the PGO profile generation is not deterministic...). rustdoc-175 diff > > > /Sundeep >> Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#200231):https://lists.openembedded.org/g/openembedded-core/message/200231 > Mute This Topic:https://lists.openembedded.org/mt/106407787/7677583 > Group Owner:openembedded-core+owner@lists.openembedded.org > Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [sundeep.kokkonda@eng.windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Yes please. This is what I've been trying to get across: try disabling the PGO once more, but do not change what gets installed, and ensure that rustdoc does get installed and packaged. Then, if that solves the reproducibility issue we need to look at why exactly chromium doesn't like it, so that it had to be reverted: https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 Alex On Mon, 3 Jun 2024 at 09:08, Sundeep KOKKONDA < Sundeep.Kokkonda@windriver.com> wrote: > > On 03-Jun-24 9:53, Sundeep KOKKONDA via lists.openembedded.org wrote: > > > On 01-Jun-24 15:57, Alexander Kanavin wrote: > > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Fri, 31 May 2024 at 15:28, Richard Purdie vialists.openembedded.org<richard.purdie=linuxfoundation.org@lists.openembedded.org> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > I've been asked what the reproducibility issue is and what data we > have. I ran a couple of test builds on the current autobuilder and the > test cluster. I added in the diff at the end of this email, just to > limit the test to what we're interested in. > > The test on the new test cluster passed: > https://valkyrie.yoctoproject.org/#/builders/37/builds/36 > > However the build on the current autobuilder did not: > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876 > > and the different binaries generated are here: > http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/ > > We need to get to the bottom of why these are different and the test is > sometimes failing. Any help appreciated! > > It is unclear what we need to do to reproduce the issue on demand. > > One possible lead is in the original attempt to fix this;https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7 > > It is not yet clear if disabling PGO makes things reproducible, as the > same commit also prevented the non reproducible binary (rustdoc) from > being installed (with the demangler line). > > PGO as the culprit should be either confirmed, or ruled out first. > > If PGO indeed helps then the next step is figure out how to disable > (or tweak) it in a way that doesn't cause issues with chromium: > https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 > > Hello Alex, > > Initially we did enabled PGO & rust-demangler options for rustdoc > reproducibility. > > https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/5 > > I did not get any info (from doc or community) on how and why optimation > is affecting rustdoc binary. > > https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/6 > > However, later we learned that the '*rustdoc*' is reproducible without > PGO also, and so removed for chromium project. And after removing PGO also > we've not seen this rustdoc repro issue on autobuilder. > > The issue is not reproduced on my end to confirm the PGO is root cause or > any recent changes on rust recipe causing the issue? > > > Further analysis on autobuilder binaries shows that the deviations are > similar to that of earlier *rustdoc *issue. The deviations are in > .llvm.<hash> suffixes. > > As discussed in issue > <https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027>, > may be we should check by disabling PGO (by default it is enabled and as in comment > the PGO profile generation is not deterministic...). > [image: rustdoc-175 diff] > > > /Sundeep > > Alex > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#200231): https://lists.openembedded.org/g/openembedded-core/message/200231 > Mute This Topic: https://lists.openembedded.org/mt/106407787/7677583 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [sundeep.kokkonda@eng.windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- > > >
Hello, I've a local reproducer for this issue. Below are a few observations with my analysis. - The deviation in rustdoc binary are because of the different hash-id generated between the build (as explained above - https://lists.openembedded.org/g/openembedded-core/message/200238) - The hash-id generates identically when the rust built from it's sources every time. - During the oe-selftest triggered, when the issue reproduced I can see there are reproducibleA & reproducibleB directories are generated. But, the rust is not built from sources on both directories beacuse of Setscene Tasks and Shared State. (reproducibleB directory has rust sources built and reproducibleA has setscene realted data copied the files into it from sstate and so, no rust sources are cloned & build into it) - The above case not always happening with oe-selftest, when the test passed I can see the reproducibleA also has rust sources pulled and built, in this case rustdoc generates identically. - When the issue reproduced, I did a check on the sstate sigdata to find any deviations but those sig's are identical. - Also, when the test failed, I tried to find the unique hash-id's generated in rustdoc in the generated rlibs of rust. I could only find one hash-id in rlibs where the rust sources are actually built. The other hash-id not found in rlibs of reproducibleA/B. - The ' diff .../reproducibleA/tmp/work/core2-64-poky-linux/rust/1.75.0/temp/depsig.do_package_write_deb .../reproducibleB/tmp/work/core2-64-poky-linux/rust/1.75.0/temp/depsig.do_package_write_deb ' shows - 8c8 < -rw- 15932884 6a6618ccfc171ea507ad39eb95bc239116b5f5c9e85a50ce9de3b43c73ff17d7./deploy-debs/core2-64/rust-dbg_1.75.0-r0_amd64.deb --- > -rw- 15932940 4dcbe007aa8d46e658f95a3651938d67ca5cd55409c503861f365cb9c38f14e7./deploy-debs/core2-64/rust-dbg_1.75.0-r0_amd64.deb 11c11 < -rw- 2902068 564925753476842ea9052311afa27cbc099136113ea9f5394b5554eccb09e8f3./deploy-debs/core2-64/rust-rustdoc_1.75.0-r0_amd64.deb --- > -rw- 2901612 c0cd05fe344400c0a9d31fecd808c3152481f35dacc0556c006b88a239209358./deploy-debs/core2-64/rust-rustdoc_1.75.0-r0_amd64.deb I am not sure of what that diff indicates. Can I get some pointers on - 1. Why only sometimes the setscene setting on reproducibleA (which causes rust sources not to rebuilt) 2. Is there any way/tool to analyze the above depsig data or what is going-on with setscene. This helps to identify the issue is happening due to sstate? if not I can look into rust.
On Mon, 24 Jun 2024 at 06:00, Sundeep KOKKONDA via
lists.openembedded.org
<sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote:
> - The hash-id generates identically when the rust built from it's sources every time.
This is not the case. I just built rust (without sstate, from sources)
in two different build directories on the same machine, and indeed
rustdoc is different:
alex@Zen2:/srv/storage/alex/yocto$ diff -uNr
build-64/tmp/work/core2-64-poky-linux/rust/1.75.0/image/
build-64-2/tmp/work/core2-64-poky-linux/rust/1.75.0/image/
Binary files build-64/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc
and build-64-2/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc
differ
Alex
On Mon, Jun 24, 2024 at 01:50 PM, Alexander Kanavin wrote: > > This is not the case. I just built rust (without sstate, from sources) > in two different build directories on the same machine, and indeed Thanks for sharing this... Unfortunately I couldn't see this happening during my testing. If this is the case, the issue might be there in rust. for me diff is not there... > diff -uNr build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/ build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/ > diff build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc > diff -uNr build-b/tmp/deploy/deb/core2-64/ build-c/tmp/deploy/deb/core2-64/
On Mon, 24 Jun 2024 at 11:07, Sundeep KOKKONDA via lists.openembedded.org <sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote: > This is not the case. I just built rust (without sstate, from sources) > in two different build directories on the same machine, and indeed > > Thanks for sharing this... Unfortunately I couldn't see this happening during my testing. If this is the case, the issue might be there in rust. > > for me diff is not there... > > diff -uNr build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/ build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/ > > diff build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc > > diff -uNr build-b/tmp/deploy/deb/core2-64/ build-c/tmp/deploy/deb/core2-64/ I can share the two different build directories for your further inspection perhaps? Let me upload the tarballs somewhere public. Alex
On Mon, 24 Jun 2024 at 11:11, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > I can share the two different build directories for your further > inspection perhaps? Let me upload the tarballs somewhere public. They're here: https://drive.google.com/file/d/1kKhHoC3oMFPFjvuLvM4glQzKX8JeXNue/view?usp=drive_link https://drive.google.com/file/d/1RCoZipt6i9wWG3j-j2rilWCNW_NMNCk9/view?usp=drive_link Please let me know when you have downloaded them because I'm nearly out of space there, and need to delete them ASAP. Alex
On 24-Jun-24 17:52, Alexander Kanavin wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Mon, 24 Jun 2024 at 11:11, Alexander Kanavin via > lists.openembedded.org<alex.kanavin=gmail.com@lists.openembedded.org> > wrote: >> I can share the two different build directories for your further >> inspection perhaps? Let me upload the tarballs somewhere public. > They're here: > https://drive.google.com/file/d/1kKhHoC3oMFPFjvuLvM4glQzKX8JeXNue/view?usp=drive_link > https://drive.google.com/file/d/1RCoZipt6i9wWG3j-j2rilWCNW_NMNCk9/view?usp=drive_link > > Please let me know when you have downloaded them because I'm nearly > out of space there, and need to delete them ASAP. Hi Alex, Copied. Thnx. I tried again with latest poky, still the /rustdoc /generated identical :( > > Alex
On Tue, 25 Jun 2024 at 05:02, Sundeep KOKKONDA <Sundeep.Kokkonda@windriver.com> wrote: > I can share the two different build directories for your further > inspection perhaps? Let me upload the tarballs somewhere public. > > They're here: > https://drive.google.com/file/d/1kKhHoC3oMFPFjvuLvM4glQzKX8JeXNue/view?usp=drive_link > https://drive.google.com/file/d/1RCoZipt6i9wWG3j-j2rilWCNW_NMNCk9/view?usp=drive_link > > Please let me know when you have downloaded them because I'm nearly > out of space there, and need to delete them ASAP. > > Hi Alex, > > Copied. Thnx. The idea is that these tarballs contain a complete build directory of rust, with all the intermediate artefacts (e.g. individual object files). So you could try to look into it to see where things start to diverge between them. It has to be something specific to rustdoc, possibly iterating over a non-deterministic list, such as a file listing that hasn't been sorted, or a list of keys in a hash table that hasn't been sorted either. Alex
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py index 97a9c3da908..62b9f912af3 100644 --- a/meta/lib/oeqa/selftest/cases/reproducible.py +++ b/meta/lib/oeqa/selftest/cases/reproducible.py @@ -16,8 +16,6 @@ import os import datetime exclude_packages = [ - 'rust-rustdoc', - 'rust-dbg' ] def is_excluded(package): @@ -135,7 +133,7 @@ class ReproducibleTests(OESelftestTestCase): max_report_size = 250 * 1024 * 1024 # targets are the things we want to test the reproducibility of - targets = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline', 'core-image-weston', 'world'] + targets = ['rust'] # sstate targets are things to pull from sstate to potentially cut build/debugging time sstate_targets = []