| Message ID | 2e75c6169cac05273b3edde52ea820506ce9d3fe.camel@linuxfoundation.org |
|---|---|
| State | New |
| Headers | show |
| Series | Rust reproducibility issue summary | expand |
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information: --- Testing patch /home/patchtest/share/mboxes/Rust-reproducibility-issue-summary.patch FAIL: test Signed-off-by presence: Mbox is missing Signed-off-by. Add it manually or with "git commit --amend -s" (test_mbox.TestMbox.test_signed_off_by_presence) FAIL: test shortlog format: Commit shortlog (first line of commit message) should follow the format "<target>: <summary>" (test_mbox.TestMbox.test_shortlog_format) PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint) PASS: test author valid (test_mbox.TestMbox.test_author_valid) PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence) PASS: test max line length (test_metadata.TestMetadata.test_max_line_length) PASS: test mbox format (test_mbox.TestMbox.test_mbox_format) PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade) PASS: test pylint (test_python_pylint.PyLint.test_pylint) PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length) PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list) SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files) SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore) SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format) SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence) SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format) SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format) SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned) SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence) SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence) SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head) SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files) SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence) --- Please address the issues identified and submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
On Fri, 31 May 2024 at 15:28, Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > I've been asked what the reproducibility issue is and what data we > have. I ran a couple of test builds on the current autobuilder and the > test cluster. I added in the diff at the end of this email, just to > limit the test to what we're interested in. > > The test on the new test cluster passed: > > https://valkyrie.yoctoproject.org/#/builders/37/builds/36 > > However the build on the current autobuilder did not: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876 > > and the different binaries generated are here: > > http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/ > > We need to get to the bottom of why these are different and the test is > sometimes failing. Any help appreciated! > > It is unclear what we need to do to reproduce the issue on demand. One possible lead is in the original attempt to fix this; https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7 It is not yet clear if disabling PGO makes things reproducible, as the same commit also prevented the non reproducible binary (rustdoc) from being installed (with the demangler line). PGO as the culprit should be either confirmed, or ruled out first. If PGO indeed helps then the next step is figure out how to disable (or tweak) it in a way that doesn't cause issues with chromium: https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 Alex
On 01-Jun-24 15:57, Alexander Kanavin wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Fri, 31 May 2024 at 15:28, Richard Purdie via > lists.openembedded.org > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: >> I've been asked what the reproducibility issue is and what data we >> have. I ran a couple of test builds on the current autobuilder and the >> test cluster. I added in the diff at the end of this email, just to >> limit the test to what we're interested in. >> >> The test on the new test cluster passed: >> >> https://valkyrie.yoctoproject.org/#/builders/37/builds/36 >> >> However the build on the current autobuilder did not: >> >> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876 >> >> and the different binaries generated are here: >> >> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/ >> >> We need to get to the bottom of why these are different and the test is >> sometimes failing. Any help appreciated! >> >> It is unclear what we need to do to reproduce the issue on demand. > One possible lead is in the original attempt to fix this; > https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7 > > It is not yet clear if disabling PGO makes things reproducible, as the > same commit also prevented the non reproducible binary (rustdoc) from > being installed (with the demangler line). > > PGO as the culprit should be either confirmed, or ruled out first. > > If PGO indeed helps then the next step is figure out how to disable > (or tweak) it in a way that doesn't cause issues with chromium: > > https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 Hello Alex, Initially we did enabled PGO & rust-demangler options for rustdoc reproducibility. https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/5 I did not get any info (from doc or community) on how and why optimation is affecting rustdoc binary. https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/6 However, later we learned that the '/rustdoc/' is reproducible without PGO also, and so removed for chromium project. And after removing PGO also we've not seen this rustdoc repro issue on autobuilder. The issue is not reproduced on my end to confirm the PGO is root cause or any recent changes on rust recipe causing the issue? /Sundeep > > Alex
On 03-Jun-24 9:53, Sundeep KOKKONDA via lists.openembedded.org wrote: > > > On 01-Jun-24 15:57, Alexander Kanavin wrote: >> CAUTION: This email comes from a non Wind River email account! >> Do not click links or open attachments unless you recognize the sender and know the content is safe. >> >> On Fri, 31 May 2024 at 15:28, Richard Purdie via >> lists.openembedded.org >> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: >>> I've been asked what the reproducibility issue is and what data we >>> have. I ran a couple of test builds on the current autobuilder and the >>> test cluster. I added in the diff at the end of this email, just to >>> limit the test to what we're interested in. >>> >>> The test on the new test cluster passed: >>> >>> https://valkyrie.yoctoproject.org/#/builders/37/builds/36 >>> >>> However the build on the current autobuilder did not: >>> >>> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876 >>> >>> and the different binaries generated are here: >>> >>> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/ >>> >>> We need to get to the bottom of why these are different and the test is >>> sometimes failing. Any help appreciated! >>> >>> It is unclear what we need to do to reproduce the issue on demand. >> One possible lead is in the original attempt to fix this; >> https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7 >> >> It is not yet clear if disabling PGO makes things reproducible, as the >> same commit also prevented the non reproducible binary (rustdoc) from >> being installed (with the demangler line). >> >> PGO as the culprit should be either confirmed, or ruled out first. >> >> If PGO indeed helps then the next step is figure out how to disable >> (or tweak) it in a way that doesn't cause issues with chromium: >> >> https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 > Hello Alex, > > Initially we did enabled PGO & rust-demangler options for rustdoc > reproducibility. > https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/5 > > I did not get any info (from doc or community) on how and why > optimation is affecting rustdoc binary. > https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/6 > > However, later we learned that the '/rustdoc/' is reproducible without > PGO also, and so removed for chromium project. And after removing PGO > also we've not seen this rustdoc repro issue on autobuilder. > > The issue is not reproduced on my end to confirm the PGO is root cause > or any recent changes on rust recipe causing the issue? Further analysis on autobuilder binaries shows that the deviations are similar to that of earlier /rustdoc /issue. The deviations are in |.llvm.<hash>| suffixes. As discussed in issue <https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027>, may be we should check by disabling PGO (by default it is enabled and as in comment <in https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/3>the PGO profile generation is not deterministic...). rustdoc-175 diff > > > /Sundeep >> Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#200231):https://lists.openembedded.org/g/openembedded-core/message/200231 > Mute This Topic:https://lists.openembedded.org/mt/106407787/7677583 > Group Owner:openembedded-core+owner@lists.openembedded.org > Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [sundeep.kokkonda@eng.windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Yes please. This is what I've been trying to get across: try disabling the PGO once more, but do not change what gets installed, and ensure that rustdoc does get installed and packaged. Then, if that solves the reproducibility issue we need to look at why exactly chromium doesn't like it, so that it had to be reverted: https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 Alex On Mon, 3 Jun 2024 at 09:08, Sundeep KOKKONDA < Sundeep.Kokkonda@windriver.com> wrote: > > On 03-Jun-24 9:53, Sundeep KOKKONDA via lists.openembedded.org wrote: > > > On 01-Jun-24 15:57, Alexander Kanavin wrote: > > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Fri, 31 May 2024 at 15:28, Richard Purdie vialists.openembedded.org<richard.purdie=linuxfoundation.org@lists.openembedded.org> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > I've been asked what the reproducibility issue is and what data we > have. I ran a couple of test builds on the current autobuilder and the > test cluster. I added in the diff at the end of this email, just to > limit the test to what we're interested in. > > The test on the new test cluster passed: > https://valkyrie.yoctoproject.org/#/builders/37/builds/36 > > However the build on the current autobuilder did not: > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876 > > and the different binaries generated are here: > http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/ > > We need to get to the bottom of why these are different and the test is > sometimes failing. Any help appreciated! > > It is unclear what we need to do to reproduce the issue on demand. > > One possible lead is in the original attempt to fix this;https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7 > > It is not yet clear if disabling PGO makes things reproducible, as the > same commit also prevented the non reproducible binary (rustdoc) from > being installed (with the demangler line). > > PGO as the culprit should be either confirmed, or ruled out first. > > If PGO indeed helps then the next step is figure out how to disable > (or tweak) it in a way that doesn't cause issues with chromium: > https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84 > > Hello Alex, > > Initially we did enabled PGO & rust-demangler options for rustdoc > reproducibility. > > https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/5 > > I did not get any info (from doc or community) on how and why optimation > is affecting rustdoc binary. > > https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/6 > > However, later we learned that the '*rustdoc*' is reproducible without > PGO also, and so removed for chromium project. And after removing PGO also > we've not seen this rustdoc repro issue on autobuilder. > > The issue is not reproduced on my end to confirm the PGO is root cause or > any recent changes on rust recipe causing the issue? > > > Further analysis on autobuilder binaries shows that the deviations are > similar to that of earlier *rustdoc *issue. The deviations are in > .llvm.<hash> suffixes. > > As discussed in issue > <https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027>, > may be we should check by disabling PGO (by default it is enabled and as in comment > the PGO profile generation is not deterministic...). > [image: rustdoc-175 diff] > > > /Sundeep > > Alex > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#200231): https://lists.openembedded.org/g/openembedded-core/message/200231 > Mute This Topic: https://lists.openembedded.org/mt/106407787/7677583 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [sundeep.kokkonda@eng.windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- > > >
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py index 97a9c3da908..62b9f912af3 100644 --- a/meta/lib/oeqa/selftest/cases/reproducible.py +++ b/meta/lib/oeqa/selftest/cases/reproducible.py @@ -16,8 +16,6 @@ import os import datetime exclude_packages = [ - 'rust-rustdoc', - 'rust-dbg' ] def is_excluded(package): @@ -135,7 +133,7 @@ class ReproducibleTests(OESelftestTestCase): max_report_size = 250 * 1024 * 1024 # targets are the things we want to test the reproducibility of - targets = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline', 'core-image-weston', 'world'] + targets = ['rust'] # sstate targets are things to pull from sstate to potentially cut build/debugging time sstate_targets = []