| Message ID | 20260302064938.2052450-1-Deepesh.Varatharajan@windriver.com |
|---|---|
| Headers | show |
| Series | Oe-selftest for Clang, LLVM, LLD | expand |
Hi All, Please ignore this patch. The llvm_sysroot_preprocess_testsuite function should be added to llvm:class-target, not llvm:class-nativesdk. I will send the corrected version in V3. Regards, Deepesh On 02-03-2026 12:19, Deepesh.Varatharajan@windriver.com wrote: > From: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com> > > This patch series introduces a test framework to run selftests for > target builds of Clang, LLVM, and LLD. > > v2 changes: > - Enable required CMake flags only when running the testsuite. > - Ensure default build time and package sizes remain unaffected. > > Summary of changes: > -Patch 1: Disable clang-tools-extra tests, as Clang's CMake cannot > detect the llvm-bcanalyzer target. > -Patch 2: Enabled the necessary CMake flags for clang, llvm & lld tests > when CLANG_ENABLE_TESTSUITE is set to "1". > -Patch 3: Selftest script for Clang, LLVM, and LLD including exclude list > -Patch 4: Fix rust build failure by replacing target llvm-config with native > > Testing: > 1. The test suite has been successfully validated on x86-64, arm64, riscv64 > and arm32 for all Clang, LLD & LLVM components. > 2. Clang and LLD tests passed successfully on x86. LLVM tests are skipped > on x86 since it's broken. A follow-up bug will be filed and addressed. > 3. Clang, LLVM & LLD all together ~80K tests are there and we currently > ~250 tests to exclude list which are failing (0.32%). > > Impact on build time: > These builds were executed in a controlled environment without poky-sstate > enabled. > > Default(without changes): > +-----------+------------+------------+------------+----------------+ > | Target | Run 1 | Run 2 | Run 3 | Average | > | --------- | ---------- | ---------- | ---------- | -------------- | > | llvm | 22m18.441s | 22m31.905s | 22m23.718s | 22m24.688s | > | clang | 30m09.774s | 30m22.680s | 30m15.312s | 30m15.922s | > | lld | 1m26.994s | 1m30.415s | 1m29.006s | 1m28.805s | > +-----------+------------+------------+------------+----------------+ > > With the testsuite changes applied: > +-----------+------------+------------+------------+----------------+ > | Target | Run 1 | Run 2 | Run 3 | Average | > | --------- | ---------- | ---------- | ---------- | -------------- | > | llvm | 23m32.104s | 23m45.887s | 23m38.157s | 23m38.716s | > | clang | 30m31.402s | 30m46.198s | 30m39.445s | 30m39.015s | > | lld | 1m29.845s | 1m34.212s | 1m31.925s | 1m31.994s | > +-----------+------------+------------+------------+----------------+ > > Comparison: > The additional overhead is minimal and primarily due to enabling > test-related build configurations. > +-----------+-------------+------------------+------------------+ > | Target | Default Avg | With Changes Avg | **Diff** | > | --------- | ----------- | ---------------- | ---------------- | > | llvm | 22m24.688s | 23m38.716s | +1m14.028s | > | clang | 30m15.922s | 30m39.015s | +0m23.093s | > | lld | 1m28.805s | 1m31.994s | +0m3.189s | > +-----------+-------------+------------------+------------------+ > > Impact on package size and Fix: > The testsuite changes has a negative impact on certain package sizes, > as below: > +---------------------+----------+----------+-----------+------------+ > | Package | Old (MB) | New (MB) | Diff (MB) | Change (%) | > +=====================+==========+==========+===========+============+ > | clang/clang-dbg | 44.99 | 59.79 | +14.81 | +32.92% | > | clang/clang-dev | 30.22 | 30.22 | -0.00 | -0.00% | > | clang/clang-tools | 5.05 | 29.58 | +24.53 | +485.87% | > | llvm/llvm-bin | 16.40 | 71.77 | +55.36 | +337.50% | > | llvm/llvm-dbg | 386.79 | 646.12 | +259.33 | +67.05% | > | llvm/llvm-dev | 32.67 | 33.67 | +1.00 | +3.05% | > | llvm/llvm-src | 387.54 | 389.44 | +1.90 | +0.49% | > | llvm/llvm-staticdev | 609.68 | 616.80 | +7.12 | +1.17% | > +---------------------+----------+----------+-----------+------------+ > > FIX: > All testsuite-related changes are now gated by the CLANG_ENABLE_TESTSUITE > variable (default "0"), ensuring: > - Test-specific CMake flags are applied only during test execution. > - Standard builds remain fully unaffected in both build time and package size. > > Deepesh Varatharajan (4): > clang-tools-extra: disable tests > clang: Use CLANG_ENABLE_TESTSUITE to enable LLVM, Clang, and LLD tests > oeqa/selftest/clang: Add oe-seltests for Clang/LLVM/LLD > rust: Fix rust build failure > > meta/lib/oeqa/selftest/cases/clang.py | 299 ++++++++++++++++++ > .../0042-guard-clang-tools-extra-test.patch | 74 +++++ > meta/recipes-devtools/clang/clang_git.bb | 8 + > meta/recipes-devtools/clang/common.inc | 1 + > meta/recipes-devtools/clang/lld_git.bb | 5 + > meta/recipes-devtools/clang/llvm_git.bb | 18 +- > meta/recipes-devtools/rust/rust_1.93.0.bb | 5 +- > 7 files changed, 407 insertions(+), 3 deletions(-) > create mode 100644 meta/lib/oeqa/selftest/cases/clang.py > create mode 100644 meta/recipes-devtools/clang/clang/0042-guard-clang-tools-extra-test.patch >
On Mon Mar 2, 2026 at 7:49 AM CET, Deepesh via lists.openembedded.org Varatharajan wrote: > From: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com> > > This patch series introduces a test framework to run selftests for > target builds of Clang, LLVM, and LLD. > > v2 changes: > - Enable required CMake flags only when running the testsuite. > - Ensure default build time and package sizes remain unaffected. > > Summary of changes: > -Patch 1: Disable clang-tools-extra tests, as Clang's CMake cannot > detect the llvm-bcanalyzer target. > -Patch 2: Enabled the necessary CMake flags for clang, llvm & lld tests > when CLANG_ENABLE_TESTSUITE is set to "1". > -Patch 3: Selftest script for Clang, LLVM, and LLD including exclude list > -Patch 4: Fix rust build failure by replacing target llvm-config with native > Hi Deepesh, Thanks for the new version, but it looks like we now have some failing tests: 2026-03-02 09:52:32,943 - oe-selftest - INFO - clang.ClangSelfTestSystemEmulated.test_clang (subunit.RemotedTestCase) 2026-03-02 09:52:32,943 - oe-selftest - INFO - ... FAIL ... AssertionError: llvm-lit failed for Clang: llvm-lit: /srv/pokybuild/yocto-worker/qemuarm-tc/build/build-st-3082983/tmp/work-shared/llvm-project-source-21.1.8-r1/sources/llvm-project-21.1.8.src/llvm/utils/lit/lit/llvm/subst.py:133: fatal: Did not find FileCheck in /srv/pokybuild/yocto-worker/qemuarm-tc/build/build-st-3082983/tmp/work/cortexa15t2hf-neon-poky-linux-gnueabi/clang/21.1.8/recipe-sysroot/usr/bin ... 2026-03-02 09:54:51,485 - oe-selftest - INFO - clang.LLVMSelfTestSystemEmulated.test_llvm (subunit.RemotedTestCase) 2026-03-02 09:54:51,485 - oe-selftest - INFO - ... FAIL ... 2026-03-02 09:55:49,997 - oe-selftest - INFO - clang.LLDSelfTestSystemEmulated.test_lld (subunit.RemotedTestCase) 2026-03-02 09:55:49,997 - oe-selftest - INFO - ... FAIL https://autobuilder.yoctoproject.org/valkyrie/#/builders/42/builds/3224 https://autobuilder.yoctoproject.org/valkyrie/#/builders/5/builds/3222 https://autobuilder.yoctoproject.org/valkyrie/#/builders/58/builds/1118 Can you have a look at these? Thanks, Mathieu
On 02-03-2026 16:18, Mathieu Dubois-Briand wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Mon Mar 2, 2026 at 7:49 AM CET, Deepesh via lists.openembedded.org Varatharajan wrote: >> From: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com> >> >> This patch series introduces a test framework to run selftests for >> target builds of Clang, LLVM, and LLD. >> >> v2 changes: >> - Enable required CMake flags only when running the testsuite. >> - Ensure default build time and package sizes remain unaffected. >> >> Summary of changes: >> -Patch 1: Disable clang-tools-extra tests, as Clang's CMake cannot >> detect the llvm-bcanalyzer target. >> -Patch 2: Enabled the necessary CMake flags for clang, llvm & lld tests >> when CLANG_ENABLE_TESTSUITE is set to "1". >> -Patch 3: Selftest script for Clang, LLVM, and LLD including exclude list >> -Patch 4: Fix rust build failure by replacing target llvm-config with native >> > Hi Deepesh, > > Thanks for the new version, but it looks like we now have some failing > tests: > > 2026-03-02 09:52:32,943 - oe-selftest - INFO - clang.ClangSelfTestSystemEmulated.test_clang (subunit.RemotedTestCase) > 2026-03-02 09:52:32,943 - oe-selftest - INFO - ... FAIL > ... > AssertionError: llvm-lit failed for Clang: llvm-lit: /srv/pokybuild/yocto-worker/qemuarm-tc/build/build-st-3082983/tmp/work-shared/llvm-project-source-21.1.8-r1/sources/llvm-project-21.1.8.src/llvm/utils/lit/lit/llvm/subst.py:133: fatal: Did not find FileCheck in /srv/pokybuild/yocto-worker/qemuarm-tc/build/build-st-3082983/tmp/work/cortexa15t2hf-neon-poky-linux-gnueabi/clang/21.1.8/recipe-sysroot/usr/bin > ... > 2026-03-02 09:54:51,485 - oe-selftest - INFO - clang.LLVMSelfTestSystemEmulated.test_llvm (subunit.RemotedTestCase) > 2026-03-02 09:54:51,485 - oe-selftest - INFO - ... FAIL > ... > 2026-03-02 09:55:49,997 - oe-selftest - INFO - clang.LLDSelfTestSystemEmulated.test_lld (subunit.RemotedTestCase) > 2026-03-02 09:55:49,997 - oe-selftest - INFO - ... FAIL > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/42/builds/3224 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/5/builds/3222 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/58/builds/1118 > > Can you have a look at these? Hi Mathieu, Thanks for pointing this out. I’ve already sent a V3 of this patch that addresses this issue. Regards, Deepesh > > Thanks, > Mathieu > > -- > Mathieu Dubois-Briand, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com >
From: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com> This patch series introduces a test framework to run selftests for target builds of Clang, LLVM, and LLD. v2 changes: - Enable required CMake flags only when running the testsuite. - Ensure default build time and package sizes remain unaffected. Summary of changes: -Patch 1: Disable clang-tools-extra tests, as Clang's CMake cannot detect the llvm-bcanalyzer target. -Patch 2: Enabled the necessary CMake flags for clang, llvm & lld tests when CLANG_ENABLE_TESTSUITE is set to "1". -Patch 3: Selftest script for Clang, LLVM, and LLD including exclude list -Patch 4: Fix rust build failure by replacing target llvm-config with native Testing: 1. The test suite has been successfully validated on x86-64, arm64, riscv64 and arm32 for all Clang, LLD & LLVM components. 2. Clang and LLD tests passed successfully on x86. LLVM tests are skipped on x86 since it's broken. A follow-up bug will be filed and addressed. 3. Clang, LLVM & LLD all together ~80K tests are there and we currently ~250 tests to exclude list which are failing (0.32%). Impact on build time: These builds were executed in a controlled environment without poky-sstate enabled. Default(without changes): +-----------+------------+------------+------------+----------------+ | Target | Run 1 | Run 2 | Run 3 | Average | | --------- | ---------- | ---------- | ---------- | -------------- | | llvm | 22m18.441s | 22m31.905s | 22m23.718s | 22m24.688s | | clang | 30m09.774s | 30m22.680s | 30m15.312s | 30m15.922s | | lld | 1m26.994s | 1m30.415s | 1m29.006s | 1m28.805s | +-----------+------------+------------+------------+----------------+ With the testsuite changes applied: +-----------+------------+------------+------------+----------------+ | Target | Run 1 | Run 2 | Run 3 | Average | | --------- | ---------- | ---------- | ---------- | -------------- | | llvm | 23m32.104s | 23m45.887s | 23m38.157s | 23m38.716s | | clang | 30m31.402s | 30m46.198s | 30m39.445s | 30m39.015s | | lld | 1m29.845s | 1m34.212s | 1m31.925s | 1m31.994s | +-----------+------------+------------+------------+----------------+ Comparison: The additional overhead is minimal and primarily due to enabling test-related build configurations. +-----------+-------------+------------------+------------------+ | Target | Default Avg | With Changes Avg | **Diff** | | --------- | ----------- | ---------------- | ---------------- | | llvm | 22m24.688s | 23m38.716s | +1m14.028s | | clang | 30m15.922s | 30m39.015s | +0m23.093s | | lld | 1m28.805s | 1m31.994s | +0m3.189s | +-----------+-------------+------------------+------------------+ Impact on package size and Fix: The testsuite changes has a negative impact on certain package sizes, as below: +---------------------+----------+----------+-----------+------------+ | Package | Old (MB) | New (MB) | Diff (MB) | Change (%) | +=====================+==========+==========+===========+============+ | clang/clang-dbg | 44.99 | 59.79 | +14.81 | +32.92% | | clang/clang-dev | 30.22 | 30.22 | -0.00 | -0.00% | | clang/clang-tools | 5.05 | 29.58 | +24.53 | +485.87% | | llvm/llvm-bin | 16.40 | 71.77 | +55.36 | +337.50% | | llvm/llvm-dbg | 386.79 | 646.12 | +259.33 | +67.05% | | llvm/llvm-dev | 32.67 | 33.67 | +1.00 | +3.05% | | llvm/llvm-src | 387.54 | 389.44 | +1.90 | +0.49% | | llvm/llvm-staticdev | 609.68 | 616.80 | +7.12 | +1.17% | +---------------------+----------+----------+-----------+------------+ FIX: All testsuite-related changes are now gated by the CLANG_ENABLE_TESTSUITE variable (default "0"), ensuring: - Test-specific CMake flags are applied only during test execution. - Standard builds remain fully unaffected in both build time and package size. Deepesh Varatharajan (4): clang-tools-extra: disable tests clang: Use CLANG_ENABLE_TESTSUITE to enable LLVM, Clang, and LLD tests oeqa/selftest/clang: Add oe-seltests for Clang/LLVM/LLD rust: Fix rust build failure meta/lib/oeqa/selftest/cases/clang.py | 299 ++++++++++++++++++ .../0042-guard-clang-tools-extra-test.patch | 74 +++++ meta/recipes-devtools/clang/clang_git.bb | 8 + meta/recipes-devtools/clang/common.inc | 1 + meta/recipes-devtools/clang/lld_git.bb | 5 + meta/recipes-devtools/clang/llvm_git.bb | 18 +- meta/recipes-devtools/rust/rust_1.93.0.bb | 5 +- 7 files changed, 407 insertions(+), 3 deletions(-) create mode 100644 meta/lib/oeqa/selftest/cases/clang.py create mode 100644 meta/recipes-devtools/clang/clang/0042-guard-clang-tools-extra-test.patch