| Message ID | 20241011122044.12222-1-mikko.rapeli@linaro.org |
|---|---|
| Headers | show |
| Series | systemd uki support | expand |
On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote: > These changes enable building systemd uki images which combine > kernel, kernel command line, initrd and possibly signatures to > a single UEFI binary. This binary can be booted with UEFI firmware > and systemd-boot. No grub is needed and UEFI firmware and/or > systemd-boot provide possibilities for boot menus. > The uki binary can also be signed for UEFI secure boot > so the secure boot extends from firmware to kernel and initrd. > Binding secure boot to full userspace is then easier since for example > kernel command line and initrd contain the support needed to mount > encrypted dm-verity etc partitions, and/or create partitions on demand > with systemd-repart using device specific TPM devices for encryption. > > Tested on qemuarm64-secureboot machine from meta-arm with changes to > support secure boot. Slightly different configuration tested on > multiple arm64 System Ready boards with UEFI firmware, real and firmware > based TPM devices. Tested with ovmf firmware on x86_64 with selftests but > without secure boot which seems to be harder to setup in ovmf. > > Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and > wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests > on a build machine with zfs filesystem. Will investigate this further. > The issue seems to be in mkfs.ext4 producing broken filesystem, and partially > in the tests which don't run the correct rootfs file (.ext4 vs .wic). > Will debug this further and it is IMO unrelated to these changes since > they reproduce on pure master branch without this series. > > v8: fixed comments from Ross Burton: debug print from warning to debug, > dropped duplicate DISTRO_FEATURE setting for systemd in tests, > removed aarch64 comment from tests which are currently x86 only. > Fixed the new aarch64 wic selftest to run on both genericarm64 > and qemuarm64 by adding bios, virtio disk driver etc settings > for runqemu (already set in genericarm64 but missing from qemuarm64). > > v7: add missing "ovmf" to runqemu argument to > test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang > > v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on > genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added > selftest to cover this wks usage on x86 and aarch64 > > v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release > dependency for all archs" since systemd-boot does not support all > architectures > > v4: handle missing runqemu variable from build config, add > python3-pefile to fast ptest list > > v3: rebased, fixed and added more sefltests, removed wic plugin side uki > support > > v2: https://lists.openembedded.org/g/openembedded-core/message/204090 > > Michelle Lin (1): > uki.bbclass: add class for building Unified Kernel Images (UKI) > > Mikko Rapeli (7): > wic bootimg-efi.py: keep timestamps and add debug prints > wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass > oeqa selftest uki.py: add tests for uki.bbclass > oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu > oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi > test > oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu > oeqa selftest wic.py: support UKIs via uki.bbclass > I'm still seeing failures in CI: https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio which is despite setting: https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6 QEMU_USE_KVM = "False" into the test to ensure it doesn't have the issue the barebox testing was seeing. I've sent a patch to try and clean up the lock error. There is also this: https://valkyrie.yoctoproject.org//#/builders/76/builds/235 https://valkyrie.yoctoproject.org//#/builders/48/builds/181 https://valkyrie.yoctoproject.org//#/builders/54/builds/230 which is due to the binaries being run "in tree" within the edk2 build as well as from the sysroot. This generates two sets of pyc files which then conflict (or not) depending on which host the build ran on and which pyc files are in sstate. We're going to have to get this fixed before it can merge, probably by deleting the pyc files at install unless we can find anything more elegant. Cheers, Richard
Hi, On Sun, Oct 13, 2024 at 08:43:15AM +0100, Richard Purdie wrote: > On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote: > > These changes enable building systemd uki images which combine > > kernel, kernel command line, initrd and possibly signatures to > > a single UEFI binary. This binary can be booted with UEFI firmware > > and systemd-boot. No grub is needed and UEFI firmware and/or > > systemd-boot provide possibilities for boot menus. > > The uki binary can also be signed for UEFI secure boot > > so the secure boot extends from firmware to kernel and initrd. > > Binding secure boot to full userspace is then easier since for example > > kernel command line and initrd contain the support needed to mount > > encrypted dm-verity etc partitions, and/or create partitions on demand > > with systemd-repart using device specific TPM devices for encryption. > > > > Tested on qemuarm64-secureboot machine from meta-arm with changes to > > support secure boot. Slightly different configuration tested on > > multiple arm64 System Ready boards with UEFI firmware, real and firmware > > based TPM devices. Tested with ovmf firmware on x86_64 with selftests but > > without secure boot which seems to be harder to setup in ovmf. > > > > Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and > > wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests > > on a build machine with zfs filesystem. Will investigate this further. > > The issue seems to be in mkfs.ext4 producing broken filesystem, and partially > > in the tests which don't run the correct rootfs file (.ext4 vs .wic). > > Will debug this further and it is IMO unrelated to these changes since > > they reproduce on pure master branch without this series. > > > > v8: fixed comments from Ross Burton: debug print from warning to debug, > > ��� dropped duplicate DISTRO_FEATURE setting for systemd in tests, > > ��� removed aarch64 comment from tests which are currently x86 only. > > ��� Fixed the new aarch64 wic selftest to run on both genericarm64 > > ��� and qemuarm64 by adding bios, virtio disk driver etc settings > > ��� for runqemu (already set in genericarm64 but missing from qemuarm64). > > > > v7: add missing "ovmf" to runqemu argument to > > ��� test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang > > > > v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on > > ��� genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added > > ��� selftest to cover this wks usage on x86 and aarch64 > > > > v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release > > ��� dependency for all archs" since systemd-boot does not support all > > ��� architectures > > > > v4: handle missing runqemu variable from build config, add > > python3-pefile to fast ptest list > > > > v3: rebased, fixed and added more sefltests, removed wic plugin side uki > > support > > > > v2: https://lists.openembedded.org/g/openembedded-core/message/204090 > > > > Michelle Lin (1): > > � uki.bbclass: add class for building Unified Kernel Images (UKI) > > > > Mikko Rapeli (7): > > � wic bootimg-efi.py: keep timestamps and add debug prints > > � wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass > > � oeqa selftest uki.py: add tests for uki.bbclass > > � oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu > > � oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi > > ��� test > > � oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu > > � oeqa selftest wic.py: support UKIs via uki.bbclass > > > > I'm still seeing failures in CI: > > https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio > > which is despite setting: > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6 > > QEMU_USE_KVM = "False" > > into the test to ensure it doesn't have the issue the barebox testing > was seeing. > > I've sent a patch to try and clean up the lock error. > > There is also this: > > https://valkyrie.yoctoproject.org//#/builders/76/builds/235 > https://valkyrie.yoctoproject.org//#/builders/48/builds/181 > https://valkyrie.yoctoproject.org//#/builders/54/builds/230 > > which is due to the binaries being run "in tree" within the edk2 build > as well as from the sysroot. This generates two sets of pyc files which > then conflict (or not) depending on which host the build ran on and > which pyc files are in sstate. > > We're going to have to get this fixed before it can merge, probably by > deleting the pyc files at install unless we can find anything more > elegant. Sent an ovmf-native patch separately for this. Cheers, -Mikko
Hi, On Mon, Oct 14, 2024 at 01:30:56PM +0300, Mikko Rapeli via lists.openembedded.org wrote: > Hi, > > On Sun, Oct 13, 2024 at 08:43:15AM +0100, Richard Purdie wrote: > > On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote: > > > These changes enable building systemd uki images which combine > > > kernel, kernel command line, initrd and possibly signatures to > > > a single UEFI binary. This binary can be booted with UEFI firmware > > > and systemd-boot. No grub is needed and UEFI firmware and/or > > > systemd-boot provide possibilities for boot menus. > > > The uki binary can also be signed for UEFI secure boot > > > so the secure boot extends from firmware to kernel and initrd. > > > Binding secure boot to full userspace is then easier since for example > > > kernel command line and initrd contain the support needed to mount > > > encrypted dm-verity etc partitions, and/or create partitions on demand > > > with systemd-repart using device specific TPM devices for encryption. > > > > > > Tested on qemuarm64-secureboot machine from meta-arm with changes to > > > support secure boot. Slightly different configuration tested on > > > multiple arm64 System Ready boards with UEFI firmware, real and firmware > > > based TPM devices. Tested with ovmf firmware on x86_64 with selftests but > > > without secure boot which seems to be harder to setup in ovmf. > > > > > > Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and > > > wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests > > > on a build machine with zfs filesystem. Will investigate this further. > > > The issue seems to be in mkfs.ext4 producing broken filesystem, and partially > > > in the tests which don't run the correct rootfs file (.ext4 vs .wic). > > > Will debug this further and it is IMO unrelated to these changes since > > > they reproduce on pure master branch without this series. > > > > > > v8: fixed comments from Ross Burton: debug print from warning to debug, > > > ��� dropped duplicate DISTRO_FEATURE setting for systemd in tests, > > > ��� removed aarch64 comment from tests which are currently x86 only. > > > ��� Fixed the new aarch64 wic selftest to run on both genericarm64 > > > ��� and qemuarm64 by adding bios, virtio disk driver etc settings > > > ��� for runqemu (already set in genericarm64 but missing from qemuarm64). > > > > > > v7: add missing "ovmf" to runqemu argument to > > > ��� test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang > > > > > > v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on > > > ��� genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added > > > ��� selftest to cover this wks usage on x86 and aarch64 > > > > > > v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release > > > ��� dependency for all archs" since systemd-boot does not support all > > > ��� architectures > > > > > > v4: handle missing runqemu variable from build config, add > > > python3-pefile to fast ptest list > > > > > > v3: rebased, fixed and added more sefltests, removed wic plugin side uki > > > support > > > > > > v2: https://lists.openembedded.org/g/openembedded-core/message/204090 > > > > > > Michelle Lin (1): > > > � uki.bbclass: add class for building Unified Kernel Images (UKI) > > > > > > Mikko Rapeli (7): > > > � wic bootimg-efi.py: keep timestamps and add debug prints > > > � wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass > > > � oeqa selftest uki.py: add tests for uki.bbclass > > > � oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu > > > � oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi > > > ��� test > > > � oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu > > > � oeqa selftest wic.py: support UKIs via uki.bbclass > > > > > > > I'm still seeing failures in CI: > > > > https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio > > > > which is despite setting: > > > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6 > > > > QEMU_USE_KVM = "False" > > > > into the test to ensure it doesn't have the issue the barebox testing > > was seeing. > > > > I've sent a patch to try and clean up the lock error. > > > > There is also this: > > > > https://valkyrie.yoctoproject.org//#/builders/76/builds/235 > > https://valkyrie.yoctoproject.org//#/builders/48/builds/181 > > https://valkyrie.yoctoproject.org//#/builders/54/builds/230 > > > > which is due to the binaries being run "in tree" within the edk2 build > > as well as from the sysroot. This generates two sets of pyc files which > > then conflict (or not) depending on which host the build ran on and > > which pyc files are in sstate. > > > > We're going to have to get this fixed before it can merge, probably by > > deleting the pyc files at install unless we can find anything more > > elegant. > > Sent an ovmf-native patch separately for this. With ovmf-native change applied to master, can this series be tried again? Or are some changes needed? Cheers, -Mikko
On Tue, 2024-10-15 at 09:44 +0300, Mikko Rapeli wrote: > Hi, > > On Mon, Oct 14, 2024 at 01:30:56PM +0300, Mikko Rapeli via lists.openembedded.org wrote: > > Hi, > > > > On Sun, Oct 13, 2024 at 08:43:15AM +0100, Richard Purdie wrote: > > > On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote: > > > > These changes enable building systemd uki images which combine > > > > kernel, kernel command line, initrd and possibly signatures to > > > > a single UEFI binary. This binary can be booted with UEFI firmware > > > > and systemd-boot. No grub is needed and UEFI firmware and/or > > > > systemd-boot provide possibilities for boot menus. > > > > The uki binary can also be signed for UEFI secure boot > > > > so the secure boot extends from firmware to kernel and initrd. > > > > Binding secure boot to full userspace is then easier since for example > > > > kernel command line and initrd contain the support needed to mount > > > > encrypted dm-verity etc partitions, and/or create partitions on demand > > > > with systemd-repart using device specific TPM devices for encryption. > > > > > > > > Tested on qemuarm64-secureboot machine from meta-arm with changes to > > > > support secure boot. Slightly different configuration tested on > > > > multiple arm64 System Ready boards with UEFI firmware, real and firmware > > > > based TPM devices. Tested with ovmf firmware on x86_64 with selftests but > > > > without secure boot which seems to be harder to setup in ovmf. > > > > > > > > Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and > > > > wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests > > > > on a build machine with zfs filesystem. Will investigate this further. > > > > The issue seems to be in mkfs.ext4 producing broken filesystem, and partially > > > > in the tests which don't run the correct rootfs file (.ext4 vs .wic). > > > > Will debug this further and it is IMO unrelated to these changes since > > > > they reproduce on pure master branch without this series. > > > > > > > > v8: fixed comments from Ross Burton: debug print from warning to debug, > > > > dropped duplicate DISTRO_FEATURE setting for systemd in tests, > > > > removed aarch64 comment from tests which are currently x86 only. > > > > Fixed the new aarch64 wic selftest to run on both genericarm64 > > > > and qemuarm64 by adding bios, virtio disk driver etc settings > > > > for runqemu (already set in genericarm64 but missing from qemuarm64). > > > > > > > > v7: add missing "ovmf" to runqemu argument to > > > > test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang > > > > > > > > v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on > > > > genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added > > > > selftest to cover this wks usage on x86 and aarch64 > > > > > > > > v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release > > > > dependency for all archs" since systemd-boot does not support all > > > > architectures > > > > > > > > v4: handle missing runqemu variable from build config, add > > > > python3-pefile to fast ptest list > > > > > > > > v3: rebased, fixed and added more sefltests, removed wic plugin side uki > > > > support > > > > > > > > v2: https://lists.openembedded.org/g/openembedded-core/message/204090 > > > > > > > > Michelle Lin (1): > > > > uki.bbclass: add class for building Unified Kernel Images (UKI) > > > > > > > > Mikko Rapeli (7): > > > > wic bootimg-efi.py: keep timestamps and add debug prints > > > > wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass > > > > oeqa selftest uki.py: add tests for uki.bbclass > > > > oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu > > > > oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi > > > > test > > > > oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu > > > > oeqa selftest wic.py: support UKIs via uki.bbclass > > > > > > > > > > I'm still seeing failures in CI: > > > > > > https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio > > > > > > which is despite setting: > > > > > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6 > > > > > > QEMU_USE_KVM = "False" > > > > > > into the test to ensure it doesn't have the issue the barebox testing > > > was seeing. > > > > > > I've sent a patch to try and clean up the lock error. > > > > > > There is also this: > > > > > > https://valkyrie.yoctoproject.org//#/builders/76/builds/235 > > > https://valkyrie.yoctoproject.org//#/builders/48/builds/181 > > > https://valkyrie.yoctoproject.org//#/builders/54/builds/230 > > > > > > which is due to the binaries being run "in tree" within the edk2 build > > > as well as from the sysroot. This generates two sets of pyc files which > > > then conflict (or not) depending on which host the build ran on and > > > which pyc files are in sstate. > > > > > > We're going to have to get this fixed before it can merge, probably by > > > deleting the pyc files at install unless we can find anything more > > > elegant. > > > > Sent an ovmf-native patch separately for this. > > With ovmf-native change applied to master, can this series be tried again? > > Or are some changes needed? This handles the pyc issue which is useful as it caused widespread build failures. The question of the first arm test failure remains :/. I will run it through testing again though. I'm juggling far too many failures at the moment, I can't tell what is "good" and what isn't very easily. Cheers, Richard
On Tue, 2024-10-15 at 09:44 +0300, Mikko Rapeli wrote: > Hi, > > On Mon, Oct 14, 2024 at 01:30:56PM +0300, Mikko Rapeli via lists.openembedded.org wrote: > > Hi, > > > > On Sun, Oct 13, 2024 at 08:43:15AM +0100, Richard Purdie wrote: > > > On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote: > > > > These changes enable building systemd uki images which combine > > > > kernel, kernel command line, initrd and possibly signatures to > > > > a single UEFI binary. This binary can be booted with UEFI firmware > > > > and systemd-boot. No grub is needed and UEFI firmware and/or > > > > systemd-boot provide possibilities for boot menus. > > > > The uki binary can also be signed for UEFI secure boot > > > > so the secure boot extends from firmware to kernel and initrd. > > > > Binding secure boot to full userspace is then easier since for example > > > > kernel command line and initrd contain the support needed to mount > > > > encrypted dm-verity etc partitions, and/or create partitions on demand > > > > with systemd-repart using device specific TPM devices for encryption. > > > > > > > > Tested on qemuarm64-secureboot machine from meta-arm with changes to > > > > support secure boot. Slightly different configuration tested on > > > > multiple arm64 System Ready boards with UEFI firmware, real and firmware > > > > based TPM devices. Tested with ovmf firmware on x86_64 with selftests but > > > > without secure boot which seems to be harder to setup in ovmf. > > > > > > > > Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and > > > > wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests > > > > on a build machine with zfs filesystem. Will investigate this further. > > > > The issue seems to be in mkfs.ext4 producing broken filesystem, and partially > > > > in the tests which don't run the correct rootfs file (.ext4 vs .wic). > > > > Will debug this further and it is IMO unrelated to these changes since > > > > they reproduce on pure master branch without this series. > > > > > > > > v8: fixed comments from Ross Burton: debug print from warning to debug, > > > > dropped duplicate DISTRO_FEATURE setting for systemd in tests, > > > > removed aarch64 comment from tests which are currently x86 only. > > > > Fixed the new aarch64 wic selftest to run on both genericarm64 > > > > and qemuarm64 by adding bios, virtio disk driver etc settings > > > > for runqemu (already set in genericarm64 but missing from qemuarm64). > > > > > > > > v7: add missing "ovmf" to runqemu argument to > > > > test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang > > > > > > > > v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on > > > > genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added > > > > selftest to cover this wks usage on x86 and aarch64 > > > > > > > > v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release > > > > dependency for all archs" since systemd-boot does not support all > > > > architectures > > > > > > > > v4: handle missing runqemu variable from build config, add > > > > python3-pefile to fast ptest list > > > > > > > > v3: rebased, fixed and added more sefltests, removed wic plugin side uki > > > > support > > > > > > > > v2: https://lists.openembedded.org/g/openembedded-core/message/204090 > > > > > > > > Michelle Lin (1): > > > > uki.bbclass: add class for building Unified Kernel Images (UKI) > > > > > > > > Mikko Rapeli (7): > > > > wic bootimg-efi.py: keep timestamps and add debug prints > > > > wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass > > > > oeqa selftest uki.py: add tests for uki.bbclass > > > > oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu > > > > oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi > > > > test > > > > oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu > > > > oeqa selftest wic.py: support UKIs via uki.bbclass > > > > > > > > > > I'm still seeing failures in CI: > > > > > > https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio > > > > > > which is despite setting: > > > > > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6 > > > > > > QEMU_USE_KVM = "False" > > > > > > into the test to ensure it doesn't have the issue the barebox testing > > > was seeing. > > > > > > I've sent a patch to try and clean up the lock error. > > > > > > There is also this: > > > > > > https://valkyrie.yoctoproject.org//#/builders/76/builds/235 > > > https://valkyrie.yoctoproject.org//#/builders/48/builds/181 > > > https://valkyrie.yoctoproject.org//#/builders/54/builds/230 > > > > > > which is due to the binaries being run "in tree" within the edk2 build > > > as well as from the sysroot. This generates two sets of pyc files which > > > then conflict (or not) depending on which host the build ran on and > > > which pyc files are in sstate. > > > > > > We're going to have to get this fixed before it can merge, probably by > > > deleting the pyc files at install unless we can find anything more > > > elegant. > > > > Sent an ovmf-native patch separately for this. > > With ovmf-native change applied to master, can this series be tried again? > > Or are some changes needed? The arm test still fails: https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio The traceback is at least cleaner now I fixed the lock error. I did test with KVM disabled. Cheers, Richard
Hi, On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote: > The arm test still fails: > > https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio > > The traceback is at least cleaner now I fixed the lock error. I did > test with KVM disabled. Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64 build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu 20.04? I will remove the tests as it is clearly too hard to get these running. It is a bit hard to figure out in which environments selftests are running. Cheers, -Mikko
Don’t remove the tests please. Without them things tend to regress. I understand it’s frustrating, but we do have arm people here that had pushed hard for testing in arm hosts, (without really thinking through how contributors are supposed to deal with such failures) and I think it’s now on them to help you out. Alex On Tue 15. Oct 2024 at 13.23, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote: > Hi, > > On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote: > > The arm test still fails: > > > > > https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio > > > > The traceback is at least cleaner now I fixed the lock error. I did > > test with KVM disabled. > > Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64 > build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu > 20.04? > > I will remove the tests as it is clearly too hard to get these running. > > It is a bit hard to figure out in which environments selftests are running. > > Cheers, > > -Mikko > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#205806): > https://lists.openembedded.org/g/openembedded-core/message/205806 > Mute This Topic: https://lists.openembedded.org/mt/108947467/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Alternatively just mark the test as specific to x86-64 host, there are several self tests that do this. Alex On Tue 15. Oct 2024 at 13.32, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > Don’t remove the tests please. Without them things tend to regress. I > understand it’s frustrating, but we do have arm people here that had pushed > hard for testing in arm hosts, (without really thinking through how > contributors are supposed to deal with such failures) and I think it’s now > on them to help you out. > > Alex > > On Tue 15. Oct 2024 at 13.23, Mikko Rapeli via lists.openembedded.org > <mikko.rapeli=linaro.org@lists.openembedded.org> wrote: > >> Hi, >> >> On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote: >> > The arm test still fails: >> > >> > >> https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio >> > >> > The traceback is at least cleaner now I fixed the lock error. I did >> > test with KVM disabled. >> >> Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64 >> build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu >> 20.04? >> >> I will remove the tests as it is clearly too hard to get these running. >> >> It is a bit hard to figure out in which environments selftests are >> running. >> >> Cheers, >> >> -Mikko >> >> >> >> > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#205807): > https://lists.openembedded.org/g/openembedded-core/message/205807 > Mute This Topic: https://lists.openembedded.org/mt/108947467/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Tue, 2024-10-15 at 14:23 +0300, Mikko Rapeli wrote: > Hi, > > On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote: > > The arm test still fails: > > > > https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio > > > > The traceback is at least cleaner now I fixed the lock error. I did > > test with KVM disabled. > > Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64 > build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu > 20.04? > > I will remove the tests as it is clearly too hard to get these > running. > > It is a bit hard to figure out in which environments selftests are > running. Please don't remove them. The automated tests are the main reason the project actually continues to function in any sane way. For that reason I really do appreciate having them and would like to see more of them. Unfortunately I don't know enough about the subject area to help on this. I have at least tried to improve the failure messages and run out the one area I knew was potentially an issue (KVM on arm). Cheers, Richard
Hi, On Tue, Oct 15, 2024 at 03:13:44PM +0100, Richard Purdie wrote: > On Tue, 2024-10-15 at 14:23 +0300, Mikko Rapeli wrote: > > Hi, > > > > On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote: > > > The arm test still fails: > > > > > > https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio > > > > > > The traceback is at least cleaner now I fixed the lock error. I did > > > test with KVM disabled. > > > > Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64 > > build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu > > 20.04? > > > > I will remove the tests as it is clearly too hard to get these > > running. > > > > It is a bit hard to figure out in which environments selftests are > > running. > > Please don't remove them. The automated tests are the main reason the > project actually continues to function in any sane way. > > For that reason I really do appreciate having them and would like to > see more of them. > > Unfortunately I don't know enough about the subject area to help on > this. I have at least tried to improve the failure messages and run out > the one area I knew was potentially an issue (KVM on arm). Did the test pass on x86_64 build host in your CI? How about I run the wic.Wic2.test_efi_plugin_plain_systemd_boot_qemu_aarch64 test only on x86_64 BUILD_ARCH and aarch64 TARGET_ARCH for now? A decorator like this could help: --- a/meta/lib/oeqa/core/decorator/data.py +++ b/meta/lib/oeqa/core/decorator/data.py @@ -228,3 +228,15 @@ class skipIfNotArch(OETestDecorator): arch = self.case.td['HOST_ARCH'] if arch not in self.archs: self.case.skipTest('Test skipped on %s' % arch) + +@registerDecorator +class skipIfNotBuildArch(OETestDecorator): + """ + Skip test if BUILD_ARCH is not present in the tuple specified. + """ + + attrs = ('archs',) + def setUpDecorator(self): + arch = self.case.td['BUILD_ARCH'] + if arch not in self.archs: + self.case.skipTest('Test skipped on %s' % arch) Cheers, -Mikko
Hi, On Tue, Oct 15, 2024 at 02:23:07PM +0300, Mikko Rapeli via lists.openembedded.org wrote: > Hi, > > On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote: > > The arm test still fails: > > > > https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio > > > > The traceback is at least cleaner now I fixed the lock error. I did > > test with KVM disabled. > > Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64 > build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu 20.04? > > I will remove the tests as it is clearly too hard to get these running. > > It is a bit hard to figure out in which environments selftests are running. I sent out a v9 which disables the test on non-x86_64 build hosts. I hope that was the only failure seen from this series. Cheers, -Mikko
These changes enable building systemd uki images which combine kernel, kernel command line, initrd and possibly signatures to a single UEFI binary. This binary can be booted with UEFI firmware and systemd-boot. No grub is needed and UEFI firmware and/or systemd-boot provide possibilities for boot menus. The uki binary can also be signed for UEFI secure boot so the secure boot extends from firmware to kernel and initrd. Binding secure boot to full userspace is then easier since for example kernel command line and initrd contain the support needed to mount encrypted dm-verity etc partitions, and/or create partitions on demand with systemd-repart using device specific TPM devices for encryption. Tested on qemuarm64-secureboot machine from meta-arm with changes to support secure boot. Slightly different configuration tested on multiple arm64 System Ready boards with UEFI firmware, real and firmware based TPM devices. Tested with ovmf firmware on x86_64 with selftests but without secure boot which seems to be harder to setup in ovmf. Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests on a build machine with zfs filesystem. Will investigate this further. The issue seems to be in mkfs.ext4 producing broken filesystem, and partially in the tests which don't run the correct rootfs file (.ext4 vs .wic). Will debug this further and it is IMO unrelated to these changes since they reproduce on pure master branch without this series. v8: fixed comments from Ross Burton: debug print from warning to debug, dropped duplicate DISTRO_FEATURE setting for systemd in tests, removed aarch64 comment from tests which are currently x86 only. Fixed the new aarch64 wic selftest to run on both genericarm64 and qemuarm64 by adding bios, virtio disk driver etc settings for runqemu (already set in genericarm64 but missing from qemuarm64). v7: add missing "ovmf" to runqemu argument to test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added selftest to cover this wks usage on x86 and aarch64 v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release dependency for all archs" since systemd-boot does not support all architectures v4: handle missing runqemu variable from build config, add python3-pefile to fast ptest list v3: rebased, fixed and added more sefltests, removed wic plugin side uki support v2: https://lists.openembedded.org/g/openembedded-core/message/204090 Michelle Lin (1): uki.bbclass: add class for building Unified Kernel Images (UKI) Mikko Rapeli (7): wic bootimg-efi.py: keep timestamps and add debug prints wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass oeqa selftest uki.py: add tests for uki.bbclass oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi test oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu oeqa selftest wic.py: support UKIs via uki.bbclass meta-selftest/wic/test_efi_plugin.wks | 5 +- .../test_efi_plugin_plain_systemd-boot.wks | 5 + meta/classes-recipe/image_types_wic.bbclass | 3 +- meta/classes-recipe/uki.bbclass | 195 ++++++++++++++++++ .../oeqa/selftest/cases/efibootpartition.py | 11 +- meta/lib/oeqa/selftest/cases/uki.py | 133 ++++++++++++ meta/lib/oeqa/selftest/cases/wic.py | 131 ++++++++++-- .../wic/canned-wks/efi-uki-bootdisk.wks.in | 3 + scripts/lib/wic/plugins/source/bootimg-efi.py | 183 +++++----------- 9 files changed, 511 insertions(+), 158 deletions(-) create mode 100644 meta-selftest/wic/test_efi_plugin_plain_systemd-boot.wks create mode 100644 meta/classes-recipe/uki.bbclass create mode 100644 meta/lib/oeqa/selftest/cases/uki.py create mode 100644 scripts/lib/wic/canned-wks/efi-uki-bootdisk.wks.in