| Message ID | 20250513213834.87830-1-adrian.freihofer@siemens.com |
|---|---|
| Headers | show |
| Series | FIT image improvements | expand |
On Tue May 13, 2025 at 11:36 PM CEST, Adrian Freihofer via lists.openembedded.org wrote: > From: Adrian Freihofer <adrian.freihofer@siemens.com> > > This patch series re-writes the FIT image related code. The goal is to > fix [YOCTO #12912] which is a long standing issue. > Hi Adrian, Thanks for your patches. I believe it breaks the build on the autobuilder. Maybe only on x86? ERROR: lib32-linux-yocto-fitimage-6.12.23+git-r0 do_compile: Error executing a python function in exec_func_python() autogenerated: ... File: '/srv/pokybuild/yocto-worker/qemux86-world/build/meta/classes-recipe/kernel-fit-image.bbclass', lineno: 57, function: do_compile ... Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/pokybuild/yocto-worker/qemux86-world/build/build/tmp/work/qemux86_64-pokymllib32-linux/lib32-linux-yocto-fitimage/6.12.23+git/lib32-recipe-sysroot/sysroot-only/linux.bin' https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/1604 https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/1486 Can you have a look at this failure please?
On Wed, 2025-05-14 at 09:43 +0200, Mathieu Dubois-Briand wrote: > On Tue May 13, 2025 at 11:36 PM CEST, Adrian Freihofer via > lists.openembedded.org wrote: > > From: Adrian Freihofer <adrian.freihofer@siemens.com> > > > > This patch series re-writes the FIT image related code. The goal is > > to > > fix [YOCTO #12912] which is a long standing issue. > > > > Hi Adrian, > > Thanks for your patches. > > I believe it breaks the build on the autobuilder. Maybe only on x86? > > ERROR: lib32-linux-yocto-fitimage-6.12.23+git-r0 do_compile: Error > executing a python function in exec_func_python() autogenerated: > ... > File: '/srv/pokybuild/yocto-worker/qemux86-world/build/meta/classes- > recipe/kernel-fit-image.bbclass', lineno: 57, function: do_compile > ... > Exception: FileNotFoundError: [Errno 2] No such file or directory: > '/srv/pokybuild/yocto-worker/qemux86- > world/build/build/tmp/work/qemux86_64-pokymllib32-linux/lib32-linux- > yocto-fitimage/6.12.23+git/lib32-recipe-sysroot/sysroot- > only/linux.bin' > > https://autobuilder.yoctoproject.org/valkyrie/api/v2/logs/2376349/raw_inline > https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/1486 > > The failing build is then still bitbake world for multilib x86-32. I was hoping that this is fixed now with the line added to multilib.bbclass by this patch https://patchwork.yoctoproject.org/project/oe-core/patch/20250513213834.87830-11-adrian.freihofer@siemens.com/ diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass index a4151658a62..ea80a6260c4 100644 --- a/meta/classes/multilib.bbclass +++ b/meta/classes/multilib.bbclass @@ -21,6 +21,7 @@ python multilib_virtclass_handler () { bpn = d.getVar("BPN") if ("virtual/kernel" in provides or bb.data.inherits_class('module-base', d) + or bb.data.inherits_class('linux-fit-image', d) or bpn in non_ml_recipes): raise bb.parse.SkipRecipe("We shouldn't have multilib variants for %s" % bpn) Looks like this is not the right or at least not the complete fix then. Thank you for the feedback. Adrian > Can you have a look at this failure please? >
From: Adrian Freihofer <adrian.freihofer@siemens.com> This patch series re-writes the FIT image related code. The goal is to fix [YOCTO #12912] which is a long standing issue. Changes in comparison to v1 --------------------------- * Exclude recipes which inherit the kernel-fit-image.bbclass from multilib builds. This fixes an issue discovered by the AB when running bitbake world with a multilib configuraton enabled. * Remove some new tests which expected openssl from the host. * Rebase the patches on the new bug-fix commit "u-boot: ensure keys are generated before assembling U-Boot FIT image" from Rogerio Guerra Borin <rogerio.borin@toradex.com> This fix required splitting the key generation into a separate recipe. Note: These patches depend on this commit, which is included in this series. As of today, the patch is already present on master-next. * Add a cover letter with correct subject for the patch series What gets fixed --------------- * sstate does not work well if a FIT image contains an initramfs. The kernel gets re-built from scratch if the build runs from an empty TMPDIR: https://lists.openembedded.org/g/openembedded-core/message/203510. This is also problematic for SDK use cases since working with the SDK is annoying if the kernel gets re-built from scratch for no good reason. * A FIT image kernel is not available as a package, but all other kernel image types are. * The its-file is generated by complicated shell code with lots of code duplications. Switching to Python simplifies the maintenance (also because of the indentation with spaces and tabs in the shell code). * Separating the FIT image related complexity from the kernel build complexity simplifies the maintenance of the kernel but also of the FIT image related code. * There is already a new (but unfortunately completely unaligned) implementation for creating FIT images in meta-openembedded: https://github.com/openembedded/meta-openembedded/blob/master/meta-oe/ classes/fitimage.bbclass. Let's hope this patch series will lay a solid foundation for a future merge of the two implementations. * The new implementation in Python is also a preparation for additional features, such as adding different types of artifacts or generating the configuration nodes with greater flexibility. Currently, exactly one configuration per device node is supported. Architectural change -------------------- The existing kernel-fitimage.bbclass is designed to be added to KERNEL_CLASSES. It appends code to the kernel's tasks and injects additional tasks between the existing ones. Some functions rely on running within the kernel's build folder structure. The new implementation introduces the kernel-fit-image.bbclass, which is intended to be inherited by an independent recipe. This recipe takes the kernel artifacts from the sstate-cache and assembles the FIT image entirely independently of the kernel's build tasks and directory structure. An example of using the new kernel-fit-image.bbclass is the linux-yocto-fitimage.bb recipe, which builds the FIT image for the linux-yocto kernel. The recipe looks like this: SUMMARY = "The Linux kernel as a FIT image (optionally with initramfs)" SECTION = "kernel" LICENSE = "GPL-2.0-with-Linux-syscall-note" LIC_FILES_CHKSUM = "\ file://${COREBASE}/meta/files/common-licenses/GPL-2.0-with-Linux-syscall-note; md5=0bad96c422c41c3a94009dcfe1bff992" inherit kernel-fit-image The configuration variables defined in the conf/image-fitimage.conf file are handled by the kernel-fit-image.bbclass in the same way as they are by the kernel-fitimage.bbclass. The new implementation is 98% backward compatible with the existing kernel-fitimage.bbclass. The existing test-suite runs with minimal chagnes. With the kernel-fitimage.bbclass, the FIT image was built as part of the kernel itself. To ensure the new recipe is built automatically, the following variables can be set, for example, in the machine configuration file: # Do not install the kernel image package RRECOMMENDS:${KERNEL_PACKAGE_NAME}-base = "" # Install the FIT image package into the rootfs (there is now a package :-) MACHINE_EXTRA_RDEPENDS += "${PREFERRED_PROVIDER_virtual/kernel}-fitimage" # Configure the image.bbclass to depend on the fitImage instead of only # the kernel to ensure the FIT image is built with the image KERNEL_DEPLOY_DEPEND = "${PREFERRED_PROVIDER_virtual/kernel}-fitimage:do_deploy" Breaking changes ---------------- * Currently it is possible to bundle an initramfs into the kernel and finally add this bundle to a FIT image. With this patch series completely merged, the INITRAMFS_IMAGE_BUNDLE variable will not have an impact on the FIT image build process. Of course, nothing changes on the kernel build code. It is still possible to bundle an initramfs into the kernel. Just adding this bundle also to the FIT image is not possible anymore. It would be possible to bring this feature back by slightly modifying the new implementation. But it would make the implementation more complex and I don't see a real use case for this feature. Bundling an initramfs and a kernel is basically what the FIT image is supposed to do natively, without the help of the kernel build code. As long as the kernel build code deals with the initramfs, using a kernel from the sstate-cache is not possible without re-building the kernel. That's why this patch series removes the support for this feature. * Building a kernel FIT image changes. Before this patch series: A configuration like: KERNEL_IMAGETYPE = "fitImage" KERNEL_CLASSES = "kernel-fitimage" and building a kernel like: bitbake linux-yocto generated a FIT image including the kernel and maybe additional artifacts. With this patch series merged, the same can be achieved like this: bitbake linux-yocto-fitimage While this simple example is even simpler than before, there are also other examples which might look more complicated with the new implementation than with the old implementation. Proposal for merging these patch series --------------------------------------- The patches are split into small steps which allow a step-by-step merging: 1. Already on master-next: * u-boot: ensure keys are generated before assembling U-Boot FIT image 2. Add the new implementation without touching the old implementation * oe-selftest: add new ext dtb recipe * oe-selftest: fitimage: test coverage for ext dtb * kernel-signing-keys-native: refactor key generation into a new recipe * kernel-uboot: do not require the kernel build folder * kernel-devicetree: export dtbs to sysroot * kernel.bbclass: stage the sysroot-only folder * kernel-uboot.bbclass: stage the kernel binary * kernel-fitimage: refactor order in its * kernel-fit-image.bbclass: add a new FIT image implementation * maintainers: add myself for linux-yocto-fitimage * oe-selftest: fitimage add tests for fitimage.py * oe-selftest: fitimage refactor kernel-fitimage.bbclass * oe-selftest: fitimage: run all tests for the FIT image recipe * oe-selftest: fitimage refactor classes 3. Refactor the old kernel-fitimage.bbclass to Python to share code with the new implementation. While not strictly required, this allows users to migrate to the new Python code with 100% backward compatibility. Both implementations could be maintained until the new one is widely accepted and tested. Note: This change does obviousely not resolve any issues as the architecture remains the same. We can also simply drop this patch. * kernel-fitimage: re-write its code in Python 4. Remove the old kernel-fitimage.bbclass and clean up the code from the initramfs bundle in the FIT image left overs. * kernel.bbclass: remove support for type fitImage * oe-selftest: fitimage: remove kernel-fitimage tests * kernel.bbclass: remove support for type fitImage * oe-selftest: fitimage: remove kernel-fitimage tests * kernel-fitimage.bbclass: remove it * kernel-uboot.bbclass: merge it into kernel-uimage.bbclass * oe-selftest: fitimage: drop initramfs bundle test * kernel-fit-image.bbclass: ignore INITRAMFS_IMAGE_BUNDLE Side note: The development of the commits took place the other way round: First the existing implementation was re-written in Python. Then the new implementation was split out and finally the independent recipe could be added to use the new kernel-fit-image.bbclass as well. Some documentation patches are under development on a branch: https://git.yoctoproject.org/poky-contrib/log/?h=adrianf/fitimage-improvements Adrian Freihofer (21): oe-selftest: add new ext dtb recipe oe-selftest: fitimage: test coverage for ext dtb kernel-signing-keys-native: refactor key generation into a new recipe kernel-uboot: do not require the kernel build folder kernel-devicetree: export dtbs to sysroot kernel.bbclass: stage the sysroot-only folder kernel-uboot.bbclass: stage the kernel binary kernel-fitimage: refactor order in its kernel-fit-image.bbclass: add a new FIT image implementation maintainers: add myself for linux-yocto-fitimage oe-selftest: fitimage add tests for fitimage.py oe-selftest: fitimage refactor kernel-fitimage.bbclass oe-selftest: fitimage: run all tests for the FIT image recipe oe-selftest: fitimage refactor classes kernel-fitimage: re-write its code in Python kernel.bbclass: remove support for type fitImage oe-selftest: fitimage: remove kernel-fitimage tests kernel-fitimage.bbclass: remove it kernel-uboot.bbclass: merge it into kernel-uimage.bbclass oe-selftest: fitimage: drop initramfs bundle test kernel-fit-image.bbclass: ignore INITRAMFS_IMAGE_BUNDLE Rogerio Guerra Borin (1): u-boot: ensure keys are generated before assembling U-Boot FIT image .../recipes-test/ext-dtb/bborg-relay-00a2.bb | 14 + .../ext-dtb/files/BBORG_RELAY-00A2.dts | 49 + meta/classes-recipe/kernel-devicetree.bbclass | 4 + meta/classes-recipe/kernel-fit-image.bbclass | 176 ++++ meta/classes-recipe/kernel-fitimage.bbclass | 839 ------------------ meta/classes-recipe/kernel-uboot.bbclass | 51 -- meta/classes-recipe/kernel-uimage.bbclass | 70 +- meta/classes-recipe/kernel.bbclass | 24 +- meta/classes-recipe/uboot-sign.bbclass | 3 + meta/classes/multilib.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/fitimage.py | 463 ++++++++++ meta/lib/oeqa/selftest/cases/fitimage.py | 284 ++++-- .../kernel-signing-keys-native.bb | 61 ++ .../linux/linux-yocto-fitimage_6.12.bb | 13 + 15 files changed, 1072 insertions(+), 981 deletions(-) create mode 100644 meta-selftest/recipes-test/ext-dtb/bborg-relay-00a2.bb create mode 100644 meta-selftest/recipes-test/ext-dtb/files/BBORG_RELAY-00A2.dts create mode 100644 meta/classes-recipe/kernel-fit-image.bbclass delete mode 100644 meta/classes-recipe/kernel-fitimage.bbclass delete mode 100644 meta/classes-recipe/kernel-uboot.bbclass create mode 100644 meta/lib/oe/fitimage.py create mode 100644 meta/recipes-kernel/kernel-signing-keys/kernel-signing-keys-native.bb create mode 100644 meta/recipes-kernel/linux/linux-yocto-fitimage_6.12.bb