| Message ID | 20250519110838.82978-1-adrian.freihofer@siemens.com |
|---|---|
| Headers | show |
| Series | FIT image improvements | expand |
Hi Adrian, thanks for your patches and sorry for jumping in late. Can you please have a look at my below questions. On 25-05-19, AdrianF wrote: > From: Adrian Freihofer <adrian.freihofer@siemens.com> ... > What gets fixed > --------------- > > * sstate does not work well if a FIT image contains an initramfs. > The kernel gets re-built from scratch if the build runs from an > empty TMPDIR: > https://lists.openembedded.org/g/openembedded-core/message/203510. > This is also problematic for SDK use cases since working with the > SDK is annoying if the kernel gets re-built from scratch for no > good reason. > * A FIT image kernel is not available as a package, but all other > kernel image types are. > * The its-file is generated by complicated shell code with lots of > code duplications. Switching to Python simplifies the maintenance > (also because of the indentation with spaces and tabs in the shell > code). > * Separating the FIT image related complexity from the kernel build > complexity simplifies the maintenance of the kernel but also of > the FIT image related code. > * There is already a new (but unfortunately completely unaligned) Can you please elaborate what you mean by 'unaligned'? > implementation for creating FIT images in meta-openembedded: > https://github.com/openembedded/meta-openembedded/blob/master/meta-oe/ > classes/fitimage.bbclass. Let's hope this patch series will lay a > solid foundation for a future merge of the two implementations. > * The new implementation in Python is also a preparation for additional > features, such as adding different types of artifacts or generating > the configuration nodes with greater flexibility. Currently, exactly > one configuration per device node is supported. All the above points are already available/fixed by the meta-openembbed fitimage.bbclass. Can you please elaborate what your class supports which isn't supported yet by the fitimage.bbclass? Regards, Marco
On Mon, 2025-05-19 at 21:18 +0200, Marco Felsch wrote: > Hi Adrian, > > thanks for your patches and sorry for jumping in late. Can you please > have a look at my below questions. > > On 25-05-19, AdrianF wrote: > > From: Adrian Freihofer <adrian.freihofer@siemens.com> > > ... > > > What gets fixed > > --------------- > > > > * sstate does not work well if a FIT image contains an initramfs. > > The kernel gets re-built from scratch if the build runs from an > > empty TMPDIR: > > > > https://lists.openembedded.org/g/openembedded-core/message/203510 > > . > > This is also problematic for SDK use cases since working with the > > SDK is annoying if the kernel gets re-built from scratch for no > > good reason. > > * A FIT image kernel is not available as a package, but all other > > kernel image types are. > > * The its-file is generated by complicated shell code with lots of > > code duplications. Switching to Python simplifies the maintenance > > (also because of the indentation with spaces and tabs in the > > shell > > code). > > * Separating the FIT image related complexity from the kernel build > > complexity simplifies the maintenance of the kernel but also of > > the FIT image related code. > > * There is already a new (but unfortunately completely unaligned) > > Can you please elaborate what you mean by 'unaligned'? There is a FIT image implementation in OE-core. It comes with tests and it is well documented. There is no need for a second implementation without tests and without documentation. > > > implementation for creating FIT images in meta-openembedded: > > > > https://github.com/openembedded/meta-openembedded/blob/master/meta-oe/ > > classes/fitimage.bbclass. Let's hope this patch series will lay a > > solid foundation for a future merge of the two implementations. > > * The new implementation in Python is also a preparation for > > additional > > features, such as adding different types of artifacts or > > generating > > the configuration nodes with greater flexibility. Currently, > > exactly > > one configuration per device node is supported. > > All the above points are already available/fixed by the > meta-openembbed fitimage.bbclass. Can you please elaborate what your > class supports which isn't supported yet by the fitimage.bbclass? > One aspect I particularly appreciate about OE/Yocto, as well as many open source projects integrated with it, is the strong focus on maintaining long-term backward compatibility. OE/Yocto has consistently provided this for many years, and like many other stakeholders, we rely on this stability. It is essential to continuously improve the existing code base without allowing it to become unmaintainable. Introducing a second implementation for functionality that already exists—without clearly defined use cases and adequate test coverage is not a sustainable approach. Simply adding, removing, or replacing code without providing a suitable migration plan for existing users is insufficient. This lack of a clear migration path is exactly what I find missing with the fitimage.bbclass. I hope you share this perspective and will support FIT image integration in OE-core. I have made every effort to align the two implementations, and I believe there is still further potential for improvement. Regards, Adrian > Regards, > Marco
On Mon, 2025-05-19 at 13:07 +0200, Adrian Freihofer via lists.openembedded.org wrote: > From: Adrian Freihofer <adrian.freihofer@siemens.com> > > This patch series re-writes the FIT image related code. The goal is > to fix [YOCTO #12912] which is a long standing issue. I ran this through testing and there was still one oe-selftest issue remaining: https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/1610 oe-selftest -r distrodata.Distrodata.test_maintainers should reproduce. I think there were challenges with the recipe sometimes being buildable and sometimes not so this might not be an entirely straight forward fix to work everywhere as I seem to remember reporting this one before on an earlier version. Cheers, Richard
On Thu, 2025-05-22 at 22:36 +0100, Richard Purdie via lists.openembedded.org wrote: > On Mon, 2025-05-19 at 13:07 +0200, Adrian Freihofer via > lists.openembedded.org wrote: > > From: Adrian Freihofer <adrian.freihofer@siemens.com> > > > > This patch series re-writes the FIT image related code. The goal is > > to fix [YOCTO #12912] which is a long standing issue. > > I ran this through testing and there was still one oe-selftest issue > remaining: > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/1610 > > oe-selftest -r distrodata.Distrodata.test_maintainers > > should reproduce. I think there were challenges with the recipe > sometimes being buildable and sometimes not so this might not be an > entirely straight forward fix to work everywhere as I seem to remember > reporting this one before on an earlier version. I realised this was just a simpler missing maintainers entry so I quickly added one for testing to master-next rather than drop the patchset again. You should probably send that patch if you're ok with it, assuming testing works out ok. Cheers, Richard
On Fri, 2025-05-23 at 06:42 +0100, Richard Purdie wrote: > On Thu, 2025-05-22 at 22:36 +0100, Richard Purdie via > lists.openembedded.org wrote: > > On Mon, 2025-05-19 at 13:07 +0200, Adrian Freihofer via > > lists.openembedded.org wrote: > > > From: Adrian Freihofer <adrian.freihofer@siemens.com> > > > > > > This patch series re-writes the FIT image related code. The goal > > > is > > > to fix [YOCTO #12912] which is a long standing issue. > > > > I ran this through testing and there was still one oe-selftest > > issue > > remaining: > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/1610 > > > > oe-selftest -r distrodata.Distrodata.test_maintainers > > > > should reproduce. I think there were challenges with the recipe > > sometimes being buildable and sometimes not so this might not be an > > entirely straight forward fix to work everywhere as I seem to > > remember > > reporting this one before on an earlier version. > > I realised this was just a simpler missing maintainers entry so I > quickly added one for testing to master-next rather than drop the > patchset again. You should probably send that patch if you're ok with > it, assuming testing works out ok. > Hi Richard, I've just submitted v5, which should be identical to what's currently in master-next. Based on your feedback, I understand that v4's only issue was the missing maintainer entry. For context: The recently merged commit "u-boot: ensure keys are generated before assembling U-Boot FIT image" reintroduced a task dependency where U-Boot depends on a kernel task. To resolve this dependency, I've moved the key generation functionality out of the kernel into a separate recipe. However, I overlooked adding a maintainer entry for this new key-generation recipe, which is now addressed in v5. Thank you! Adrian > Cheers, > > Richard
On Fri, 2025-05-23 at 09:55 +0000, Freihofer, Adrian wrote: > I've just submitted v5, which should be identical to what's currently > in master-next. Based on your feedback, I understand that v4's only > issue was the missing maintainer entry. > > For context: The recently merged commit "u-boot: ensure keys are > generated before assembling U-Boot FIT image" reintroduced a task > dependency where U-Boot depends on a kernel task. To resolve this > dependency, I've moved the key generation functionality out of the > kernel into a separate recipe. However, I overlooked adding a > maintainer entry for this new key-generation recipe, which is now > addressed in v5. Thanks, v5 passes testing and is queued for final review/merge. I understand larger patch series are a bit more complex to merge, this does look like a good and much needed improvement to me. Cheers, Richard
From: Adrian Freihofer <adrian.freihofer@siemens.com> This patch series re-writes the FIT image related code. The goal is to fix [YOCTO #12912] which is a long standing issue. Changes in comparison to v3 --------------------------- * Fix ERROR: linux-yocto-6.12.27+git-r0 do_uboot_mkimage: Execution... * Pass the kernel artifacts via the deploy directory. This makes the task dependencies slightly more complicated. But passing the artifacts via sysroot requires having some artifacts twice in the sstate-cache (once in sysroot and once in deploy-dir). * Support the bundled kernel+initramfs in FIT image again. After some discussions, it turned out that some BSPs make use of this unusual combination and dropping support for it would probably lead to issues. Re-introducing this was also part of the motivation for passing the artifacts via deploy rather than via sysroot. * Drop the patch which merged kernel-uboot.bbclass into kernel-uimage.bbclass. This can be done later if it seems useful. * Drop doc patches again Changes in comparison to v2 --------------------------- * Fix the multilib exclusion check linux-fit-image -> kernel-fit-image This should solve the issue with bitbake world and multilib builds Changes in comparison to v1 --------------------------- * Exclude recipes which inherit the kernel-fit-image.bbclass from multilib builds. This fixes an issue discovered by the AB when running bitbake world with a multilib configuraton enabled. * Remove some new tests which expected openssl from the host. * Rebase the patches on the new bug-fix commit "u-boot: ensure keys are generated before assembling U-Boot FIT image" from Rogerio Guerra Borin <rogerio.borin@toradex.com> This fix required splitting the key generation into a separate recipe. Note: These patches depend on this commit, which is included in this series. As of today, the patch is already present on master-next. * Add a cover letter with correct subject for the patch series What gets fixed --------------- * sstate does not work well if a FIT image contains an initramfs. The kernel gets re-built from scratch if the build runs from an empty TMPDIR: https://lists.openembedded.org/g/openembedded-core/message/203510. This is also problematic for SDK use cases since working with the SDK is annoying if the kernel gets re-built from scratch for no good reason. * A FIT image kernel is not available as a package, but all other kernel image types are. * The its-file is generated by complicated shell code with lots of code duplications. Switching to Python simplifies the maintenance (also because of the indentation with spaces and tabs in the shell code). * Separating the FIT image related complexity from the kernel build complexity simplifies the maintenance of the kernel but also of the FIT image related code. * There is already a new (but unfortunately completely unaligned) implementation for creating FIT images in meta-openembedded: https://github.com/openembedded/meta-openembedded/blob/master/meta-oe/ classes/fitimage.bbclass. Let's hope this patch series will lay a solid foundation for a future merge of the two implementations. * The new implementation in Python is also a preparation for additional features, such as adding different types of artifacts or generating the configuration nodes with greater flexibility. Currently, exactly one configuration per device node is supported. Architectural change -------------------- The existing kernel-fitimage.bbclass is designed to be added to KERNEL_CLASSES. It appends code to the kernel's tasks and injects additional tasks between the existing ones. Some functions rely on running within the kernel's build folder structure. The new implementation introduces the kernel-fit-image.bbclass, which is intended to be inherited by an independent recipe. This recipe takes the kernel artifacts from the sstate-cache and assembles the FIT image entirely independently of the kernel's build tasks and directory structure. An example of using the new kernel-fit-image.bbclass is the linux-yocto-fitimage.bb recipe, which builds the FIT image for the linux-yocto kernel. The recipe looks like this: SUMMARY = "The Linux kernel as a FIT image (optionally with initramfs)" SECTION = "kernel" LICENSE = "GPL-2.0-with-Linux-syscall-note" LIC_FILES_CHKSUM = "\ file://${COREBASE}/meta/files/common-licenses/GPL-2.0-with-Linux-syscall-note; md5=0bad96c422c41c3a94009dcfe1bff992" inherit kernel-fit-image The configuration variables defined in the conf/image-fitimage.conf file are handled by the kernel-fit-image.bbclass in the same way as they are by the kernel-fitimage.bbclass. The new implementation is 99% backward compatible with the existing kernel-fitimage.bbclass. The existing test-suite runs with minimal chagnes. With the kernel-fitimage.bbclass, the FIT image was built as part of the kernel itself. To ensure the new recipe is built automatically, the following variables can be set, for example, in the machine configuration file: # Do not install the kernel image package RRECOMMENDS:${KERNEL_PACKAGE_NAME}-base = "" # Install the FIT image package into the rootfs (there is now a package :-) MACHINE_EXTRA_RDEPENDS += "${PREFERRED_PROVIDER_virtual/kernel}-fitimage" # Configure the image.bbclass to depend on the fitImage instead of only # the kernel to ensure the FIT image is built with the image KERNEL_DEPLOY_DEPEND = "${PREFERRED_PROVIDER_virtual/kernel}-fitimage:do_deploy" Breaking changes ---------------- * Building a kernel FIT image changes. Before this patch series: A configuration like: KERNEL_IMAGETYPE = "fitImage" KERNEL_CLASSES = "kernel-fitimage" and building a kernel like: bitbake linux-yocto generated a FIT image including the kernel and maybe additional artifacts. With this patch series merged, the same can be achieved like this: bitbake linux-yocto-fitimage While this simple example is even simpler than before, there are also other examples which might look more complicated with the new implementation than with the old implementation. Proposal for merging these patch series --------------------------------------- The patches are split into small steps which allow a step-by-step merging: 1. New tests without impact on code: * oe-selftest: add new ext dtb recipe * oe-selftest: fitimage: test coverage for ext dtb 2. Add the new implementation without changing the existing implementation * kernel-signing-keys-native: refactor key generation into a new recipe * kernel-uboot.bbclass: do not require the kernel build folder * kernel-uboot.bbclass: deploy the vmlinux kernel binary * kernel-fitimage: refactor order in its * kernel-fit-image.bbclass: add a new FIT image implementation * maintainers: add myself for linux-yocto-fitimage * oe-selftest: fitimage add tests for fitimage.py * oe-selftest: fitimage support new FIT recipe as well * oe-selftest: fitimage: run all tests for both FIT implementations * oe-selftest: fitimage refactor classes 3. Refactor the old kernel-fitimage.bbclass to Python to share code with the new implementation. While not strictly required, this allows users to migrate to the new Python code with 100% backward compatibility. Both implementations could be maintained until the new one is widely accepted and tested. Note: This change does obviousely not resolve any issues as the architecture remains the same. We can also simply drop this patch. * kernel-fitimage: re-write its code in Python 4. Remove the old kernel-fitimage.bbclass and clean up the code from the initramfs bundle in the FIT image left overs. * oe-selftest: fitimage: remove kernel-fitimage tests * kernel.bbclass: remove support for type fitImage * kernel-fitimage.bbclass: remove it Side note: The development of the commits took place the other way round: First the existing implementation was re-written in Python. Then the new implementation was split out and finally the independent recipe could be added to use the new kernel-fit-image.bbclass as well. Adrian Freihofer (16): oe-selftest: add new ext dtb recipe oe-selftest: fitimage: test coverage for ext dtb kernel-signing-keys-native: refactor key generation into a new recipe kernel-uboot.bbclass: do not require the kernel build folder kernel-uboot.bbclass: deploy the vmlinux kernel binary kernel-fitimage: refactor order in its kernel-fit-image.bbclass: add a new FIT image implementation maintainers: add myself for linux-yocto-fitimage oe-selftest: fitimage add tests for fitimage.py oe-selftest: fitimage support new FIT recipe as well oe-selftest: fitimage: run all tests for both FIT implementations oe-selftest: fitimage refactor classes kernel-fitimage: re-write its code in Python oe-selftest: fitimage: remove kernel-fitimage tests kernel.bbclass: remove support for type fitImage kernel-fitimage.bbclass: remove it .../recipes-test/ext-dtb/bborg-relay-00a2.bb | 14 + .../ext-dtb/files/BBORG_RELAY-00A2.dts | 49 + meta/classes-recipe/kernel-fit-image.bbclass | 184 ++++ meta/classes-recipe/kernel-fitimage.bbclass | 839 ------------------ meta/classes-recipe/kernel-uboot.bbclass | 47 +- meta/classes-recipe/kernel-uimage.bbclass | 3 +- meta/classes-recipe/kernel.bbclass | 20 +- meta/classes-recipe/uboot-sign.bbclass | 5 +- meta/classes/multilib.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/fitimage.py | 463 ++++++++++ meta/lib/oeqa/selftest/cases/fitimage.py | 249 +++++- .../kernel-signing-keys-native.bb | 75 ++ .../linux/linux-yocto-fitimage_6.12.bb | 13 + 14 files changed, 1060 insertions(+), 903 deletions(-) create mode 100644 meta-selftest/recipes-test/ext-dtb/bborg-relay-00a2.bb create mode 100644 meta-selftest/recipes-test/ext-dtb/files/BBORG_RELAY-00A2.dts create mode 100644 meta/classes-recipe/kernel-fit-image.bbclass delete mode 100644 meta/classes-recipe/kernel-fitimage.bbclass create mode 100644 meta/lib/oe/fitimage.py create mode 100644 meta/recipes-kernel/kernel-signing-keys/kernel-signing-keys-native.bb create mode 100644 meta/recipes-kernel/linux/linux-yocto-fitimage_6.12.bb