diff mbox series

[1/2] kernel-fitimage: add support for custom ITS file override

Message ID 20250919073335.2069114-2-kavinaya@qti.qualcomm.com
State New
Headers show
Series [1/2] kernel-fitimage: add support for custom ITS file override | expand

Commit Message

Kavinaya S Sept. 19, 2025, 7:33 a.m. UTC 
Update kernel-fitimage.bbclass to use CUSTOM_ITS_FILE when provided.
If the file exists, it will be used instead of generating an ITS file.
This improves flexibility for advanced use cases while maintaining
backward compatibility.

Signed-off-by: Kavinaya S <kavinaya@qti.qualcomm.com>
---
 meta/classes-recipe/kernel-fit-image.bbclass | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

Comments

AdrianF Sept. 21, 2025, 11:40 a.m. UTC | #1 
On Fri, 2025-09-19 at 13:03 +0530, Kavinaya S via
lists.openembedded.org wrote:
> Update kernel-fitimage.bbclass to use CUSTOM_ITS_FILE when provided.
> If the file exists, it will be used instead of generating an ITS
> file.
> This improves flexibility for advanced use cases while maintaining
> backward compatibility.

Hi Kavinaya

Thank you for the patch. LGTM.

But, there is a but. While I acknowledge the concept of 'improved
flexibility for advanced use cases,' it's vital to consider a common
pitfall. Many existing BSPs have unfortunately led to user frustration.
This is predominantly due to vendors integrating considerable amounts
of downstream code that, despite intentions, often proves incompatible
with future Yocto updates.

The direct consequence is that users face significant hurdles in
updating their products, leading to either stagnation or high costs
associated with fixing or circumventing these incompatibilities.

This problem not only impacts users directly but also casts a shadow on
the Yocto Project, which can be perceived as culpable. In reality, such
extensions often bypass upstream maintainers entirely, lacking the
critical upstream test coverage necessary for long-term stability.

My recommendation, therefore, is to proactively prevent such
frustration by officially supporting these 'advanced' use cases and
ensuring they have dedicated test coverage. This approach is preferable
to merely contributing API extensions that might enable all sorts of
potentially unexpected and untested downstream implementations.

Kind regards,
Adrian

> 
> Signed-off-by: Kavinaya S <kavinaya@qti.qualcomm.com>
> ---
>  meta/classes-recipe/kernel-fit-image.bbclass | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes-recipe/kernel-fit-image.bbclass
> b/meta/classes-recipe/kernel-fit-image.bbclass
> index f04aee1807..9258af938c 100644
> --- a/meta/classes-recipe/kernel-fit-image.bbclass
> +++ b/meta/classes-recipe/kernel-fit-image.bbclass
> @@ -139,8 +139,15 @@ python do_compile() {
>      # Generate the configuration section
>     
> root_node.fitimage_emit_section_config(d.getVar("FIT_CONF_DEFAULT_DTB
> "))
>  
> -    # Write the its file
> -    root_node.write_its_file(itsfile)
> +    # Resolve the full path to the custom ITS file if one is
> specified.
> +    # If CUSTOM_ITS_FILE is not set, the ITS file will be generated
> dynamically
> +    custom_its_file = d.getVar('CUSTOM_ITS_FILE')
> +    if custom_its_file:
> +        custom_its_file = os.path.join(kernel_deploydir,
> custom_its_file)
> +        bb.note(f"Using custom ITS file: {custom_its_file}")
> +        shutil.copyfile(custom_its_file, itsfile)
> +    else:
> +        root_node.write_its_file(itsfile)
>  
>      # Assemble the FIT image
>      root_node.run_mkimage_assemble(itsfile, fitname)
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#223768):
> https://lists.openembedded.org/g/openembedded-core/message/223768
> Mute This Topic: https://lists.openembedded.org/mt/115333710/3616858
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe:
> https://lists.openembedded.org/g/openembedded-core/unsub [
> adrian.freihofer@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Kavinaya S Sept. 22, 2025, 8:41 a.m. UTC | #2 
On Sun, Sep 21, 2025 at 05:10 PM, Adrian Freihofer wrote:

> 
> Hi Kavinaya
> 
> Thank you for the patch. LGTM.
> 
> But, there is a but. While I acknowledge the concept of 'improved
> flexibility for advanced use cases,' it's vital to consider a common
> pitfall. Many existing BSPs have unfortunately led to user frustration.
> This is predominantly due to vendors integrating considerable amounts
> of downstream code that, despite intentions, often proves incompatible
> with future Yocto updates.
> 
> The direct consequence is that users face significant hurdles in
> updating their products, leading to either stagnation or high costs
> associated with fixing or circumventing these incompatibilities.
> 
> This problem not only impacts users directly but also casts a shadow on
> the Yocto Project, which can be perceived as culpable. In reality, such
> extensions often bypass upstream maintainers entirely, lacking the
> critical upstream test coverage necessary for long-term stability.
> 
> My recommendation, therefore, is to proactively prevent such
> frustration by officially supporting these 'advanced' use cases and
> ensuring they have dedicated test coverage. This approach is preferable
> to merely contributing API extensions that might enable all sorts of
> potentially unexpected and untested downstream implementations.
> 
> Kind regards,
> Adrian

Hi Adrian,

This change proposes an enhancement to the kernel-FIT image generation process within OpenEmbedded—specifically, the ability to optionally provide a manual.its file in place of the dynamically generated version from kernel-fit-image.bbclass.

This request is motivated by practical challenges encountered in supporting multiple SoC and board variants using a unified DTS file. The current dynamic generation mechanism does not accommodate the level of specificity required for our deployment scenarios. Below are the key reasons for this proposal:

*1. Variant-Specific Compatibility Strings*
Our DTS files support multiple board variants, but the compatible string format required by our UEFI implementation is more granular than what dynamic generation allows. The format we rely on is:
compatible = "qcom,<SoC>[-<soc_version>][-<foundry_id>]-<board>[/<subtype>][-<pmic>][-<board_version>]"
This structure enables precise identification of the appropriate DTB for each board variant, which is critical for correct boot behavior.

*2. Overlay DTBO Integration*
As demonstrated in the upstream kernel Makefile for QCOM platforms (link), overlays are used to generate composite DTBs:

qcs6490-rb3gen2-vision-mezzanine-dtbs := qcs6490-rb3gen2.dtb qcs6490-rb3gen2-vision-mezzanine.dtbo  
qcs6490-rb3gen2-industrial-mezzanine-dtbs := qcs6490-rb3gen2.dtb qcs6490-rb3gen2-industrial-mezzanine.dtbo

In both cases, the base DTB remains the same, but the overlays differ. Without manual.its file support, we cannot assign unique compatible strings to these variants, which limits UEFI’s ability to distinguish between them.

*3. Scalability and Maintainability*
As the number of supported variants grows, maintaining compatibility and configuration logic solely through dynamic generation becomes increasingly complex and error-prone. Manual.its files offer a scalable and maintainable alternative.

Therefore, we suggest introducing a conditional override mechanism in kernel-fit-image.bbclass that allows developers to supply a manual.its file when needed. This approach preserves existing behavior while enabling advanced use cases.

We understand the importance of maintaining upstream compatibility and minimizing user frustration. To that end, we propose this enhancement as an *opt-in mechanism* , with the intention of collaborating on upstream test coverage and documentation to ensure long-term stability. We believe this strikes a balance between flexibility for advanced use cases and the robustness expected from the Yocto Project.

Thanks,

Kavinaya
Alexander Kanavin Sept. 22, 2025, 9:58 a.m. UTC | #3 
On Mon, 22 Sept 2025 at 10:41, kavinaya via lists.openembedded.org
<kavinaya=qti.qualcomm.com@lists.openembedded.org> wrote:
> We understand the importance of maintaining upstream compatibility and minimizing user frustration. To that end, we propose this enhancement as an opt-in mechanism, with the intention of collaborating on upstream test coverage and documentation to ensure long-term stability. We believe this strikes a balance between flexibility for advanced use cases and the robustness expected from the Yocto Project.
I have no idea about this .its stuff, but adding hooks for replacing
existing, not-quite-perfect solutions with proprietary magic is not
particularly welcome in open source projects. You should work towards
improving what already exists.

Alex
AdrianF Sept. 22, 2025, 8:58 p.m. UTC | #4 
Hi Kavinaya

I've had another look, and I'd like to share some more detailed
feedback that I believe will help us refine it further.

First, two simple findings from today's review meeting:
 * Squash Commits: Please consolidate the two patches into a single
   commit for better clarity and history.
 * Namespace Adherence: The new variable CUSTOM_ITS_FILE should align
   with our existing naming conventions. Please rename it to
   FIT_CUSTOM_ITS_FILE.

Now, I have a more fundamental question about the current
implementation, which suggests we might need to explore a different
approach for this functionality. My concern stems from how the
do_compile() function handles the ITS file generation:
   
do_compile () {
root_node = 700 lines of Python code prepare the its data
structure ready to be written into a file.
    
# Then just ignore all this and use another (hard-coded?) its file
custom_its_file = d.getVar('CUSTOM_ITS_FILE')
if custom_its_file:
custom_its_file = os.path.join(kernel_deploydir,
custom_its_file)
bb.note(f"Using custom ITS file: {custom_its_file}")
shutil.copyfile(custom_its_file, itsfile)
else:
root_node.write_its_file(itsfile)

# Assemble the FIT image
root_node.run_mkimage_assemble(itsfile, fitname)

# Sign the FIT image if required
root_node.run_mkimage_sign(fitimage)
}

As I understand it, the current logic (depending on FIT_* variable
configurations) appears to execute a significant portion of the
existing ITS data preparation code—potentially up to 700 lines—only to
then overwrite the generated output with a custom, possibly hard-coded,
ITS file. This pattern seems to bypass the intended use of the existing
generation logic.

Could you clarify the specific need for this custom ITS file? Perhaps
the desired outcome could be achieved by leveraging the existing
KERNEL_DEVICETREE_* variables without needing to introduce a separate
custom file that overrides the generated one.

I would like to understand:
 * What specific limitations or requirements does the generated ITS
   file not meet for your use case?
 * Could you share an example of how your custom ITS file differs from
   what's generated?

Let's discuss this further to find the most efficient and maintainable
solution. I would also prefer an implementation that allows writing an
oe-selftest which clearly covers your use case. I do not see how this
could work with this approach.

Best regards,
Adrian



On Sun, 2025-09-21 at 11:40 +0000, Adrian Freihofer via
lists.openembedded.org wrote:
> On Fri, 2025-09-19 at 13:03 +0530, Kavinaya S via
> lists.openembedded.org wrote:
> > Update kernel-fitimage.bbclass to use CUSTOM_ITS_FILE when
> > provided.
> > If the file exists, it will be used instead of generating an ITS
> > file.
> > This improves flexibility for advanced use cases while maintaining
> > backward compatibility.
> 
> Hi Kavinaya
> 
> Thank you for the patch. LGTM.
> 
> But, there is a but. While I acknowledge the concept of 'improved
> flexibility for advanced use cases,' it's vital to consider a common
> pitfall. Many existing BSPs have unfortunately led to user
> frustration.
> This is predominantly due to vendors integrating considerable amounts
> of downstream code that, despite intentions, often proves
> incompatible
> with future Yocto updates.
> 
> The direct consequence is that users face significant hurdles in
> updating their products, leading to either stagnation or high costs
> associated with fixing or circumventing these incompatibilities.
> 
> This problem not only impacts users directly but also casts a shadow
> on
> the Yocto Project, which can be perceived as culpable. In reality,
> such
> extensions often bypass upstream maintainers entirely, lacking the
> critical upstream test coverage necessary for long-term stability.
> 
> My recommendation, therefore, is to proactively prevent such
> frustration by officially supporting these 'advanced' use cases and
> ensuring they have dedicated test coverage. This approach is
> preferable
> to merely contributing API extensions that might enable all sorts of
> potentially unexpected and untested downstream implementations.
> 
> Kind regards,
> Adrian
> 
> > 
> > Signed-off-by: Kavinaya S <kavinaya@qti.qualcomm.com>
> > ---
> >  meta/classes-recipe/kernel-fit-image.bbclass | 11 +++++++++--
> >  1 file changed, 9 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/classes-recipe/kernel-fit-image.bbclass
> > b/meta/classes-recipe/kernel-fit-image.bbclass
> > index f04aee1807..9258af938c 100644
> > --- a/meta/classes-recipe/kernel-fit-image.bbclass
> > +++ b/meta/classes-recipe/kernel-fit-image.bbclass
> > @@ -139,8 +139,15 @@ python do_compile() {
> >      # Generate the configuration section
> >     
> > root_node.fitimage_emit_section_config(d.getVar("FIT_CONF_DEFAULT_D
> > TB
> > "))
> >  
> > -    # Write the its file
> > -    root_node.write_its_file(itsfile)
> > +    # Resolve the full path to the custom ITS file if one is
> > specified.
> > +    # If CUSTOM_ITS_FILE is not set, the ITS file will be
> > generated
> > dynamically
> > +    custom_its_file = d.getVar('CUSTOM_ITS_FILE')
> > +    if custom_its_file:
> > +        custom_its_file = os.path.join(kernel_deploydir,
> > custom_its_file)
> > +        bb.note(f"Using custom ITS file: {custom_its_file}")
> > +        shutil.copyfile(custom_its_file, itsfile)
> > +    else:
> > +        root_node.write_its_file(itsfile)
> >  
> >      # Assemble the FIT image
> >      root_node.run_mkimage_assemble(itsfile, fitname)
> > 
> > 
> > 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#223790):
> https://lists.openembedded.org/g/openembedded-core/message/223790
> Mute This Topic: https://lists.openembedded.org/mt/115333710/3616858
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe:
> https://lists.openembedded.org/g/openembedded-core/unsub [
> adrian.freihofer@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Kavinaya S Sept. 24, 2025, 8:30 a.m. UTC | #5 
On Tue, Sep 23, 2025 at 02:28 AM, Adrian Freihofer wrote:

>
> Could you clarify the specific need for this custom ITS file? Perhaps
> the desired outcome could be achieved by leveraging the existing
> KERNEL_DEVICETREE_* variables without needing to introduce a separate
> custom file that overrides the generated one.
> 
> I would like to understand:
> * What specific limitations or requirements does the generated ITS
> file not meet for your use case?
> * Could you share an example of how your custom ITS file differs from
> what's generated?
>
Hi Adrian,

1.The compatible string format required by our UEFI implementation is different from what dynamic generation allows. The format we rely on is:
   compatible = "qcom,<SoC>[-<soc_version>][-<foundry_id>]-<board>[/<subtype>][-<pmic>][-<board_version>]"
   This structure enables precise identification of the appropriate DTB for each board variant.

2.As in the upstream kernel Makefile for QCOM platforms (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/qcom/Makefile ), overlays are used to generate composite DTBs:
 
   dtb-$(CONFIG_ARCH_QCOM)	+= qcs6490-rb3gen2.dtb
   qcs6490-rb3gen2-vision-mezzanine-dtbs := qcs6490-rb3gen2.dtb qcs6490-rb3gen2-vision-mezzanine.dtbo
   qcs6490-rb3gen2-industrial-mezzanine-dtbs := qcs6490-rb3gen2.dtb qcs6490-rb3gen2-industrial-mezzanine.dtbo

   In the above cases, the base DTB remains the same, but the overlays differ. Without manual .its file support, we cannot assign unique compatible strings to these variants.

The ITS file generated dynamically will look like:
             configurations {
                default = "conf-qcs6490-rb3gen2.dtb";
                conf-qcs6490-rb3gen2.dtb {
                        description = "1 Linux kernel, FDT blob";
                        kernel = "kernel-1";
                        fdt = "fdt-qcs6490-rb3gen2.dtb";
                        compatible = "qcom,qcs6490-rb3gen2", "qcom,qcm6490";
                        hash-1 {
                                algo = "sha256";
                        };
                };
                conf-qcs6490-rb3gen2-vision-mezzanine.dtb {
                        description = "0 Linux kernel, FDT blob";
                        kernel = "kernel-1";
                        fdt = "fdt-qcs6490-rb3gen2-vision-mezzanine.dtb";
                        compatible = "qcom,qcs6490-rb3gen2", "qcom,qcm6490";
                        hash-1 {
                                algo = "sha256";
                        };
                };
                conf-qcs6490-rb3gen2-industrial-mezzanine.dtb {
                        description = "0 Linux kernel, FDT blob";
                        kernel = "kernel-1";
                        fdt = "fdt-qcs6490-rb3gen2-industrial-mezzanine.dtb";
                        compatible = "qcom,qcs6490-rb3gen2", "qcom,qcm6490";
                        hash-1 {
                                algo = "sha256";
                };
               };




The ITS file that would meet our requirements will look like:
        configurations {
                default = "conf-qcs6490-rb3gen2.dtb";
                conf-qcs6490-rb3gen2.dtb {
                        description = "1 Linux kernel, FDT blob";
                        kernel = "kernel-1";
                        fdt = "fdt-qcs6490-rb3gen2.dtb";
                        compatible = "qcom,qcs6490-rb3gen2", "qcom,qcm6490";
                        hash-1 {
                                algo = "sha256";
                        };
                };
                conf-qcs6490-rb3gen2-vision-mezzanine.dtb {
                        description = "0 Linux kernel, FDT blob";
                        kernel = "kernel-1";
                        fdt = "fdt-qcs6490-rb3gen2-vision-mezzanine.dtb";
                        compatible = "qcom,qcs6490-rb-subtype5";
                        hash-1 {
                                algo = "sha256";
                        };
                };
                conf-qcs6490-rb3gen2-industrial-mezzanine.dtb {
                        description = "0 Linux kernel, FDT blob";
                        kernel = "kernel-1";
                        fdt = "fdt-qcs6490-rb3gen2-industrial-mezzanine.dtb";
                        compatible = "qcom,qcs6490-rb-subtype2";
                        hash-1 {
                                algo = "sha256";
                };
               };
Please look into the difference in the compatible string in both cases.

Thanks,
Kavinaya
Kavinaya S Sept. 24, 2025, 9:46 a.m. UTC | #6 
On Tue, Sep 23, 2025 at 02:28 AM, Adrian Freihofer wrote:

>
> First, two simple findings from today's review meeting:
> * Squash Commits: Please consolidate the two patches into a single
> commit for better clarity and history.
> * Namespace Adherence: The new variable CUSTOM_ITS_FILE should align
> with our existing naming conventions. Please rename it to
> FIT_CUSTOM_ITS_FILE.

Sure. I will consolidate the two patches into a single patch and rename CUSTOM_ITS_FILE to FIT_CUSTOM_ITS_FILE.
> 
> Now, I have a more fundamental question about the current
> implementation, which suggests we might need to explore a different
> approach for this functionality. My concern stems from how the
> do_compile() function handles the ITS file generation:
> 
> do_compile () {
> root_node = 700 lines of Python code prepare the its data
> structure ready to be written into a file.
> 
> # Then just ignore all this and use another (hard-coded?) its file
> custom_its_file = d.getVar('CUSTOM_ITS_FILE')
> if custom_its_file:
> custom_its_file = os.path.join(kernel_deploydir,
> custom_its_file)
> bb.note(f"Using custom ITS file: {custom_its_file}")
> shutil.copyfile(custom_its_file, itsfile)
> else:
> root_node.write_its_file(itsfile)
> 
> # Assemble the FIT image
> root_node.run_mkimage_assemble(itsfile, fitname)
> 
> # Sign the FIT image if required
> root_node.run_mkimage_sign(fitimage)
> }
> 
> As I understand it, the current logic (depending on FIT_* variable
> configurations) appears to execute a significant portion of the
> existing ITS data preparation code—potentially up to 700 lines—only to
> then overwrite the generated output with a custom, possibly hard-coded,
> ITS file. This pattern seems to bypass the intended use of the existing
> generation logic.
>
The existing logic is retained because it performs essential preparation steps beyond ITS generation, such as copying DTBs and other components into the FIT image assembly directory. These steps are required for successful image assembly even when a custom ITS file is used.

Thanks,
Kavinaya
Kavinaya S Sept. 26, 2025, 3:42 a.m. UTC | #7 
Hi Adrian,

Thanks for your question. To clarify:

We are not hardcoding anything or bypassing upstream logic. All BitBake variables and artifact handling remain unchanged. The only difference is that if FIT_CUSTOM_ITS_FILE is set, we use that file instead of generating one dynamically.
This ensures we still rely on the existing process for preparing kernel, DTBs, and other artifacts.
Regarding extending the ITS generator: yes, we considered it, but our required compatible string format is very specific and would need major changes to the generator, adding complexity and maintenance overhead. Using a custom ITS file is simpler, low-impact, and keeps upstream behavior intact while meeting our UEFI compatibility needs.

Let me know if you’d like more details or if you see a better alternative.

Thanks,
Kavinaya
Alexander Kanavin Sept. 26, 2025, 7:31 a.m. UTC | #8 
On Fri, 26 Sept 2025 at 05:42, kavinaya via lists.openembedded.org
<kavinaya=qti.qualcomm.com@lists.openembedded.org> wrote:
> We are not hardcoding anything or bypassing upstream logic. All BitBake variables and artifact handling remain unchanged. The only difference is that if FIT_CUSTOM_ITS_FILE is set, we use that file instead of generating one dynamically.
> This ensures we still rely on the existing process for preparing kernel, DTBs, and other artifacts.
> Regarding extending the ITS generator: yes, we considered it, but our required compatible string format is very specific and would need major changes to the generator, adding complexity and maintenance overhead. Using a custom ITS file is simpler, low-impact, and keeps upstream behavior intact while meeting our UEFI compatibility needs.

Using custom files with unclear, unspecified ways to produce them also
fragments the ecosystem and inserts vendor magic and
non-reproducibility into the build process. That is a negative impact
on the project, and goes against its principles.

Fixing the ITS generator is still the preferred solution, even if it
requires more work on your part, possibly including writing tests.

Alex
Kavinaya S Sept. 29, 2025, 2:26 p.m. UTC | #9 
On Fri, Sep 26, 2025 at 01:01 PM, Alexander Kanavin wrote:

>
> Using custom files with unclear, unspecified ways to produce them also
> fragments the ecosystem and inserts vendor magic and
> non-reproducibility into the build process. That is a negative impact
> on the project, and goes against its principles.
> 
> Fixing the ITS generator is still the preferred solution, even if it
> requires more work on your part, possibly including writing tests.
>
Hi Alexander,

Thanks for the feedback. I’ll address these concerns by extending the ITS generator instead of using custom ITS files, ensuring the 
solution remains reproducible and aligned with project principles. I’ll also add tests as needed.

Thanks,
Kavinaya
diff mbox series

Patch

diff --git a/meta/classes-recipe/kernel-fit-image.bbclass b/meta/classes-recipe/kernel-fit-image.bbclass
index f04aee1807..9258af938c 100644
--- a/meta/classes-recipe/kernel-fit-image.bbclass
+++ b/meta/classes-recipe/kernel-fit-image.bbclass
@@ -139,8 +139,15 @@  python do_compile() {
     # Generate the configuration section
     root_node.fitimage_emit_section_config(d.getVar("FIT_CONF_DEFAULT_DTB"))
 
-    # Write the its file
-    root_node.write_its_file(itsfile)
+    # Resolve the full path to the custom ITS file if one is specified.
+    # If CUSTOM_ITS_FILE is not set, the ITS file will be generated dynamically
+    custom_its_file = d.getVar('CUSTOM_ITS_FILE')
+    if custom_its_file:
+        custom_its_file = os.path.join(kernel_deploydir, custom_its_file)
+        bb.note(f"Using custom ITS file: {custom_its_file}")
+        shutil.copyfile(custom_its_file, itsfile)
+    else:
+        root_node.write_its_file(itsfile)
 
     # Assemble the FIT image
     root_node.run_mkimage_assemble(itsfile, fitname)