mbox series

[v2,0/6] Use the kernel from sstate when building fitImages

Message ID 20240715141448.2158477-1-adrian.freihofer@gmail.com
Headers show
Series Use the kernel from sstate when building fitImages | expand

Message

Adrian Freihofer July 15, 2024, 2:10 p.m. UTC
Changes in comparison to v1:
- Add the missing dependency from do_image_wic to
  initramfs.do_image_complete which caused the build failure on the AB.
- Support symbolic links for $dtb_path again, as suggested by Mark.
- Simplify the refactoring of the uboot_prep_kimage function.
- Evaluate staging via sysroot instead of taking the kernel artifacts
  from the deploy folder. But there is an issue with this approach: It
  does not allow to remove the dependency from do_deploy on do_install.
- Rebasing to latest master

If the KERNEL_IMAGETYPES(S) contains fitImage, the kernel is always
rebuilt when something changes in the initramfs frequently.
This is even worse if the build runs from an empty TMPDIR. The kernel
re-build starts by fetching the large kernel git repository and
recompiling it from scratch.

This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel
Makefile is needed to generate the initramfs bundle the kernel build
folder is required.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the
build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs which are available in the deploy
directory. That means creating the fitImage can be done with artifacts
which are already cached by the sstate.

There is an extra commit providing a html file. This html file provides
some graphics which show the changes in the kernel's task dependencies.
I hope this simplifies the re-view of this patch series.

Testing:
- oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg \
     reproducible -T yocto-mirrors -T machine -T toolchain-user \
     -T toolchain-system
- Build for core-image-minimal for MACHINE = "genericarm64"
  (failure from AB was reproducible and is fixed now)
- Build clean followed by build from sstate
  Note: Adding this test to oe-selftest failed because sstate is read only
  - Clean build config:
    KERNEL_IMAGETYPE = "Image"
    KERNEL_IMAGETYPES += " fitImage "
    KERNEL_CLASSES = " kernel-fitimage "
    IMAGE_FSTYPES += "cpio.gz"
    INITRAMFS_IMAGE = "core-image-minimal"
    IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
    UBOOT_RD_LOADADDRESS = "0x88000000"
    UBOOT_RD_ENTRYPOINT = "0x88000000"
    UBOOT_LOADADDRESS = "0x80080000"
    UBOOT_ENTRYPOINT = "0x80080000"
    FIT_DESC = "A model description"
    FOO_VAR = "1"
    INHERIT += "image-buildinfo"
    IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
  - Append the following and rebuild with sstate:
    FOO_VAR = "2"
    TMPDIR = "${TOPDIR}/tmp-2"
  - Check the log files:
    - tmp/log/cooker/qemux86-64 contains:
      linux-yocto-6.6.35+git-r0: task do_deploy: Succeeded
      linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
    - tmp-2/log/cooker/qemux86-64 contains:
      linux-yocto-6.6.35+git-r0: task do_deploy_setscene: Succeeded
      linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
- To re-view the chagnes in the task dependencies, the script bellow
  has been used. It confirms:
  - For builds with fitImage and unbundled initramfs:
    - the do_deploy_fitimage_unbundled task runs after do_deploy
    - do_assemble_fitimage_initramfs is not executed
    - do_bundle_initramfs is not executed
    - It works for fitImage in KERNEL_IMAGETYPE as well as for fitImage
      in KERNEL_IMAGETYPES
  - For builds with fitImage and bundled initramfs: No changes
  - For builds with fitImage, without initramfs:
    - do_assemble_fitimage_initramfs is not executed
    - do_bundle_initramfs is not executed

OUTPUT_FILE=task-depends.md
GIT_BRANCH=master
GIT_BRANCH_NEXT=adrianf/kernel-fitimage-sstate

echo "# Task dependeny changes" > "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"

run_bitbake(){
    echo "$1" >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"

    bitbake virtual/kernel -g
    grep -E '(digraph depends|linux-yocto|\})' task-depends.dot \
    | grep -v -E '(spdx|do_kernel_configcheck|do_prepare_recipe_sysroot|do_populate_lic|native.do_populate_sysroot)' \
    | grep -v -E '(depmodwrapper-cross.do_populate_sysroot|binutils-cross-x86_64.do_populate_sysroot|do_package)' \
    | grep -v -E '(do_validate_branches|do_unpack|do_recipe_qa|do_patch|do_kernel_checkout|do_kernel_configme|do_kernel_metadata)' \
    | grep -v -E '(do_sizecheck|do_strip|do_compile_kernelmodules|do_shared_workdir|do_transform_kernel|do_kernel_link_images)' \
    | grep -v -E '(do_kernel_version_sanity_check|do_symlink_kernsrc|do_deploy_source_date_epoch|do_configure|do_fetch)' \
    | sed -e 's;\\n\:.*.bb;;g' -e 's;linux-yocto[. ];;g' > task-depends-filtered.dot

    echo '```plantuml' >> "$OUTPUT_FILE"
    cat task-depends-filtered.dot >> "$OUTPUT_FILE"
    echo '```' >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"
}

run_bitbake_per_branch(){
    echo "## Configuration: $1" >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"
    echo '```raw' >> "$OUTPUT_FILE"
    cat build/conf/auto.conf >> "$OUTPUT_FILE"
    echo '```' >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"

    git checkout $GIT_BRANCH
    run_bitbake "### branch: $GIT_BRANCH"
    mv -f task-depends-filtered.dot task-depends-filtered-old.dot

    git checkout $GIT_BRANCH_NEXT
    run_bitbake "### branch: $GIT_BRANCH_NEXT"

    echo "## Diff" >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"
    echo '```patch' >> "$OUTPUT_FILE"
    diff task-depends-filtered-old.dot task-depends-filtered.dot >> "$OUTPUT_FILE"
    echo '```' >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"
}

cat << EOF > build/conf/auto.conf
KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
EOF
run_bitbake_per_branch "image, fitimage, initramfs, unbundled"

cat << EOF > build/conf/auto.conf
KERNEL_IMAGETYPE:forcevariable = "fitImage"
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
EOF
run_bitbake_per_branch "fitimage, initramfs, unbundled"

cat << EOF > build/conf/auto.conf
KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
INITRAMFS_IMAGE_BUNDLE = "1"
EOF
run_bitbake_per_branch "image, fitimage, initramfs, bundled"

cat << EOF > build/conf/auto.conf
KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
EOF
run_bitbake_per_branch "image, fitimage"

rm -f build/conf/auto.conf task-depends-filtered-old.dot task-depends-filtered.dot task-depends.dot

Adrian Freihofer (6):
  kernel-fitimage: fix intentation
  kernel-fitimage: fix external dtb check
  kernel: refactor linux compression
  kernel-fitimage: refactor fitimage_assemble
  kernel: refactor fitimage
  kernel-fitimage: run unbundled fitimage after deploy

 meta/classes-recipe/image.bbclass           |  12 +-
 meta/classes-recipe/kernel-fitimage.bbclass | 188 ++++++++++++--------
 meta/classes-recipe/kernel-uboot.bbclass    |   1 +
 meta/classes-recipe/kernel.bbclass          |  31 ++--
 4 files changed, 148 insertions(+), 84 deletions(-)

Comments

Adrian Freihofer July 15, 2024, 2:22 p.m. UTC | #1
task-depends      Task dependeny changes 

Configuration: image, fitimage, initramfs, unbundled 

KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
branch: master 





depends
undefined


core-image-minimal.do_build
undefined
core-image-minimal.do_build




do_deploy
undefined
do_deploy




core-image-minimal.do_build->do_deploy
undefined
undefined



do_assemble_fitimage_initramfs
undefined
do_assemble_fitimage_initramfs




do_deploy->do_assemble_fitimage_initramfs
undefined
undefined



do_bundle_initramfs
undefined
do_bundle_initramfs




do_deploy->do_bundle_initramfs
undefined
undefined



do_populate_sysroot
undefined
do_populate_sysroot




do_deploy->do_populate_sysroot
undefined
undefined



do_assemble_fitimage
undefined
do_assemble_fitimage




do_compile
undefined
do_compile




do_assemble_fitimage->do_compile
undefined
undefined



do_kernel_generate_rsa_keys
undefined
do_kernel_generate_rsa_keys




do_assemble_fitimage->do_kernel_generate_rsa_keys
undefined
undefined



do_kernel_generate_rsa_keys->do_compile
undefined
undefined



core-image-minimal.do_image_complete
undefined
core-image-minimal.do_image_complete




do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_assemble_fitimage_initramfs->do_bundle_initramfs
undefined
undefined



do_bundle_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_install
undefined
do_install




do_bundle_initramfs->do_install
undefined
undefined



do_install->do_assemble_fitimage
undefined
undefined



do_install->do_compile
undefined
undefined



do_populate_sysroot->do_install
undefined
undefined



branch: adrianf/kernel-fitimage-sstate 





depends
undefined


core-image-minimal.do_build
undefined
core-image-minimal.do_build




do_deploy
undefined
do_deploy




core-image-minimal.do_build->do_deploy
undefined
undefined



do_deploy_fitimage_unbundled
undefined
do_deploy_fitimage_unbundled




core-image-minimal.do_build->do_deploy_fitimage_unbundled
undefined
undefined



core-image-minimal.do_image_complete
undefined
core-image-minimal.do_image_complete




do_deploy->core-image-minimal.do_image_complete
undefined
undefined



do_install
undefined
do_install




do_deploy->do_install
undefined
undefined



do_populate_sysroot
undefined
do_populate_sysroot




do_deploy->do_populate_sysroot
undefined
undefined



do_deploy_fitimage_unbundled->do_deploy
undefined
undefined



do_deploy_fitimage_unbundled->core-image-minimal.do_image_complete
undefined
undefined



do_assemble_fitimage
undefined
do_assemble_fitimage




do_compile
undefined
do_compile




do_assemble_fitimage->do_compile
undefined
undefined



do_kernel_generate_rsa_keys
undefined
do_kernel_generate_rsa_keys




do_assemble_fitimage->do_kernel_generate_rsa_keys
undefined
undefined



do_kernel_generate_rsa_keys->do_compile
undefined
undefined



do_install->do_assemble_fitimage
undefined
undefined



do_install->do_compile
undefined
undefined



do_populate_sysroot->do_install
undefined
undefined



Diff 

2a3
> "core-image-minimal.do_build" -> "do_deploy_fitimage_unbundled"
6,11d6
< "do_assemble_fitimage_initramfs"
[label="do_assemble_fitimage_initramfs"]
< "do_assemble_fitimage_initramfs" -> "core-image-
minimal.do_image_complete"
< "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs"
< "do_bundle_initramfs" [label="do_bundle_initramfs"]
< "do_bundle_initramfs" -> "core-image-minimal.do_image_complete"
< "do_bundle_initramfs" -> "do_install"
14,15c9,10
< "do_deploy" -> "do_assemble_fitimage_initramfs"
< "do_deploy" -> "do_bundle_initramfs"
---
> "do_deploy" -> "core-image-minimal.do_image_complete"
> "do_deploy" -> "do_install"
16a12,14
> "do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"]
> "do_deploy_fitimage_unbundled" -> "core-image-
minimal.do_image_complete"
> "do_deploy_fitimage_unbundled" -> "do_deploy"
Configuration: fitimage, initramfs, unbundled 

KERNEL_IMAGETYPE:forcevariable = "fitImage"
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
branch: master 





depends
undefined


core-image-minimal.do_build
undefined
core-image-minimal.do_build




do_deploy
undefined
do_deploy




core-image-minimal.do_build->do_deploy
undefined
undefined



do_assemble_fitimage_initramfs
undefined
do_assemble_fitimage_initramfs




do_deploy->do_assemble_fitimage_initramfs
undefined
undefined



do_bundle_initramfs
undefined
do_bundle_initramfs




do_deploy->do_bundle_initramfs
undefined
undefined



do_populate_sysroot
undefined
do_populate_sysroot




do_deploy->do_populate_sysroot
undefined
undefined



do_assemble_fitimage
undefined
do_assemble_fitimage




do_compile
undefined
do_compile




do_assemble_fitimage->do_compile
undefined
undefined



do_kernel_generate_rsa_keys
undefined
do_kernel_generate_rsa_keys




do_assemble_fitimage->do_kernel_generate_rsa_keys
undefined
undefined



do_kernel_generate_rsa_keys->do_compile
undefined
undefined



core-image-minimal.do_image_complete
undefined
core-image-minimal.do_image_complete




do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_assemble_fitimage_initramfs->do_bundle_initramfs
undefined
undefined



do_bundle_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_install
undefined
do_install




do_bundle_initramfs->do_install
undefined
undefined



do_install->do_assemble_fitimage
undefined
undefined



do_install->do_compile
undefined
undefined



do_populate_sysroot->do_install
undefined
undefined



branch: adrianf/kernel-fitimage-sstate 





depends
undefined


core-image-minimal.do_build
undefined
core-image-minimal.do_build




do_deploy
undefined
do_deploy




core-image-minimal.do_build->do_deploy
undefined
undefined



do_deploy_fitimage_unbundled
undefined
do_deploy_fitimage_unbundled




core-image-minimal.do_build->do_deploy_fitimage_unbundled
undefined
undefined



core-image-minimal.do_image_complete
undefined
core-image-minimal.do_image_complete




do_deploy->core-image-minimal.do_image_complete
undefined
undefined



do_install
undefined
do_install




do_deploy->do_install
undefined
undefined



do_populate_sysroot
undefined
do_populate_sysroot




do_deploy->do_populate_sysroot
undefined
undefined



do_deploy_fitimage_unbundled->do_deploy
undefined
undefined



do_deploy_fitimage_unbundled->core-image-minimal.do_image_complete
undefined
undefined



do_assemble_fitimage
undefined
do_assemble_fitimage




do_compile
undefined
do_compile




do_assemble_fitimage->do_compile
undefined
undefined



do_kernel_generate_rsa_keys
undefined
do_kernel_generate_rsa_keys




do_assemble_fitimage->do_kernel_generate_rsa_keys
undefined
undefined



do_kernel_generate_rsa_keys->do_compile
undefined
undefined



do_install->do_assemble_fitimage
undefined
undefined



do_install->do_compile
undefined
undefined



do_populate_sysroot->do_install
undefined
undefined



Diff 

2a3
> "core-image-minimal.do_build" -> "do_deploy_fitimage_unbundled"
6,11d6
< "do_assemble_fitimage_initramfs"
[label="do_assemble_fitimage_initramfs"]
< "do_assemble_fitimage_initramfs" -> "core-image-
minimal.do_image_complete"
< "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs"
< "do_bundle_initramfs" [label="do_bundle_initramfs"]
< "do_bundle_initramfs" -> "core-image-minimal.do_image_complete"
< "do_bundle_initramfs" -> "do_install"
14,15c9,10
< "do_deploy" -> "do_assemble_fitimage_initramfs"
< "do_deploy" -> "do_bundle_initramfs"
---
> "do_deploy" -> "core-image-minimal.do_image_complete"
> "do_deploy" -> "do_install"
16a12,14
> "do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"]
> "do_deploy_fitimage_unbundled" -> "core-image-
minimal.do_image_complete"
> "do_deploy_fitimage_unbundled" -> "do_deploy"
Configuration: image, fitimage, initramfs, bundled 

KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
INITRAMFS_IMAGE_BUNDLE = "1"
branch: master 





depends
undefined


core-image-minimal.do_build
undefined
core-image-minimal.do_build




do_deploy
undefined
do_deploy




core-image-minimal.do_build->do_deploy
undefined
undefined



do_assemble_fitimage_initramfs
undefined
do_assemble_fitimage_initramfs




do_deploy->do_assemble_fitimage_initramfs
undefined
undefined



do_bundle_initramfs
undefined
do_bundle_initramfs




do_deploy->do_bundle_initramfs
undefined
undefined



do_populate_sysroot
undefined
do_populate_sysroot




do_deploy->do_populate_sysroot
undefined
undefined



do_transform_bundled_initramfs
undefined
do_transform_bundled_initramfs




do_deploy->do_transform_bundled_initramfs
undefined
undefined



do_assemble_fitimage
undefined
do_assemble_fitimage




do_compile
undefined
do_compile




do_assemble_fitimage->do_compile
undefined
undefined



do_kernel_generate_rsa_keys
undefined
do_kernel_generate_rsa_keys




do_assemble_fitimage->do_kernel_generate_rsa_keys
undefined
undefined



do_kernel_generate_rsa_keys->do_compile
undefined
undefined



core-image-minimal.do_image_complete
undefined
core-image-minimal.do_image_complete




do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_assemble_fitimage_initramfs->do_bundle_initramfs
undefined
undefined



do_bundle_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_install
undefined
do_install




do_bundle_initramfs->do_install
undefined
undefined



do_install->do_assemble_fitimage
undefined
undefined



do_install->do_compile
undefined
undefined



do_populate_sysroot->do_install
undefined
undefined



do_transform_bundled_initramfs->do_bundle_initramfs
undefined
undefined



branch: adrianf/kernel-fitimage-sstate 





depends
undefined


core-image-minimal.do_build
undefined
core-image-minimal.do_build




do_deploy
undefined
do_deploy




core-image-minimal.do_build->do_deploy
undefined
undefined



do_assemble_fitimage_initramfs
undefined
do_assemble_fitimage_initramfs




do_deploy->do_assemble_fitimage_initramfs
undefined
undefined



do_bundle_initramfs
undefined
do_bundle_initramfs




do_deploy->do_bundle_initramfs
undefined
undefined



do_populate_sysroot
undefined
do_populate_sysroot




do_deploy->do_populate_sysroot
undefined
undefined



do_transform_bundled_initramfs
undefined
do_transform_bundled_initramfs




do_deploy->do_transform_bundled_initramfs
undefined
undefined



do_assemble_fitimage
undefined
do_assemble_fitimage




do_compile
undefined
do_compile




do_assemble_fitimage->do_compile
undefined
undefined



do_kernel_generate_rsa_keys
undefined
do_kernel_generate_rsa_keys




do_assemble_fitimage->do_kernel_generate_rsa_keys
undefined
undefined



do_kernel_generate_rsa_keys->do_compile
undefined
undefined



core-image-minimal.do_image_complete
undefined
core-image-minimal.do_image_complete




do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_assemble_fitimage_initramfs->do_bundle_initramfs
undefined
undefined



do_bundle_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_install
undefined
do_install




do_bundle_initramfs->do_install
undefined
undefined



do_install->do_assemble_fitimage
undefined
undefined



do_install->do_compile
undefined
undefined



do_populate_sysroot->do_install
undefined
undefined



do_transform_bundled_initramfs->do_bundle_initramfs
undefined
undefined



Diff 

Configuration: image, fitimage 

KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
branch: master 





depends
undefined


core-image-minimal.do_build
undefined
core-image-minimal.do_build




do_deploy
undefined
do_deploy




core-image-minimal.do_build->do_deploy
undefined
undefined



do_assemble_fitimage_initramfs
undefined
do_assemble_fitimage_initramfs




do_deploy->do_assemble_fitimage_initramfs
undefined
undefined



do_bundle_initramfs
undefined
do_bundle_initramfs




do_deploy->do_bundle_initramfs
undefined
undefined



do_populate_sysroot
undefined
do_populate_sysroot




do_deploy->do_populate_sysroot
undefined
undefined



do_assemble_fitimage
undefined
do_assemble_fitimage




do_compile
undefined
do_compile




do_assemble_fitimage->do_compile
undefined
undefined



do_kernel_generate_rsa_keys
undefined
do_kernel_generate_rsa_keys




do_assemble_fitimage->do_kernel_generate_rsa_keys
undefined
undefined



do_kernel_generate_rsa_keys->do_compile
undefined
undefined



core-image-minimal.do_image_complete
undefined
core-image-minimal.do_image_complete




do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_assemble_fitimage_initramfs->do_bundle_initramfs
undefined
undefined



do_bundle_initramfs->core-image-minimal.do_image_complete
undefined
undefined



do_install
undefined
do_install




do_bundle_initramfs->do_install
undefined
undefined



do_install->do_assemble_fitimage
undefined
undefined



do_install->do_compile
undefined
undefined



do_populate_sysroot->do_install
undefined
undefined



branch: adrianf/kernel-fitimage-sstate 





depends
undefined


core-image-minimal.do_build
undefined
core-image-minimal.do_build




do_deploy
undefined
do_deploy




core-image-minimal.do_build->do_deploy
undefined
undefined



do_deploy_fitimage_unbundled
undefined
do_deploy_fitimage_unbundled




core-image-minimal.do_build->do_deploy_fitimage_unbundled
undefined
undefined



core-image-minimal.do_image_complete
undefined
core-image-minimal.do_image_complete




do_deploy->core-image-minimal.do_image_complete
undefined
undefined



do_install
undefined
do_install




do_deploy->do_install
undefined
undefined



do_populate_sysroot
undefined
do_populate_sysroot




do_deploy->do_populate_sysroot
undefined
undefined



do_deploy_fitimage_unbundled->do_deploy
undefined
undefined



do_deploy_fitimage_unbundled->core-image-minimal.do_image_complete
undefined
undefined



do_assemble_fitimage
undefined
do_assemble_fitimage




do_compile
undefined
do_compile




do_assemble_fitimage->do_compile
undefined
undefined



do_kernel_generate_rsa_keys
undefined
do_kernel_generate_rsa_keys




do_assemble_fitimage->do_kernel_generate_rsa_keys
undefined
undefined



do_kernel_generate_rsa_keys->do_compile
undefined
undefined



do_install->do_assemble_fitimage
undefined
undefined



do_install->do_compile
undefined
undefined



do_populate_sysroot->do_install
undefined
undefined



Diff 

2a3
> "core-image-minimal.do_build" -> "do_deploy_fitimage_unbundled"
6,11d6
< "do_assemble_fitimage_initramfs"
[label="do_assemble_fitimage_initramfs"]
< "do_assemble_fitimage_initramfs" -> "core-image-
minimal.do_image_complete"
< "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs"
< "do_bundle_initramfs" [label="do_bundle_initramfs"]
< "do_bundle_initramfs" -> "core-image-minimal.do_image_complete"
< "do_bundle_initramfs" -> "do_install"
14,15c9,10
< "do_deploy" -> "do_assemble_fitimage_initramfs"
< "do_deploy" -> "do_bundle_initramfs"
---
> "do_deploy" -> "core-image-minimal.do_image_complete"
> "do_deploy" -> "do_install"
16a12,14
> "do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"]
> "do_deploy_fitimage_unbundled" -> "core-image-
minimal.do_image_complete"
> "do_deploy_fitimage_unbundled" -> "do_deploy"
Martin Jansa July 15, 2024, 2:32 p.m. UTC | #2
Doesn't it still rebuild from scratch when IMAGE_VERSION_SUFFIX changes?

To be able to re-use the kernel from sstate even when
IMAGE_VERSION_SUFFIX changes I've implemented:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
which was unfortunately rejected due to lack of interest from other
people outside LGE, but maybe you could take advantage of that as
well.

On Mon, Jul 15, 2024 at 4:22 PM Adrian Freihofer via
lists.openembedded.org
<adrian.freihofer=gmail.com@lists.openembedded.org> wrote:
>
> Task dependeny changes
>
> Configuration: image, fitimage, initramfs, unbundled
>
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal-initramfs"
>
> branch: master
>
> core-image-minimal.do_build do_deploy do_assemble_fitimage_initramfs do_bundle_initramfs do_populate_sysroot do_assemble_fitimage do_compile do_kernel_generate_rsa_keys core-image-minimal.do_image_complete do_install
>
> branch: adrianf/kernel-fitimage-sstate
>
> core-image-minimal.do_build do_deploy do_deploy_fitimage_unbundled core-image-minimal.do_image_complete do_install do_populate_sysroot do_assemble_fitimage do_compile do_kernel_generate_rsa_keys
>
> Diff
>
> 2a3
> > "core-image-minimal.do_build" -> "do_deploy_fitimage_unbundled"
> 6,11d6
> < "do_assemble_fitimage_initramfs" [label="do_assemble_fitimage_initramfs"]
> < "do_assemble_fitimage_initramfs" -> "core-image-minimal.do_image_complete"
> < "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs"
> < "do_bundle_initramfs" [label="do_bundle_initramfs"]
> < "do_bundle_initramfs" -> "core-image-minimal.do_image_complete"
> < "do_bundle_initramfs" -> "do_install"
> 14,15c9,10
> < "do_deploy" -> "do_assemble_fitimage_initramfs"
> < "do_deploy" -> "do_bundle_initramfs"
> ---
> > "do_deploy" -> "core-image-minimal.do_image_complete"
> > "do_deploy" -> "do_install"
> 16a12,14
> > "do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"]
> > "do_deploy_fitimage_unbundled" -> "core-image-minimal.do_image_complete"
> > "do_deploy_fitimage_unbundled" -> "do_deploy"
>
> Configuration: fitimage, initramfs, unbundled
>
> KERNEL_IMAGETYPE:forcevariable = "fitImage"
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal-initramfs"
>
> branch: master
>
> core-image-minimal.do_build do_deploy do_assemble_fitimage_initramfs do_bundle_initramfs do_populate_sysroot do_assemble_fitimage do_compile do_kernel_generate_rsa_keys core-image-minimal.do_image_complete do_install
>
> branch: adrianf/kernel-fitimage-sstate
>
> core-image-minimal.do_build do_deploy do_deploy_fitimage_unbundled core-image-minimal.do_image_complete do_install do_populate_sysroot do_assemble_fitimage do_compile do_kernel_generate_rsa_keys
>
> Diff
>
> 2a3
> > "core-image-minimal.do_build" -> "do_deploy_fitimage_unbundled"
> 6,11d6
> < "do_assemble_fitimage_initramfs" [label="do_assemble_fitimage_initramfs"]
> < "do_assemble_fitimage_initramfs" -> "core-image-minimal.do_image_complete"
> < "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs"
> < "do_bundle_initramfs" [label="do_bundle_initramfs"]
> < "do_bundle_initramfs" -> "core-image-minimal.do_image_complete"
> < "do_bundle_initramfs" -> "do_install"
> 14,15c9,10
> < "do_deploy" -> "do_assemble_fitimage_initramfs"
> < "do_deploy" -> "do_bundle_initramfs"
> ---
> > "do_deploy" -> "core-image-minimal.do_image_complete"
> > "do_deploy" -> "do_install"
> 16a12,14
> > "do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"]
> > "do_deploy_fitimage_unbundled" -> "core-image-minimal.do_image_complete"
> > "do_deploy_fitimage_unbundled" -> "do_deploy"
>
> Configuration: image, fitimage, initramfs, bundled
>
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal-initramfs"
> INITRAMFS_IMAGE_BUNDLE = "1"
>
> branch: master
>
> core-image-minimal.do_build do_deploy do_assemble_fitimage_initramfs do_bundle_initramfs do_populate_sysroot do_transform_bundled_initramfs do_assemble_fitimage do_compile do_kernel_generate_rsa_keys core-image-minimal.do_image_complete do_install
>
> branch: adrianf/kernel-fitimage-sstate
>
> core-image-minimal.do_build do_deploy do_assemble_fitimage_initramfs do_bundle_initramfs do_populate_sysroot do_transform_bundled_initramfs do_assemble_fitimage do_compile do_kernel_generate_rsa_keys core-image-minimal.do_image_complete do_install
>
> Diff
>
> Configuration: image, fitimage
>
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
>
> branch: master
>
> core-image-minimal.do_build do_deploy do_assemble_fitimage_initramfs do_bundle_initramfs do_populate_sysroot do_assemble_fitimage do_compile do_kernel_generate_rsa_keys core-image-minimal.do_image_complete do_install
>
> branch: adrianf/kernel-fitimage-sstate
>
> core-image-minimal.do_build do_deploy do_deploy_fitimage_unbundled core-image-minimal.do_image_complete do_install do_populate_sysroot do_assemble_fitimage do_compile do_kernel_generate_rsa_keys
>
> Diff
>
> 2a3
> > "core-image-minimal.do_build" -> "do_deploy_fitimage_unbundled"
> 6,11d6
> < "do_assemble_fitimage_initramfs" [label="do_assemble_fitimage_initramfs"]
> < "do_assemble_fitimage_initramfs" -> "core-image-minimal.do_image_complete"
> < "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs"
> < "do_bundle_initramfs" [label="do_bundle_initramfs"]
> < "do_bundle_initramfs" -> "core-image-minimal.do_image_complete"
> < "do_bundle_initramfs" -> "do_install"
> 14,15c9,10
> < "do_deploy" -> "do_assemble_fitimage_initramfs"
> < "do_deploy" -> "do_bundle_initramfs"
> ---
> > "do_deploy" -> "core-image-minimal.do_image_complete"
> > "do_deploy" -> "do_install"
> 16a12,14
> > "do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"]
> > "do_deploy_fitimage_unbundled" -> "core-image-minimal.do_image_complete"
> > "do_deploy_fitimage_unbundled" -> "do_deploy"
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#201933): https://lists.openembedded.org/g/openembedded-core/message/201933
> Mute This Topic: https://lists.openembedded.org/mt/107231736/3617156
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [martin.jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Adrian Freihofer July 18, 2024, 8 a.m. UTC | #3
Hi Martin

Thank you for looking at my patches.

On Mon, 2024-07-15 at 16:32 +0200, Martin Jansa wrote:
> Doesn't it still rebuild from scratch when IMAGE_VERSION_SUFFIX
> changes?

I wonder why this happens in WebOS. I think that this line
kernel_do_deploy[vardepsexclude] = "DATETIME"
prevents such unnecessary rebuilds.

So far we haven't had any problems with rebuilding kernels. The trouble
started when we started using fitImages with initramfs for some
devices. It is also worth noting that we build from sstate, but with an
empty TMPDIR.

In such a scenario bitbake does:

 * The initramfs gets assembled. This is expected because images are
   not sstate cached. This runs quickly because all the packages which
   are required to build the initramfs come from sstate-cache.
 * Since the initramfs changes, the fitImage which includes it must be
   re-assembled as well.
 * Now the hassle starts: The do_assemble_fitimage_initramfs needs the
   kernel binaries (linux.bin, DTBs...) which are not available from
   sttate. Bitbake has to start with kernel do_fetch in case of an
   empty TMPDIR just to get the fitImage assembled.

With my patches the kernel binaries used for the fitImage come from
sstate (if the initramfs is not bundled).

I tested this like that:

cat build/conf/local.conf

KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "and RAM disk
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal"
IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
UBOOT_RD_LOADADDRESS = "0x88000000"
UBOOT_RD_ENTRYPOINT = "0x88000000"
UBOOT_LOADADDRESS = "0x80080000"
UBOOT_ENTRYPOINT = "0x80080000"
FIT_DESC = "A model description"

# Allow to change the initramfs
FOO_VAR = "1"
INHERIT += "image-buildinfo"
IMAGE_BUILDINFO_VARS:append = " FOO_VAR"


bitbake virtual/kernel

mv build/tmp build/tmp-old-1

bitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimal do_rootfs ...
- kernel do_deploy (from sstate)
- kernel do_deploy_fitimage_unbundled


bitbake virtual/kernel
-> bitbake does: nothing


# Enforce a re-build of the initramfs
mv build/tmp build/tmp-old-2
echo 'FOO_VAR = "1"' >> build/conf/local.conf

bitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimal do_rootfs ...
- kernel do_deploy (from sstate)
- kernel do_deploy_fitimage_unbundled


bitbake virtual/kernel
-> bitbake does: nothing


diff \
<(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
<(ls build/tmp/deploy/images/qemux86-64/ -1)
4,10c4,10
< core-image-minimal-qemux86-64-20240718065105.cpio.gz
< core-image-minimal-qemux86-64-20240718065105.ext4
< core-image-minimal-qemux86-64-20240718065105.manifest
< core-image-minimal-qemux86-64-20240718065105.qemuboot.conf
< core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst
< core-image-minimal-qemux86-64-20240718065105.tar.bz2
< core-image-minimal-qemux86-64-20240718065105.testdata.json
---
> core-image-minimal-qemux86-64-20240718073341.cpio.gz
> core-image-minimal-qemux86-64-20240718073341.ext4
> core-image-minimal-qemux86-64-20240718073341.manifest
> core-image-minimal-qemux86-64-20240718073341.qemuboot.conf
> core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst
> core-image-minimal-qemux86-64-20240718073341.tar.bz2
> core-image-minimal-qemux86-64-20240718073341.testdata.json
20c20
< fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718065105.bin
---
> fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.bin
22c22
< fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718065105.its
---
> fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.its



The answer is then: No it does not re-build. At least not in the
scenario which I would like to fix it, it does not rebuild from
scratch.

Does that make sense?

Regards,
Adrian

> > 
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#201933):
> > https://lists.openembedded.org/g/openembedded-core/message/201933
> > Mute This Topic:
> > https://lists.openembedded.org/mt/107231736/3617156
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe:
> > https://lists.openembedded.org/g/openembedded-core/unsub [
> > martin.jansa@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
Martin Jansa July 18, 2024, 8:39 a.m. UTC | #4
On Thu, Jul 18, 2024 at 10:00 AM Adrian Freihofer
<adrian.freihofer@gmail.com> wrote:
>
> Hi Martin
>
> Thank you for looking at my patches.
>
> On Mon, 2024-07-15 at 16:32 +0200, Martin Jansa wrote:
> > Doesn't it still rebuild from scratch when IMAGE_VERSION_SUFFIX
> > changes?
>
> I wonder why this happens in WebOS. I think that this line
> kernel_do_deploy[vardepsexclude] = "DATETIME"
> prevents such unnecessary rebuilds.

Hi Adrian,

yes DATETIME is excluded by default, but it's useful to use different
suffix in IMAGE_VERSION_SUFFIX (e.g. BUILD_NUMBER from CI on jenkins
or release version when building release) and in such cases you don't
want to vardepexclude it, because it's useful to produce matching
version in images as well as kernel (and other artifacts you might
have), so your 1.2.1 release images doesn't end with 1.2.0 kernel
artifacts just because kernel sstate signature didn't change in 1.2.1
bugfix release.

Does this make sense? I bet it would happen in your builds as well.

Whole point of:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
is to allow "renaming" (adding more hardlinks) artifacts with matching
IMAGE_VERSION_SUFFIX without the need of rebuilding anything (e.g.
version-less kernel artifacts are re-used from sstate and then only
quick deploy-links task is executed to add right versioned hardlinks
in deploy directory.)

Cheers,

> So far we haven't had any problems with rebuilding kernels. The trouble
> started when we started using fitImages with initramfs for some
> devices. It is also worth noting that we build from sstate, but with an
> empty TMPDIR.
>
> In such a scenario bitbake does:
>
>  * The initramfs gets assembled. This is expected because images are
>    not sstate cached. This runs quickly because all the packages which
>    are required to build the initramfs come from sstate-cache.
>  * Since the initramfs changes, the fitImage which includes it must be
>    re-assembled as well.
>  * Now the hassle starts: The do_assemble_fitimage_initramfs needs the
>    kernel binaries (linux.bin, DTBs...) which are not available from
>    sttate. Bitbake has to start with kernel do_fetch in case of an
>    empty TMPDIR just to get the fitImage assembled.
>
> With my patches the kernel binaries used for the fitImage come from
> sstate (if the initramfs is not bundled).
>
> I tested this like that:
>
> cat build/conf/local.conf
>
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "and RAM disk
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal"
> IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
> UBOOT_RD_LOADADDRESS = "0x88000000"
> UBOOT_RD_ENTRYPOINT = "0x88000000"
> UBOOT_LOADADDRESS = "0x80080000"
> UBOOT_ENTRYPOINT = "0x80080000"
> FIT_DESC = "A model description"
>
> # Allow to change the initramfs
> FOO_VAR = "1"
> INHERIT += "image-buildinfo"
> IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
>
>
> bitbake virtual/kernel
>
> mv build/tmp build/tmp-old-1
>
> bitbake virtual/kernel
> -> bitbake does:
> - setscene tasks
> - core-image-minimal do_rootfs ...
> - kernel do_deploy (from sstate)
> - kernel do_deploy_fitimage_unbundled
>
>
> bitbake virtual/kernel
> -> bitbake does: nothing
>
>
> # Enforce a re-build of the initramfs
> mv build/tmp build/tmp-old-2
> echo 'FOO_VAR = "1"' >> build/conf/local.conf
>
> bitbake virtual/kernel
> -> bitbake does:
> - setscene tasks
> - core-image-minimal do_rootfs ...
> - kernel do_deploy (from sstate)
> - kernel do_deploy_fitimage_unbundled
>
>
> bitbake virtual/kernel
> -> bitbake does: nothing
>
>
> diff \
> <(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
> <(ls build/tmp/deploy/images/qemux86-64/ -1)
> 4,10c4,10
> < core-image-minimal-qemux86-64-20240718065105.cpio.gz
> < core-image-minimal-qemux86-64-20240718065105.ext4
> < core-image-minimal-qemux86-64-20240718065105.manifest
> < core-image-minimal-qemux86-64-20240718065105.qemuboot.conf
> < core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst
> < core-image-minimal-qemux86-64-20240718065105.tar.bz2
> < core-image-minimal-qemux86-64-20240718065105.testdata.json
> ---
> > core-image-minimal-qemux86-64-20240718073341.cpio.gz
> > core-image-minimal-qemux86-64-20240718073341.ext4
> > core-image-minimal-qemux86-64-20240718073341.manifest
> > core-image-minimal-qemux86-64-20240718073341.qemuboot.conf
> > core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst
> > core-image-minimal-qemux86-64-20240718073341.tar.bz2
> > core-image-minimal-qemux86-64-20240718073341.testdata.json
> 20c20
> < fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> 20240718065105.bin
> ---
> > fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> 20240718073341.bin
> 22c22
> < fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> 20240718065105.its
> ---
> > fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> 20240718073341.its
>
>
>
> The answer is then: No it does not re-build. At least not in the
> scenario which I would like to fix it, it does not rebuild from
> scratch.
>
> Does that make sense?
>
> Regards,
> Adrian
>
> > >
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > Links: You receive all messages sent to this group.
> > > View/Reply Online (#201933):
> > > https://lists.openembedded.org/g/openembedded-core/message/201933
> > > Mute This Topic:
> > > https://lists.openembedded.org/mt/107231736/3617156
> > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > Unsubscribe:
> > > https://lists.openembedded.org/g/openembedded-core/unsub [
> > > martin.jansa@gmail.com]
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > >
>
Jose Quaresma July 18, 2024, 10:28 a.m. UTC | #5
Adrian Freihofer via lists.openembedded.org <adrian.freihofer=
gmail.com@lists.openembedded.org> escreveu (quinta, 18/07/2024 à(s) 09:00):

> Hi Martin
>
> Thank you for looking at my patches.
>
> On Mon, 2024-07-15 at 16:32 +0200, Martin Jansa wrote:
> > Doesn't it still rebuild from scratch when IMAGE_VERSION_SUFFIX
> > changes?
>
> I wonder why this happens in WebOS. I think that this line
> kernel_do_deploy[vardepsexclude] = "DATETIME"
> prevents such unnecessary rebuilds.
>
> So far we haven't had any problems with rebuilding kernels. The trouble
> started when we started using fitImages with initramfs for some
> devices. It is also worth noting that we build from sstate, but with an
> empty TMPDIR.
>
> In such a scenario bitbake does:
>
>  * The initramfs gets assembled. This is expected because images are
>    not sstate cached. This runs quickly because all the packages which
>    are required to build the initramfs come from sstate-cache.
>  * Since the initramfs changes, the fitImage which includes it must be
>    re-assembled as well.
>

I have faced this problem and sent a patch [1] to support storing the full
images on the sstate.
As it was not a tested and widely used case, it was rejected.
We continue using this in our distro but without the patch we need to
change more parts on our side.

[1] https://lists.openembedded.org/g/openembedded-core/message/184243


>  * Now the hassle starts: The do_assemble_fitimage_initramfs needs the
>    kernel binaries (linux.bin, DTBs...) which are not available from
>    sttate. Bitbake has to start with kernel do_fetch in case of an
>    empty TMPDIR just to get the fitImage assembled.
>
> With my patches the kernel binaries used for the fitImage come from
> sstate (if the initramfs is not bundled).
>

Anyway your overwall patch set is still a good improvement to not
rebuilding the kernel.
That was my concern also and the main reason for my patch.

Jose


>
> I tested this like that:
>
> cat build/conf/local.conf
>
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "and RAM disk
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal"
> IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
> UBOOT_RD_LOADADDRESS = "0x88000000"
> UBOOT_RD_ENTRYPOINT = "0x88000000"
> UBOOT_LOADADDRESS = "0x80080000"
> UBOOT_ENTRYPOINT = "0x80080000"
> FIT_DESC = "A model description"
>
> # Allow to change the initramfs
> FOO_VAR = "1"
> INHERIT += "image-buildinfo"
> IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
>
>
> bitbake virtual/kernel
>
> mv build/tmp build/tmp-old-1
>
> bitbake virtual/kernel
> -> bitbake does:
> - setscene tasks
> - core-image-minimal do_rootfs ...
> - kernel do_deploy (from sstate)
> - kernel do_deploy_fitimage_unbundled
>
>
> bitbake virtual/kernel
> -> bitbake does: nothing
>
>
> # Enforce a re-build of the initramfs
> mv build/tmp build/tmp-old-2
> echo 'FOO_VAR = "1"' >> build/conf/local.conf
>
> bitbake virtual/kernel
> -> bitbake does:
> - setscene tasks
> - core-image-minimal do_rootfs ...
> - kernel do_deploy (from sstate)
> - kernel do_deploy_fitimage_unbundled
>
>
> bitbake virtual/kernel
> -> bitbake does: nothing
>
>
> diff \
> <(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
> <(ls build/tmp/deploy/images/qemux86-64/ -1)
> 4,10c4,10
> < core-image-minimal-qemux86-64-20240718065105.cpio.gz
> < core-image-minimal-qemux86-64-20240718065105.ext4
> < core-image-minimal-qemux86-64-20240718065105.manifest
> < core-image-minimal-qemux86-64-20240718065105.qemuboot.conf
> < core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst
> < core-image-minimal-qemux86-64-20240718065105.tar.bz2
> < core-image-minimal-qemux86-64-20240718065105.testdata.json
> ---
> > core-image-minimal-qemux86-64-20240718073341.cpio.gz
> > core-image-minimal-qemux86-64-20240718073341.ext4
> > core-image-minimal-qemux86-64-20240718073341.manifest
> > core-image-minimal-qemux86-64-20240718073341.qemuboot.conf
> > core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst
> > core-image-minimal-qemux86-64-20240718073341.tar.bz2
> > core-image-minimal-qemux86-64-20240718073341.testdata.json
> 20c20
> < fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> 20240718065105.bin
> ---
> > fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> 20240718073341.bin
> 22c22
> < fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> 20240718065105.its
> ---
> > fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> 20240718073341.its
>
>
>
> The answer is then: No it does not re-build. At least not in the
> scenario which I would like to fix it, it does not rebuild from
> scratch.
>
> Does that make sense?
>
> Regards,
> Adrian
>
> > >
> > >
> > >
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#202191):
> https://lists.openembedded.org/g/openembedded-core/message/202191
> Mute This Topic: https://lists.openembedded.org/mt/107231736/5052612
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> quaresma.jose@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
Adrian Freihofer July 18, 2024, 12:37 p.m. UTC | #6
On Thu, 2024-07-18 at 10:39 +0200, Martin Jansa wrote:
> On Thu, Jul 18, 2024 at 10:00 AM Adrian Freihofer
> <adrian.freihofer@gmail.com> wrote:
> > 
> > Hi Martin
> > 
> > Thank you for looking at my patches.
> > 
> > On Mon, 2024-07-15 at 16:32 +0200, Martin Jansa wrote:
> > > Doesn't it still rebuild from scratch when IMAGE_VERSION_SUFFIX
> > > changes?
> > 
> > I wonder why this happens in WebOS. I think that this line
> > kernel_do_deploy[vardepsexclude] = "DATETIME"
> > prevents such unnecessary rebuilds.
> 
> Hi Adrian,
> 
> yes DATETIME is excluded by default, but it's useful to use different
> suffix in IMAGE_VERSION_SUFFIX (e.g. BUILD_NUMBER from CI on jenkins
> or release version when building release) and in such cases you don't
> want to vardepexclude it, because it's useful to produce matching
> version in images as well as kernel (and other artifacts you might
> have), so your 1.2.1 release images doesn't end with 1.2.0 kernel
> artifacts just because kernel sstate signature didn't change in 1.2.1
> bugfix release.
> 
> Does this make sense? I bet it would happen in your builds as well.

I try to optimize everything towards binary-reproducible builds. This
means that we remove all kinds of variables that change because the
time changes or because the CI build ID changes or something like that.
I see no benefit in compiling such information into the firmware.
But the problem I see is that such ideas clash with efficient and
binary-reproducible builds.

That means that I don't see a reason for changing the
IMAGE_VERSION_SUFFIX variable in such a way.

> 
> Whole point of:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
> is to allow "renaming" (adding more hardlinks) artifacts with
> matching
> IMAGE_VERSION_SUFFIX without the need of rebuilding anything (e.g.
> version-less kernel artifacts are re-used from sstate and then only
> quick deploy-links task is executed to add right versioned hardlinks
> in deploy directory.)
> 

Well, yes, it would. But, I am confident that it is a better strategy
to simply not do this than to support it.

If such non-reproducible information is needed anyway, it can be
archived independently of Bitbake.

Adrian

> Cheers,
> 
> > So far we haven't had any problems with rebuilding kernels. The
> > trouble
> > started when we started using fitImages with initramfs for some
> > devices. It is also worth noting that we build from sstate, but
> > with an
> > empty TMPDIR.
> > 
> > In such a scenario bitbake does:
> > 
> >  * The initramfs gets assembled. This is expected because images
> > are
> >    not sstate cached. This runs quickly because all the packages
> > which
> >    are required to build the initramfs come from sstate-cache.
> >  * Since the initramfs changes, the fitImage which includes it must
> > be
> >    re-assembled as well.
> >  * Now the hassle starts: The do_assemble_fitimage_initramfs needs
> > the
> >    kernel binaries (linux.bin, DTBs...) which are not available
> > from
> >    sttate. Bitbake has to start with kernel do_fetch in case of an
> >    empty TMPDIR just to get the fitImage assembled.
> > 
> > With my patches the kernel binaries used for the fitImage come from
> > sstate (if the initramfs is not bundled).
> > 
> > I tested this like that:
> > 
> > cat build/conf/local.conf
> > 
> > KERNEL_IMAGETYPE = "Image"
> > KERNEL_IMAGETYPES += " fitImage "
> > KERNEL_CLASSES = " kernel-fitimage "and RAM disk
> > IMAGE_FSTYPES += "cpio.gz"
> > INITRAMFS_IMAGE = "core-image-minimal"
> > IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
> > UBOOT_RD_LOADADDRESS = "0x88000000"
> > UBOOT_RD_ENTRYPOINT = "0x88000000"
> > UBOOT_LOADADDRESS = "0x80080000"
> > UBOOT_ENTRYPOINT = "0x80080000"
> > FIT_DESC = "A model description"
> > 
> > # Allow to change the initramfs
> > FOO_VAR = "1"
> > INHERIT += "image-buildinfo"
> > IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
> > 
> > 
> > bitbake virtual/kernel
> > 
> > mv build/tmp build/tmp-old-1
> > 
> > bitbake virtual/kernel
> > -> bitbake does:
> > - setscene tasks
> > - core-image-minimal do_rootfs ...
> > - kernel do_deploy (from sstate)
> > - kernel do_deploy_fitimage_unbundled
> > 
> > 
> > bitbake virtual/kernel
> > -> bitbake does: nothing
> > 
> > 
> > # Enforce a re-build of the initramfs
> > mv build/tmp build/tmp-old-2
> > echo 'FOO_VAR = "1"' >> build/conf/local.conf
> > 
> > bitbake virtual/kernel
> > -> bitbake does:
> > - setscene tasks
> > - core-image-minimal do_rootfs ...
> > - kernel do_deploy (from sstate)
> > - kernel do_deploy_fitimage_unbundled
> > 
> > 
> > bitbake virtual/kernel
> > -> bitbake does: nothing
> > 
> > 
> > diff \
> > <(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
> > <(ls build/tmp/deploy/images/qemux86-64/ -1)
> > 4,10c4,10
> > < core-image-minimal-qemux86-64-20240718065105.cpio.gz
> > < core-image-minimal-qemux86-64-20240718065105.ext4
> > < core-image-minimal-qemux86-64-20240718065105.manifest
> > < core-image-minimal-qemux86-64-20240718065105.qemuboot.conf
> > < core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst
> > < core-image-minimal-qemux86-64-20240718065105.tar.bz2
> > < core-image-minimal-qemux86-64-20240718065105.testdata.json
> > ---
> > > core-image-minimal-qemux86-64-20240718073341.cpio.gz
> > > core-image-minimal-qemux86-64-20240718073341.ext4
> > > core-image-minimal-qemux86-64-20240718073341.manifest
> > > core-image-minimal-qemux86-64-20240718073341.qemuboot.conf
> > > core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst
> > > core-image-minimal-qemux86-64-20240718073341.tar.bz2
> > > core-image-minimal-qemux86-64-20240718073341.testdata.json
> > 20c20
> > < fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> > 20240718065105.bin
> > ---
> > > fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> > 20240718073341.bin
> > 22c22
> > < fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-
> > qemux86-64-
> > 20240718065105.its
> > ---
> > > fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-
> > > qemux86-64-
> > 20240718073341.its
> > 
> > 
> > 
> > The answer is then: No it does not re-build. At least not in the
> > scenario which I would like to fix it, it does not rebuild from
> > scratch.
> > 
> > Does that make sense?
> > 
> > Regards,
> > Adrian
> > 
> > > > 
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > Links: You receive all messages sent to this group.
> > > > View/Reply Online (#201933):
> > > > https://lists.openembedded.org/g/openembedded-core/message/201933
> > > > Mute This Topic:
> > > > https://lists.openembedded.org/mt/107231736/3617156
> > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > Unsubscribe:
> > > > https://lists.openembedded.org/g/openembedded-core/unsub [
> > > > martin.jansa@gmail.com]
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > 
> >
Adrian Freihofer July 18, 2024, 12:44 p.m. UTC | #7
On Thu, 2024-07-18 at 11:28 +0100, Jose Quaresma wrote:
> 
> 
> Adrian Freihofer via lists.openembedded.org
> <adrian.freihofer=gmail.com@lists.openembedded.org> escreveu (quinta,
> 18/07/2024 à(s) 09:00):
> > Hi Martin
> > 
> > Thank you for looking at my patches.
> > 
> > On Mon, 2024-07-15 at 16:32 +0200, Martin Jansa wrote:
> > > Doesn't it still rebuild from scratch when IMAGE_VERSION_SUFFIX
> > > changes?
> > 
> > I wonder why this happens in WebOS. I think that this line
> > kernel_do_deploy[vardepsexclude] = "DATETIME"
> > prevents such unnecessary rebuilds.
> > 
> > So far we haven't had any problems with rebuilding kernels. The
> > trouble
> > started when we started using fitImages with initramfs for some
> > devices. It is also worth noting that we build from sstate, but
> > with an
> > empty TMPDIR.
> > 
> > In such a scenario bitbake does:
> > 
> >  * The initramfs gets assembled. This is expected because images
> > are
> >    not sstate cached. This runs quickly because all the packages
> > which
> >    are required to build the initramfs come from sstate-cache.
> >  * Since the initramfs changes, the fitImage which includes it must
> > be
> >    re-assembled as well.
> > 
> 
> 
> I have faced this problem and sent a patch [1] to support storing the
> full images on the sstate.
> As it was not a tested and widely used case, it was rejected.
> We continue using this in our distro but without the patch we need to
> change more parts on our side.

I don't think adding images to the sstate cache can help much. But it
comes at a high cost. It definitely wouldn't help in the case that
should be optimized with my patches here:
 * If the initramfs changes it needs to be re-built and cannot be taken
   from the sstate-cache anyway.
 * The issue is: If the initramfs changes, the kernel gets re-built.
   What we need is an sstate for the kernel artifacts but not for the
   initramfs artifacts.

Adrian

> 
> [1] https://lists.openembedded.org/g/openembedded-core/message/184243
>  
> >  * Now the hassle starts: The do_assemble_fitimage_initramfs needs
> > the
> >    kernel binaries (linux.bin, DTBs...) which are not available
> > from
> >    sttate. Bitbake has to start with kernel do_fetch in case of an
> >    empty TMPDIR just to get the fitImage assembled.
> > 
> > With my patches the kernel binaries used for the fitImage come from
> > sstate (if the initramfs is not bundled).
> > 
> 
> 
> Anyway your overwall patch set is still a good improvement to not
> rebuilding the kernel.
> That was my concern also and the main reason for my patch.
>  
> Jose
>  
> > 
> > I tested this like that:
> > 
> > cat build/conf/local.conf
> > 
> > KERNEL_IMAGETYPE = "Image"
> > KERNEL_IMAGETYPES += " fitImage "
> > KERNEL_CLASSES = " kernel-fitimage "and RAM disk
> > IMAGE_FSTYPES += "cpio.gz"
> > INITRAMFS_IMAGE = "core-image-minimal"
> > IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
> > UBOOT_RD_LOADADDRESS = "0x88000000"
> > UBOOT_RD_ENTRYPOINT = "0x88000000"
> > UBOOT_LOADADDRESS = "0x80080000"
> > UBOOT_ENTRYPOINT = "0x80080000"
> > FIT_DESC = "A model description"
> > 
> > # Allow to change the initramfs
> > FOO_VAR = "1"
> > INHERIT += "image-buildinfo"
> > IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
> > 
> > 
> > bitbake virtual/kernel
> > 
> > mv build/tmp build/tmp-old-1
> > 
> > bitbake virtual/kernel
> > -> bitbake does:
> > - setscene tasks
> > - core-image-minimal do_rootfs ...
> > - kernel do_deploy (from sstate)
> > - kernel do_deploy_fitimage_unbundled
> > 
> > 
> > bitbake virtual/kernel
> > -> bitbake does: nothing
> > 
> > 
> > # Enforce a re-build of the initramfs
> > mv build/tmp build/tmp-old-2
> > echo 'FOO_VAR = "1"' >> build/conf/local.conf
> > 
> > bitbake virtual/kernel
> > -> bitbake does:
> > - setscene tasks
> > - core-image-minimal do_rootfs ...
> > - kernel do_deploy (from sstate)
> > - kernel do_deploy_fitimage_unbundled
> > 
> > 
> > bitbake virtual/kernel
> > -> bitbake does: nothing
> > 
> > 
> > diff \
> > <(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
> > <(ls build/tmp/deploy/images/qemux86-64/ -1)
> > 4,10c4,10
> > < core-image-minimal-qemux86-64-20240718065105.cpio.gz
> > < core-image-minimal-qemux86-64-20240718065105.ext4
> > < core-image-minimal-qemux86-64-20240718065105.manifest
> > < core-image-minimal-qemux86-64-20240718065105.qemuboot.conf
> > < core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst
> > < core-image-minimal-qemux86-64-20240718065105.tar.bz2
> > < core-image-minimal-qemux86-64-20240718065105.testdata.json
> > ---
> > > core-image-minimal-qemux86-64-20240718073341.cpio.gz
> > > core-image-minimal-qemux86-64-20240718073341.ext4
> > > core-image-minimal-qemux86-64-20240718073341.manifest
> > > core-image-minimal-qemux86-64-20240718073341.qemuboot.conf
> > > core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst
> > > core-image-minimal-qemux86-64-20240718073341.tar.bz2
> > > core-image-minimal-qemux86-64-20240718073341.testdata.json
> > 20c20
> > < fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> > 20240718065105.bin
> > ---
> > > fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
> > 20240718073341.bin
> > 22c22
> > < fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-
> > qemux86-64-
> > 20240718065105.its
> > ---
> > > fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-
> > > qemux86-64-
> > 20240718073341.its
> > 
> > 
> > 
> > The answer is then: No it does not re-build. At least not in the
> > scenario which I would like to fix it, it does not rebuild from
> > scratch.
> > 
> > Does that make sense?
> > 
> > Regards,
> > Adrian
> > 
> > > > 
> > > > 
> > > > 
> > 
> > 
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#202191):
> > https://lists.openembedded.org/g/openembedded-core/message/202191
> > Mute This Topic:
> > https://lists.openembedded.org/mt/107231736/5052612
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-
> > core/unsub [quaresma.jose@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> > 
> 
> 
> -- 
> Best regards,
> 
> José Quaresma
Martin Jansa July 18, 2024, 1:29 p.m. UTC | #8
On Thu, Jul 18, 2024 at 2:37 PM Adrian Freihofer
<adrian.freihofer@gmail.com> wrote:
>
> On Thu, 2024-07-18 at 10:39 +0200, Martin Jansa wrote:
> > On Thu, Jul 18, 2024 at 10:00 AM Adrian Freihofer
> > <adrian.freihofer@gmail.com> wrote:
> > >
> > > Hi Martin
> > >
> > > Thank you for looking at my patches.
> > >
> > > On Mon, 2024-07-15 at 16:32 +0200, Martin Jansa wrote:
> > > > Doesn't it still rebuild from scratch when IMAGE_VERSION_SUFFIX
> > > > changes?
> > >
> > > I wonder why this happens in WebOS. I think that this line
> > > kernel_do_deploy[vardepsexclude] = "DATETIME"
> > > prevents such unnecessary rebuilds.
> >
> > Hi Adrian,
> >
> > yes DATETIME is excluded by default, but it's useful to use different
> > suffix in IMAGE_VERSION_SUFFIX (e.g. BUILD_NUMBER from CI on jenkins
> > or release version when building release) and in such cases you don't
> > want to vardepexclude it, because it's useful to produce matching
> > version in images as well as kernel (and other artifacts you might
> > have), so your 1.2.1 release images doesn't end with 1.2.0 kernel
> > artifacts just because kernel sstate signature didn't change in 1.2.1
> > bugfix release.
> >
> > Does this make sense? I bet it would happen in your builds as well.
>
> I try to optimize everything towards binary-reproducible builds. This
> means that we remove all kinds of variables that change because the
> time changes or because the CI build ID changes or something like that.
> I see no benefit in compiling such information into the firmware.
> But the problem I see is that such ideas clash with efficient and
> binary-reproducible builds.
>
> That means that I don't see a reason for changing the
> IMAGE_VERSION_SUFFIX variable in such a way.
>
> >
> > Whole point of:
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
> > is to allow "renaming" (adding more hardlinks) artifacts with
> > matching
> > IMAGE_VERSION_SUFFIX without the need of rebuilding anything (e.g.
> > version-less kernel artifacts are re-used from sstate and then only
> > quick deploy-links task is executed to add right versioned hardlinks
> > in deploy directory.)
> >
>
> Well, yes, it would. But, I am confident that it is a better strategy
> to simply not do this than to support it.
>
> If such non-reproducible information is needed anyway, it can be
> archived independently of Bitbake.

Well the changes from:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
make the builds _more_ reproducible, because they move the DATETIME or
whatever IMAGE_VERSION_SUFFIX out of the artifacts in sstate and move
creating versioned names to separate task which can be disabled (by
setting IMAGE_VERSION_SUFFIX to empty if you prefer not to create
them).
Alexandre Belloni July 24, 2024, 3:54 p.m. UTC | #9
Hello,

I'm pretty sure this causes this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/151/builds/2005/steps/12/logs/stdio

And also those oe-selftest failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6945/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6987/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/7000/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/7010/steps/14/logs/stdio


On 15/07/2024 16:10:37+0200, Adrian Freihofer wrote:
> Changes in comparison to v1:
> - Add the missing dependency from do_image_wic to
>   initramfs.do_image_complete which caused the build failure on the AB.
> - Support symbolic links for $dtb_path again, as suggested by Mark.
> - Simplify the refactoring of the uboot_prep_kimage function.
> - Evaluate staging via sysroot instead of taking the kernel artifacts
>   from the deploy folder. But there is an issue with this approach: It
>   does not allow to remove the dependency from do_deploy on do_install.
> - Rebasing to latest master
> 
> If the KERNEL_IMAGETYPES(S) contains fitImage, the kernel is always
> rebuilt when something changes in the initramfs frequently.
> This is even worse if the build runs from an empty TMPDIR. The kernel
> re-build starts by fetching the large kernel git repository and
> recompiling it from scratch.
> 
> This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel
> Makefile is needed to generate the initramfs bundle the kernel build
> folder is required.
> But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the
> build folder is not needed. Creating the initramfs bundle requires:
> linux.bin, DTBs and the initramfs which are available in the deploy
> directory. That means creating the fitImage can be done with artifacts
> which are already cached by the sstate.
> 
> There is an extra commit providing a html file. This html file provides
> some graphics which show the changes in the kernel's task dependencies.
> I hope this simplifies the re-view of this patch series.
> 
> Testing:
> - oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg \
>      reproducible -T yocto-mirrors -T machine -T toolchain-user \
>      -T toolchain-system
> - Build for core-image-minimal for MACHINE = "genericarm64"
>   (failure from AB was reproducible and is fixed now)
> - Build clean followed by build from sstate
>   Note: Adding this test to oe-selftest failed because sstate is read only
>   - Clean build config:
>     KERNEL_IMAGETYPE = "Image"
>     KERNEL_IMAGETYPES += " fitImage "
>     KERNEL_CLASSES = " kernel-fitimage "
>     IMAGE_FSTYPES += "cpio.gz"
>     INITRAMFS_IMAGE = "core-image-minimal"
>     IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
>     UBOOT_RD_LOADADDRESS = "0x88000000"
>     UBOOT_RD_ENTRYPOINT = "0x88000000"
>     UBOOT_LOADADDRESS = "0x80080000"
>     UBOOT_ENTRYPOINT = "0x80080000"
>     FIT_DESC = "A model description"
>     FOO_VAR = "1"
>     INHERIT += "image-buildinfo"
>     IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
>   - Append the following and rebuild with sstate:
>     FOO_VAR = "2"
>     TMPDIR = "${TOPDIR}/tmp-2"
>   - Check the log files:
>     - tmp/log/cooker/qemux86-64 contains:
>       linux-yocto-6.6.35+git-r0: task do_deploy: Succeeded
>       linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
>     - tmp-2/log/cooker/qemux86-64 contains:
>       linux-yocto-6.6.35+git-r0: task do_deploy_setscene: Succeeded
>       linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
> - To re-view the chagnes in the task dependencies, the script bellow
>   has been used. It confirms:
>   - For builds with fitImage and unbundled initramfs:
>     - the do_deploy_fitimage_unbundled task runs after do_deploy
>     - do_assemble_fitimage_initramfs is not executed
>     - do_bundle_initramfs is not executed
>     - It works for fitImage in KERNEL_IMAGETYPE as well as for fitImage
>       in KERNEL_IMAGETYPES
>   - For builds with fitImage and bundled initramfs: No changes
>   - For builds with fitImage, without initramfs:
>     - do_assemble_fitimage_initramfs is not executed
>     - do_bundle_initramfs is not executed
> 
> OUTPUT_FILE=task-depends.md
> GIT_BRANCH=master
> GIT_BRANCH_NEXT=adrianf/kernel-fitimage-sstate
> 
> echo "# Task dependeny changes" > "$OUTPUT_FILE"
> echo "" >> "$OUTPUT_FILE"
> 
> run_bitbake(){
>     echo "$1" >> "$OUTPUT_FILE"
>     echo "" >> "$OUTPUT_FILE"
> 
>     bitbake virtual/kernel -g
>     grep -E '(digraph depends|linux-yocto|\})' task-depends.dot \
>     | grep -v -E '(spdx|do_kernel_configcheck|do_prepare_recipe_sysroot|do_populate_lic|native.do_populate_sysroot)' \
>     | grep -v -E '(depmodwrapper-cross.do_populate_sysroot|binutils-cross-x86_64.do_populate_sysroot|do_package)' \
>     | grep -v -E '(do_validate_branches|do_unpack|do_recipe_qa|do_patch|do_kernel_checkout|do_kernel_configme|do_kernel_metadata)' \
>     | grep -v -E '(do_sizecheck|do_strip|do_compile_kernelmodules|do_shared_workdir|do_transform_kernel|do_kernel_link_images)' \
>     | grep -v -E '(do_kernel_version_sanity_check|do_symlink_kernsrc|do_deploy_source_date_epoch|do_configure|do_fetch)' \
>     | sed -e 's;\\n\:.*.bb;;g' -e 's;linux-yocto[. ];;g' > task-depends-filtered.dot
> 
>     echo '```plantuml' >> "$OUTPUT_FILE"
>     cat task-depends-filtered.dot >> "$OUTPUT_FILE"
>     echo '```' >> "$OUTPUT_FILE"
>     echo "" >> "$OUTPUT_FILE"
> }
> 
> run_bitbake_per_branch(){
>     echo "## Configuration: $1" >> "$OUTPUT_FILE"
>     echo "" >> "$OUTPUT_FILE"
>     echo '```raw' >> "$OUTPUT_FILE"
>     cat build/conf/auto.conf >> "$OUTPUT_FILE"
>     echo '```' >> "$OUTPUT_FILE"
>     echo "" >> "$OUTPUT_FILE"
> 
>     git checkout $GIT_BRANCH
>     run_bitbake "### branch: $GIT_BRANCH"
>     mv -f task-depends-filtered.dot task-depends-filtered-old.dot
> 
>     git checkout $GIT_BRANCH_NEXT
>     run_bitbake "### branch: $GIT_BRANCH_NEXT"
> 
>     echo "## Diff" >> "$OUTPUT_FILE"
>     echo "" >> "$OUTPUT_FILE"
>     echo '```patch' >> "$OUTPUT_FILE"
>     diff task-depends-filtered-old.dot task-depends-filtered.dot >> "$OUTPUT_FILE"
>     echo '```' >> "$OUTPUT_FILE"
>     echo "" >> "$OUTPUT_FILE"
> }
> 
> cat << EOF > build/conf/auto.conf
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal-initramfs"
> EOF
> run_bitbake_per_branch "image, fitimage, initramfs, unbundled"
> 
> cat << EOF > build/conf/auto.conf
> KERNEL_IMAGETYPE:forcevariable = "fitImage"
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal-initramfs"
> EOF
> run_bitbake_per_branch "fitimage, initramfs, unbundled"
> 
> cat << EOF > build/conf/auto.conf
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal-initramfs"
> INITRAMFS_IMAGE_BUNDLE = "1"
> EOF
> run_bitbake_per_branch "image, fitimage, initramfs, bundled"
> 
> cat << EOF > build/conf/auto.conf
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
> EOF
> run_bitbake_per_branch "image, fitimage"
> 
> rm -f build/conf/auto.conf task-depends-filtered-old.dot task-depends-filtered.dot task-depends.dot
> 
> Adrian Freihofer (6):
>   kernel-fitimage: fix intentation
>   kernel-fitimage: fix external dtb check
>   kernel: refactor linux compression
>   kernel-fitimage: refactor fitimage_assemble
>   kernel: refactor fitimage
>   kernel-fitimage: run unbundled fitimage after deploy
> 
>  meta/classes-recipe/image.bbclass           |  12 +-
>  meta/classes-recipe/kernel-fitimage.bbclass | 188 ++++++++++++--------
>  meta/classes-recipe/kernel-uboot.bbclass    |   1 +
>  meta/classes-recipe/kernel.bbclass          |  31 ++--
>  4 files changed, 148 insertions(+), 84 deletions(-)
> 
> -- 
> 2.45.2
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#201926): https://lists.openembedded.org/g/openembedded-core/message/201926
> Mute This Topic: https://lists.openembedded.org/mt/107231736/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>