classes: add rootfs-footprint

The rootfs-footprint class can be inherited by image recipes to generate
a .json footprint file. This file contains a list of all installed packages,
with the following format:

{
    "package_name": "busybox-udhcpc",
    "version": "1.37.0",
    "total_size": 2701,
    "installed_files": {
        "/etc/udhcpc.d/50default": 2652,
        "/usr/share/udhcpc/default.script": 49
    }
}

The data can be used with other tools/scripts to generate detailed
rootfs reports or perform relevant comparisons.

Signed-off-by: Beniamin Sandu <beniamin.sandu@windriver.com>
---
 meta/classes/rootfs-footprint.bbclass | 54 +++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100644 meta/classes/rootfs-footprint.bbclass

Isn't this what spdx already does?
E.g. tmp/deploy/spdx/3.0.1/qemux86_64/rootfs/core-image-full-cmdline-qemux86-64-rootfs.spdx.json

Alex

On Fri, 24 Jan 2025 at 17:29, Beniamin Sandu via
lists.openembedded.org
<beniamin.sandu=windriver.com@lists.openembedded.org> wrote:
>
> The rootfs-footprint class can be inherited by image recipes to generate
> a .json footprint file. This file contains a list of all installed packages,
> with the following format:
>
> {
>     "package_name": "busybox-udhcpc",
>     "version": "1.37.0",
>     "total_size": 2701,
>     "installed_files": {
>         "/etc/udhcpc.d/50default": 2652,
>         "/usr/share/udhcpc/default.script": 49
>     }
> }
>
> The data can be used with other tools/scripts to generate detailed
> rootfs reports or perform relevant comparisons.
>
> Signed-off-by: Beniamin Sandu <beniamin.sandu@windriver.com>
> ---
>  meta/classes/rootfs-footprint.bbclass | 54 +++++++++++++++++++++++++++
>  1 file changed, 54 insertions(+)
>  create mode 100644 meta/classes/rootfs-footprint.bbclass
>
> diff --git a/meta/classes/rootfs-footprint.bbclass b/meta/classes/rootfs-footprint.bbclass
> new file mode 100644
> index 0000000000..247db0b709
> --- /dev/null
> +++ b/meta/classes/rootfs-footprint.bbclass
> @@ -0,0 +1,54 @@
> +# Generates a json footprint file with the list of packages installed in the rootfs image,
> +# with the following format:
> +#
> +# {
> +#     "package_name": "busybox-syslog",
> +#     "version": "1.37.0",
> +#     "total_size": 2824,
> +#     "installed_files": {
> +#         "/etc/init.d/syslog": 2104,
> +#         "/etc/syslog-startup.conf": 651,
> +#         "/etc/syslog.conf": 69
> +#     }
> +# }
> +#
> +# Copyright (C) 2024 Wind River Systems, Inc.
> +# Author: Beniamin Sandu <beniamin.sandu@windriver.com>
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +
> +ROOTFS_POSTPROCESS_COMMAND:append = " create_rootfs_footprint; "
> +ROOTFS_FOOTPRINT_FILE ?= "${IMGDEPLOYDIR}/${IMAGE_NAME}.footprint.json"
> +
> +python create_rootfs_footprint() {
> +    from oe.rootfs import image_list_installed_packages
> +    import oe.packagedata
> +    import json
> +
> +    footprint_file = d.getVar('ROOTFS_FOOTPRINT_FILE')
> +    pkg_info_dic = {}
> +    pkg_dic = dict(package_name="", version="", total_size=0, installed_files="")
> +    pkg_list = sorted(image_list_installed_packages(d))
> +
> +    pkg_data_list = []
> +
> +    for pkg in pkg_list:
> +        pkg_info = os.path.join(d.getVar('PKGDATA_DIR'), 'runtime-reverse', pkg)
> +        pkg_name = os.path.basename(os.readlink(pkg_info))
> +        pkg_info_dic[pkg_name] = oe.packagedata.read_pkgdatafile(pkg_info)
> +
> +        pkg_dic["package_name"] = pkg_name
> +        pkg_dic["version"] = pkg_info_dic[pkg_name]["PV"]
> +        pkg_dic["total_size"] = int(pkg_info_dic[pkg_name]["PKGSIZE:" + pkg_name])
> +        pkg_dic["installed_files"] = json.loads(pkg_info_dic[pkg_name]["FILES_INFO:" + pkg_name])
> +
> +        pkg_data_list.append(pkg_dic.copy())
> +
> +    with open(footprint_file, 'w') as f:
> +        json.dump(pkg_data_list, f, indent=4)
> +
> +    footprint_link = os.path.join(d.getVar('IMGDEPLOYDIR'), "%s.footprint.json" % d.getVar('IMAGE_LINK_NAME'))
> +    if os.path.lexists(footprint_link):
> +        os.remove(footprint_link)
> +    os.symlink(os.path.basename(footprint_file), footprint_link)
> +}
> --
> 2.39.5
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#210236): https://lists.openembedded.org/g/openembedded-core/message/210236
> Mute This Topic: https://lists.openembedded.org/mt/110791960/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Hi Beniamin

I was also thinking about buildhistory and spdx when reading this.

Regards,
Adrian

Alexander Kanavin via lists.openembedded.org <alex.kanavin=
gmail.com@lists.openembedded.org> schrieb am Fr., 24. Jan. 2025, 20:30:

> Isn't this what spdx already does?
> E.g.
> tmp/deploy/spdx/3.0.1/qemux86_64/rootfs/core-image-full-cmdline-qemux86-64-rootfs.spdx.json
>
> Alex
>
> On Fri, 24 Jan 2025 at 17:29, Beniamin Sandu via
> lists.openembedded.org
> <beniamin.sandu=windriver.com@lists.openembedded.org> wrote:
> >
> > The rootfs-footprint class can be inherited by image recipes to generate
> > a .json footprint file. This file contains a list of all installed
> packages,
> > with the following format:
> >
> > {
> >     "package_name": "busybox-udhcpc",
> >     "version": "1.37.0",
> >     "total_size": 2701,
> >     "installed_files": {
> >         "/etc/udhcpc.d/50default": 2652,
> >         "/usr/share/udhcpc/default.script": 49
> >     }
> > }
> >
> > The data can be used with other tools/scripts to generate detailed
> > rootfs reports or perform relevant comparisons.
> >
> > Signed-off-by: Beniamin Sandu <beniamin.sandu@windriver.com>
> > ---
> >  meta/classes/rootfs-footprint.bbclass | 54 +++++++++++++++++++++++++++
> >  1 file changed, 54 insertions(+)
> >  create mode 100644 meta/classes/rootfs-footprint.bbclass
> >
> > diff --git a/meta/classes/rootfs-footprint.bbclass
> b/meta/classes/rootfs-footprint.bbclass
> > new file mode 100644
> > index 0000000000..247db0b709
> > --- /dev/null
> > +++ b/meta/classes/rootfs-footprint.bbclass
> > @@ -0,0 +1,54 @@
> > +# Generates a json footprint file with the list of packages installed
> in the rootfs image,
> > +# with the following format:
> > +#
> > +# {
> > +#     "package_name": "busybox-syslog",
> > +#     "version": "1.37.0",
> > +#     "total_size": 2824,
> > +#     "installed_files": {
> > +#         "/etc/init.d/syslog": 2104,
> > +#         "/etc/syslog-startup.conf": 651,
> > +#         "/etc/syslog.conf": 69
> > +#     }
> > +# }
> > +#
> > +# Copyright (C) 2024 Wind River Systems, Inc.
> > +# Author: Beniamin Sandu <beniamin.sandu@windriver.com>
> > +#
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +
> > +ROOTFS_POSTPROCESS_COMMAND:append = " create_rootfs_footprint; "
> > +ROOTFS_FOOTPRINT_FILE ?= "${IMGDEPLOYDIR}/${IMAGE_NAME}.footprint.json"
> > +
> > +python create_rootfs_footprint() {
> > +    from oe.rootfs import image_list_installed_packages
> > +    import oe.packagedata
> > +    import json
> > +
> > +    footprint_file = d.getVar('ROOTFS_FOOTPRINT_FILE')
> > +    pkg_info_dic = {}
> > +    pkg_dic = dict(package_name="", version="", total_size=0,
> installed_files="")
> > +    pkg_list = sorted(image_list_installed_packages(d))
> > +
> > +    pkg_data_list = []
> > +
> > +    for pkg in pkg_list:
> > +        pkg_info = os.path.join(d.getVar('PKGDATA_DIR'),
> 'runtime-reverse', pkg)
> > +        pkg_name = os.path.basename(os.readlink(pkg_info))
> > +        pkg_info_dic[pkg_name] =
> oe.packagedata.read_pkgdatafile(pkg_info)
> > +
> > +        pkg_dic["package_name"] = pkg_name
> > +        pkg_dic["version"] = pkg_info_dic[pkg_name]["PV"]
> > +        pkg_dic["total_size"] = int(pkg_info_dic[pkg_name]["PKGSIZE:" +
> pkg_name])
> > +        pkg_dic["installed_files"] =
> json.loads(pkg_info_dic[pkg_name]["FILES_INFO:" + pkg_name])
> > +
> > +        pkg_data_list.append(pkg_dic.copy())
> > +
> > +    with open(footprint_file, 'w') as f:
> > +        json.dump(pkg_data_list, f, indent=4)
> > +
> > +    footprint_link = os.path.join(d.getVar('IMGDEPLOYDIR'),
> "%s.footprint.json" % d.getVar('IMAGE_LINK_NAME'))
> > +    if os.path.lexists(footprint_link):
> > +        os.remove(footprint_link)
> > +    os.symlink(os.path.basename(footprint_file), footprint_link)
> > +}
> > --
> > 2.39.5
> >
> >
> >
> >
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#210246):
> https://lists.openembedded.org/g/openembedded-core/message/210246
> Mute This Topic: https://lists.openembedded.org/mt/110791960/4454582
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> adrian.freihofer@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

Hey folks,

Thanks for the feedback.

The main idea behind this class, was to have an easy way to extract rootfs information, mostly focused on sizes.  A common scenario on my side was upgrading a set of layers to a different yocto version, building for a system, and ending up with quite different and usually bigger final image. On systems where rootfs size is still a top priority, this can add on quite some tedious extra work of identifying where all the extra space is coming from.

I had a quick look at the current classes, and some issues that I found (especially for this use case), are:

For spdx:
- there is a lot of information generated, and could not find any on package/file sizes
- spread over a lot of files
- impossible to read with the naked eye

For buildhistory:
- a lot of information, spread over a lot of files
- different output formats in the generated data

To that end, I wanted something that:
- is fast to generate
- is easy to read/follow with the naked eye if needed
- all the relevant information is in one file
- has a consistent and easy to parse format

Now a script with relatively minimal logic (parsing a single json file) can generate very detailed rootfs reports, e.g. sorting packages by size, information on added/removed pacakges or files,  other changes in list of installed files for a specific package, etc.

Let me know what you think.

Cheers,
Beni

You probably should look into improving buildhistory. Or-core needs
maintainers for things that already exist, not addition of new things that
partially overlap in functionality.

Alex

On Fri 24. Jan 2025 at 22.25, Sandu, Beniamin <Beniamin.Sandu@windriver.com>
wrote:

> Hey folks,
>
> Thanks for the feedback.
>
> The main idea behind this class, was to have an easy way to extract rootfs
> information, mostly focused on sizes.  A common scenario on my side was
> upgrading a set of layers to a different yocto version, building for a
> system, and ending up with quite different and usually bigger final image.
> On systems where rootfs size is still a top priority, this can add on quite
> some tedious extra work of identifying where all the extra space is coming
> from.
>
> I had a quick look at the current classes, and some issues that I found
> (especially for this use case), are:
>
> For spdx:
> - there is a lot of information generated, and could not find any on
> package/file sizes
> - spread over a lot of files
> - impossible to read with the naked eye
>
> For buildhistory:
> - a lot of information, spread over a lot of files
> - different output formats in the generated data
>
> To that end, I wanted something that:
> - is fast to generate
> - is easy to read/follow with the naked eye if needed
> - all the relevant information is in one file
> - has a consistent and easy to parse format
>
> Now a script with relatively minimal logic (parsing a single json file)
> can generate very detailed rootfs reports, e.g. sorting packages by size,
> information on added/removed pacakges or files,  other changes in list of
> installed files for a specific package, etc.
>
> Let me know what you think.
>
> Cheers,
> Beni
> ________________________________________
> From: Adrian Freihofer <adrian.freihofer@gmail.com>
> Sent: 24 January 2025 21:41
> To: Alexander Kanavin <alex.kanavin@gmail.com>
> Cc: Sandu, Beniamin <Beniamin.Sandu@windriver.com>; OE-core <
> openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH] classes: add rootfs-footprint
>
> Hi Beniamin
>
> I was also thinking about buildhistory and spdx when reading this.
>
> Regards,
> Adrian
>
> Alexander Kanavin via lists.openembedded.org <alex.kanavin=
> gmail.com@lists.openembedded.org> schrieb am Fr., 24. Jan. 2025, 20:30:
> Isn't this what spdx already does?
> E.g.
> tmp/deploy/spdx/3.0.1/qemux86_64/rootfs/core-image-full-cmdline-qemux86-64-rootfs.spdx.json
>
> Alex
>
> On Fri, 24 Jan 2025 at 17:29, Beniamin Sandu via
> lists.openembedded.org
> <beniamin.sandu=windriver.com@lists.openembedded.org> wrote:
> >
> > The rootfs-footprint class can be inherited by image recipes to generate
> > a .json footprint file. This file contains a list of all installed
> packages,
> > with the following format:
> >
> > {
> >     "package_name": "busybox-udhcpc",
> >     "version": "1.37.0",
> >     "total_size": 2701,
> >     "installed_files": {
> >         "/etc/udhcpc.d/50default": 2652,
> >         "/usr/share/udhcpc/default.script": 49
> >     }
> > }
> >
> > The data can be used with other tools/scripts to generate detailed
> > rootfs reports or perform relevant comparisons.
> >
> > Signed-off-by: Beniamin Sandu <beniamin.sandu@windriver.com>
> > ---
> >  meta/classes/rootfs-footprint.bbclass | 54 +++++++++++++++++++++++++++
> >  1 file changed, 54 insertions(+)
> >  create mode 100644 meta/classes/rootfs-footprint.bbclass
> >
> > diff --git a/meta/classes/rootfs-footprint.bbclass
> b/meta/classes/rootfs-footprint.bbclass
> > new file mode 100644
> > index 0000000000..247db0b709
> > --- /dev/null
> > +++ b/meta/classes/rootfs-footprint.bbclass
> > @@ -0,0 +1,54 @@
> > +# Generates a json footprint file with the list of packages installed
> in the rootfs image,
> > +# with the following format:
> > +#
> > +# {
> > +#     "package_name": "busybox-syslog",
> > +#     "version": "1.37.0",
> > +#     "total_size": 2824,
> > +#     "installed_files": {
> > +#         "/etc/init.d/syslog": 2104,
> > +#         "/etc/syslog-startup.conf": 651,
> > +#         "/etc/syslog.conf": 69
> > +#     }
> > +# }
> > +#
> > +# Copyright (C) 2024 Wind River Systems, Inc.
> > +# Author: Beniamin Sandu <beniamin.sandu@windriver.com>
> > +#
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +
> > +ROOTFS_POSTPROCESS_COMMAND:append = " create_rootfs_footprint; "
> > +ROOTFS_FOOTPRINT_FILE ?= "${IMGDEPLOYDIR}/${IMAGE_NAME}.footprint.json"
> > +
> > +python create_rootfs_footprint() {
> > +    from oe.rootfs import image_list_installed_packages
> > +    import oe.packagedata
> > +    import json
> > +
> > +    footprint_file = d.getVar('ROOTFS_FOOTPRINT_FILE')
> > +    pkg_info_dic = {}
> > +    pkg_dic = dict(package_name="", version="", total_size=0,
> installed_files="")
> > +    pkg_list = sorted(image_list_installed_packages(d))
> > +
> > +    pkg_data_list = []
> > +
> > +    for pkg in pkg_list:
> > +        pkg_info = os.path.join(d.getVar('PKGDATA_DIR'),
> 'runtime-reverse', pkg)
> > +        pkg_name = os.path.basename(os.readlink(pkg_info))
> > +        pkg_info_dic[pkg_name] =
> oe.packagedata.read_pkgdatafile(pkg_info)
> > +
> > +        pkg_dic["package_name"] = pkg_name
> > +        pkg_dic["version"] = pkg_info_dic[pkg_name]["PV"]
> > +        pkg_dic["total_size"] = int(pkg_info_dic[pkg_name]["PKGSIZE:" +
> pkg_name])
> > +        pkg_dic["installed_files"] =
> json.loads(pkg_info_dic[pkg_name]["FILES_INFO:" + pkg_name])
> > +
> > +        pkg_data_list.append(pkg_dic.copy())
> > +
> > +    with open(footprint_file, 'w') as f:
> > +        json.dump(pkg_data_list, f, indent=4)
> > +
> > +    footprint_link = os.path.join(d.getVar('IMGDEPLOYDIR'),
> "%s.footprint.json" % d.getVar('IMAGE_LINK_NAME'))
> > +    if os.path.lexists(footprint_link):
> > +        os.remove(footprint_link)
> > +    os.symlink(os.path.basename(footprint_file), footprint_link)
> > +}
> > --
> > 2.39.5
> >
> >
> >
> >
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#210246):
> https://lists.openembedded.org/g/openembedded-core/message/210246
> Mute This Topic: https://lists.openembedded.org/mt/110791960/4454582
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> adrian.freihofer@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>