[v2] runqemu: Add support for running compressed .zst rootfs images

From: Lamine REHAHLIA <lamine.rehahlia@smile.fr>

Enhance runqemu to detect and decompress .zst-compressed rootfs images
(e.g. ext4.zst, wic.zst) automatically. If a decompressed image already
exists in the original directory, it will be reused to avoid overwriting
build artifacts. Otherwise, the image is decompressed and removed after
the QEMU session ends.

This allows runqemu to be used seamlessly with compressed image formats
generated by the build system or during releases.

Note: support for .zst images is only available when snapshot mode is
enabled

IMPORTANT:
This patch assumes that the original directory of the .zst-compressed
image is writable. If, for some reason, the path passed from CI or
another system to the script is read-only, the decompression step will
fail when trying to write the uncompressed image to the same directory.

Signed-off-by: Lamine REHAHLIA <lamine.rehahlia@smile.fr>
---
 scripts/runqemu | 40 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 39 insertions(+), 1 deletion(-)

This is better than v1 (enabling .zst only in snapshot mode), but it
would help to explain to the user why snapshot must be used.

Also .zst is a somewhat arbitrary choice, other compression options
exist, so why not those?

I'd rather just keep things as they are: if you're downloading a
compressed image, decompress it manually.

Alex

On Fri, 25 Jul 2025 at 16:30, lamine.rehahlia via
lists.openembedded.org
<lamine.rehahlia=smile.fr@lists.openembedded.org> wrote:
>
> From: Lamine REHAHLIA <lamine.rehahlia@smile.fr>
>
> Enhance runqemu to detect and decompress .zst-compressed rootfs images
> (e.g. ext4.zst, wic.zst) automatically. If a decompressed image already
> exists in the original directory, it will be reused to avoid overwriting
> build artifacts. Otherwise, the image is decompressed and removed after
> the QEMU session ends.
>
> This allows runqemu to be used seamlessly with compressed image formats
> generated by the build system or during releases.
>
> Note: support for .zst images is only available when snapshot mode is
> enabled
>
> IMPORTANT:
> This patch assumes that the original directory of the .zst-compressed
> image is writable. If, for some reason, the path passed from CI or
> another system to the script is read-only, the decompression step will
> fail when trying to write the uncompressed image to the same directory.
>
> Signed-off-by: Lamine REHAHLIA <lamine.rehahlia@smile.fr>
> ---
>  scripts/runqemu | 40 +++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 39 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/runqemu b/scripts/runqemu
> index 3d77046972..30954ba332 100755
> --- a/scripts/runqemu
> +++ b/scripts/runqemu
> @@ -376,7 +376,45 @@ class BaseConfig(object):
>               re.search('fitImage', p) or re.search('uImage', p):
>              self.kernel =  p
>          elif os.path.isfile(p) and ('-image-' in os.path.basename(p) or '.rootfs.' in os.path.basename(p)):
> -            self.rootfs = p
> +
> +            # Decompress ZST image if needed
> +            if p.endswith('.zst'):
> +                # Ensure snapshot mode is active before allowing decompression.
> +                if not self.snapshot:
> +                        raise RunQemuError(".zst images are only supported when snapshot mode is enabled. ")
> +                # Get the real path to the image to avoid issues when a symbolic link is passed.
> +                # This ensures we always operate on the actual file.
> +                image_path = os.path.realpath(p)
> +                # Extract target filename by removing .zst
> +                image_dir = os.path.dirname(image_path)
> +                uncompressed_name = os.path.basename(image_path).replace(".zst", "")
> +                uncompressed_path = os.path.join(image_dir, uncompressed_name)
> +
> +                # If the decompressed image already exists (e.g., in the deploy directory),
> +                # we use it directly to avoid overwriting artifacts generated by the build system.
> +                # This prevents redundant decompression and preserves build outputs.
> +                if os.path.exists(uncompressed_path):
> +                    logger.warning(f"Found existing decompressed image: {uncompressed_path}, Using it directly.")
> +                else:
> +                    logger.info(f"Decompressing {p} to {uncompressed_path}")
> +                    # Ensure the 'zstd' tool is installed before attempting to decompress.
> +                    if not shutil.which('zstd'):
> +                        raise RunQemuError(f"'zstd' is required to decompress {p} but was not found in PATH")
> +                    try:
> +                        with open(uncompressed_path, 'wb') as out_file:
> +                            subprocess.check_call(['zstd', '-d', '-c', image_path], stdout=out_file)
> +                    except subprocess.CalledProcessError as e:
> +                        self.cleanup_files.append(uncompressed_path)
> +                        raise RunQemuError(f"Failed to decompress {p}: {e}")
> +
> +                    # Mark for deletion at the end
> +                    self.cleanup_files.append(uncompressed_path)
> +
> +                # Use the decompressed image as the rootfs
> +                self.rootfs = uncompressed_path
> +
> +            else:
> +                self.rootfs = p
>              # Check filename against self.fstypes can handle <file>.cpio.gz,
>              # otherwise, its type would be "gz", which is incorrect.
>              fst = ""
> --
> 2.43.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#220914): https://lists.openembedded.org/g/openembedded-core/message/220914
> Mute This Topic: https://lists.openembedded.org/mt/114338431/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On Fri, 2025-07-25 at 16:30 +0200, lamine.rehahlia via
lists.openembedded.org wrote:
> From: Lamine REHAHLIA <lamine.rehahlia@smile.fr>
> 
> Enhance runqemu to detect and decompress .zst-compressed rootfs
> images
> (e.g. ext4.zst, wic.zst) automatically. If a decompressed image
> already
> exists in the original directory, it will be reused to avoid
> overwriting
> build artifacts. Otherwise, the image is decompressed and removed
> after
> the QEMU session ends.
> 
> This allows runqemu to be used seamlessly with compressed image
> formats
> generated by the build system or during releases.
> 
> Note: support for .zst images is only available when snapshot mode is
> enabled
> 
> IMPORTANT:
> This patch assumes that the original directory of the .zst-compressed
> image is writable. If, for some reason, the path passed from CI or
> another system to the script is read-only, the decompression step
> will
> fail when trying to write the uncompressed image to the same
> directory.
> 
> Signed-off-by: Lamine REHAHLIA <lamine.rehahlia@smile.fr>
> ---
>  scripts/runqemu | 40 +++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 39 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/runqemu b/scripts/runqemu
> index 3d77046972..30954ba332 100755
> --- a/scripts/runqemu
> +++ b/scripts/runqemu
> @@ -376,7 +376,45 @@ class BaseConfig(object):
>               re.search('fitImage', p) or re.search('uImage', p):
>              self.kernel =  p
>          elif os.path.isfile(p) and ('-image-' in os.path.basename(p)
> or '.rootfs.' in os.path.basename(p)):
> -            self.rootfs = p
> +
> +            # Decompress ZST image if needed
> +            if p.endswith('.zst'):
> +                # Ensure snapshot mode is active before allowing
> decompression.
> +                if not self.snapshot:
> +                        raise RunQemuError(".zst images are only
> supported when snapshot mode is enabled. ")
> +                # Get the real path to the image to avoid issues
> when a symbolic link is passed.
> +                # This ensures we always operate on the actual file.
> +                image_path = os.path.realpath(p)
> +                # Extract target filename by removing .zst
> +                image_dir = os.path.dirname(image_path)
> +                uncompressed_name =
> os.path.basename(image_path).replace(".zst", "")
> +                uncompressed_path = os.path.join(image_dir,
> uncompressed_name)
> +
> +                # If the decompressed image already exists (e.g., in
> the deploy directory),
> +                # we use it directly to avoid overwriting artifacts
> generated by the build system.
> +                # This prevents redundant decompression and
> preserves build outputs.
> +                if os.path.exists(uncompressed_path):
> +                    logger.warning(f"Found existing decompressed
> image: {uncompressed_path}, Using it directly.")
> +                else:
> +                    logger.info(f"Decompressing {p} to
> {uncompressed_path}")
> +                    # Ensure the 'zstd' tool is installed before
> attempting to decompress.
> +                    if not shutil.which('zstd'):
> +                        raise RunQemuError(f"'zstd' is required to
> decompress {p} but was not found in PATH")
> +                    try:
> +                        with open(uncompressed_path, 'wb') as
> out_file:
> +                            subprocess.check_call(['zstd', '-d', '-
> c', image_path], stdout=out_file)

Usually, these image files are sparse files. When you compress and then
decompress them, they may become regular files, which can be
significantly larger than the original sparse files. This can
negatively impact performance and increase the required storage.

Acc. to the man page of zstd, sparse files are supported:
--[no-]sparse
  enable / disable sparse FS support, to make files with many
  zeroes smaller on disk. Creating sparse files may save disk
  space and speed up decompression by reducing the amount of disk
  I/O. default: enabled when output is into a file, and disabled
  when output is stdout. This setting overrides default and can
  force sparse mode over stdout.

To address this, the code may need improvements. You could pass `--
sparse` to ensure a sparse file is written to stdout, or, preferably,
use the `-o` option to write directly to a file.

Adrian

> +                    except subprocess.CalledProcessError as e:
> +                        self.cleanup_files.append(uncompressed_path)
> +                        raise RunQemuError(f"Failed to decompress
> {p}: {e}")
> +
> +                    # Mark for deletion at the end
> +                    self.cleanup_files.append(uncompressed_path)
> +
> +                # Use the decompressed image as the rootfs
> +                self.rootfs = uncompressed_path
> +
> +            else:
> +                self.rootfs = p
>              # Check filename against self.fstypes can handle
> <file>.cpio.gz,
>              # otherwise, its type would be "gz", which is incorrect.
>              fst = ""
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#220914):
> https://lists.openembedded.org/g/openembedded-core/message/220914
> Mute This Topic: https://lists.openembedded.org/mt/114338431/4454582
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe:
> https://lists.openembedded.org/g/openembedded-core/unsub [
> adrian.freihofer@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-