From patchwork Thu Jul 31 08:27:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yoann Congal X-Patchwork-Id: 67813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC2B7C87FD2 for ; Thu, 31 Jul 2025 08:27:28 +0000 (UTC) Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by mx.groups.io with SMTP id smtpd.web10.56902.1753950444302269539 for ; Thu, 31 Jul 2025 01:27:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@smile.fr header.s=google header.b=nr10QnlJ; spf=pass (domain: smile.fr, ip: 209.85.128.65, mailfrom: yoann.congal@smile.fr) Received: by mail-wm1-f65.google.com with SMTP id 5b1f17b1804b1-4561514c7f0so1281385e9.0 for ; Thu, 31 Jul 2025 01:27:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smile.fr; s=google; t=1753950442; x=1754555242; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ocBHOlOu/xMMMJiOpYeIMVLPGLzAfwmp0e76wraa93w=; b=nr10QnlJdMdHW47CGT4BGqQbyz/OwsL71CDdrcOOf4haHmNEt9qYaAIYPLFZ3Q9o58 PC4BjYK+G7erPVePXHFhyHYUoVsIrYZfMRs4mc51ThXaby9qExb6c9tTpucnimUJWVi8 qKdzlHsMvfeznEiZQTbzBpHMeEV3snjc6pxZQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753950442; x=1754555242; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ocBHOlOu/xMMMJiOpYeIMVLPGLzAfwmp0e76wraa93w=; b=DYrAZnyrlG2DSmTmr2lYK6Xl0U9cJ5DlDcRNlEz/VSCQaRcJUXls2zOcCEmNQ1IZO3 GdDdBLEmAzv3uaRg4A/pl3up3hr2P+oBHRBaebSKdLS3usfXFyTu3HUu9E6pzdj5bfhB 4+0kNwGR0C2gJQ7khCMzKxEh5g5bFyJceQ0l9NMWqWX6JB0HAlIpluAGNHdzTe4FCkz4 EPtCifyYYa7zUzTd7ne5HU/aFuCVOnKy8IEpamD8n5zqYjtMln9b1iDHYyvkUVjJz28v xAWEEej4JqphN/Q1bUBgvuJ3m7m8whBUbp17AbhcAqh0MA5iUL6fzQ8knR6X1y23LH5/ rb3w== X-Gm-Message-State: AOJu0YwdJjoHIPwIHjDuOpvfY2oYlzifWd1GZyeLCRy8wtXoGaOKjXA8 7/xmwcWT+x2UuviC4er2YzH3mlNwi3lMmOMO2AsJE3ILCGWWuuaZcfJo2/IoPr5nPShloPfSpME ppUUuBQjVpYiH X-Gm-Gg: ASbGncvzrEnmKuO0DQv0oLABvI0sJ3TQyD8SUvZfIiwPEoZMdKiCbXE7d4GhJJIdyFc bRp+p9qSSWQHGHrC+VyjdOdy1UVgFRfPOtifl8Yn3dNw9MNB+3X2+R3BYfH6Y3l1XPVhF4195J+ mb/sY02TZ6E7LEbkZBs/Yi6L+2ZMeiMv9l2vMXROMUEPja/0jBbcPXybl/pJbBkW2aDhwDb2vrc 0aze3Fdx4fAZxnxONdw7XVCbwlMaFlmjxHVmRs6MbDGWk7hWAcFIFxihTXKdxarp3Q2QYnnfxRe 8+Uxz0Kb25pncQKNVMwseOqX0/8ii6S0hNWShD3C4yHZzv3L9/XjuNW+WzKNpM9Bo1LYQJl3qyb cjRRBr3EbSAx5ay/i32UICdUlnFPI9OALLxYseU4S5vnHGDdGBCKDgdGhLyeRImquHBwn0qv62U ySqNnl5OG7GlIbyqlAP1p8c68QALcH9XaEdfk= X-Google-Smtp-Source: AGHT+IGCe5jCNWufW9khPW5aaGluxNa8RQvwylgQsTihPBm4D8ZLjbijcwvkOPQnaz70MsgKn/0j5g== X-Received: by 2002:a05:600c:3ac9:b0:453:2066:4a26 with SMTP id 5b1f17b1804b1-45892bbf7e6mr62771165e9.16.1753950441975; Thu, 31 Jul 2025 01:27:21 -0700 (PDT) Received: from P-ASN-ECS-830T8C3.idf.intranet (static-css-ccs-204145.business.bouyguestelecom.com. [176.157.204.145]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3b79c47908esm1562672f8f.59.2025.07.31.01.27.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 31 Jul 2025 01:27:21 -0700 (PDT) From: Yoann Congal To: openembedded-core@lists.openembedded.org Cc: Yoann Congal , Lamine REHAHLIA Subject: [PATCH v3 1/4] runqemu: Add support for running compressed .zst rootfs images Date: Thu, 31 Jul 2025 10:27:04 +0200 Message-Id: <20250731082707.3481718-1-yoann.congal@smile.fr> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 31 Jul 2025 08:27:28 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221176 From: Lamine REHAHLIA Enhance runqemu to detect and decompress .zst-compressed rootfs images (e.g. ext4.zst, wic.zst) automatically. If a decompressed image already exists in the original directory, it will be reused to avoid overwriting build artifacts. Otherwise, the image is decompressed and removed after the QEMU session ends. This allows runqemu to be used seamlessly with compressed image formats generated by the build system or during releases. Note: support for .zst images is only available when snapshot mode is enabled IMPORTANT: This patch assumes that the original directory of the .zst-compressed image is writable. If, for some reason, the path passed from CI or another system to the script is read-only, the decompression step will fail when trying to write the uncompressed image to the same directory. Signed-off-by: Lamine REHAHLIA Signed-off-by: Yoann Congal --- v4: moved the feature to support calling "runqemu " and support usign snapshot after the image. Also improved the error message when a compressed image is used without snapshot mode. --- scripts/runqemu | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/scripts/runqemu b/scripts/runqemu index 3d77046972..c668906bdd 100755 --- a/scripts/runqemu +++ b/scripts/runqemu @@ -197,7 +197,7 @@ class BaseConfig(object): self.portlocks = {} self.bitbake_e = '' self.snapshot = False - self.wictypes = ('wic', 'wic.vmdk', 'wic.qcow2', 'wic.vdi', "wic.vhd", "wic.vhdx") + self.wictypes = ('wic.zst', 'wic', 'wic.vmdk', 'wic.qcow2', 'wic.vdi', "wic.vhd", "wic.vhdx") self.fstypes = ('ext2', 'ext3', 'ext4', 'jffs2', 'nfs', 'btrfs', 'cpio.gz', 'cpio', 'ramfs', 'tar.bz2', 'tar.gz', 'squashfs', 'squashfs-xz', 'squashfs-lzo', @@ -418,6 +418,48 @@ class BaseConfig(object): else: raise RunQemuError("Unknown path arg %s" % p) + def uncompress_rootfs(self): + """Decompress ZST rootfs image if needed""" + if not self.rootfs or not self.fstype.endswith('.zst'): + return + + # Ensure snapshot mode is active before allowing decompression. + if not self.snapshot: + raise RunQemuError(".zst images are only supported with snapshot mode. " \ + "You can either use the \"snapshot\" option or use an uncompressed image.") + + # Get the real path to the image to avoid issues when a symbolic link is passed. + # This ensures we always operate on the actual file. + image_path = os.path.realpath(self.rootfs) + # Extract target filename by removing .zst + image_dir = os.path.dirname(image_path) + uncompressed_name = os.path.basename(image_path).replace(".zst", "") + uncompressed_path = os.path.join(image_dir, uncompressed_name) + + # If the decompressed image already exists (e.g., in the deploy directory), + # we use it directly to avoid overwriting artifacts generated by the build system. + # This prevents redundant decompression and preserves build outputs. + if os.path.exists(uncompressed_path): + logger.warning(f"Found existing decompressed image: {uncompressed_path}, Using it directly.") + else: + logger.info(f"Decompressing {self.rootfs} to {uncompressed_path}") + # Ensure the 'zstd' tool is installed before attempting to decompress. + if not shutil.which('zstd'): + raise RunQemuError(f"'zstd' is required to decompress {self.rootfs} but was not found in PATH") + try: + with open(uncompressed_path, 'wb') as out_file: + subprocess.check_call(['zstd', '-d', '-c', image_path], stdout=out_file) + except subprocess.CalledProcessError as e: + self.cleanup_files.append(uncompressed_path) + raise RunQemuError(f"Failed to decompress {self.rootfs}: {e}") + + # Mark for deletion at the end + self.cleanup_files.append(uncompressed_path) + + # Use the decompressed image as the rootfs + self.rootfs = uncompressed_path + self.fstype = self.fstype.removesuffix(".zst") + def check_arg_machine(self, arg): """Check whether it is a machine""" if self.get('MACHINE') == arg: @@ -1762,6 +1804,7 @@ def main(): config.check_args() config.read_qemuboot() config.check_and_set() + config.uncompress_rootfs() # Check whether the combos is valid or not config.validate_combos() config.print_config()