| Message ID | 20251003203130.23843-1-daniel.dragomir@windriver.com |
|---|---|
| State | New |
| Headers | show |
| Series | wic/engine: fix copying directories into wic image with ext* partition | expand |
On 3 Oct 2025, at 21:31, Dragomir, Daniel via lists.openembedded.org <daniel.dragomir=windriver.com@lists.openembedded.org> wrote: > > wic uses debugfs to write on ext* partitions, but debugfs can only > write to the current working directory and it cannot copy complete > directory trees. Running 'wic ls' on a copied directory show this: > -l: Ext2 inode is not a directory > > Fix this by creating a command list for debugfs (-f parameter) when > recursive parsing the host directory in order to create a similar > directory structure (mkdir) and copy files (write) on each level > into the destination directory from the wic's ext* partition. This is great, but could you please add a test case to the wic selftests to exercise this and ensure it doesn’t break in the future? Thanks, Ross
On Mon, 2025-10-06 at 20:08 +0000, Ross Burton via lists.openembedded.org wrote: > On 3 Oct 2025, at 21:31, Dragomir, Daniel via lists.openembedded.org > <daniel.dragomir=windriver.com@lists.openembedded.org> wrote: > > > > wic uses debugfs to write on ext* partitions, but debugfs can only > > write to the current working directory and it cannot copy complete > > directory trees. Running 'wic ls' on a copied directory show this: > > -l: Ext2 inode is not a directory > > > > Fix this by creating a command list for debugfs (-f parameter) when > > recursive parsing the host directory in order to create a similar > > directory structure (mkdir) and copy files (write) on each level > > into the destination directory from the wic's ext* partition. > > This is great, but could you please add a test case to the wic > selftests to exercise this and ensure it doesn’t break in the future? This has been waiting on the tests for a while in master-next. Is that something you are able to do? Cheers, Richard
Sorry for the late response... I’m working this week on a selftest for the case fixed by this patch. Meanwhile, I was focused on another aspect related to this patch. Our client reported different behavior when copying complex directory structures onto a WIC image using different servers. It appears that the issue is related to the debugfs version (e2fsprogs package). On Ubuntu 24.04 with e2fsprogs 4.17, all files are copied correctly, while on Ubuntu 20.04 with e2fsprogs 1.45, the copy operation fails to copy all files and does not create all directories. This failure is silent, with no error reported by debugfs. I tried several approaches to solve this (splitting the command list in max 100 instances and calling debugfs 1.45 multiple times), but without success. For our client, we decided to add a debugfs version check in wic code and to add a warning if the version is < 1.47. On those specific servers, they will use wic with the -n/--native-sysroot option, pointing to a scarthgap SDK that contains 1.47 version of the e2fsprogs package. How should we handle this exception? This was fixed in newer debugfs version, but should we add a debugfs version warning when broken version of debugfs is used on host? Regards, Daniel ------------------------------------------------------------------------ *From:* Ross Burton <Ross.Burton@arm.com> *Sent:* Monday, October 6, 2025 11:08 PM *To:* Dragomir, Daniel <Daniel.Dragomir@windriver.com> *Cc:* openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> *Subject:* Re: [OE-core][PATCH] wic/engine: fix copying directories into wic image with ext* partition CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe. On 3 Oct 2025, at 21:31, Dragomir, Daniel via lists.openembedded.org <daniel.dragomir=windriver.com@lists.openembedded.org> wrote: > > wic uses debugfs to write on ext* partitions, but debugfs can only > write to the current working directory and it cannot copy complete > directory trees. Running 'wic ls' on a copied directory show this: > -l: Ext2 inode is not a directory > > Fix this by creating a command list for debugfs (-f parameter) when > recursive parsing the host directory in order to create a similar > directory structure (mkdir) and copy files (write) on each level > into the destination directory from the wic's ext* partition. This is great, but could you please add a test case to the wic selftests to exercise this and ensure it doesn’t break in the future? Thanks, Ross
On Wed, 2025-12-17 at 15:15 +0200, Dragomir, Daniel via lists.openembedded.org wrote: > How should we handle this exception? This was fixed in newer debugfs version, but should we add a debugfs version warning when broken version of debugfs is used on host? > > Thanks for the selftest, that helps a lot. For the wic issue, I think a version check inside wic would be fine. Please just make it clear in the commit message this is for standalone use and we don't see this in OE since our version is new enough. Cheers, Richard
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py index b9e60cbe4e..9d596be3a7 100644 --- a/scripts/lib/wic/engine.py +++ b/scripts/lib/wic/engine.py @@ -345,29 +345,64 @@ class Disk: path)) def copy(self, src, dest): - """Copy partition image into wic image.""" - pnum = dest.part if isinstance(src, str) else src.part + """Copy files or directories to/from the vfat or ext* partition.""" + pnum = dest.part if isinstance(src, str) else src.part + partimg = self._get_part_image(pnum) if self.partitions[pnum].fstype.startswith('ext'): - if isinstance(src, str): - cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\ - format(os.path.dirname(dest.path), src, os.path.basename(src), - self.debugfs, self._get_part_image(pnum)) - else: # copy from wic - # run both dump and rdump to support both files and directory + if isinstance(src, str): # host to image case + if os.path.isdir(src): + base = os.path.abspath(src) + base_parent = os.path.dirname(base) + cmds = [] + made = set() + + for root, dirs, files in os.walk(base): + for fname in files: + host_file = os.path.join(root, fname) + rel = os.path.relpath(host_file, base_parent) + dest_file = os.path.join(dest.path, rel) + dest_dir = os.path.dirname(dest_file) + + # create dir structure (mkdir -p) + parts = dest_dir.strip('/').split('/') + cur = '' + for p in parts: + cur = cur + '/' + p + if cur not in made: + cmds.append(f'mkdir "{cur}"') + made.add(cur) + + cmds.append(f'write "{host_file}" "{dest_file}"') + + # write script to a temp file + with tempfile.NamedTemporaryFile(mode='w', delete=False, + prefix='wic-debugfs-') as tf: + for line in cmds: + tf.write(line + '\n') + scriptname = tf.name + + cmd = f"{self.debugfs} -w -f {scriptname} {partimg}" + + else: # single file + cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\ + format(os.path.dirname(dest.path), src, + os.path.basename(src), self.debugfs, partimg) + + else: # image to host case cmd = "printf 'cd {}\ndump /{} {}\nrdump /{} {}\n' | {} {}".\ format(os.path.dirname(src.path), src.path, - dest, src.path, dest, self.debugfs, - self._get_part_image(pnum)) + dest, src.path, dest, self.debugfs, partimg) + else: # fat if isinstance(src, str): cmd = "{} -i {} -snop {} ::{}".format(self.mcopy, - self._get_part_image(pnum), - src, dest.path) + partimg, + src, dest.path) else: cmd = "{} -i {} -snop ::{} {}".format(self.mcopy, - self._get_part_image(pnum), - src.path, dest) + partimg, + src.path, dest) exec_cmd(cmd, as_shell=True) self._put_part_image(pnum)
wic uses debugfs to write on ext* partitions, but debugfs can only write to the current working directory and it cannot copy complete directory trees. Running 'wic ls' on a copied directory show this: -l: Ext2 inode is not a directory Fix this by creating a command list for debugfs (-f parameter) when recursive parsing the host directory in order to create a similar directory structure (mkdir) and copy files (write) on each level into the destination directory from the wic's ext* partition. Signed-off-by: Daniel Dragomir <daniel.dragomir@windriver.com> --- scripts/lib/wic/engine.py | 63 ++++++++++++++++++++++++++++++--------- 1 file changed, 49 insertions(+), 14 deletions(-)