| Message ID | 20250708124711.3541367-1-martin.jansa@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | fetch2: zstd: use --force to allow decompressing symlinks | expand |
On Tue, 8 Jul 2025 at 14:47, Martin Jansa via lists.openembedded.org <martin.jansa=gmail.com@lists.openembedded.org> wrote: > - cmd = 'zstd --decompress --stdout %s | %s -f -' % (file, tar_cmd) > + cmd = 'zstd --force --decompress --stdout %s | %s -f -' % (file, tar_cmd) > elif file.endswith('.zst'): > - cmd = 'zstd --decompress --stdout %s > %s' % (file, efile) > + cmd = 'zstd --force --decompress --stdout %s > %s' % (file, efile) • -f, --force: disable input and output checks. Allows overwriting existing files, input from console, output to stdout, operating on links, block devices, etc. During decompression and when the output destination is stdout, pass-through unrecognized formats as-is. I'm not sure using this --force option to fix this one particular issue is right. It will mask other problems that should stop the build at the decompression point. Alex
On Tue, Jul 8, 2025 at 4:45 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > On Tue, 8 Jul 2025 at 14:47, Martin Jansa via lists.openembedded.org > <martin.jansa=gmail.com@lists.openembedded.org> wrote: > > - cmd = 'zstd --decompress --stdout %s | %s -f -' % (file, tar_cmd) > > + cmd = 'zstd --force --decompress --stdout %s | %s -f -' % (file, tar_cmd) > > elif file.endswith('.zst'): > > - cmd = 'zstd --decompress --stdout %s > %s' % (file, efile) > > + cmd = 'zstd --force --decompress --stdout %s > %s' % (file, efile) > > > • -f, --force: disable input and output checks. Allows > overwriting existing files, input from console, output to stdout, > operating on links, block devices, etc. During decompression and when > the output > destination is stdout, pass-through unrecognized formats as-is. > > I'm not sure using this --force option to fix this one particular > issue is right. It will mask other problems that should stop the build > at the decompression point. So we're not going to fix the known issue, because there could be unknown potential issues? If zstd decompresses garbage, then tar will fail like it does now when zstd decompresses the symlinked archive to: "Warning : /OE/downloads/some-ver.tar.zst is a symbolic link, ignoring" message which tar doesn't recognize. and the symlink is created by the default bitbake fetcher behavior when MIRROR is over file:// (e.g. nfs mount). Luckily only a few recipes are using .zst (my DL_DIR only has one after building world with many layers).
On Wed, 16 Jul 2025 at 10:16, Martin Jansa <martin.jansa@gmail.com> wrote: > So we're not going to fix the known issue, because there could be > unknown potential issues? That's not what I said. I said we should seek a better solution than this. A patch to add --follow-symlinks, submitted upstream to zstd, would be much better received. Or some way to avoid having symlinks in the first place. Alex
On Wed, 16 Jul 2025 at 10:37, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > That's not what I said. I said we should seek a better solution than this. > > A patch to add --follow-symlinks, submitted upstream to zstd, would be > much better received. Or some way to avoid having symlinks in the > first place. I mean, we could also resolve the symlink before giving it to zstd, right? Trivially done in python. Alex
On Wed, Jul 16, 2025 at 10:37 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > On Wed, 16 Jul 2025 at 10:16, Martin Jansa <martin.jansa@gmail.com> wrote: > > So we're not going to fix the known issue, because there could be > > unknown potential issues? > > That's not what I said. I said we should seek a better solution than this. > > A patch to add --follow-symlinks, submitted upstream to zstd, would be > much better received. Or some way to avoid having symlinks in the > first place. --force works perfectly fine and I'm not too afraid of "overwriting existing files (as we're using --stdout anyway), input from console (as bitbake -s passing a file from SRC_URI), output to stdout (that's what we need), operating on links (that's what we need), block devices (if there is a recipe with block device ending with .zst in SRC_URI?), etc." I'm not going to spend my time adding --follow-symlinks in zstd-native unless you've an example where --force might make it worse than it currently is, because I won't be able to persuade upstream to accept that patch when I don't agree it's needed. Cheers,
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 0ad987c59..b3bf4d0b0 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1547,9 +1547,9 @@ class FetchMethod(object): elif file.endswith('.7z'): cmd = '7za x -y %s 1>/dev/null' % file elif file.endswith('.tzst') or file.endswith('.tar.zst'): - cmd = 'zstd --decompress --stdout %s | %s -f -' % (file, tar_cmd) + cmd = 'zstd --force --decompress --stdout %s | %s -f -' % (file, tar_cmd) elif file.endswith('.zst'): - cmd = 'zstd --decompress --stdout %s > %s' % (file, efile) + cmd = 'zstd --force --decompress --stdout %s > %s' % (file, efile) elif file.endswith('.zip') or file.endswith('.jar'): try: dos = bb.utils.to_boolean(urldata.parm.get('dos'), False)