| Message ID | 20240502142716.2112863-1-philip.lorenz@bmw.de |
|---|---|
| State | New |
| Headers | show |
| Series | ipk: Do not hardcode payload compression algorithm | expand |
Hi Philip, On 5/2/24 4:27 PM, Philip Lorenz via lists.openembedded.org wrote: > [You don't often get email from philip.lorenz=bmw.de@lists.openembedded.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] > > The chosen payload compression algorithm can be changed by overriding > `OPKGBUILDCMD`. Ensure that package extraction deals with this by > globbing for "data.tar.*" to select the actual payload tarball. > > Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> > --- > meta/lib/oe/package_manager/ipk/__init__.py | 8 +++++++- > 1 file changed, 7 insertions(+), 1 deletion(-) > > diff --git a/meta/lib/oe/package_manager/ipk/__init__.py b/meta/lib/oe/package_manager/ipk/__init__.py > index 8cc9953a027..0f0038d00d9 100644 > --- a/meta/lib/oe/package_manager/ipk/__init__.py > +++ b/meta/lib/oe/package_manager/ipk/__init__.py > @@ -4,6 +4,7 @@ > # SPDX-License-Identifier: GPL-2.0-only > # > > +import glob > import re > import shutil > import subprocess > @@ -134,11 +135,16 @@ class OpkgDpkgPM(PackageManager): > tmp_dir = tempfile.mkdtemp() > current_dir = os.getcwd() > os.chdir(tmp_dir) > - data_tar = 'data.tar.zst' > > try: > cmd = [ar_cmd, 'x', pkg_path] > output = subprocess.check_output(cmd, stderr=subprocess.STDOUT) > + data_tar = glob.glob("data.tar.*") What happens if we have two consecutive builds with the same build directory but different compression algorithms? Would we have two data.tar.* matches? Basically, I'm wondering here if we shouldn't have a way to extract the compression algorithm from a variable instead of trying to guess? Cheers, Quentin
Hi Quentin, On 02.05.24 19:19, Quentin Schulz wrote: > Sent from outside the BMW organization - be CAUTIOUS, particularly > with links and attachments. > Absender außerhalb der BMW Organisation - Bitte VORSICHT beim Öffnen > von Links und Anhängen. > ----------------------------------------------------------------------------------------------- > > > Hi Philip, > > On 5/2/24 4:27 PM, Philip Lorenz via lists.openembedded.org wrote: >> [You don't often get email from >> philip.lorenz=bmw.de@lists.openembedded.org. Learn why this is >> important at https://aka.ms/LearnAboutSenderIdentification ] >> >> The chosen payload compression algorithm can be changed by overriding >> `OPKGBUILDCMD`. Ensure that package extraction deals with this by >> globbing for "data.tar.*" to select the actual payload tarball. >> >> Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> >> --- >> meta/lib/oe/package_manager/ipk/__init__.py | 8 +++++++- >> 1 file changed, 7 insertions(+), 1 deletion(-) >> >> diff --git a/meta/lib/oe/package_manager/ipk/__init__.py >> b/meta/lib/oe/package_manager/ipk/__init__.py >> index 8cc9953a027..0f0038d00d9 100644 >> --- a/meta/lib/oe/package_manager/ipk/__init__.py >> +++ b/meta/lib/oe/package_manager/ipk/__init__.py >> @@ -4,6 +4,7 @@ >> # SPDX-License-Identifier: GPL-2.0-only >> # >> >> +import glob >> import re >> import shutil >> import subprocess >> @@ -134,11 +135,16 @@ class OpkgDpkgPM(PackageManager): >> tmp_dir = tempfile.mkdtemp() >> current_dir = os.getcwd() >> os.chdir(tmp_dir) >> - data_tar = 'data.tar.zst' >> >> try: >> cmd = [ar_cmd, 'x', pkg_path] >> output = subprocess.check_output(cmd, >> stderr=subprocess.STDOUT) >> + data_tar = glob.glob("data.tar.*") > > What happens if we have two consecutive builds with the same build > directory but different compression algorithms? Would we have two > data.tar.* matches? Each call to extract() creates a new temporary directory into which the content of that particular IPK are extracted. IPKs should contain only one data tarball and as such the described scenario should never occur (and the length check I added is a merely a safety measure). I initially considered introducing a dedicated variable to specify the IPK compression algorithm, but this would have also required to introduce * another variable for the parametrization of said algorithm (see ZSTD_DEFAULTS, XZ_DEFAULTS) * a mapping from opkg-build algorithm name to the actual file extension (see [1]) This seems overly complex when a glob achieves the same effect so I went for the submitted solution instead. Br, Philip [1] https://git.yoctoproject.org/opkg-utils/tree/opkg-build#n157
Hi Philip, On 5/2/24 8:00 PM, Philip Lorenz wrote: > Hi Quentin, > > On 02.05.24 19:19, Quentin Schulz wrote: >> Sent from outside the BMW organization - be CAUTIOUS, particularly >> with links and attachments. >> Absender außerhalb der BMW Organisation - Bitte VORSICHT beim Öffnen >> von Links und Anhängen. >> ----------------------------------------------------------------------------------------------- >> >> Hi Philip, >> >> On 5/2/24 4:27 PM, Philip Lorenz via lists.openembedded.org wrote: >>> [You don't often get email from >>> philip.lorenz=bmw.de@lists.openembedded.org. Learn why this is >>> important at https://aka.ms/LearnAboutSenderIdentification ] >>> >>> The chosen payload compression algorithm can be changed by overriding >>> `OPKGBUILDCMD`. Ensure that package extraction deals with this by >>> globbing for "data.tar.*" to select the actual payload tarball. >>> >>> Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> >>> --- >>> meta/lib/oe/package_manager/ipk/__init__.py | 8 +++++++- >>> 1 file changed, 7 insertions(+), 1 deletion(-) >>> >>> diff --git a/meta/lib/oe/package_manager/ipk/__init__.py >>> b/meta/lib/oe/package_manager/ipk/__init__.py >>> index 8cc9953a027..0f0038d00d9 100644 >>> --- a/meta/lib/oe/package_manager/ipk/__init__.py >>> +++ b/meta/lib/oe/package_manager/ipk/__init__.py >>> @@ -4,6 +4,7 @@ >>> # SPDX-License-Identifier: GPL-2.0-only >>> # >>> >>> +import glob >>> import re >>> import shutil >>> import subprocess >>> @@ -134,11 +135,16 @@ class OpkgDpkgPM(PackageManager): >>> tmp_dir = tempfile.mkdtemp() >>> current_dir = os.getcwd() >>> os.chdir(tmp_dir) >>> - data_tar = 'data.tar.zst' >>> >>> try: >>> cmd = [ar_cmd, 'x', pkg_path] >>> output = subprocess.check_output(cmd, >>> stderr=subprocess.STDOUT) >>> + data_tar = glob.glob("data.tar.*") >> >> What happens if we have two consecutive builds with the same build >> directory but different compression algorithms? Would we have two >> data.tar.* matches? > > Each call to extract() creates a new temporary directory into which the > content of that particular IPK are extracted. IPKs should contain only > one data tarball and as such the described scenario should never occur > (and the length check I added is a merely a safety measure). > This makes sense, thanks for taking the time to explain. Cheers, Quentin
diff --git a/meta/lib/oe/package_manager/ipk/__init__.py b/meta/lib/oe/package_manager/ipk/__init__.py index 8cc9953a027..0f0038d00d9 100644 --- a/meta/lib/oe/package_manager/ipk/__init__.py +++ b/meta/lib/oe/package_manager/ipk/__init__.py @@ -4,6 +4,7 @@ # SPDX-License-Identifier: GPL-2.0-only # +import glob import re import shutil import subprocess @@ -134,11 +135,16 @@ class OpkgDpkgPM(PackageManager): tmp_dir = tempfile.mkdtemp() current_dir = os.getcwd() os.chdir(tmp_dir) - data_tar = 'data.tar.zst' try: cmd = [ar_cmd, 'x', pkg_path] output = subprocess.check_output(cmd, stderr=subprocess.STDOUT) + data_tar = glob.glob("data.tar.*") + if len(data_tar) != 1: + bb.fatal("Unable to extract %s package. Failed to identify " + "data tarball (found tarballs '%s').", + pkg_path, data_tar) + data_tar = data_tar[0] cmd = [tar_cmd, 'xf', data_tar] output = subprocess.check_output(cmd, stderr=subprocess.STDOUT) except subprocess.CalledProcessError as e:
The chosen payload compression algorithm can be changed by overriding `OPKGBUILDCMD`. Ensure that package extraction deals with this by globbing for "data.tar.*" to select the actual payload tarball. Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> --- meta/lib/oe/package_manager/ipk/__init__.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)