| Message ID | 20231020124450.9736-4-niko.mauno@vaisala.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/4] package_rpm.bbclass: Fix some pycodestyle issues | expand |
On Fri, 2023-10-20 at 12:44 +0000, Niko Mauno via lists.openembedded.org wrote: > From: Niko Mauno <niko.mauno@vaisala.com> > > Commit 4a4d5f78a6962dda5f63e9891825c80a8a87bf66 ("package_rpm: use zstd > instead of xz") changed the rpm package compressor from 'xz' to 'zstd' > which results in decompression failure with BusyBox-provided 'rpm2cpio' > applet and 'rpm' applet when given the '-i' (Install package) option: > > rpm2cpio: no gzip/bzip2/xz magic > > In order to maintain compatibility with BusyBox introduce new variable > which can be overridden like: > > RPM_COMPRESSION = "xz" > > to enable rpm decompression without including the full rpm package in > the resulting root filesystem. Is it common for people to need to manipulate rpms on target without rpm being present using busybox? Do you know if busybox plans to add zstd support? As far as I could tell when we looked at this, the rpm world was moving over to zstd so adding in conditional xz support for a limited use case probably just creates a maintenance headache going forward as it isn't something we test or plan to test? Are you using this on an LTS release or master? Cheers, Richard
On 20.10.2023 16.00, Richard Purdie wrote: > Is it common for people to need to manipulate rpms on target without > rpm being present using busybox? Do you know if busybox plans to add > zstd support? > > As far as I could tell when we looked at this, the rpm world was moving > over to zstd so adding in conditional xz support for a limited use case > probably just creates a maintenance headache going forward as it isn't > something we test or plan to test? > > Are you using this on an LTS release or master? Hi Richard, After moving to Yocto Kirkstone (LTS) which we currently use, we started getting developer reports of 'no gzip/bzip2/xz magic' error from BusyBox applet. For example developer uploads an rpm file to target device and then unpacks it in runtime using 'rpm2cpio <somepkg.rpm> | cpio -idmv' command. We don't include rpm package on target filesystems as that would increase rootfs footprint approximately 5 MB. Including only rpm2cpio from rpm package needs still accompanying libraries, increasing the size by 2.5 MB approximately which also feels too big. I checked the discussions from BusyBox mailing list and found that in 2021 the maintainer seemed reluctant to accept the proposed zstd support because it was not optimized for embedded systems. http://lists.busybox.net/pipermail/busybox/2021-September/089179.html BR, Niko
On 10/20/23 6:42 AM, Niko Mauno via lists.openembedded.org wrote: > > > On 20.10.2023 16.00, Richard Purdie wrote: >> Is it common for people to need to manipulate rpms on target without >> rpm being present using busybox? Do you know if busybox plans to add >> zstd support? >> >> As far as I could tell when we looked at this, the rpm world was moving >> over to zstd so adding in conditional xz support for a limited use case >> probably just creates a maintenance headache going forward as it isn't >> something we test or plan to test? >> >> Are you using this on an LTS release or master? > > Hi Richard, > > After moving to Yocto Kirkstone (LTS) which we currently use, we started > getting developer reports of 'no gzip/bzip2/xz magic' error from BusyBox > applet. For example developer uploads an rpm file to target device and > then unpacks it in runtime using 'rpm2cpio <somepkg.rpm> | cpio -idmv' > command. > > We don't include rpm package on target filesystems as that would > increase rootfs footprint approximately 5 MB. Including only rpm2cpio > from rpm package needs still accompanying libraries, increasing the size > by 2.5 MB approximately which also feels too big. > it seems you are quite sensitive to size, I wonder if opkg backend is better suited for your usecase than rpm. > I checked the discussions from BusyBox mailing list and found that in > 2021 the maintainer seemed reluctant to accept the proposed zstd support > because it was not optimized for embedded systems. > http://lists.busybox.net/pipermail/busybox/2021-September/089179.html > > BR, Niko > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#189532): https://lists.openembedded.org/g/openembedded-core/message/189532 > Mute This Topic: https://lists.openembedded.org/mt/102080114/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 10/20/23 16:00, Richard Purdie wrote: > As far as I could tell when we looked at this, the rpm world was moving > over to zstd so adding in conditional xz support for a limited use case > probably just creates a maintenance headache going forward as it isn't > something we test or plan to test? I now submitted a v2 for your consideration, but assuming it still introduces a maintenance challenge for YP, we will look into other means to work around the issue. -Niko
On 20.10.2023 19.34, Khem Raj wrote: > it seems you are quite sensitive to size, I wonder if opkg backend is > better suited for your usecase than rpm. Hi Khem, thanks for the idea. We used opkg a few years ago, however certain technical reasons were in favor of rpm which we have used since. Perhaps we have a bit different view, smaller image means for us spending less time building, booting and flashing firmware (i.e. not just disk space consumption alone). -Niko
On Tue, Oct 24, 2023 at 5:49 AM Niko Mauno <niko.mauno@vaisala.com> wrote: > > On 20.10.2023 19.34, Khem Raj wrote: > > it seems you are quite sensitive to size, I wonder if opkg backend is > > better suited for your usecase than rpm. > > Hi Khem, thanks for the idea. We used opkg a few years ago, however > certain technical reasons were in favor of rpm which we have used since. > Perhaps we have a bit different view, smaller image means for us > spending less time building, booting and flashing firmware (i.e. not > just disk space consumption alone). Using xz certainly is slower than using zstd so it certainly will regress on your first consideration about build time. booting is perhaps not impacted unless compressed file system is used and you are changing the compression type for that. flashing again is subjective, if you carry package metadata into images it might matter otherwise I would think your images with opkg or rpm should be about the same size if you include packager into your image then opkg might be slightly smaller. > > -Niko
diff --git a/meta/classes-global/package_rpm.bbclass b/meta/classes-global/package_rpm.bbclass index 246106ea4f..da25b7682c 100644 --- a/meta/classes-global/package_rpm.bbclass +++ b/meta/classes-global/package_rpm.bbclass @@ -13,6 +13,9 @@ RPMBUILD = "rpmbuild" PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms" +# Override variable to use alternative 'xz' or 'none' compression +RPM_COMPRESSION ?= "zstd" + # Maintaining the perfile dependencies has significant overhead when writing the # packages. When set, this value merges them for efficiency. MERGEPERFILEDEPS = "1" @@ -659,6 +662,7 @@ python do_package_rpm () { # Setup the rpmbuild arguments... rpmbuild = d.getVar('RPMBUILD') + rpmcomp = d.getVar('RPM_COMPRESSION') # Too many places in dnf stack assume that arch-independent packages are "noarch". # Let's not fight against this. @@ -682,8 +686,16 @@ python do_package_rpm () { cmd = cmd + " --define '_use_internal_dependency_generator 0'" cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'" cmd = cmd + " --define '_build_id_links none'" - cmd = cmd + " --define '_binary_payload w19T%d.zstdio'" % int(d.getVar("ZSTD_THREADS")) - cmd = cmd + " --define '_source_payload w19T%d.zstdio'" % int(d.getVar("ZSTD_THREADS")) + if rpmcomp == "zstd": + cmd = cmd + " --define '_binary_payload w19T%d.zstdio'" % int(d.getVar("ZSTD_THREADS")) + cmd = cmd + " --define '_source_payload w19T%d.zstdio'" % int(d.getVar("ZSTD_THREADS")) + elif rpmcomp == 'xz': + cmd = cmd + " --define '_binary_payload w6T%d.xzdio'" % int(d.getVar("XZ_THREADS")) + cmd = cmd + " --define '_source_payload w6T%d.xzdio'" % int(d.getVar("XZ_THREADS")) + elif rpmcomp == 'none': + pass + else: + bb.fatal('unsupported rpm compression method: "%s"' % rpmcomp) cmd = cmd + " --define 'clamp_mtime_to_source_date_epoch 1'" cmd = cmd + " --define 'use_source_date_epoch_as_buildtime 1'" cmd = cmd + " --define '_buildhost reproducible'"