| Message ID | 20230530131708.1916975-1-mingli.yu@eng.windriver.com |
|---|---|
| State | New |
| Headers | show |
| Series | qemu: Split the qemu package | expand |
On Tue, 2023-05-30 at 21:17 +0800, Yu, Mingli wrote: > From: Mingli Yu <mingli.yu@windriver.com> > > Currently all files packaged into one package such as qemu-7.2.0-*.rpm. > After the qemu package installed on the target, it will take up 464M > which includes not only the one matches the arch of the target but aslo > all available built qemu targets which set by QEMU_TARGETS. > > Split the qemu package into qemu-7.2.0-*.rpm, qemu-aarch64-7.2.0*.rpm, > qemu-arm-7.2.0*.rpm, qemu-x86_64-7.2.0*.rpm and etc. And let user can only > choose the corresponding qemu arch package they want to install should ease > the concerns who cares much about the size in embedded device as it > decreases the qemu rpm(qemu-7.2.0*.rpm) size from about 65M to about 17M > and the size of the extracted qemu RPM decreased from about 464M to about > 248M. > > Signed-off-by: Mingli Yu <mingli.yu@windriver.com> > --- > meta/recipes-devtools/qemu/qemu.inc | 19 +++++++++++++++++++ > 1 file changed, 19 insertions(+) > > diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc > index a87dee5c99..7302d63747 100644 > --- a/meta/recipes-devtools/qemu/qemu.inc > +++ b/meta/recipes-devtools/qemu/qemu.inc > @@ -230,6 +230,25 @@ INSANE_SKIP:${PN} = "arch" > > FILES:${PN} += "${datadir}/icons" > > +python(){ > + allarchs = d.getVar('QEMU_TARGETS').split() > + packages = d.getVar('PACKAGES').split() > + pn = d.getVar('PN') > + > + newpackages=[] > + for arch in allarchs: > + archpackage = 'qemu-' + arch > + if archpackage not in packages: > + newpackages.append(archpackage) Are any of the architectures already in packages? I'm a little curious if there are and why? > + d.setVar('FILES:' + pn + "-" + arch, '${bindir}/qemu-' + arch) > + d.appendVar('FILES:' + pn + "-" + arch, ' ' + '${bindir}/qemu-system-' + arch) Why set a variable, then append to it? > + if arch == "mips": > + d.appendVar('RDEPENDS:' + pn + '-' + arch, ' ' + 'bash') May as well just do: RDEPENDS:${PN}-mips += "bash" ? > + packages = newpackages + packages > + d.setVar('PACKAGES', ' '.join(packages)) > +} Also, should the main qemu package pull in all the other subpackages? Cheers, Richard
On Tue, May 30, 2023 at 9:17 AM Yu, Mingli <mingli.yu@eng.windriver.com> wrote: > > From: Mingli Yu <mingli.yu@windriver.com> > > Currently all files packaged into one package such as qemu-7.2.0-*.rpm. > After the qemu package installed on the target, it will take up 464M > which includes not only the one matches the arch of the target but aslo > all available built qemu targets which set by QEMU_TARGETS. > > Split the qemu package into qemu-7.2.0-*.rpm, qemu-aarch64-7.2.0*.rpm, > qemu-arm-7.2.0*.rpm, qemu-x86_64-7.2.0*.rpm and etc. And let user can only > choose the corresponding qemu arch package they want to install should ease > the concerns who cares much about the size in embedded device as it > decreases the qemu rpm(qemu-7.2.0*.rpm) size from about 65M to about 17M > and the size of the extracted qemu RPM decreased from about 464M to about > 248M. I've been splitting the qemu packages in meta-virtualization for years now. I find the python more difficult to read than just using overrides, as it is much more explicit. Is there a reason why that approach didn't work in this scenario ? As Richard mentioned, you should also pull in all the subpackages by default with the main qemu package, to avoid regressions. Also, in the experience of meta-virt, you only want this splitting on target builds, not native/host. Bruce > > Signed-off-by: Mingli Yu <mingli.yu@windriver.com> > --- > meta/recipes-devtools/qemu/qemu.inc | 19 +++++++++++++++++++ > 1 file changed, 19 insertions(+) > > diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc > index a87dee5c99..7302d63747 100644 > --- a/meta/recipes-devtools/qemu/qemu.inc > +++ b/meta/recipes-devtools/qemu/qemu.inc > @@ -230,6 +230,25 @@ INSANE_SKIP:${PN} = "arch" > > FILES:${PN} += "${datadir}/icons" > > +python(){ > + allarchs = d.getVar('QEMU_TARGETS').split() > + packages = d.getVar('PACKAGES').split() > + pn = d.getVar('PN') > + > + newpackages=[] > + for arch in allarchs: > + archpackage = 'qemu-' + arch > + if archpackage not in packages: > + newpackages.append(archpackage) > + d.setVar('FILES:' + pn + "-" + arch, '${bindir}/qemu-' + arch) > + d.appendVar('FILES:' + pn + "-" + arch, ' ' + '${bindir}/qemu-system-' + arch) > + if arch == "mips": > + d.appendVar('RDEPENDS:' + pn + '-' + arch, ' ' + 'bash') > + packages = newpackages + packages > + d.setVar('PACKAGES', ' '.join(packages)) > +} > + > + > # Put the guest agent in a separate package > PACKAGES =+ "${PN}-guest-agent" > SUMMARY:${PN}-guest-agent = "QEMU guest agent" > -- > 2.25.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#181922): https://lists.openembedded.org/g/openembedded-core/message/181922 > Mute This Topic: https://lists.openembedded.org/mt/99219254/1050810 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
I might be missing something here, but can the free-form, anonymous python code block be avoided? Don't we have PACKAGES_DYNAMIC for this purpose? Alex On Tue, 30 May 2023 at 15:51, Bruce Ashfield <bruce.ashfield@gmail.com> wrote: > > On Tue, May 30, 2023 at 9:17 AM Yu, Mingli <mingli.yu@eng.windriver.com> wrote: > > > > From: Mingli Yu <mingli.yu@windriver.com> > > > > Currently all files packaged into one package such as qemu-7.2.0-*.rpm. > > After the qemu package installed on the target, it will take up 464M > > which includes not only the one matches the arch of the target but aslo > > all available built qemu targets which set by QEMU_TARGETS. > > > > Split the qemu package into qemu-7.2.0-*.rpm, qemu-aarch64-7.2.0*.rpm, > > qemu-arm-7.2.0*.rpm, qemu-x86_64-7.2.0*.rpm and etc. And let user can only > > choose the corresponding qemu arch package they want to install should ease > > the concerns who cares much about the size in embedded device as it > > decreases the qemu rpm(qemu-7.2.0*.rpm) size from about 65M to about 17M > > and the size of the extracted qemu RPM decreased from about 464M to about > > 248M. > > I've been splitting the qemu packages in meta-virtualization for years now. > > I find the python more difficult to read than just using overrides, as > it is much more explicit. > > Is there a reason why that approach didn't work in this scenario ? > > As Richard mentioned, you should also pull in all the subpackages by > default with the main qemu package, to avoid regressions. > > Also, in the experience of meta-virt, you only want this splitting on > target builds, not native/host. > > Bruce > > > > > Signed-off-by: Mingli Yu <mingli.yu@windriver.com> > > --- > > meta/recipes-devtools/qemu/qemu.inc | 19 +++++++++++++++++++ > > 1 file changed, 19 insertions(+) > > > > diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc > > index a87dee5c99..7302d63747 100644 > > --- a/meta/recipes-devtools/qemu/qemu.inc > > +++ b/meta/recipes-devtools/qemu/qemu.inc > > @@ -230,6 +230,25 @@ INSANE_SKIP:${PN} = "arch" > > > > FILES:${PN} += "${datadir}/icons" > > > > +python(){ > > + allarchs = d.getVar('QEMU_TARGETS').split() > > + packages = d.getVar('PACKAGES').split() > > + pn = d.getVar('PN') > > + > > + newpackages=[] > > + for arch in allarchs: > > + archpackage = 'qemu-' + arch > > + if archpackage not in packages: > > + newpackages.append(archpackage) > > + d.setVar('FILES:' + pn + "-" + arch, '${bindir}/qemu-' + arch) > > + d.appendVar('FILES:' + pn + "-" + arch, ' ' + '${bindir}/qemu-system-' + arch) > > + if arch == "mips": > > + d.appendVar('RDEPENDS:' + pn + '-' + arch, ' ' + 'bash') > > + packages = newpackages + packages > > + d.setVar('PACKAGES', ' '.join(packages)) > > +} > > + > > + > > # Put the guest agent in a separate package > > PACKAGES =+ "${PN}-guest-agent" > > SUMMARY:${PN}-guest-agent = "QEMU guest agent" > > -- > > 2.25.1 > > > > > > > > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#181924): https://lists.openembedded.org/g/openembedded-core/message/181924 > Mute This Topic: https://lists.openembedded.org/mt/99219254/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Tue, 2023-05-30 at 16:33 +0200, Alexander Kanavin wrote: > I might be missing something here, but can the free-form, anonymous > python code block be avoided? Don't we have PACKAGES_DYNAMIC for this > purpose? PACKAGES_DYNAMIC is for when we can't predict the packages a recipe might generate. A good example might be kernel modules. You're right that we could add a do_split_packages() call to the qemu recipe have have it generate these dynamically. The downside would be the namespacing as dynamic packages need to have specific namespaces (e.g. kernel-module-XXX). This means qemu-mips wouldn't be an option (conflicts with non dynamic packages like qemu- dbg). We could use a more specific prefix like qemu-system-XXX and qemu-user- XXX and use do_split_packages I did also wonder about using more specific inline python for some of this, things along the lines of: PACKAGES += '${@" ".join("qemu-system-" + x for x in d.getVar('QEMU_TARGETS').split())}' I'm also not a fan of the python code block. We do use do_split_packages() in other recipes like gstreamer to handle things like this. Cheers, Richard
On Tue, May 30, 2023 at 10:54 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Tue, 2023-05-30 at 16:33 +0200, Alexander Kanavin wrote: > > I might be missing something here, but can the free-form, anonymous > > python code block be avoided? Don't we have PACKAGES_DYNAMIC for this > > purpose? > > PACKAGES_DYNAMIC is for when we can't predict the packages a recipe > might generate. A good example might be kernel modules. > > You're right that we could add a do_split_packages() call to the qemu > recipe have have it generate these dynamically. > > The downside would be the namespacing as dynamic packages need to have > specific namespaces (e.g. kernel-module-XXX). This means qemu-mips > wouldn't be an option (conflicts with non dynamic packages like qemu- > dbg). > > We could use a more specific prefix like qemu-system-XXX and qemu-user- > XXX and use do_split_packages > > I did also wonder about using more specific inline python for some of > this, things along the lines of: > > PACKAGES += '${@" ".join("qemu-system-" + x for x in d.getVar('QEMU_TARGETS').split())}' > > I'm also not a fan of the python code block. > > We do use do_split_packages() in other recipes like gstreamer to handle > things like this. > And in case anyone hasn't looked it up, this is the meta-virt solution: https://git.yoctoproject.org/meta-virtualization/tree/recipes-devtools/qemu/qemu-package-split.inc Which I'll have to re-work once (if) something lands in core. It isn't suitable as-is, but it doesn't need any python code to suit the on-target system emulation needs of meta-virt. Bruce > Cheers, > > Richard >
On 5/30/23 23:09, Bruce Ashfield wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Tue, May 30, 2023 at 10:54 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: >> >> On Tue, 2023-05-30 at 16:33 +0200, Alexander Kanavin wrote: >>> I might be missing something here, but can the free-form, anonymous >>> python code block be avoided? Don't we have PACKAGES_DYNAMIC for this >>> purpose? >> >> PACKAGES_DYNAMIC is for when we can't predict the packages a recipe >> might generate. A good example might be kernel modules. >> >> You're right that we could add a do_split_packages() call to the qemu >> recipe have have it generate these dynamically. >> >> The downside would be the namespacing as dynamic packages need to have >> specific namespaces (e.g. kernel-module-XXX). This means qemu-mips >> wouldn't be an option (conflicts with non dynamic packages like qemu- >> dbg). >> >> We could use a more specific prefix like qemu-system-XXX and qemu-user- >> XXX and use do_split_packages >> >> I did also wonder about using more specific inline python for some of >> this, things along the lines of: >> >> PACKAGES += '${@" ".join("qemu-system-" + x for x in d.getVar('QEMU_TARGETS').split())}' >> >> I'm also not a fan of the python code block. >> >> We do use do_split_packages() in other recipes like gstreamer to handle >> things like this. >> > > And in case anyone hasn't looked it up, this is the meta-virt solution: > > https://git.yoctoproject.org/meta-virtualization/tree/recipes-devtools/qemu/qemu-package-split.inc Thanks! I did see this before I send out https://patchwork.yoctoproject.org/project/oe-core/patch/20230530131708.1916975-1-mingli.yu@eng.windriver.com. Considering to dynamically generate the sub-packages via QEMU_TARGETS, so I use a python block and don't need the change the code even QEMU_TARGETS has some change. But I should let all sub-package to depend the qemu-7.2.0*.rpm. Thanks, > > Which I'll have to re-work once (if) something lands in core. > > It isn't suitable as-is, but it doesn't need any python code to suit > the on-target system emulation needs of meta-virt. > > Bruce > >> Cheers, >> >> Richard >> > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II
On Thu, Jun 1, 2023 at 5:33 AM Yu, Mingli <mingli.yu@windriver.com> wrote: > > > > On 5/30/23 23:09, Bruce Ashfield wrote: > > CAUTION: This email comes from a non Wind River email account! > > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > On Tue, May 30, 2023 at 10:54 AM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > >> > >> On Tue, 2023-05-30 at 16:33 +0200, Alexander Kanavin wrote: > >>> I might be missing something here, but can the free-form, anonymous > >>> python code block be avoided? Don't we have PACKAGES_DYNAMIC for this > >>> purpose? > >> > >> PACKAGES_DYNAMIC is for when we can't predict the packages a recipe > >> might generate. A good example might be kernel modules. > >> > >> You're right that we could add a do_split_packages() call to the qemu > >> recipe have have it generate these dynamically. > >> > >> The downside would be the namespacing as dynamic packages need to have > >> specific namespaces (e.g. kernel-module-XXX). This means qemu-mips > >> wouldn't be an option (conflicts with non dynamic packages like qemu- > >> dbg). > >> > >> We could use a more specific prefix like qemu-system-XXX and qemu-user- > >> XXX and use do_split_packages > >> > >> I did also wonder about using more specific inline python for some of > >> this, things along the lines of: > >> > >> PACKAGES += '${@" ".join("qemu-system-" + x for x in d.getVar('QEMU_TARGETS').split())}' > >> > >> I'm also not a fan of the python code block. > >> > >> We do use do_split_packages() in other recipes like gstreamer to handle > >> things like this. > >> > > > > And in case anyone hasn't looked it up, this is the meta-virt solution: > > > > https://git.yoctoproject.org/meta-virtualization/tree/recipes-devtools/qemu/qemu-package-split.inc > > Thanks! I did see this before I send out > https://patchwork.yoctoproject.org/project/oe-core/patch/20230530131708.1916975-1-mingli.yu@eng.windriver.com. > > > Considering to dynamically generate the sub-packages via QEMU_TARGETS, > so I use a python block and don't need the change the code even > QEMU_TARGETS has some change. QEMU_TARGETS is not something that changes very often. I still think that there needs to be a way to opt-out of the split packages, since there are some split requirements for virtualization, etc, that don't follow the relatively simple pattern being introduced here. With the python code and the :prepend, I don't see how I'll be able to opt out of it in meta-virtualization. Also, the dependency I see in the v2 patch isn't what I'd expect to keep existing use cases working. i.e. For kernel modules we have the package that rdepends on all the split packages, since there are expectations in places that all the qemu-system packages are installed. Bruce > > But I should let all sub-package to depend the qemu-7.2.0*.rpm. > > Thanks, > > > > > Which I'll have to re-work once (if) something lands in core. > > > > It isn't suitable as-is, but it doesn't need any python code to suit > > the on-target system emulation needs of meta-virt. > > > > Bruce > > > >> Cheers, > >> > >> Richard > >> > > > > > > -- > > - Thou shalt not follow the NULL pointer, for chaos and madness await > > thee at its end > > - "Use the force Harry" - Gandalf, Star Trek II -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc index a87dee5c99..7302d63747 100644 --- a/meta/recipes-devtools/qemu/qemu.inc +++ b/meta/recipes-devtools/qemu/qemu.inc @@ -230,6 +230,25 @@ INSANE_SKIP:${PN} = "arch" FILES:${PN} += "${datadir}/icons" +python(){ + allarchs = d.getVar('QEMU_TARGETS').split() + packages = d.getVar('PACKAGES').split() + pn = d.getVar('PN') + + newpackages=[] + for arch in allarchs: + archpackage = 'qemu-' + arch + if archpackage not in packages: + newpackages.append(archpackage) + d.setVar('FILES:' + pn + "-" + arch, '${bindir}/qemu-' + arch) + d.appendVar('FILES:' + pn + "-" + arch, ' ' + '${bindir}/qemu-system-' + arch) + if arch == "mips": + d.appendVar('RDEPENDS:' + pn + '-' + arch, ' ' + 'bash') + packages = newpackages + packages + d.setVar('PACKAGES', ' '.join(packages)) +} + + # Put the guest agent in a separate package PACKAGES =+ "${PN}-guest-agent" SUMMARY:${PN}-guest-agent = "QEMU guest agent"