| Message ID | 20250423083634.137495-1-mikko.rapeli@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | [v2] oeqa selftest: read qemu options from TEST_RUNQEMUPARAMS | expand |
On Wed, 2025-04-23 at 11:36 +0300, Mikko Rapeli via lists.openembedded.org wrote: > To support "slirp" userspace networking which works more easily > on various build machines, also without root and sudo rights > to setup loop interfaces. > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> > --- > meta/lib/oeqa/selftest/cases/barebox.py | 5 +++-- > meta/lib/oeqa/selftest/cases/debuginfod.py | 4 +++- > meta/lib/oeqa/selftest/cases/devtool.py | 6 ++++-- > .../oeqa/selftest/cases/efibootpartition.py | 3 ++- > meta/lib/oeqa/selftest/cases/gcc.py | 3 ++- > meta/lib/oeqa/selftest/cases/gdbserver.py | 3 ++- > meta/lib/oeqa/selftest/cases/glibc.py | 3 ++- > meta/lib/oeqa/selftest/cases/imagefeatures.py | 3 ++- > meta/lib/oeqa/selftest/cases/locales.py | 5 +++-- > meta/lib/oeqa/selftest/cases/overlayfs.py | 19 ++++++++++++------- > meta/lib/oeqa/selftest/cases/package.py | 6 ++++-- > meta/lib/oeqa/selftest/cases/runtime_test.py | 6 ++++-- > meta/lib/oeqa/selftest/cases/rust.py | 3 ++- > meta/lib/oeqa/selftest/cases/uboot.py | 5 +++-- > 14 files changed, 48 insertions(+), 26 deletions(-) > > v2: added missing get_bb_var import to u-boot.py > > v1: https://lists.openembedded.org/g/openembedded-core/message/215225 > > diff --git a/meta/lib/oeqa/selftest/cases/barebox.py b/meta/lib/oeqa/selftest/cases/barebox.py > index 3f8f232432..b4d8310666 100644 > --- a/meta/lib/oeqa/selftest/cases/barebox.py > +++ b/meta/lib/oeqa/selftest/cases/barebox.py > @@ -6,7 +6,7 @@ > # > > from oeqa.selftest.case import OESelftestTestCase > -from oeqa.utils.commands import bitbake, runqemu > +from oeqa.utils.commands import bitbake, runqemu, get_bb_var > from oeqa.core.decorator.data import skipIfNotArch > from oeqa.core.decorator import OETestTag > > @@ -34,7 +34,8 @@ QEMU_USE_KVM = "False" > > bitbake("virtual/bootloader core-image-minimal") > > - with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic', > + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" > + with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic %s' % (runqemu_params), > boot_patterns=barebox_boot_patterns) as qemu: > > # test if barebox console works This has sat in master-next for a long time, mostly because I've been putting off trying to write down my thoughts on this. The challenge with this change is that it makes slirp work in some cases but not others. We've been very clear up to now that tap/tun is our preferred way to run the tests. If this merges, people will start to expect slirp to become a first class citizen and file bugs against the tests which don't use it, or submit patches with only the slirp tests passing and them complain that we shouldn't have the other tests. I'm very wary of having "two ways" of doing things, particularly as one half always ends up with subtle breakage. I'm also wary of adding feature support by stealth, which this has potential to become. That said, I can totally understand why running the tests which can use slirp can be useful, particularly as there are some environments where tun/tap isn't possible. This means I am torn on the patch despite my reservations. There also isn't any testing data included. Did you run all the tests you are patching and are confirming they work with slirp or was this just a way to pass the option (and potentially other options) to all tests? Do we know how many tests work with slirp and how many don't? From a practicality standpoint, I do have an issue with the implementation since it duplicates the image name for the get_bb_var and the runqemu call and I'm not sure I like that implementation detail. That is a solvable problem but the decision above on whether we want to do this at all remains. I am also worried that users will set things in this variable which are incompatible with tests and it can potentially make it harder to debug user reported issues as we will have another setting we need them to confirm whether they're setting or not. There were ideas about decorators for tests in the patch review call and there is potential in that idea but again, it adds two ways to do things and is effectively making it a first class feature which I'm a little wary of. We do want users to be able to run the tests and help with with issues too though, so we come back to be being torn on it. So, I've written my thoughts down. I'm not sure that helps us much on deciding whether to merge it or not. Cheers, Richard
Hi, On Tue, May 20, 2025 at 10:30:04AM +0100, Richard Purdie wrote: > On Wed, 2025-04-23 at 11:36 +0300, Mikko Rapeli via lists.openembedded.org wrote: > > To support "slirp" userspace networking which works more easily > > on various build machines, also without root and sudo rights > > to setup loop interfaces. > > > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> > > --- > > �meta/lib/oeqa/selftest/cases/barebox.py������ |� 5 +++-- > > �meta/lib/oeqa/selftest/cases/debuginfod.py��� |� 4 +++- > > �meta/lib/oeqa/selftest/cases/devtool.py������ |� 6 ++++-- > > �.../oeqa/selftest/cases/efibootpartition.py�� |� 3 ++- > > �meta/lib/oeqa/selftest/cases/gcc.py���������� |� 3 ++- > > �meta/lib/oeqa/selftest/cases/gdbserver.py���� |� 3 ++- > > �meta/lib/oeqa/selftest/cases/glibc.py�������� |� 3 ++- > > �meta/lib/oeqa/selftest/cases/imagefeatures.py |� 3 ++- > > �meta/lib/oeqa/selftest/cases/locales.py������ |� 5 +++-- > > �meta/lib/oeqa/selftest/cases/overlayfs.py���� | 19 ++++++++++++------- > > �meta/lib/oeqa/selftest/cases/package.py������ |� 6 ++++-- > > �meta/lib/oeqa/selftest/cases/runtime_test.py� |� 6 ++++-- > > �meta/lib/oeqa/selftest/cases/rust.py��������� |� 3 ++- > > �meta/lib/oeqa/selftest/cases/uboot.py�������� |� 5 +++-- > > �14 files changed, 48 insertions(+), 26 deletions(-) > > > > v2: added missing get_bb_var import to u-boot.py > > > > v1: https://lists.openembedded.org/g/openembedded-core/message/215225 > > > > diff --git a/meta/lib/oeqa/selftest/cases/barebox.py b/meta/lib/oeqa/selftest/cases/barebox.py > > index 3f8f232432..b4d8310666 100644 > > --- a/meta/lib/oeqa/selftest/cases/barebox.py > > +++ b/meta/lib/oeqa/selftest/cases/barebox.py > > @@ -6,7 +6,7 @@ > > �# > > � > > �from oeqa.selftest.case import OESelftestTestCase > > -from oeqa.utils.commands import bitbake, runqemu > > +from oeqa.utils.commands import bitbake, runqemu, get_bb_var > > �from oeqa.core.decorator.data import skipIfNotArch > > �from oeqa.core.decorator import OETestTag > > � > > @@ -34,7 +34,8 @@ QEMU_USE_KVM = "False" > > � > > �������� bitbake("virtual/bootloader core-image-minimal") > > � > > -������� with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic', > > +������� runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" > > +������� with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic %s' % (runqemu_params), > > ��������������������� boot_patterns=barebox_boot_patterns) as qemu: > > � > > ������������ # test if barebox console works > > This has sat in master-next for a long time, mostly because I've been > putting off trying to write down my thoughts on this. > > The challenge with this change is that it makes slirp work in some > cases but not others. We've been very clear up to now that tap/tun is > our preferred way to run the tests. If this merges, people will start > to expect slirp to become a first class citizen and file bugs against > the tests which don't use it, or submit patches with only the slirp > tests passing and them complain that we shouldn't have the other tests. > > I'm very wary of having "two ways" of doing things, particularly as one > half always ends up with subtle breakage. I'm also wary of adding > feature support by stealth, which this has potential to become. > > That said, I can totally understand why running the tests which can use > slirp can be useful, particularly as there are some environments where > tun/tap isn't possible. This means I am torn on the patch despite my > reservations. > > There also isn't any testing data included. Did you run all the tests > you are patching and are confirming they work with slirp or was this > just a way to pass the option (and potentially other options) to all > tests? Do we know how many tests work with slirp and how many don't? I would like to run selftests on genericarm64 target machine and aarch64 build host. They don't pass yet. This patch is one of the things I would need, currently, to work on the other problems. A full series is a lot of work and I don't think I'm up to it atm, but maybe over time... > From a practicality standpoint, I do have an issue with the > implementation since it duplicates the image name for the get_bb_var > and the runqemu call and I'm not sure I like that implementation > detail. That is a solvable problem but the decision above on whether we > want to do this at all remains. I can fix this if needed. Some tests used variable for image name while many did not. > I am also worried that users will set things in this variable which are > incompatible with tests and it can potentially make it harder to debug > user reported issues as we will have another setting we need them to > confirm whether they're setting or not. > > There were ideas about decorators for tests in the patch review call > and there is potential in that idea but again, it adds two ways to do > things and is effectively making it a first class feature which I'm a > little wary of. > > We do want users to be able to run the tests and help with with issues > too though, so we come back to be being torn on it. > > So, I've written my thoughts down. I'm not sure that helps us much on > deciding whether to merge it or not. I'm relatively new to selftests and the environments and configs they run. slirp and support for it like in this patch is one thing, but there are others. Which build machine hosts and distributions are supported? I've heard that x86_64 and aarch64 hosts are used with the supported set of distros. Then the config inside of distros is where tun/tap and slirp are the alternatives. Yocto CI runs with tun/tap, but this may be harder to setup for us developers contributing changes to classes and need to adapt the tests. Thus slirp support is nice to have. There was something more regards to graphics support and qemu, maybe even a HW dependency, at least on x86_64. Then build target machines. I presume only machines from oe-core are supported and meta-yocto-bsp machines like genericarm64 not. What about fixing issues when oe-selftests are run with genericarm64 machine config? Usually there are just a few simple things to fix, like dependencies and configs to select. Often these can be the same as on qemuarm64 since qemu is the target to boot images in tests. Target distro then are AFAIK poky and poky-altcfg so custom distros will likely break a number of things which can not be fixed upstream. So when a user builds upstream supported machines and distro, they can expect the tests to pass. This allows contributing changes and fixing the tests on their machines before submitting things to review and yocto project CI runs where things may break. Though testing all configurations has proven very hard. slirp support makes testing a bit simpler, for me at least, but I understand your pain in upstream. For my changes into UKI support, the slirp support was already fixed in wic and uki selftests. Thus I'd like to support in other tests. But if it's too much work for you to maintain, I get it and can apply fixes locally in my tree to test. Cheers, -Mikko
On Tue, 2025-05-20 at 11:59 +0100, Mikko Rapeli wrote: > On Tue, May 20, 2025 at 10:30:04AM +0100, Richard Purdie wrote: > > > > This has sat in master-next for a long time, mostly because I've been > > putting off trying to write down my thoughts on this. > > > > The challenge with this change is that it makes slirp work in some > > cases but not others. We've been very clear up to now that tap/tun is > > our preferred way to run the tests. If this merges, people will start > > to expect slirp to become a first class citizen and file bugs against > > the tests which don't use it, or submit patches with only the slirp > > tests passing and them complain that we shouldn't have the other tests. > > > > I'm very wary of having "two ways" of doing things, particularly as one > > half always ends up with subtle breakage. I'm also wary of adding > > feature support by stealth, which this has potential to become. > > > > That said, I can totally understand why running the tests which can use > > slirp can be useful, particularly as there are some environments where > > tun/tap isn't possible. This means I am torn on the patch despite my > > reservations. > > > > There also isn't any testing data included. Did you run all the tests > > you are patching and are confirming they work with slirp or was this > > just a way to pass the option (and potentially other options) to all > > tests? Do we know how many tests work with slirp and how many don't? > > I would like to run selftests on genericarm64 target machine and aarch64 build > host. They don't pass yet. This patch is one of the things I would need, > currently, to work on the other problems. A full series is a lot of work > and I don't think I'm up to it atm, but maybe over time... The autobuilder does run oe-selftest on aarch64 build hosts but it uses qemuarm64 for MACHINE. I'd hope there isn't that much difference between qemuarm64 and genericarm64 but that should be the only difference in your config besides slirp vs tun/tap. > > From a practicality standpoint, I do have an issue with the > > implementation since it duplicates the image name for the get_bb_var > > and the runqemu call and I'm not sure I like that implementation > > detail. That is a solvable problem but the decision above on whether we > > want to do this at all remains. > > I can fix this if needed. Some tests used variable for image name > while many did not. It may be better to add something into the runqemu function itself to query the variable but I didn't look into the code implications of that. > > I'm relatively new to selftests and the environments and configs > they run. > > slirp and support for it like in this patch is one thing, but there > are others. > > Which build machine hosts and distributions are supported? > I've heard that x86_64 and aarch64 hosts are used with the supported > set of distros. https://autobuilder.yoctoproject.org/valkyrie/#/workers gives an idea of which host distros and architectures we have in the autobuilder. "arm" in the name means it is an aarch64 system so those are all ubuntus of three different versions. > Then the config inside of distros is where tun/tap and slirp are the alternatives. > Yocto CI runs with tun/tap, but this may be harder to setup for us developers > contributing changes to classes and need to adapt the tests. Thus slirp support > is nice to have. There was something more regards to graphics support and > qemu, maybe even a HW dependency, at least on x86_64. x86_64 tries to do GL passthrough on some distros. I don't think we have that enabled for arm. The tests should get skipped if not supported and include detection code. > Then build target machines. I presume only machines from oe-core are > supported and meta-yocto-bsp machines like genericarm64 not. What about > fixing issues when oe-selftests are run with genericarm64 machine config? Correct, we only test using MACHINEs from those layers and only under qemu. There is no reason genericarm64 shouldn't work, it just hasn't been tested/fixed/enabled. > Usually there are just a few simple things to fix, like dependencies and > configs to select. Often these can be the same as on qemuarm64 since > qemu is the target to boot images in tests. > Target distro then are AFAIK poky and poky-altcfg so custom distros > will likely break a number of things which can not be fixed upstream. The tests can check the config and skip if some config is not supported. Cheers, Richard
Hi, On Tue, May 20, 2025 at 12:10:43PM +0100, Richard Purdie wrote: > On Tue, 2025-05-20 at 11:59 +0100, Mikko Rapeli wrote: > > On Tue, May 20, 2025 at 10:30:04AM +0100, Richard Purdie wrote: > > > > > > This has sat in master-next for a long time, mostly because I've been > > > putting off trying to write down my thoughts on this. > > > > > > The challenge with this change is that it makes slirp work in some > > > cases but not others. We've been very clear up to now that tap/tun is > > > our preferred way to run the tests. If this merges, people will start > > > to expect slirp to become a first class citizen and file bugs against > > > the tests which don't use it, or submit patches with only the slirp > > > tests passing and them complain that we shouldn't have the other tests. > > > > > > I'm very wary of having "two ways" of doing things, particularly as one > > > half always ends up with subtle breakage. I'm also wary of adding > > > feature support by stealth, which this has potential to become. > > > > > > That said, I can totally understand why running the tests which can use > > > slirp can be useful, particularly as there are some environments where > > > tun/tap isn't possible. This means I am torn on the patch despite my > > > reservations. > > > > > > There also isn't any testing data included. Did you run all the tests > > > you are patching and are confirming they work with slirp or was this > > > just a way to pass the option (and potentially other options) to all > > > tests? Do we know how many tests work with slirp and how many don't? > > > > I would like to run selftests on genericarm64 target machine and aarch64 build > > host. They don't pass yet. This patch is one of the things I would need, > > currently, to work on the other problems. A full series is a lot of work > > and I don't think I'm up to it atm, but maybe over time... > > The autobuilder does run oe-selftest on aarch64 build hosts but it uses > qemuarm64 for MACHINE. I'd hope there isn't that much difference > between qemuarm64 and genericarm64 but that should be the only > difference in your config besides slirp vs tun/tap. There are a few more but really, genericarm64 should IMO replace qemuarm64 since both run qemu and former also runs on various other boards and is a good start for developing products on arm64/aarch64. The fixes are trivial once I figure out which approaches are acceptable to you and Ross. For example qemu testing requires a firmware from u-boot, but genericarm64 on purpose doesn't define virtual/bootloader in machine config. Should the tests define u-boot dependency in local.conf or bitbake command line for genericarm64 then? Practically this could be a config snippet added to local.conf after genericarm64 defaults, like the clearing of SANITY_TESTED_DISTROS. > > > From a practicality standpoint, I do have an issue with the > > > implementation since it duplicates the image name for the get_bb_var > > > and the runqemu call and I'm not sure I like that implementation > > > detail. That is a solvable problem but the decision above on whether we > > > want to do this at all remains. > > > > I can fix this if needed. Some tests used variable for image name > > while many did not. > > It may be better to add something into the runqemu function itself to > query the variable but I didn't look into the code implications of > that. Good idea, I'll try this approach. > > I'm relatively new to selftests and the environments and configs > > they run. > > > > slirp and support for it like in this patch is one thing, but there > > are others. > > > > Which build machine hosts and distributions are supported? > > I've heard that x86_64 and aarch64 hosts are used with the supported > > set of distros. > > https://autobuilder.yoctoproject.org/valkyrie/#/workers gives an idea > of which host distros and architectures we have in the autobuilder. > "arm" in the name means it is an aarch64 system so those are all > ubuntus of three different versions. And those are subset of SANITY_TESTED_DISTROS in poky. > > Then the config inside of distros is where tun/tap and slirp are the alternatives. > > Yocto CI runs with tun/tap, but this may be harder to setup for us developers > > contributing changes to classes and need to adapt the tests. Thus slirp support > > is nice to have. There was something more regards to graphics support and > > qemu, maybe even a HW dependency, at least on x86_64. > > x86_64 tries to do GL passthrough on some distros. I don't think we > have that enabled for arm. The tests should get skipped if not > supported and include detection code. Yes, the qemu GL graphics and screenshot testcase. > > Then build target machines. I presume only machines from oe-core are > > supported and meta-yocto-bsp machines like genericarm64 not. What about > > fixing issues when oe-selftests are run with genericarm64 machine config? > > Correct, we only test using MACHINEs from those layers and only under > qemu. There is no reason genericarm64 shouldn't work, it just hasn't > been tested/fixed/enabled. Yes, I can work on this. > > Usually there are just a few simple things to fix, like dependencies and > > configs to select. Often these can be the same as on qemuarm64 since > > qemu is the target to boot images in tests. > > > Target distro then are AFAIK poky and poky-altcfg so custom distros > > will likely break a number of things which can not be fixed upstream. > > The tests can check the config and skip if some config is not > supported. There are some assumptions which are true on qemu but not on other machines and which then impact for example the qemu config (which backend block device emulation to use) and which drivers need to be compiled into the kernel built-in, or if boot with initrd is needed to load modules and to find the real rootfs. Cheers, -Mikko
On 20 May 2025, at 14:17, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote: > > There are a few more but really, genericarm64 should IMO replace qemuarm64 > since both run qemu and former also runs on various other boards and is > a good start for developing products on arm64/aarch64. There’s quite a large conceptual gap between the two: qemuarm64 is a machine that is explicitly and specifically for booting inside a qemu, ideally with KVM. genericarm64 has a much wider scope, assumes SystemReady firmware and real hardware. There’s also a case to be made for merging sbsa-ref (from meta-arm) and qemuarm64, but again I’d suggest not: sbsa-ref explicitly can’t do KVM whereas qemuarm64 can. From a performance-of-testing-userspace perspective, this is an important distinction. Ross
Hi, On Tue, May 20, 2025 at 01:24:41PM +0000, Ross Burton wrote: > On 20 May 2025, at 14:17, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote: > > > > There are a few more but really, genericarm64 should IMO replace qemuarm64 > > since both run qemu and former also runs on various other boards and is > > a good start for developing products on arm64/aarch64. > > There’s quite a large conceptual gap between the two: qemuarm64 is a machine that is explicitly and specifically for booting inside a qemu, ideally with KVM. genericarm64 has a much wider scope, assumes SystemReady firmware and real hardware. qemuarm64 does have a bit odd kernel config, IMO, for pure KVM support. And does the qemu-boot-with-KVM support really need a full machine config, or could genericarm64 kernel and optimized images be sufficient? For example include an initrd with plain KVM and rootfs storage backend support, nothing generic. We've used qemuarm64 and meta-arm side qemuarm64-secureboot machines for userspace on a number of boards and setups without issues but rolled out custom BSP, usually just kernel config and initrd. genericarm64 easily replaces these. These then required a separate build of the SystemReady firmware or support from board vendors which frequently is there already. > There’s also a case to be made for merging sbsa-ref (from meta-arm) and qemuarm64, but again I’d suggest not: sbsa-ref explicitly can’t do KVM whereas qemuarm64 can. From a performance-of-testing-userspace perspective, this is an important distinction. Thanks, I've been wondering what that machine was for. Now back to the selftests and genericarm64. Does it make sense to you to fix oe-core selftests to work on genericarm64? If yes, how should the qemu firmware dependency to u-boot be filled. Manually per build via local.conf or bitbake command line to also build u-boot? Or something else? Cheers, -Mikko
diff --git a/meta/lib/oeqa/selftest/cases/barebox.py b/meta/lib/oeqa/selftest/cases/barebox.py index 3f8f232432..b4d8310666 100644 --- a/meta/lib/oeqa/selftest/cases/barebox.py +++ b/meta/lib/oeqa/selftest/cases/barebox.py @@ -6,7 +6,7 @@ # from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, runqemu +from oeqa.utils.commands import bitbake, runqemu, get_bb_var from oeqa.core.decorator.data import skipIfNotArch from oeqa.core.decorator import OETestTag @@ -34,7 +34,8 @@ QEMU_USE_KVM = "False" bitbake("virtual/bootloader core-image-minimal") - with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic', + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic %s' % (runqemu_params), boot_patterns=barebox_boot_patterns) as qemu: # test if barebox console works diff --git a/meta/lib/oeqa/selftest/cases/debuginfod.py b/meta/lib/oeqa/selftest/cases/debuginfod.py index 46c0cd87bb..22d198b65b 100644 --- a/meta/lib/oeqa/selftest/cases/debuginfod.py +++ b/meta/lib/oeqa/selftest/cases/debuginfod.py @@ -147,10 +147,12 @@ CORE_IMAGE_EXTRA_INSTALL += "elfutils xz" """) bitbake("core-image-minimal elfutils-native:do_addto_recipe_sysroot xz:do_localpkgfeed") + runqemu_params = get_bb_var("TEST_RUNQEMUPARAMS", "core-image-minimal") or "" + try: self.start_debuginfod(get_bb_var("LOCALPKGFEED_DIR", "xz")) - with runqemu("core-image-minimal", runqemuparams="nographic") as qemu: + with runqemu("core-image-minimal", runqemuparams="nographic %s" % runqemu_params) as qemu: cmd = "DEBUGINFOD_URLS=http://%s:%d/ debuginfod-find debuginfo /usr/bin/xz" % (qemu.server_ip, self.port) self.logger.info(f"Starting client {cmd}") status, output = qemu.run_serial(cmd) diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index 0155ee62ee..c9081264e8 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -1836,8 +1836,9 @@ class DevtoolDeployTargetTests(DevtoolBase): # First try a dry-run of deploy-target result = runCmd('devtool deploy-target -n %s root@localhost' % testrecipe) self.assertIn(' %s' % testfile, result.output) + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', testimage) or "" # Boot the image - with runqemu(testimage) as qemu: + with runqemu(testimage, runqemuparams=runqemu_params) as qemu: # Now really test deploy-target result = runCmd('devtool deploy-target -c %s root@%s' % (testrecipe, qemu.ip)) # Run a test command to see if it was installed properly @@ -2810,7 +2811,8 @@ class DevtoolIdeSdkTests(DevtoolBase): # Verify deployment to Qemu (system mode) works bitbake(testimage) - with runqemu(testimage, runqemuparams="nographic") as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu(testimage, runqemuparams="nographic %s" % runqemu_params) as qemu: # cmake-example recipe recipe_name = "cmake-example" example_exe = "cmake-example" diff --git a/meta/lib/oeqa/selftest/cases/efibootpartition.py b/meta/lib/oeqa/selftest/cases/efibootpartition.py index fcfcdaf7e4..b831aec81a 100644 --- a/meta/lib/oeqa/selftest/cases/efibootpartition.py +++ b/meta/lib/oeqa/selftest/cases/efibootpartition.py @@ -30,5 +30,6 @@ IMAGE_INSTALL:append = " grub-efi kernel-image-bzimage" % (image)) bitbake(image + " ovmf") - with runqemu(image, ssh=False, launch_cmd=cmd) as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', image) or "" + with runqemu(image, ssh=False, launch_cmd=cmd, runqemuparams=runqemu_params) as qemu: self.assertTrue(qemu.runner.logged, "Failed: %s" % cmd) diff --git a/meta/lib/oeqa/selftest/cases/gcc.py b/meta/lib/oeqa/selftest/cases/gcc.py index 1bda29a72b..0d27e9b2e6 100644 --- a/meta/lib/oeqa/selftest/cases/gcc.py +++ b/meta/lib/oeqa/selftest/cases/gcc.py @@ -79,7 +79,8 @@ class GccSelfTestBase(OESelftestTestCase, OEPTestResultTestCase): bitbake("core-image-minimal") # wrap the execution with a qemu instance - with runqemu("core-image-minimal", runqemuparams = "nographic") as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu("core-image-minimal", runqemuparams = "nographic %s" % runqemu_params) as qemu: # validate that SSH is working status, _ = qemu.run("uname") self.assertEqual(status, 0) diff --git a/meta/lib/oeqa/selftest/cases/gdbserver.py b/meta/lib/oeqa/selftest/cases/gdbserver.py index b6b7c5c473..99c9e5ed59 100644 --- a/meta/lib/oeqa/selftest/cases/gdbserver.py +++ b/meta/lib/oeqa/selftest/cases/gdbserver.py @@ -42,7 +42,8 @@ CORE_IMAGE_EXTRA_INSTALL = "gdbserver" filename = os.path.join(bb_vars['DEPLOY_DIR_IMAGE'], "%s.tar.bz2" % bb_vars['IMAGE_LINK_NAME']) shutil.unpack_archive(filename, debugfs) - with runqemu("core-image-minimal", runqemuparams="nographic") as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu("core-image-minimal", runqemuparams="nographic %s" % runqemu_params) as qemu: status, output = qemu.run_serial("kmod --help") self.assertIn("modprobe", output) diff --git a/meta/lib/oeqa/selftest/cases/glibc.py b/meta/lib/oeqa/selftest/cases/glibc.py index bd56b2f6e7..89114d72ba 100644 --- a/meta/lib/oeqa/selftest/cases/glibc.py +++ b/meta/lib/oeqa/selftest/cases/glibc.py @@ -70,7 +70,8 @@ class GlibcSelfTestBase(OESelftestTestCase, OEPTestResultTestCase): bitbake("core-image-minimal") # start runqemu - qemu = s.enter_context(runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 1024")) + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + qemu = s.enter_context(runqemu("core-image-minimal", runqemuparams = "nographic %s" % runqemu_params, qemuparams = "-m 1024")) # validate that SSH is working status, _ = qemu.run("uname") diff --git a/meta/lib/oeqa/selftest/cases/imagefeatures.py b/meta/lib/oeqa/selftest/cases/imagefeatures.py index 94d01ba116..c46022c49f 100644 --- a/meta/lib/oeqa/selftest/cases/imagefeatures.py +++ b/meta/lib/oeqa/selftest/cases/imagefeatures.py @@ -36,7 +36,8 @@ class ImageFeatures(OESelftestTestCase): # Build a core-image-minimal bitbake('core-image-minimal') - with runqemu("core-image-minimal") as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu("core-image-minimal", runqemuparams=runqemu_params) as qemu: # Attempt to ssh with each user into qemu with empty password for user in [self.root_user, self.test_user]: ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user=user) diff --git a/meta/lib/oeqa/selftest/cases/locales.py b/meta/lib/oeqa/selftest/cases/locales.py index ac4888ef66..372421a3dc 100644 --- a/meta/lib/oeqa/selftest/cases/locales.py +++ b/meta/lib/oeqa/selftest/cases/locales.py @@ -4,7 +4,7 @@ from oeqa.selftest.case import OESelftestTestCase from oeqa.core.decorator import OETestTag -from oeqa.utils.commands import bitbake, runqemu +from oeqa.utils.commands import bitbake, runqemu, get_bb_var class LocalesTest(OESelftestTestCase): @@ -25,7 +25,8 @@ class LocalesTest(OESelftestTestCase): # Build a core-image-minimal bitbake('core-image-minimal') - with runqemu("core-image-minimal", ssh=False, runqemuparams='nographic') as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu("core-image-minimal", ssh=False, runqemuparams='nographic %s' % runqemu_params) as qemu: cmd = "locale -a" status, output = qemu.run_serial(cmd) # output must includes fr_FR or fr_FR.UTF-8 diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py b/meta/lib/oeqa/selftest/cases/overlayfs.py index 580fbdcb9c..9601cbef4a 100644 --- a/meta/lib/oeqa/selftest/cases/overlayfs.py +++ b/meta/lib/oeqa/selftest/cases/overlayfs.py @@ -5,7 +5,7 @@ # from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, runqemu, get_bb_vars +from oeqa.utils.commands import bitbake, runqemu, get_bb_var, get_bb_vars from oeqa.core.decorator import OETestTag from oeqa.core.decorator.data import skipIfNotMachine @@ -181,7 +181,8 @@ EOT bitbake('core-image-minimal') - with runqemu('core-image-minimal') as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', runqemuparams=runqemu_params) as qemu: # Check that application service started status, output = qemu.run_serial("systemctl status my-application") self.assertTrue("active (exited)" in output, msg=output) @@ -360,7 +361,8 @@ OVERLAYFS_ROOTFS_TYPE = "ext4" bitbake('core-image-minimal') - with runqemu('core-image-minimal', image_fstype='wic') as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', runqemuparams=runqemu_params, image_fstype='wic') as qemu: status, output = qemu.run_serial("/bin/mount") line = getline_qemu(output, "upperdir=/data/overlay-etc/upper") @@ -399,7 +401,8 @@ OVERLAYFS_ROOTFS_TYPE = "ext4" bitbake('core-image-minimal') testFile = "/etc/my-test-data" - with runqemu('core-image-minimal', image_fstype='wic', discard_writes=False) as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', runqemuparams=runqemu_params, image_fstype='wic', discard_writes=False) as qemu: status, output = qemu.run_serial("/bin/mount") line = getline_qemu(output, "/dev/sda3") @@ -420,7 +423,7 @@ OVERLAYFS_ROOTFS_TYPE = "ext4" self.assertTrue(line and line.startswith(testFile), msg=output) # Check that file exists in /etc after reboot - with runqemu('core-image-minimal', image_fstype='wic') as qemu: + with runqemu('core-image-minimal', runqemuparams=runqemu_params, image_fstype='wic') as qemu: status, output = qemu.run_serial("ls -1 " + testFile) line = getline_qemu(output, testFile) self.assertTrue(line and line.startswith(testFile), msg=output) @@ -454,7 +457,8 @@ IMAGE_INSTALL:append = " overlayfs-user" self.append_config(configLower) bitbake('core-image-minimal') - with runqemu('core-image-minimal', image_fstype='wic') as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', runqemuparams=runqemu_params, image_fstype='wic') as qemu: status, output = qemu.run_serial("echo \"Modified in upper\" > /etc/" + testFile) status, output = qemu.run_serial("diff /etc/" + testFile + " /data/overlay-etc/lower/" + testFile) line = getline_qemu(output, "Modified in upper") @@ -500,7 +504,8 @@ IMAGE_INSTALL:append = " overlayfs-user" res = bitbake('core-image-minimal') - with runqemu('core-image-minimal', image_fstype='wic') as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', runqemuparams=runqemu_params, image_fstype='wic') as qemu: for filename in ("rootfs", "delayed-a", "delayed-b"): status, output = qemu.run_serial("test -f %s && echo found" % os.path.join(targettestdir, filename)) self.assertIn("found", output, "%s was not present on boot" % filename) diff --git a/meta/lib/oeqa/selftest/cases/package.py b/meta/lib/oeqa/selftest/cases/package.py index 38ed7173fe..44e30cec50 100644 --- a/meta/lib/oeqa/selftest/cases/package.py +++ b/meta/lib/oeqa/selftest/cases/package.py @@ -177,7 +177,8 @@ class PackageTests(OESelftestTestCase): self.logger.error("GDB result:\n%d: %s", status, output) return False - with runqemu('core-image-minimal') as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', runqemuparams=runqemu_params) as qemu: for binary in ['/usr/bin/hello1', '/usr/bin/hello2', '/usr/libexec/hello3', @@ -202,7 +203,8 @@ class PackageTests(OESelftestTestCase): self.fail("Cannot parse output: " + output) sysconfdir = get_bb_var('sysconfdir', 'selftest-chown') - with runqemu('core-image-minimal') as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', runqemuparams=runqemu_params) as qemu: for path in [ sysconfdir + "/selftest-chown/file", sysconfdir + "/selftest-chown/dir", sysconfdir + "/selftest-chown/symlink", diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py index 4843d810a2..8c9cfa0c54 100644 --- a/meta/lib/oeqa/selftest/cases/runtime_test.py +++ b/meta/lib/oeqa/selftest/cases/runtime_test.py @@ -42,7 +42,8 @@ class TestExport(OESelftestTestCase): isdir = os.path.isdir(testexport_dir) self.assertEqual(True, isdir, 'Failed to create testexport dir: %s' % testexport_dir) - with runqemu('core-image-minimal') as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', runqemuparams=runqemu_params) as qemu: # Attempt to run runexported.py to perform ping test test_path = os.path.join(testexport_dir, "oe-test") data_file = os.path.join(testexport_dir, 'data', 'testdata.json') @@ -320,7 +321,8 @@ class Postinst(OESelftestTestCase): self.assertTrue(os.path.isfile(os.path.join(hosttestdir, "rootfs")), "rootfs state file was not created") - with runqemu('core-image-minimal') as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', runqemuparams=runqemu_params) as qemu: # Make the test echo a string and search for that as # run_serial()'s status code is useless.' for filename in ("rootfs", "delayed-a", "delayed-b"): diff --git a/meta/lib/oeqa/selftest/cases/rust.py b/meta/lib/oeqa/selftest/cases/rust.py index 1098e902fc..f84b87d0f5 100644 --- a/meta/lib/oeqa/selftest/cases/rust.py +++ b/meta/lib/oeqa/selftest/cases/rust.py @@ -99,7 +99,8 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): # wrap the execution with a qemu instance. # Tests are run with 512 tasks in parallel to execute all tests very quickly - with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 512") as qemu: + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu("core-image-minimal", runqemuparams = "nographic %s" % runqemu_params, qemuparams = "-m 512") as qemu: # Copy remote-test-server to image through scp host_sys = get_bb_var("RUST_BUILD_SYS", "rust") ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root") diff --git a/meta/lib/oeqa/selftest/cases/uboot.py b/meta/lib/oeqa/selftest/cases/uboot.py index 96da4efb06..39e48f8bdb 100644 --- a/meta/lib/oeqa/selftest/cases/uboot.py +++ b/meta/lib/oeqa/selftest/cases/uboot.py @@ -6,7 +6,7 @@ # from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, runqemu +from oeqa.utils.commands import bitbake, runqemu, get_bb_var from oeqa.core.decorator.data import skipIfNotArch from oeqa.core.decorator import OETestTag @@ -33,7 +33,8 @@ QEMU_USE_KVM = "False" """) bitbake("virtual/bootloader core-image-minimal") - with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic', + runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', "core-image-minimal") or "" + with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic %s' % runqemu_params, boot_patterns=uboot_boot_patterns) as qemu: # test if u-boot console works
To support "slirp" userspace networking which works more easily on various build machines, also without root and sudo rights to setup loop interfaces. Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> --- meta/lib/oeqa/selftest/cases/barebox.py | 5 +++-- meta/lib/oeqa/selftest/cases/debuginfod.py | 4 +++- meta/lib/oeqa/selftest/cases/devtool.py | 6 ++++-- .../oeqa/selftest/cases/efibootpartition.py | 3 ++- meta/lib/oeqa/selftest/cases/gcc.py | 3 ++- meta/lib/oeqa/selftest/cases/gdbserver.py | 3 ++- meta/lib/oeqa/selftest/cases/glibc.py | 3 ++- meta/lib/oeqa/selftest/cases/imagefeatures.py | 3 ++- meta/lib/oeqa/selftest/cases/locales.py | 5 +++-- meta/lib/oeqa/selftest/cases/overlayfs.py | 19 ++++++++++++------- meta/lib/oeqa/selftest/cases/package.py | 6 ++++-- meta/lib/oeqa/selftest/cases/runtime_test.py | 6 ++++-- meta/lib/oeqa/selftest/cases/rust.py | 3 ++- meta/lib/oeqa/selftest/cases/uboot.py | 5 +++-- 14 files changed, 48 insertions(+), 26 deletions(-) v2: added missing get_bb_var import to u-boot.py v1: https://lists.openembedded.org/g/openembedded-core/message/215225