| Message ID | 20220221085118.2267900-1-quaresma.jose@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v2,1/2] meson.bbclass: add support for running the meson tests at build time | expand |
There are *so many* limitations of qemu-user I'm not sure this is actually useful. (speaking as someone who tried this with Python a while ago). What recipes has this successfully worked on? Ross On Mon, 21 Feb 2022 at 08:51, Jose Quaresma <quaresma.jose@gmail.com> wrote: > > The generic exe_wrapper introcuced in 40349dc5 can be used to run the > meson tests when build the receipe. This patch add the meson_test task > on the meson bbclass that adds this possibility. > > The exe_wrapper will run using the qemu usermode and because of that it > has some limitations, so don't expect the same results when it runs > natively on target. > > Running the tests at build time is upstream way for testing but is not > possible in OE as we always croos-compile. This can be useful in some > integration processes and for debuging and it adds the possibility to > run the test in an interactive way using the devtool build for example. > > There are two variable for that: > MESON_QEMU_WRAPPER_TEST_ENABLED: this enables the test when it's "1" > EXTRA_OEMESON_TEST: this is used to add extra args > > Some results when using this patch in the bigest meson project > on OE-core are the folloing: > > cat <<'EOF' >>conf/local.conf > MESON_QEMU_WRAPPER_TEST_ENABLED:pn-systemd-boot = "1" > EOF > > bitbake -c do_meson_test -v systemd-boot > > Ok: 821 > Expected Fail: 0 > Fail: 95 > Unexpected Pass: 0 > Skipped: 33 > Timeout: 0 > > cat <<'EOF' >>conf/local.conf > MESON_QEMU_WRAPPER_TEST_ENABLED:pn-glib-2.0 = "1" > EOF > > bitbake -c do_meson_test -v glib-2.0 > > Ok: 197 > Expected Fail: 0 > Fail: 76 > Unexpected Pass: 0 > Skipped: 0 > Timeout: 2 > > cat <<'EOF' >>conf/local.conf > MESON_QEMU_WRAPPER_TEST_ENABLED:pn-gstreamer1.0 = "1" > EOF > > bitbake -c do_meson_test -v gstreamer1.0 > > Ok: 95 > Expected Fail: 0 > Fail: 10 > Unexpected Pass: 0 > Skipped: 1 > Timeout: 0 > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > --- > meta/classes/meson.bbclass | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) > > diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass > index 0bfe945811..020a59a3c0 100644 > --- a/meta/classes/meson.bbclass > +++ b/meta/classes/meson.bbclass > @@ -179,3 +179,19 @@ meson_do_install() { > } > > EXPORT_FUNCTIONS do_configure do_compile do_install > + > +EXTRA_OEMESON_TEST ?= "" > + > +do_meson_test() { > + bbwarn "don't expect the same results when it runs natively as it will run in qemu usermode (that has its own limitations)" > + meson test -C ${B} ${EXTRA_OEMESON_TEST} > +} > +do_meson_test[vardeps] += "EXTRA_OEMESON_TEST" > + > +MESON_QEMU_WRAPPER_RUNTEST = "" > +MESON_QEMU_WRAPPER_RUNTEST:class-target = "${@d.getVar('MESON_QEMU_WRAPPER_TEST_ENABLED') == '1' and d.getVar('EXEWRAPPER_ENABLED') == 'True'}" > + > +python() { > + if d.getVar('MESON_QEMU_WRAPPER_RUNTEST') == 'True': > + bb.build.addtask('meson_test', 'do_install', 'do_compile', d) > +} > -- > 2.35.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#162019): https://lists.openembedded.org/g/openembedded-core/message/162019 > Mute This Topic: https://lists.openembedded.org/mt/89289917/1676615 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ross@burtonini.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Ross, Ross Burton <ross@burtonini.com> escreveu no dia segunda, 21/02/2022 à(s) 14:24: > There are *so many* limitations of qemu-user I'm not sure this is > actually useful. > > (speaking as someone who tried this with Python a while ago). > Can you please point me to your previous work if it is public, I would like to see what has already been done in this regard. > What recipes has this successfully worked on? > It works on pulseaudio that has only one test and some others that I don't remember now, but the tests run successfully only when there are few. I use it extensively with gstreamer and the failures is mainly because of files that are not found inside the qemu user mode. I think that this feature is useful for some other people, it helps me a lot in debug some tests in gstreamer. Anyway the feature is disabled by default so for sure it will not introduce any regression on OE-core. Jose > > Ross > > On Mon, 21 Feb 2022 at 08:51, Jose Quaresma <quaresma.jose@gmail.com> > wrote: > > > > The generic exe_wrapper introcuced in 40349dc5 can be used to run the > > meson tests when build the receipe. This patch add the meson_test task > > on the meson bbclass that adds this possibility. > > > > The exe_wrapper will run using the qemu usermode and because of that it > > has some limitations, so don't expect the same results when it runs > > natively on target. > > > > Running the tests at build time is upstream way for testing but is not > > possible in OE as we always croos-compile. This can be useful in some > > integration processes and for debuging and it adds the possibility to > > run the test in an interactive way using the devtool build for example. > > > > There are two variable for that: > > MESON_QEMU_WRAPPER_TEST_ENABLED: this enables the test when it's "1" > > EXTRA_OEMESON_TEST: this is used to add extra args > > > > Some results when using this patch in the bigest meson project > > on OE-core are the folloing: > > > > cat <<'EOF' >>conf/local.conf > > MESON_QEMU_WRAPPER_TEST_ENABLED:pn-systemd-boot = "1" > > EOF > > > > bitbake -c do_meson_test -v systemd-boot > > > > Ok: 821 > > Expected Fail: 0 > > Fail: 95 > > Unexpected Pass: 0 > > Skipped: 33 > > Timeout: 0 > > > > cat <<'EOF' >>conf/local.conf > > MESON_QEMU_WRAPPER_TEST_ENABLED:pn-glib-2.0 = "1" > > EOF > > > > bitbake -c do_meson_test -v glib-2.0 > > > > Ok: 197 > > Expected Fail: 0 > > Fail: 76 > > Unexpected Pass: 0 > > Skipped: 0 > > Timeout: 2 > > > > cat <<'EOF' >>conf/local.conf > > MESON_QEMU_WRAPPER_TEST_ENABLED:pn-gstreamer1.0 = "1" > > EOF > > > > bitbake -c do_meson_test -v gstreamer1.0 > > > > Ok: 95 > > Expected Fail: 0 > > Fail: 10 > > Unexpected Pass: 0 > > Skipped: 1 > > Timeout: 0 > > > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > > --- > > meta/classes/meson.bbclass | 16 ++++++++++++++++ > > 1 file changed, 16 insertions(+) > > > > diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass > > index 0bfe945811..020a59a3c0 100644 > > --- a/meta/classes/meson.bbclass > > +++ b/meta/classes/meson.bbclass > > @@ -179,3 +179,19 @@ meson_do_install() { > > } > > > > EXPORT_FUNCTIONS do_configure do_compile do_install > > + > > +EXTRA_OEMESON_TEST ?= "" > > + > > +do_meson_test() { > > + bbwarn "don't expect the same results when it runs natively as it > will run in qemu usermode (that has its own limitations)" > > + meson test -C ${B} ${EXTRA_OEMESON_TEST} > > +} > > +do_meson_test[vardeps] += "EXTRA_OEMESON_TEST" > > + > > +MESON_QEMU_WRAPPER_RUNTEST = "" > > +MESON_QEMU_WRAPPER_RUNTEST:class-target = > "${@d.getVar('MESON_QEMU_WRAPPER_TEST_ENABLED') == '1' and > d.getVar('EXEWRAPPER_ENABLED') == 'True'}" > > + > > +python() { > > + if d.getVar('MESON_QEMU_WRAPPER_RUNTEST') == 'True': > > + bb.build.addtask('meson_test', 'do_install', 'do_compile', d) > > +} > > -- > > 2.35.1 > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#162019): > https://lists.openembedded.org/g/openembedded-core/message/162019 > > Mute This Topic: https://lists.openembedded.org/mt/89289917/1676615 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > ross@burtonini.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > >
On Mon, 21 Feb 2022 at 16:00, Jose Quaresma <quaresma.jose@gmail.com> wrote: > It works on pulseaudio that has only one test and some others that I don't remember now, > but the tests run successfully only when there are few. > I use it extensively with gstreamer and the failures is mainly because of files that are not > found inside the qemu user mode. > > I think that this feature is useful for some other people, it helps me a lot in debug some tests in gstreamer. > Anyway the feature is disabled by default so for sure it will not introduce any regression on OE-core. What prevents you from running the tests in the target image with full system qemu? Alex
Hi Alexander, Alexander Kanavin <alex.kanavin@gmail.com> escreveu no dia segunda, 21/02/2022 à(s) 15:30: > On Mon, 21 Feb 2022 at 16:00, Jose Quaresma <quaresma.jose@gmail.com> > wrote: > > It works on pulseaudio that has only one test and some others that I > don't remember now, > > but the tests run successfully only when there are few. > > I use it extensively with gstreamer and the failures is mainly because > of files that are not > > found inside the qemu user mode. > > > > I think that this feature is useful for some other people, it helps me a > lot in debug some tests in gstreamer. > > Anyway the feature is disabled by default so for sure it will not > introduce any regression on OE-core. > > What prevents you from running the tests in the target image with full > system qemu? > > Alex > Nothing and this is the supported and tested way for running the test with PTEST. It can be only a different workflow, afaik for run the test with system qemu we need to: 1) build the recipe with the tested code 2) create an image with the test that will run installed 3) launch the image with runqemu 4) run the test With this approach we only need: 1) bitbake recipe Jose
On Mon, 21 Feb 2022 at 15:00, Jose Quaresma <quaresma.jose@gmail.com> wrote: >> There are *so many* limitations of qemu-user I'm not sure this is >> actually useful. >> >> (speaking as someone who tried this with Python a while ago). > > Can you please point me to your previous work if it is public, > I would like to see what has already been done in this regard. I doubt the branch is still around. I was just running make check inside a qemu-user, and noticed that so much broke. I think at the time it was fork/exec that wasn't usable under qemu-user, which breaks most test frameworks. Ross
diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass index 0bfe945811..020a59a3c0 100644 --- a/meta/classes/meson.bbclass +++ b/meta/classes/meson.bbclass @@ -179,3 +179,19 @@ meson_do_install() { } EXPORT_FUNCTIONS do_configure do_compile do_install + +EXTRA_OEMESON_TEST ?= "" + +do_meson_test() { + bbwarn "don't expect the same results when it runs natively as it will run in qemu usermode (that has its own limitations)" + meson test -C ${B} ${EXTRA_OEMESON_TEST} +} +do_meson_test[vardeps] += "EXTRA_OEMESON_TEST" + +MESON_QEMU_WRAPPER_RUNTEST = "" +MESON_QEMU_WRAPPER_RUNTEST:class-target = "${@d.getVar('MESON_QEMU_WRAPPER_TEST_ENABLED') == '1' and d.getVar('EXEWRAPPER_ENABLED') == 'True'}" + +python() { + if d.getVar('MESON_QEMU_WRAPPER_RUNTEST') == 'True': + bb.build.addtask('meson_test', 'do_install', 'do_compile', d) +}
The generic exe_wrapper introcuced in 40349dc5 can be used to run the meson tests when build the receipe. This patch add the meson_test task on the meson bbclass that adds this possibility. The exe_wrapper will run using the qemu usermode and because of that it has some limitations, so don't expect the same results when it runs natively on target. Running the tests at build time is upstream way for testing but is not possible in OE as we always croos-compile. This can be useful in some integration processes and for debuging and it adds the possibility to run the test in an interactive way using the devtool build for example. There are two variable for that: MESON_QEMU_WRAPPER_TEST_ENABLED: this enables the test when it's "1" EXTRA_OEMESON_TEST: this is used to add extra args Some results when using this patch in the bigest meson project on OE-core are the folloing: cat <<'EOF' >>conf/local.conf MESON_QEMU_WRAPPER_TEST_ENABLED:pn-systemd-boot = "1" EOF bitbake -c do_meson_test -v systemd-boot Ok: 821 Expected Fail: 0 Fail: 95 Unexpected Pass: 0 Skipped: 33 Timeout: 0 cat <<'EOF' >>conf/local.conf MESON_QEMU_WRAPPER_TEST_ENABLED:pn-glib-2.0 = "1" EOF bitbake -c do_meson_test -v glib-2.0 Ok: 197 Expected Fail: 0 Fail: 76 Unexpected Pass: 0 Skipped: 0 Timeout: 2 cat <<'EOF' >>conf/local.conf MESON_QEMU_WRAPPER_TEST_ENABLED:pn-gstreamer1.0 = "1" EOF bitbake -c do_meson_test -v gstreamer1.0 Ok: 95 Expected Fail: 0 Fail: 10 Unexpected Pass: 0 Skipped: 1 Timeout: 0 Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> --- meta/classes/meson.bbclass | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)