[meta-oe,04/11] meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution

Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-oe-ptest-image.bb for an image per ptest in the form of
meta-oe-ptest-image-XXX using class extensions.

* Add meta-oe-ptest-all-image.bb depends on ALL meta-oe-ptest-image-XXX in meta-oe
* Add meta-oe-ptest-fast-image.bb depends on meta-oe ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating
this number of class extensions is slow but making it easier to execute
and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are
checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-oe-ptest-image-XXX can be built with:
$ bitbake meta-oe-ptest-all-image

Subsequently, all the ptest images can be run with:
$ bitbake -c testimage meta-oe-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous
QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresult.json

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../images/meta-oe-ptest-all-image.bb         | 25 ++++++++++++
 .../images/meta-oe-ptest-fast-image.bb        |  6 +++
 .../images/meta-oe-ptest-image.bb             | 38 ++++++++++++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb

Thanks for this patchset Tim, really appreciate it. I have few comments.

On Sun, Apr 2, 2023 at 3:30 PM Tim Orling <ticotimo@gmail.com> wrote:
>
> Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"
>
> Refactor meta-oe-ptest-image.bb for an image per ptest in the form of
> meta-oe-ptest-image-XXX using class extensions.
>
> * Add meta-oe-ptest-all-image.bb depends on ALL meta-oe-ptest-image-XXX in meta-oe
> * Add meta-oe-ptest-fast-image.bb depends on meta-oe ptests which take less than ~30s
>
> This allows parallel execution of the tests.
>
> The downside to this approach is the parsing time of an image generating
> this number of class extensions is slow but making it easier to execute
> and collect test data should outweigh that.
>
> A useful advantage is that the dependencies of each ptest package are
> checked/tested individually.
>
> Add to local.conf:
> IMAGE_CLASSES += "testimage"
>
> All the meta-oe-ptest-image-XXX can be built with:
> $ bitbake meta-oe-ptest-all-image
>
> Subsequently, all the ptest images can be run with:
> $ bitbake -c testimage meta-oe-ptest-all-image
>
> You will likely want to limit the number of parallel image tests:
>
> BB_NUMBER_THREADS = "4"
>
> based on the number of TAP devices you have defined or the number of simultaneous
> QEMU sessions your server can support.
>
> Results can be summarized with:
>
> $ resulttool report tmp/log/oeqa/testresult.json
>
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
>  .../images/meta-oe-ptest-all-image.bb         | 25 ++++++++++++
>  .../images/meta-oe-ptest-fast-image.bb        |  6 +++
>  .../images/meta-oe-ptest-image.bb             | 38 ++++++++++++++++++-
>  3 files changed, 68 insertions(+), 1 deletion(-)
>  create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
>  create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb

I think these images also need ( and also the ones in meta-python and
meta-perl too I think )

IMAGE_FEATURES += "debug-tweaks"

since some of them add ssh and it will be helpful to be able to do
passwordless access. and I was also wondering
if meta-oe-image-base.bb is really needed. We could simply include
core-image-base.bb here. This change also makes
meta-oe-image.bb and packagegroup-meta-oe.bb also questionable. Since
build part can be taken care by bitbake world.
but we can address that later.

>
> diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> new file mode 100644
> index 000000000..0ea03dd2e
> --- /dev/null
> +++ b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> @@ -0,0 +1,25 @@
> +DESCRIPTION = "Recipe to trigger execution of all meta-oe ptest images."
> +HOMEPAGE = "https://www.openembedded.org/"
> +
> +LICENSE = "MIT"
> +
> +inherit features_check nopackages
> +REQUIRED_DISTRO_FEATURES = "ptest"
> +
> +require conf/include/ptest-packagelists-meta-oe.inc
> +
> +# Include the full set of ptests
> +PTESTS_META_OE = "${PTESTS_FAST_META_OE} ${PTESTS_SLOW_META_OE}"
> +
> +do_testimage[noexec] = "1"
> +do_testimage[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_OE').split()])}"
> +
> +do_build[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_build' for x in d.getVar('PTESTS_META_OE').split()])}"
> +
> +# normally image.bbclass would do this
> +EXCLUDE_FROM_WORLD = "1"
> +
> +python () {
> +    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
> +        bb.build.addtask("do_testimage", "", "", d)
> +}
> diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> new file mode 100644
> index 000000000..f56e841ff
> --- /dev/null
> +++ b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> @@ -0,0 +1,6 @@
> +require meta-oe-ptest-all-image.bb
> +
> +DESCRIPTION = "Recipe to trigger execution of all fast meta-oe ptest images."
> +
> +PTESTS_META_OE = "${PTESTS_FAST_META_OE}"
> +
> diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> index 0166b9c41..951d91f82 100644
> --- a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> +++ b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> @@ -1,5 +1,41 @@
> +inherit features_check
> +REQUIRED_DISTRO_FEATURES = "ptest"
> +
> +require conf/include/ptest-packagelists-meta-oe.inc
> +
>  require  meta-oe-image-base.bb
>
>  SUMMARY = "meta-oe ptest test image"
>
> -IMAGE_INSTALL += "packagegroup-meta-oe"
> +DESCRIPTION += "Also including the ${MCNAME} ptest package."
> +HOMEPAGE = "https://www.openembedded.org/"
> +
> +PTESTS_META_OE = "${PTESTS_SLOW_META_OE} ${PTESTS_FAST_META_OE}"
> +
> +IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
> +
> +BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS_META_OE').split()])}"
> +
> +# The image can be sufficiently large (~1.8GB) that we need to be careful that it fits in a live
> +# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
> +# box) and explicitly add up to 1500MB.
> +IMAGE_OVERHEAD_FACTOR = "1.0"
> +IMAGE_ROOTFS_EXTRA_SPACE = "324288"
> +# If a particular ptest needs more space, it can be customized:
> +#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
> +
> +# ptests need more memory than standard to avoid the OOM killer
> +QB_MEM = "-m 1024"
> +# If a particular ptest needs more memroy, it can be customized:
> +#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
> +
> +TEST_SUITES = "ping ssh parselogs ptest"
> +
> +# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
> +PTEST_EXPECT_FAILURE = "1"
> +
> +python () {
> +    if not d.getVar("MCNAME"):
> +        raise bb.parse.SkipRecipe("No class extension set")
> +}
> +
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#101882): https://lists.openembedded.org/g/openembedded-devel/message/101882
> Mute This Topic: https://lists.openembedded.org/mt/98023314/1997914
> Group Owner: openembedded-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On Sun, Apr 2, 2023 at 9:13 PM Khem Raj <raj.khem@gmail.com> wrote:

> Thanks for this patchset Tim, really appreciate it. I have few comments.
>
> On Sun, Apr 2, 2023 at 3:30 PM Tim Orling <ticotimo@gmail.com> wrote:
> >
> > Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"
> >
> > Refactor meta-oe-ptest-image.bb for an image per ptest in the form of
> > meta-oe-ptest-image-XXX using class extensions.
> >
> > * Add meta-oe-ptest-all-image.bb depends on ALL meta-oe-ptest-image-XXX
> in meta-oe
> > * Add meta-oe-ptest-fast-image.bb depends on meta-oe ptests which take
> less than ~30s
> >
> > This allows parallel execution of the tests.
> >
> > The downside to this approach is the parsing time of an image generating
> > this number of class extensions is slow but making it easier to execute
> > and collect test data should outweigh that.
> >
> > A useful advantage is that the dependencies of each ptest package are
> > checked/tested individually.
> >
> > Add to local.conf:
> > IMAGE_CLASSES += "testimage"
> >
> > All the meta-oe-ptest-image-XXX can be built with:
> > $ bitbake meta-oe-ptest-all-image
> >
> > Subsequently, all the ptest images can be run with:
> > $ bitbake -c testimage meta-oe-ptest-all-image
> >
> > You will likely want to limit the number of parallel image tests:
> >
> > BB_NUMBER_THREADS = "4"
> >
> > based on the number of TAP devices you have defined or the number of
> simultaneous
> > QEMU sessions your server can support.
> >
> > Results can be summarized with:
> >
> > $ resulttool report tmp/log/oeqa/testresult.json
> >
> > Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> > ---
> >  .../images/meta-oe-ptest-all-image.bb         | 25 ++++++++++++
> >  .../images/meta-oe-ptest-fast-image.bb        |  6 +++
> >  .../images/meta-oe-ptest-image.bb             | 38 ++++++++++++++++++-
> >  3 files changed, 68 insertions(+), 1 deletion(-)
> >  create mode 100644 meta-oe/recipes-core/images/
> meta-oe-ptest-all-image.bb
> >  create mode 100644 meta-oe/recipes-core/images/
> meta-oe-ptest-fast-image.bb
>
> I think these images also need ( and also the ones in meta-python and
> meta-perl too I think )
>
> IMAGE_FEATURES += "debug-tweaks"
>
> since some of them add ssh and it will be helpful to be able to do
> passwordless access.


I was mirroring the setup in oe-core. The default local.conf.sample in poky
enables
debug-tweaks and that is what I used as well as what the AutoBuilder uses.

The question is do we want to enforce a hard requirement on 'debug-tweaks'
when
some users may want to test in a different environment with a defined
user/password or
with ssh keys added to known_hosts, etc. Food for thought and I am open to
the change.


> and I was also wondering
> if meta-oe-image-base.bb is really needed. We could simply include
> core-image-base.bb here.


I was trying to do minimal change to what was already there. Open to the
change.


> This change also makes
> meta-oe-image.bb and packagegroup-meta-oe.bb also questionable. Since
> build part can be taken care by bitbake world.
> but we can address that later.


Again, I was trying to make only the changes to enable this new "per image
ptest" methodology. I personally
never used meta-oe-image.bb, meta-python-image.bb nor meta-perl-image.bb. I
am open to dropping them if
we decide to drop the packagegroups (which are TBH a bit of a challenge
since they are not necessarily always
up to date). In case others were using these for their workflows, I did not
want to cause breakage without some
discussion.

I will note that meta-oe-ptest-image.bb, meta-python-ptest-image.bb and
meta-perl-ptest-image.bb are all currently
needed in order to have layer isolation of the ptests. In
core-image-ptest.bb, the PTESTS variable is set and
I did not want to clobber that. Hence the new PTESTS_META_OE and friends
variables.

I'll also note that I tried to enable the "missing-ptest" QA check from
ptest.bbclass, but this is a per recipe check (not
an image level check) and is currently hard coded to only look at
meta/conf/distro/include/ptest-packagelists.inc and the
PTESTS_FAST and PTESTS_SLOW variables defined there. I didn't immediately
come up with a way to modularize that
behavior.


>
>
> > diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> > new file mode 100644
> > index 000000000..0ea03dd2e
> > --- /dev/null
> > +++ b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> > @@ -0,0 +1,25 @@
> > +DESCRIPTION = "Recipe to trigger execution of all meta-oe ptest images."
> > +HOMEPAGE = "https://www.openembedded.org/"
> > +
> > +LICENSE = "MIT"
> > +
> > +inherit features_check nopackages
> > +REQUIRED_DISTRO_FEATURES = "ptest"
> > +
> > +require conf/include/ptest-packagelists-meta-oe.inc
> > +
> > +# Include the full set of ptests
> > +PTESTS_META_OE = "${PTESTS_FAST_META_OE} ${PTESTS_SLOW_META_OE}"
> > +
> > +do_testimage[noexec] = "1"
> > +do_testimage[depends] = "${@'
> '.join(['meta-oe-ptest-image-'+x+':do_testimage' for x in
> d.getVar('PTESTS_META_OE').split()])}"
> > +
> > +do_build[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_build'
> for x in d.getVar('PTESTS_META_OE').split()])}"
> > +
> > +# normally image.bbclass would do this
> > +EXCLUDE_FROM_WORLD = "1"
> > +
> > +python () {
> > +    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
> > +        bb.build.addtask("do_testimage", "", "", d)
> > +}
> > diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> > new file mode 100644
> > index 000000000..f56e841ff
> > --- /dev/null
> > +++ b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> > @@ -0,0 +1,6 @@
> > +require meta-oe-ptest-all-image.bb
> > +
> > +DESCRIPTION = "Recipe to trigger execution of all fast meta-oe ptest
> images."
> > +
> > +PTESTS_META_OE = "${PTESTS_FAST_META_OE}"
> > +
> > diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> > index 0166b9c41..951d91f82 100644
> > --- a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> > +++ b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> > @@ -1,5 +1,41 @@
> > +inherit features_check
> > +REQUIRED_DISTRO_FEATURES = "ptest"
> > +
> > +require conf/include/ptest-packagelists-meta-oe.inc
> > +
> >  require  meta-oe-image-base.bb
> >
> >  SUMMARY = "meta-oe ptest test image"
> >
> > -IMAGE_INSTALL += "packagegroup-meta-oe"
> > +DESCRIPTION += "Also including the ${MCNAME} ptest package."
> > +HOMEPAGE = "https://www.openembedded.org/"
> > +
> > +PTESTS_META_OE = "${PTESTS_SLOW_META_OE} ${PTESTS_FAST_META_OE}"
> > +
> > +IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
> > +
> > +BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in
> d.getVar('PTESTS_META_OE').split()])}"
> > +
> > +# The image can be sufficiently large (~1.8GB) that we need to be
> careful that it fits in a live
> > +# image (which has a 4GB limit), so nullify the overhead factor (1.3x
> out of the
> > +# box) and explicitly add up to 1500MB.
> > +IMAGE_OVERHEAD_FACTOR = "1.0"
> > +IMAGE_ROOTFS_EXTRA_SPACE = "324288"
> > +# If a particular ptest needs more space, it can be customized:
> > +#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
> > +
> > +# ptests need more memory than standard to avoid the OOM killer
> > +QB_MEM = "-m 1024"
> > +# If a particular ptest needs more memroy, it can be customized:
> > +#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
> > +
> > +TEST_SUITES = "ping ssh parselogs ptest"
> > +
> > +# Sadly at the moment the full set of ptests is not robust enough and
> sporadically fails in random places
> > +PTEST_EXPECT_FAILURE = "1"
> > +
> > +python () {
> > +    if not d.getVar("MCNAME"):
> > +        raise bb.parse.SkipRecipe("No class extension set")
> > +}
> > +
> > --
> > 2.34.1
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#101882):
> https://lists.openembedded.org/g/openembedded-devel/message/101882
> > Mute This Topic: https://lists.openembedded.org/mt/98023314/1997914
> > Group Owner: openembedded-devel+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [
> raj.khem@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>