[08/23] oeqa/sdk: add helpers to check for and install packages

The existing tests simply look at the manifest to determine if a test
should be ran or not based on dependencies. Whilst this works for
Traditional SDKs, it fails for Extensible SDKs if they've been built in
minimal mode, where the manifest will be empty.  However, minimal eSDKs
might well have available sstate to install the missing dependencies.

Add a pair of helper functions to ensure that a package is available, or
skip the test.  This handles nativesdk- vs -native (SDK vs eSDK) and
will try to sdk-install missing dependencies into an eSDK if they're not
already installed.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/lib/oeqa/sdk/case.py | 58 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

Hi Ross,

I have a problem when running oe-selftest after this commit.
When I updated past this commit and commit which uses it:
https://git.openembedded.org/openembedded-core/commit/?id=11277efd057685558a744e98082b5709e849dd2a

I get following failure (on Debian12 VM):
===
Traceback (most recent call last):
  File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/core/case.py", line 54, in _oeSetUp
    self.setUpMethod()
  File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/sdk/cases/maturin.py", line 35, in setUp
    self.ensure_host_package("python3-maturin")
  File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/sdk/case.py", line 47, in ensure_host_package
    self._run('devtool sdk-install %s' % recipe)
  File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/sdk/case.py", line 17, in _run
    return subprocess.check_output(". %s > /dev/null; %s;" % \
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
oeqa.utils.subprocesstweak.OETestCalledProcessError: <exception str() failed>

Stdout:
Trying to install python3-maturin-native...
===

See below for comment in code which I see as problematic.

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Ross Burton via
> lists.openembedded.org
> Sent: Saturday, May 10, 2025 10:44
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH 08/23] oeqa/sdk: add helpers to check for and install
> packages
> 
> The existing tests simply look at the manifest to determine if a test
> should be ran or not based on dependencies. Whilst this works for
> Traditional SDKs, it fails for Extensible SDKs if they've been built in
> minimal mode, where the manifest will be empty.  However, minimal eSDKs
> might well have available sstate to install the missing dependencies.
> 
> Add a pair of helper functions to ensure that a package is available, or
> skip the test.  This handles nativesdk- vs -native (SDK vs eSDK) and
> will try to sdk-install missing dependencies into an eSDK if they're not
> already installed.
> 
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  meta/lib/oeqa/sdk/case.py | 58
> +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 58 insertions(+)
> 
> diff --git a/meta/lib/oeqa/sdk/case.py b/meta/lib/oeqa/sdk/case.py
> index 46a3789f572..1fd3b3b5695 100644
> --- a/meta/lib/oeqa/sdk/case.py
> +++ b/meta/lib/oeqa/sdk/case.py
> @@ -7,8 +7,10 @@
>  import os
>  import subprocess
>  import shutil
> +import unittest
> 
>  from oeqa.core.case import OETestCase
> +from oeqa.sdkext.context import OESDKExtTestContext
> 
>  class OESDKTestCase(OETestCase):
>      def _run(self, cmd):
> @@ -16,6 +18,62 @@ class OESDKTestCase(OETestCase):
>                  (self.tc.sdk_env, cmd), shell=True, executable="/bin/bash",
>                  stderr=subprocess.STDOUT, universal_newlines=True)
> 
> +    def ensure_host_package(self, *packages, recipe=None):
> +        """
> +        Check that the host variation of one of the packages listed is available
> +        in the SDK (nativesdk-foo for SDK, foo-native for eSDK). The package is
> +        a list for the case where debian-renaming may have occured, and the
> +        manifest could contain 'foo' or 'libfoo'.
> +
> +        If testing an eSDK and the package is not found, then try to install the
> +        specified recipe to install it from sstate.
> +        """
> +
> +        # In a SDK the manifest is correct. In an eSDK the manifest may be
> +        # correct (type=full) or not include packages that exist in sstate but
> +        # not installed yet (minimal) so we should try to install the recipe.
> +        for package in packages:
> +            if isinstance(self.tc, OESDKExtTestContext):
> +                package = package + "-native"
> +            else:
> +                package = "nativesdk-" + package
> +
> +            if self.tc.hasHostPackage(package):
> +                break
> +        else:
> +            if isinstance(self.tc, OESDKExtTestContext):
> +                recipe = (recipe or packages[0]) + "-native"
> +                print("Trying to install %s..." % recipe)
> +                self._run('devtool sdk-install %s' % recipe)


The comment above says "try to install", however there is no check for failure and skipping the test in such case.
Before using this code the testscases were skipped unconditionally and didn't try to install anything.
Here it looks like it should always fail if the package is not part of sdk (and python3-maturin is not part of minimal/sato sdk images on autobuilder).

I'm struggling to understand how this passes autobuilder tests green and my pribate builds are red.
Autobuilder logs show that the maturin tests were skipped and python3-maturin is never built.
Do you have a hint for me how this code can skip the recipe?

I was thinking about installing it on my host, but python3-maturin is only available from Debian 13 (which is not released yet).

Would it be fine if I submit a patch doing try/except around this line?
Or am I just missing something vital to understand the code and failure?

> +            else:
> +                raise unittest.SkipTest("Test %s needs one of %s" % (self.id(), ",
> ".join(packages)))
> +
> +    def ensure_target_package(self, *packages, multilib=False, recipe=None):
> +        """
> +        Check that at least one of the packages listed is available in the SDK,
> +        adding the multilib prefix if required. The target package is a list for
> +        the case where debian-renaming may have occured, and the manifest could
> +        contain 'foo' or 'libfoo'.
> +
> +        If testing an eSDK and the package is not found, then try to install the
> +        specified recipe to install it from sstate.
> +        """
> +
> +        # In a SDK the manifest is correct. In an eSDK the manifest may be
> +        # correct (type=full) or not include packages that exist in sstate but
> +        # not installed yet (minimal) so we should try to install the recipe.
> +        for package in packages:
> +            if self.tc.hasTargetPackage(package, multilib=multilib):
> +                break
> +        else:
> +            if isinstance(self.tc, OESDKExtTestContext):
> +                recipe = recipe or packages[0]
> +                print("Trying to install %s..." % recipe)
> +                self._run('devtool sdk-install %s' % recipe)
> +            else:
> +                raise unittest.SkipTest("Test %s needs one of %s" % (self.id(), ",
> ".join(packages)))
> +
> +
>      def fetch(self, workdir, dl_dir, url, archive=None):
>          if not archive:
>              from urllib.parse import urlparse
> --
> 2.43.0

Khem,

you mentioned the maturin test failure problem on IRC yesterday.
This is the commit which broke it.
I'll propose a patch shortly.

Peter

> -----Original Message-----
> From: Marko, Peter (FT D EU SK BFS1)
> Sent: Wednesday, July 16, 2025 15:54
> To: ross.burton@arm.com; openembedded-core@lists.openembedded.org
> Subject: RE: [OE-core] [PATCH 08/23] oeqa/sdk: add helpers to check for and
> install packages
> 
> Hi Ross,
> 
> I have a problem when running oe-selftest after this commit.
> When I updated past this commit and commit which uses it:
> https://git.openembedded.org/openembedded-
> core/commit/?id=11277efd057685558a744e98082b5709e849dd2a
> 
> I get following failure (on Debian12 VM):
> ===
> Traceback (most recent call last):
>   File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/core/case.py", line 54,
> in _oeSetUp
>     self.setUpMethod()
>   File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/sdk/cases/maturin.py",
> line 35, in setUp
>     self.ensure_host_package("python3-maturin")
>   File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/sdk/case.py", line 47,
> in ensure_host_package
>     self._run('devtool sdk-install %s' % recipe)
>   File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/sdk/case.py", line 17,
> in _run
>     return subprocess.check_output(". %s > /dev/null; %s;" % \
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
>     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/subprocess.py", line 571, in run
>     raise CalledProcessError(retcode, process.args,
> oeqa.utils.subprocesstweak.OETestCalledProcessError: <exception str() failed>
> 
> Stdout:
> Trying to install python3-maturin-native...
> ===
> 
> See below for comment in code which I see as problematic.
> 
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Ross Burton via
> > lists.openembedded.org
> > Sent: Saturday, May 10, 2025 10:44
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [PATCH 08/23] oeqa/sdk: add helpers to check for and install
> > packages
> >
> > The existing tests simply look at the manifest to determine if a test
> > should be ran or not based on dependencies. Whilst this works for
> > Traditional SDKs, it fails for Extensible SDKs if they've been built in
> > minimal mode, where the manifest will be empty.  However, minimal eSDKs
> > might well have available sstate to install the missing dependencies.
> >
> > Add a pair of helper functions to ensure that a package is available, or
> > skip the test.  This handles nativesdk- vs -native (SDK vs eSDK) and
> > will try to sdk-install missing dependencies into an eSDK if they're not
> > already installed.
> >
> > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > ---
> >  meta/lib/oeqa/sdk/case.py | 58
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 58 insertions(+)
> >
> > diff --git a/meta/lib/oeqa/sdk/case.py b/meta/lib/oeqa/sdk/case.py
> > index 46a3789f572..1fd3b3b5695 100644
> > --- a/meta/lib/oeqa/sdk/case.py
> > +++ b/meta/lib/oeqa/sdk/case.py
> > @@ -7,8 +7,10 @@
> >  import os
> >  import subprocess
> >  import shutil
> > +import unittest
> >
> >  from oeqa.core.case import OETestCase
> > +from oeqa.sdkext.context import OESDKExtTestContext
> >
> >  class OESDKTestCase(OETestCase):
> >      def _run(self, cmd):
> > @@ -16,6 +18,62 @@ class OESDKTestCase(OETestCase):
> >                  (self.tc.sdk_env, cmd), shell=True, executable="/bin/bash",
> >                  stderr=subprocess.STDOUT, universal_newlines=True)
> >
> > +    def ensure_host_package(self, *packages, recipe=None):
> > +        """
> > +        Check that the host variation of one of the packages listed is available
> > +        in the SDK (nativesdk-foo for SDK, foo-native for eSDK). The package is
> > +        a list for the case where debian-renaming may have occured, and the
> > +        manifest could contain 'foo' or 'libfoo'.
> > +
> > +        If testing an eSDK and the package is not found, then try to install the
> > +        specified recipe to install it from sstate.
> > +        """
> > +
> > +        # In a SDK the manifest is correct. In an eSDK the manifest may be
> > +        # correct (type=full) or not include packages that exist in sstate but
> > +        # not installed yet (minimal) so we should try to install the recipe.
> > +        for package in packages:
> > +            if isinstance(self.tc, OESDKExtTestContext):
> > +                package = package + "-native"
> > +            else:
> > +                package = "nativesdk-" + package
> > +
> > +            if self.tc.hasHostPackage(package):
> > +                break
> > +        else:
> > +            if isinstance(self.tc, OESDKExtTestContext):
> > +                recipe = (recipe or packages[0]) + "-native"
> > +                print("Trying to install %s..." % recipe)
> > +                self._run('devtool sdk-install %s' % recipe)
> 
> 
> The comment above says "try to install", however there is no check for failure and
> skipping the test in such case.
> Before using this code the testscases were skipped unconditionally and didn't try
> to install anything.
> Here it looks like it should always fail if the package is not part of sdk (and
> python3-maturin is not part of minimal/sato sdk images on autobuilder).
> 
> I'm struggling to understand how this passes autobuilder tests green and my
> pribate builds are red.
> Autobuilder logs show that the maturin tests were skipped and python3-maturin is
> never built.
> Do you have a hint for me how this code can skip the recipe?
> 
> I was thinking about installing it on my host, but python3-maturin is only available
> from Debian 13 (which is not released yet).
> 
> Would it be fine if I submit a patch doing try/except around this line?
> Or am I just missing something vital to understand the code and failure?
> 
> > +            else:
> > +                raise unittest.SkipTest("Test %s needs one of %s" % (self.id(), ",
> > ".join(packages)))
> > +
> > +    def ensure_target_package(self, *packages, multilib=False, recipe=None):
> > +        """
> > +        Check that at least one of the packages listed is available in the SDK,
> > +        adding the multilib prefix if required. The target package is a list for
> > +        the case where debian-renaming may have occured, and the manifest
> could
> > +        contain 'foo' or 'libfoo'.
> > +
> > +        If testing an eSDK and the package is not found, then try to install the
> > +        specified recipe to install it from sstate.
> > +        """
> > +
> > +        # In a SDK the manifest is correct. In an eSDK the manifest may be
> > +        # correct (type=full) or not include packages that exist in sstate but
> > +        # not installed yet (minimal) so we should try to install the recipe.
> > +        for package in packages:
> > +            if self.tc.hasTargetPackage(package, multilib=multilib):
> > +                break
> > +        else:
> > +            if isinstance(self.tc, OESDKExtTestContext):
> > +                recipe = recipe or packages[0]
> > +                print("Trying to install %s..." % recipe)
> > +                self._run('devtool sdk-install %s' % recipe)
> > +            else:
> > +                raise unittest.SkipTest("Test %s needs one of %s" % (self.id(), ",
> > ".join(packages)))
> > +
> > +
> >      def fetch(self, workdir, dl_dir, url, archive=None):
> >          if not archive:
> >              from urllib.parse import urlparse
> > --
> > 2.43.0

On Tue, Jul 22, 2025 at 11:14 PM Marko, Peter <Peter.Marko@siemens.com> wrote:
>
> Khem,
>
> you mentioned the maturin test failure problem on IRC yesterday.
> This is the commit which broke it.

most likely seems to be the culprit.

> I'll propose a patch shortly.
>

Thanks for help

> Peter
>
> > -----Original Message-----
> > From: Marko, Peter (FT D EU SK BFS1)
> > Sent: Wednesday, July 16, 2025 15:54
> > To: ross.burton@arm.com; openembedded-core@lists.openembedded.org
> > Subject: RE: [OE-core] [PATCH 08/23] oeqa/sdk: add helpers to check for and
> > install packages
> >
> > Hi Ross,
> >
> > I have a problem when running oe-selftest after this commit.
> > When I updated past this commit and commit which uses it:
> > https://git.openembedded.org/openembedded-
> > core/commit/?id=11277efd057685558a744e98082b5709e849dd2a
> >
> > I get following failure (on Debian12 VM):
> > ===
> > Traceback (most recent call last):
> >   File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/core/case.py", line 54,
> > in _oeSetUp
> >     self.setUpMethod()
> >   File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/sdk/cases/maturin.py",
> > line 35, in setUp
> >     self.ensure_host_package("python3-maturin")
> >   File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/sdk/case.py", line 47,
> > in ensure_host_package
> >     self._run('devtool sdk-install %s' % recipe)
> >   File "/home/projects/builds/poky-build/poky/meta/lib/oeqa/sdk/case.py", line 17,
> > in _run
> >     return subprocess.check_output(". %s > /dev/null; %s;" % \
> >            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >   File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
> >     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
> >            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >   File "/usr/lib/python3.11/subprocess.py", line 571, in run
> >     raise CalledProcessError(retcode, process.args,
> > oeqa.utils.subprocesstweak.OETestCalledProcessError: <exception str() failed>
> >
> > Stdout:
> > Trying to install python3-maturin-native...
> > ===
> >
> > See below for comment in code which I see as problematic.
> >
> > > -----Original Message-----
> > > From: openembedded-core@lists.openembedded.org <openembedded-
> > > core@lists.openembedded.org> On Behalf Of Ross Burton via
> > > lists.openembedded.org
> > > Sent: Saturday, May 10, 2025 10:44
> > > To: openembedded-core@lists.openembedded.org
> > > Subject: [OE-core] [PATCH 08/23] oeqa/sdk: add helpers to check for and install
> > > packages
> > >
> > > The existing tests simply look at the manifest to determine if a test
> > > should be ran or not based on dependencies. Whilst this works for
> > > Traditional SDKs, it fails for Extensible SDKs if they've been built in
> > > minimal mode, where the manifest will be empty.  However, minimal eSDKs
> > > might well have available sstate to install the missing dependencies.
> > >
> > > Add a pair of helper functions to ensure that a package is available, or
> > > skip the test.  This handles nativesdk- vs -native (SDK vs eSDK) and
> > > will try to sdk-install missing dependencies into an eSDK if they're not
> > > already installed.
> > >
> > > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > > ---
> > >  meta/lib/oeqa/sdk/case.py | 58
> > > +++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 58 insertions(+)
> > >
> > > diff --git a/meta/lib/oeqa/sdk/case.py b/meta/lib/oeqa/sdk/case.py
> > > index 46a3789f572..1fd3b3b5695 100644
> > > --- a/meta/lib/oeqa/sdk/case.py
> > > +++ b/meta/lib/oeqa/sdk/case.py
> > > @@ -7,8 +7,10 @@
> > >  import os
> > >  import subprocess
> > >  import shutil
> > > +import unittest
> > >
> > >  from oeqa.core.case import OETestCase
> > > +from oeqa.sdkext.context import OESDKExtTestContext
> > >
> > >  class OESDKTestCase(OETestCase):
> > >      def _run(self, cmd):
> > > @@ -16,6 +18,62 @@ class OESDKTestCase(OETestCase):
> > >                  (self.tc.sdk_env, cmd), shell=True, executable="/bin/bash",
> > >                  stderr=subprocess.STDOUT, universal_newlines=True)
> > >
> > > +    def ensure_host_package(self, *packages, recipe=None):
> > > +        """
> > > +        Check that the host variation of one of the packages listed is available
> > > +        in the SDK (nativesdk-foo for SDK, foo-native for eSDK). The package is
> > > +        a list for the case where debian-renaming may have occured, and the
> > > +        manifest could contain 'foo' or 'libfoo'.
> > > +
> > > +        If testing an eSDK and the package is not found, then try to install the
> > > +        specified recipe to install it from sstate.
> > > +        """
> > > +
> > > +        # In a SDK the manifest is correct. In an eSDK the manifest may be
> > > +        # correct (type=full) or not include packages that exist in sstate but
> > > +        # not installed yet (minimal) so we should try to install the recipe.
> > > +        for package in packages:
> > > +            if isinstance(self.tc, OESDKExtTestContext):
> > > +                package = package + "-native"
> > > +            else:
> > > +                package = "nativesdk-" + package
> > > +
> > > +            if self.tc.hasHostPackage(package):
> > > +                break
> > > +        else:
> > > +            if isinstance(self.tc, OESDKExtTestContext):
> > > +                recipe = (recipe or packages[0]) + "-native"
> > > +                print("Trying to install %s..." % recipe)
> > > +                self._run('devtool sdk-install %s' % recipe)
> >
> >
> > The comment above says "try to install", however there is no check for failure and
> > skipping the test in such case.
> > Before using this code the testscases were skipped unconditionally and didn't try
> > to install anything.
> > Here it looks like it should always fail if the package is not part of sdk (and
> > python3-maturin is not part of minimal/sato sdk images on autobuilder).
> >
> > I'm struggling to understand how this passes autobuilder tests green and my
> > pribate builds are red.
> > Autobuilder logs show that the maturin tests were skipped and python3-maturin is
> > never built.
> > Do you have a hint for me how this code can skip the recipe?
> >
> > I was thinking about installing it on my host, but python3-maturin is only available
> > from Debian 13 (which is not released yet).
> >
> > Would it be fine if I submit a patch doing try/except around this line?
> > Or am I just missing something vital to understand the code and failure?
> >
> > > +            else:
> > > +                raise unittest.SkipTest("Test %s needs one of %s" % (self.id(), ",
> > > ".join(packages)))
> > > +
> > > +    def ensure_target_package(self, *packages, multilib=False, recipe=None):
> > > +        """
> > > +        Check that at least one of the packages listed is available in the SDK,
> > > +        adding the multilib prefix if required. The target package is a list for
> > > +        the case where debian-renaming may have occured, and the manifest
> > could
> > > +        contain 'foo' or 'libfoo'.
> > > +
> > > +        If testing an eSDK and the package is not found, then try to install the
> > > +        specified recipe to install it from sstate.
> > > +        """
> > > +
> > > +        # In a SDK the manifest is correct. In an eSDK the manifest may be
> > > +        # correct (type=full) or not include packages that exist in sstate but
> > > +        # not installed yet (minimal) so we should try to install the recipe.
> > > +        for package in packages:
> > > +            if self.tc.hasTargetPackage(package, multilib=multilib):
> > > +                break
> > > +        else:
> > > +            if isinstance(self.tc, OESDKExtTestContext):
> > > +                recipe = recipe or packages[0]
> > > +                print("Trying to install %s..." % recipe)
> > > +                self._run('devtool sdk-install %s' % recipe)
> > > +            else:
> > > +                raise unittest.SkipTest("Test %s needs one of %s" % (self.id(), ",
> > > ".join(packages)))
> > > +
> > > +
> > >      def fetch(self, workdir, dl_dir, url, archive=None):
> > >          if not archive:
> > >              from urllib.parse import urlparse
> > > --
> > > 2.43.0
>