| Message ID | 20250416164134.399-1-thune.a.tran@boeing.com |
|---|---|
| State | New |
| Headers | show |
| Series | oeqa/sdk/sdktest: Add test discovery across layers | expand |
The same issue is also present for sdkext - since you are already on this, do you think you could make that work too? On 4/16/25 18:41, Thune Tran via lists.openembedded.org wrote: > The testsdk task only loads SDK tests from the openembedded-core > layer's meta/lib/oeqa/sdk/cases/. > > This patch implements test discovery to include SDK tests from all > configured layers that mirror <LAYER_DIR>/lib/oeqa/sdk/cases/. The > implementation mirrors get_runtime_paths() in meta/classes-recipe/testimage. > > Tested by: > > 1. Adding SDK test cases to <LAYER_DIR>/lib/oeqa/sdk/cases/ and > verifying they are discovered and executed with no errors. > 2. Verifying existing SDK test cases in openembedded-core layer > are discovered and executed with no errors. > > Signed-off-by: Thune Tran <thune.a.tran@boeing.com> > Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com> > --- > meta/lib/oeqa/sdk/testsdk.py | 16 +++++++++++++++- > 1 file changed, 15 insertions(+), 1 deletion(-) > > diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py > index 518b09febb..505add8f8c 100644 > --- a/meta/lib/oeqa/sdk/testsdk.py > +++ b/meta/lib/oeqa/sdk/testsdk.py > @@ -31,6 +31,20 @@ class TestSDK(TestSDKBase): > context_class = OESDKTestContext > test_type = 'sdk' > > + def get_sdk_paths(self, d): > + """ > + Returns a list of paths where SDK test must reside. > + > + SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/ > + """ > + paths = [] > + > + for layer in d.getVar('BBLAYERS').split(): > + path = os.path.join(layer, 'lib/oeqa/sdk/cases') > + if os.path.isdir(path): > + paths.append(path) > + return paths > + > def get_tcname(self, d): > """ > Get the name of the SDK file > @@ -114,7 +128,7 @@ class TestSDK(TestSDKBase): > host_pkg_manifest=host_pkg_manifest, **context_args) > > try: > - tc.loadTests(self.context_executor_class.default_cases) > + tc.loadTests(self.get_sdk_paths(d)) > except Exception as e: > import traceback > bb.fatal("Loading tests failed:\n%s" % traceback.format_exc()) > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#215001): https://lists.openembedded.org/g/openembedded-core/message/215001 > Mute This Topic: https://lists.openembedded.org/mt/112298220/6084445 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [skandigraun@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
> The same issue is also present for sdkext - since you are already on > this, do you think you could make that work too? Thanks for pointing that out. A quick look at sdkext. It's possible to make it work for it. I'm open to submitting a similar patch for sdkext. I'm not immediately set up to test it. Would you be able to help verify the patch for sdkext and also sign off on it? > On 4/16/25 18:41, Thune Tran via lists.openembedded.org wrote: >> The testsdk task only loads SDK tests from the openembedded-core >> layer's meta/lib/oeqa/sdk/cases/. >> >> This patch implements test discovery to include SDK tests from all >> configured layers that mirror <LAYER_DIR>/lib/oeqa/sdk/cases/. The >> implementation mirrors get_runtime_paths() in meta/classes- >> recipe/testimage. >> >> Tested by: >> >> 1. Adding SDK test cases to <LAYER_DIR>/lib/oeqa/sdk/cases/ and >> verifying they are discovered and executed with no errors. 2. >> Verifying existing SDK test cases in openembedded-core layer are >> discovered and executed with no errors. >> Signed-off-by: Thune Tran <thune.a.tran@boeing.com> >> Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com> >> --- >> meta/lib/oeqa/sdk/testsdk.py | 16 +++++++++++++++- >> 1 file changed, 15 insertions(+), 1 deletion(-) >> diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py >> index 518b09febb..505add8f8c 100644 >> --- a/meta/lib/oeqa/sdk/testsdk.py >> +++ b/meta/lib/oeqa/sdk/testsdk.py >> @@ -31,6 +31,20 @@ class TestSDK(TestSDKBase): >> context_class = OESDKTestContext >> test_type = 'sdk' >> + def get_sdk_paths(self, d): >> + """ >> + Returns a list of paths where SDK test must reside. >> + >> + SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/ >> + """ >> + paths = [] >> + >> + for layer in d.getVar('BBLAYERS').split(): >> + path = os.path.join(layer, 'lib/oeqa/sdk/cases') >> + if os.path.isdir(path): >> + paths.append(path) >> + return paths >> + >> def get_tcname(self, d): >> """ >> Get the name of the SDK file >> @@ -114,7 +128,7 @@ class TestSDK(TestSDKBase): >> host_pkg_manifest=host_pkg_manifest, **context_args) >> try: >> - tc.loadTests(self.context_executor_class.default_cases) >> + tc.loadTests(self.get_sdk_paths(d)) >> except Exception as e: >> import traceback >> bb.fatal("Loading tests failed:\n%s" % traceback.format_exc()) >> >>
On 4/16/25 22:54, Tran (US), Thune A wrote: >> The same issue is also present for sdkext - since you are already on >> this, do you think you could make that work too? > Thanks for pointing that out. A quick look at sdkext. It's possible to make > it work for it. I'm open to submitting a similar patch for sdkext. I'm not > immediately set up to test it. Would you be able to help verify the patch > for sdkext and also sign off on it? [Just to have it here also, not only in private messages] Yeah, I think I should be able to do that - I have an extensible sdk set up, and some classes also that I can use for testing. >> On 4/16/25 18:41, Thune Tran via lists.openembedded.org wrote: >>> The testsdk task only loads SDK tests from the openembedded-core >>> layer's meta/lib/oeqa/sdk/cases/. >>> >>> This patch implements test discovery to include SDK tests from all >>> configured layers that mirror <LAYER_DIR>/lib/oeqa/sdk/cases/. The >>> implementation mirrors get_runtime_paths() in meta/classes- >>> recipe/testimage. >>> >>> Tested by: >>> >>> 1. Adding SDK test cases to <LAYER_DIR>/lib/oeqa/sdk/cases/ and >>> verifying they are discovered and executed with no errors. 2. >>> Verifying existing SDK test cases in openembedded-core layer are >>> discovered and executed with no errors. >>> Signed-off-by: Thune Tran <thune.a.tran@boeing.com> >>> Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com> >>> --- >>> meta/lib/oeqa/sdk/testsdk.py | 16 +++++++++++++++- >>> 1 file changed, 15 insertions(+), 1 deletion(-) >>> diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py >>> index 518b09febb..505add8f8c 100644 >>> --- a/meta/lib/oeqa/sdk/testsdk.py >>> +++ b/meta/lib/oeqa/sdk/testsdk.py >>> @@ -31,6 +31,20 @@ class TestSDK(TestSDKBase): >>> context_class = OESDKTestContext >>> test_type = 'sdk' >>> + def get_sdk_paths(self, d): >>> + """ >>> + Returns a list of paths where SDK test must reside. >>> + >>> + SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/ >>> + """ >>> + paths = [] >>> + >>> + for layer in d.getVar('BBLAYERS').split(): >>> + path = os.path.join(layer, 'lib/oeqa/sdk/cases') >>> + if os.path.isdir(path): >>> + paths.append(path) >>> + return paths >>> + >>> def get_tcname(self, d): >>> """ >>> Get the name of the SDK file >>> @@ -114,7 +128,7 @@ class TestSDK(TestSDKBase): >>> host_pkg_manifest=host_pkg_manifest, **context_args) >>> try: >>> - tc.loadTests(self.context_executor_class.default_cases) >>> + tc.loadTests(self.get_sdk_paths(d)) >>> except Exception as e: >>> import traceback >>> bb.fatal("Loading tests failed:\n%s" % traceback.format_exc()) >>> >>> >
On Wed, 2025-04-16 at 16:41 +0000, Thune Tran via lists.openembedded.org wrote: > The testsdk task only loads SDK tests from the openembedded-core > layer's meta/lib/oeqa/sdk/cases/. > > This patch implements test discovery to include SDK tests from all > configured layers that mirror <LAYER_DIR>/lib/oeqa/sdk/cases/. The > implementation mirrors get_runtime_paths() in meta/classes-recipe/testimage. > > Tested by: > > 1. Adding SDK test cases to <LAYER_DIR>/lib/oeqa/sdk/cases/ and > verifying they are discovered and executed with no errors. > 2. Verifying existing SDK test cases in openembedded-core layer > are discovered and executed with no errors. > > Signed-off-by: Thune Tran <thune.a.tran@boeing.com> > Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com> > --- > meta/lib/oeqa/sdk/testsdk.py | 16 +++++++++++++++- > 1 file changed, 15 insertions(+), 1 deletion(-) > > diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py > index 518b09febb..505add8f8c 100644 > --- a/meta/lib/oeqa/sdk/testsdk.py > +++ b/meta/lib/oeqa/sdk/testsdk.py > @@ -31,6 +31,20 @@ class TestSDK(TestSDKBase): > context_class = OESDKTestContext > test_type = 'sdk' > > + def get_sdk_paths(self, d): > + """ > + Returns a list of paths where SDK test must reside. > + > + SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/ > + """ > + paths = [] > + > + for layer in d.getVar('BBLAYERS').split(): > + path = os.path.join(layer, 'lib/oeqa/sdk/cases') > + if os.path.isdir(path): > + paths.append(path) > + return paths > + > def get_tcname(self, d): > """ > Get the name of the SDK file > @@ -114,7 +128,7 @@ class TestSDK(TestSDKBase): > host_pkg_manifest=host_pkg_manifest, **context_args) > > try: > - tc.loadTests(self.context_executor_class.default_cases) > + tc.loadTests(self.get_sdk_paths(d)) > except Exception as e: > import traceback > bb.fatal("Loading tests failed:\n%s" % traceback.format_exc()) > I suspect this causes some regressions in our automated testing unfortunately: buildtools: https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1396 meta-mingw: https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/1430 Cheers, Richard
On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: %< SNIP $< > > I suspect this causes some regressions in our automated testing > unfortunately: > > buildtools: > https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1396 > > meta-mingw: > https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/1430 Would it make more sense to add self.context_executor_class.default_cases to the list of paths returned by get_sdk_paths? ..Ch:W..
On 4/17/25 06:39, Chuck Wolber via lists.openembedded.org wrote: > On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via > lists.openembedded.org <http://lists.openembedded.org> > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > %< SNIP $< > > > I suspect this causes some regressions in our automated testing > unfortunately: > > buildtools: > https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1396 > > meta-mingw: > https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/1430 > > > Would it make more sense to add self.context_executor_class.default_cases > to the list of paths returned by get_sdk_paths? > Well, the missing tests is one thing, but the other one is that the errors are thrown by tests that were not meant to be executed (at least when it comes to buildtools - haven't checked meta-mingw yet). > ..Ch:W.. > > > -- > * Opinions expressed are my own. | > --------------------------------|------ "Perfection must be reached > by degrees > she requires the slow hand of time." > | -Voltaire*
On Wed, 2025-04-16 at 21:39 -0700, Chuck Wolber wrote: > On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via > lists.openembedded.org > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > %< SNIP $< > > > > I suspect this causes some regressions in our automated testing > > unfortunately: > > > > buildtools: > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1396 > > > > meta-mingw: > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/1430 > > > > > Would it make more sense to add > self.context_executor_class.default_cases > to the list of paths returned by get_sdk_paths? meta/recipes-core/meta/buildtools-tarball.bb does: cases_path = os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__file__)), d.getVar("TESTSDK_CASES")) testsdk.context_executor_class.default_cases = cases_path mingw does things in a different place, setting: default_cases = [os.path.join(os.path.abspath(os.path.dirname(__file__)), 'cases')] in meta-mingw/lib/oeqa/sdkmingw/contex.py I think we need to add something which represents the name of the directory for cases, i.e. "sdk", "sdkming", "sdk/buildtools-cases" and then allow this to be passed into your function so regardless, it allows customisation within a layer. Just adding default cases to anything in the layer is going to always add the layer's test cases which isn't what we want (i.e. your layer would add to buildtools, mingw and sdk). Cheers, Richard
> On Wed, 2025-04-16 at 21:39 -0700, Chuck Wolber wrote: >> On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via >> lists.openembedded.org >> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: >> >> %< SNIP $< >>> >>> I suspect this causes some regressions in our automated testing >>> unfortunately: >>> >>> buildtools: >>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1 >>> 396 >>> >>> meta-mingw: >>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/14 >>> 30 >>> >> >> >> Would it make more sense to add >> self.context_executor_class.default_cases >> to the list of paths returned by get_sdk_paths? > > meta/recipes-core/meta/buildtools-tarball.bb does: > > cases_path = > os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__file__)), > d.getVar("TESTSDK_CASES")) > testsdk.context_executor_class.default_cases = cases_path > mingw does things in a different place, setting: > > default_cases = [os.path.join(os.path.abspath(os.path.dirname(__file__)), > 'cases')] > > in meta-mingw/lib/oeqa/sdkmingw/contex.py > > I think we need to add something which represents the name of the > directory for cases, i.e. "sdk", "sdkming", "sdk/buildtools-cases" and > then allow this to be passed into your function so regardless, it allows > customisation within a layer. We can match the test directory pattern (sdk, sdkming, sdk/buildtools-cases) found in default_cases. The test discovery will then be based on the matched pattern. > > Just adding default cases to anything in the layer is going to always add the > layer's test cases which isn't what we want (i.e. your layer would add to > buildtools, mingw and sdk). > > Cheers, > > Richard
On Thu, 2025-04-17 at 12:54 +0000, Thune Tran via lists.openembedded.org wrote: > > On Wed, 2025-04-16 at 21:39 -0700, Chuck Wolber wrote: > > > On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via > > > lists.openembedded.org > > > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > > > > > %< SNIP $< > > > > > > > > I suspect this causes some regressions in our automated testing > > > > unfortunately: > > > > > > > > buildtools: > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1 > > > > 396 > > > > > > > > meta-mingw: > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/14 > > > > 30 > > > > > > > > > > > > > Would it make more sense to add > > > self.context_executor_class.default_cases > > > to the list of paths returned by get_sdk_paths? > > > > meta/recipes-core/meta/buildtools-tarball.bb does: > > > > cases_path = > > os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__file__)), > > d.getVar("TESTSDK_CASES")) > > testsdk.context_executor_class.default_cases = cases_path > > mingw does things in a different place, setting: > > > > default_cases = [os.path.join(os.path.abspath(os.path.dirname(__file__)), > > 'cases')] > > > > in meta-mingw/lib/oeqa/sdkmingw/contex.py > > > > I think we need to add something which represents the name of the > > directory for cases, i.e. "sdk", "sdkming", "sdk/buildtools-cases" and > > then allow this to be passed into your function so regardless, it allows > > customisation within a layer. > We can match the test directory pattern (sdk, sdkming, sdk/buildtools-cases) > found in default_cases. The test discovery will then be based on the matched pattern. > We could do that but in this case I think it might be better to improve the code and make it a bit cleaner if we can, even if that is a little more invasive. This area of the code could do with simplification/cleanup... Cheers, Richard
> On Thu, 2025-04-17 at 12:54 +0000, Thune Tran via lists.openembedded.org > wrote: >>> On Wed, 2025-04-16 at 21:39 -0700, Chuck Wolber wrote: >>>> On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via >>>> lists.openembedded.org >>>> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: >>>> >>>> %< SNIP $< >>>>> >>>>> I suspect this causes some regressions in our automated testing >>>>> unfortunately: >>>>> >>>>> buildtools: >>>>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/buil >>>>> ds/1 >>>>> 396 >>>>> >>>>> meta-mingw: >>>>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/build >>>>> s/14 >>>>> 30 >>>>> >>>> >>>> >>>> Would it make more sense to add >>>> self.context_executor_class.default_cases >>>> to the list of paths returned by get_sdk_paths? >>> >>> meta/recipes-core/meta/buildtools-tarball.bb does: >>> >>> cases_path = >>> os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__file >>> __)), >>> d.getVar("TESTSDK_CASES")) >>> testsdk.context_executor_class.default_cases = cases_path mingw >>> does things in a different place, setting: >>> >>> default_cases = >>> [os.path.join(os.path.abspath(os.path.dirname(__file__)), >>> 'cases')] >>> >>> in meta-mingw/lib/oeqa/sdkmingw/contex.py >>> >>> I think we need to add something which represents the name of the >>> directory for cases, i.e. "sdk", "sdkming", "sdk/buildtools-cases" >>> and then allow this to be passed into your function so regardless, >>> it allows customisation within a layer. >> We can match the test directory pattern (sdk, sdkming, >> sdk/buildtools-cases) found in default_cases. The test discovery will >> then be based on the matched pattern. >> > > We could do that but in this case I think it might be better to improve > the code and make it a bit cleaner if we can, even if that is a little > more invasive. > > This area of the code could do with simplification/cleanup... Makes sense. We'll look into how the code can be improved as we continue to work in and become more familiar with this area of the code. > Cheers, > > Richard
diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py index 518b09febb..505add8f8c 100644 --- a/meta/lib/oeqa/sdk/testsdk.py +++ b/meta/lib/oeqa/sdk/testsdk.py @@ -31,6 +31,20 @@ class TestSDK(TestSDKBase): context_class = OESDKTestContext test_type = 'sdk' + def get_sdk_paths(self, d): + """ + Returns a list of paths where SDK test must reside. + + SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/ + """ + paths = [] + + for layer in d.getVar('BBLAYERS').split(): + path = os.path.join(layer, 'lib/oeqa/sdk/cases') + if os.path.isdir(path): + paths.append(path) + return paths + def get_tcname(self, d): """ Get the name of the SDK file @@ -114,7 +128,7 @@ class TestSDK(TestSDKBase): host_pkg_manifest=host_pkg_manifest, **context_args) try: - tc.loadTests(self.context_executor_class.default_cases) + tc.loadTests(self.get_sdk_paths(d)) except Exception as e: import traceback bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())