| Message ID | 20250605221301.890797-1-thune.a.tran@boeing.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v2] oeqa/sdk: Simplify test specification and discovery | expand |
On 5 Jun 2025, at 23:13, Thune Tran via lists.openembedded.org <thune.a.tran=boeing.com@lists.openembedded.org> wrote: > Simplify how tests are specified and discovered for different SDK configurations > to allow per-layer customization. Nice! > * Introduce `TESTSDK_CASE_DIRS` variable to specify test directory types, > replacing the need to modify the default_cases class member Is there a need for this to be _DIRS and not _DIR? Is this just being generalised because you can, or do you have internal uses where there are multiple case directory names? > meta/lib/oeqa/ > ├── sdk/cases/ # (default) Standard SDK: dirname="sdk" > ├── sdkbuildtools/cases/ # Buildtools: dirname="sdkbuildtools" > └── sdkbuildtools-docs/cases/ # Buildtools-docs: dirname="sdkbuildtools-docs" Can you drop the sdk prefix from sdkbuildtools and sdkbuildtools-docs? > * Add mingw sdk compatibility until meta-mingw migrates to TESTSDK_CASE_DIRS I’m tempted to say don’t bother having mingw compatibility, but if you can send a patch to migrate meta-mingw then we can just merge them both at the same time. Thanks, Ross
On Tue, 2025-06-24 at 11:59 +0000, Ross Burton via lists.openembedded.org wrote: > On 5 Jun 2025, at 23:13, Thune Tran via lists.openembedded.org > <thune.a.tran=boeing.com@lists.openembedded.org> wrote: > > Simplify how tests are specified and discovered for different SDK > > configurations > > to allow per-layer customization. > > Nice! > > > * Introduce `TESTSDK_CASE_DIRS` variable to specify test directory > > types, > > replacing the need to modify the default_cases class member > > Is there a need for this to be _DIRS and not _DIR? > > Is this just being generalised because you can, or do you have > internal uses where there are multiple case directory names? > > > meta/lib/oeqa/ > > ├── sdk/cases/ # (default) Standard SDK: > > dirname="sdk" > > ├── sdkbuildtools/cases/ # Buildtools: > > dirname="sdkbuildtools" > > └── sdkbuildtools-docs/cases/ # Buildtools-docs: > > dirname="sdkbuildtools-docs" > > Can you drop the sdk prefix from sdkbuildtools and sdkbuildtools- > docs? > > > * Add mingw sdk compatibility until meta-mingw migrates to > > TESTSDK_CASE_DIRS > > I’m tempted to say don’t bother having mingw compatibility, but if > you can send a patch to migrate meta-mingw then we can just merge > them both at the same time. Personally, I don't mind the multiple test case dirs but just wanted to say I agree with the rest of the feedback and it matches what I'd mentioned on the review call. The patch in general is great to see, we just need to tweak the details slightly, thanks! I'm happy to merge a tweak to mingw at the same time. Cheers, Richard
> On Tue, 2025-06-24 at 11:59 +0000, Ross Burton via lists.openembedded.org > wrote: >>> * Introduce `TESTSDK_CASE_DIRS` variable to specify test directory >>> types, >>> replacing the need to modify the default_cases class member >> >> Is there a need for this to be _DIRS and not _DIR? >> >> Is this just being generalised because you can, or do you have >> internal uses where there are multiple case directory names? We currently don't have any use cases requiring multiple case dirs, just following patterns of other list variables. >> Can you drop the sdk prefix from sdkbuildtools and sdkbuildtools- >> docs? Yes >> I’m tempted to say don’t bother having mingw compatibility, but if you >> can send a patch to migrate meta-mingw then we can just merge them >> both at the same time. I'll remove the mingw compatibility and submit a separate patch to meta-mingw so both can be merged together. > Personally, I don't mind the multiple test case dirs but just wanted to say I > agree with the rest of the feedback and it matches what I'd mentioned on the > review call. The patch in general is great to see, we just need to tweak the > details slightly, thanks! Thanks for the feedback! I'll send v3 with these changes.
diff --git a/meta/classes-recipe/testsdk.bbclass b/meta/classes-recipe/testsdk.bbclass index 59d2834c99..b1c4fa67e6 100644 --- a/meta/classes-recipe/testsdk.bbclass +++ b/meta/classes-recipe/testsdk.bbclass @@ -19,6 +19,7 @@ TESTSDK_SUITES ?= "" TESTSDK_CLASS_NAME ?= "oeqa.sdk.testsdk.TestSDK" TESTSDKEXT_CLASS_NAME ?= "oeqa.sdkext.testsdk.TestSDKExt" +TESTSDK_CASE_DIRS ?= "sdk" def import_and_run(name, d): import importlib diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py index 52b702b6a2..2507cb035f 100644 --- a/meta/lib/oeqa/sdk/testsdk.py +++ b/meta/lib/oeqa/sdk/testsdk.py @@ -6,6 +6,13 @@ from oeqa.sdk.context import OESDKTestContext, OESDKTestContextExecutor +# TODO: drop once the meta-mingw layer migrates to TESTSDK_CASE_DIRS +def mingw_default_dir(d): + sdkmach = d.getVar("SDKMACHINE") or "" + if sdkmach.endswith("mingw32"): + return "sdkmingw" + return "" + class TestSDKBase(object): @staticmethod def get_sdk_configuration(d, test_type): @@ -31,6 +38,32 @@ class TestSDK(TestSDKBase): context_class = OESDKTestContext test_type = 'sdk' + def sdk_dir_names(self, d): + """Return list from TESTSDK_CASE_DIRS.""" + mingw_dir = mingw_default_dir(d) + if mingw_dir: + return mingw_dir.split() + + testdirs = d.getVar("TESTSDK_CASE_DIRS") + if testdirs: + return testdirs.split() + + bb.fatal("TESTSDK_CASE_DIRS unset, can't find SDK test directories.") + + def get_sdk_paths(self, d): + """ + Return a list of paths where SDK test cases reside. + + SDK tests are expected in <LAYER_DIR>/lib/oeqa/<dirname>/cases + """ + paths = [] + for layer in d.getVar("BBLAYERS").split(): + for dirname in self.sdk_dir_names(d): + case_path = os.path.join(layer, "lib", "oeqa", dirname, "cases") + if os.path.isdir(case_path): + paths.append(case_path) + return paths + def get_tcname(self, d): """ Get the name of the SDK file @@ -115,7 +148,7 @@ class TestSDK(TestSDKBase): try: modules = (d.getVar("TESTSDK_SUITES") or "").split() - tc.loadTests(self.context_executor_class.default_cases, modules) + tc.loadTests(self.get_sdk_paths(d), modules) except Exception as e: import traceback bb.fatal("Loading tests failed:\n%s" % traceback.format_exc()) diff --git a/meta/lib/oeqa/sdk/buildtools-docs-cases/README b/meta/lib/oeqa/sdkbuildtools-docs/cases/README similarity index 100% rename from meta/lib/oeqa/sdk/buildtools-docs-cases/README rename to meta/lib/oeqa/sdkbuildtools-docs/cases/README diff --git a/meta/lib/oeqa/sdk/buildtools-docs-cases/build.py b/meta/lib/oeqa/sdkbuildtools-docs/cases/build.py similarity index 100% rename from meta/lib/oeqa/sdk/buildtools-docs-cases/build.py rename to meta/lib/oeqa/sdkbuildtools-docs/cases/build.py diff --git a/meta/lib/oeqa/sdk/buildtools-cases/README b/meta/lib/oeqa/sdkbuildtools/cases/README similarity index 100% rename from meta/lib/oeqa/sdk/buildtools-cases/README rename to meta/lib/oeqa/sdkbuildtools/cases/README diff --git a/meta/lib/oeqa/sdk/buildtools-cases/build.py b/meta/lib/oeqa/sdkbuildtools/cases/build.py similarity index 100% rename from meta/lib/oeqa/sdk/buildtools-cases/build.py rename to meta/lib/oeqa/sdkbuildtools/cases/build.py diff --git a/meta/lib/oeqa/sdk/buildtools-cases/gcc.py b/meta/lib/oeqa/sdkbuildtools/cases/gcc.py similarity index 100% rename from meta/lib/oeqa/sdk/buildtools-cases/gcc.py rename to meta/lib/oeqa/sdkbuildtools/cases/gcc.py diff --git a/meta/lib/oeqa/sdk/buildtools-cases/https.py b/meta/lib/oeqa/sdkbuildtools/cases/https.py similarity index 100% rename from meta/lib/oeqa/sdk/buildtools-cases/https.py rename to meta/lib/oeqa/sdkbuildtools/cases/https.py diff --git a/meta/lib/oeqa/sdk/buildtools-cases/sanity.py b/meta/lib/oeqa/sdkbuildtools/cases/sanity.py similarity index 100% rename from meta/lib/oeqa/sdk/buildtools-cases/sanity.py rename to meta/lib/oeqa/sdkbuildtools/cases/sanity.py diff --git a/meta/recipes-core/meta/buildtools-docs-tarball.bb b/meta/recipes-core/meta/buildtools-docs-tarball.bb index b9ef68eb6d..210c111a3c 100644 --- a/meta/recipes-core/meta/buildtools-docs-tarball.bb +++ b/meta/recipes-core/meta/buildtools-docs-tarball.bb @@ -16,4 +16,5 @@ TOOLCHAIN_OUTPUTNAME = "${SDK_ARCH}-buildtools-docs-nativesdk-standalone-${DISTR SDK_TITLE = "Docs Build tools tarball" -TESTSDK_CASES = "buildtools-docs-cases" +# Directory that contains testcases +TESTSDK_CASE_DIRS = "sdkbuildtools-docs" \ No newline at end of file diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb index 6fa6d93a3d..c5ddb39f5b 100644 --- a/meta/recipes-core/meta/buildtools-tarball.bb +++ b/meta/recipes-core/meta/buildtools-tarball.bb @@ -124,22 +124,7 @@ TOOLCHAIN_NEED_CONFIGSITE_CACHE = "" # The recipe doesn't need any default deps INHIBIT_DEFAULT_DEPS = "1" -# Directory in testsdk that contains testcases -TESTSDK_CASES = "buildtools-cases" +inherit testsdk -# We have our own code, avoid deferred inherit -SDK_CLASSES:remove = "testsdk" - -python do_testsdk() { - import oeqa.sdk.testsdk - testsdk = oeqa.sdk.testsdk.TestSDK() - - cases_path = os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__file__)), d.getVar("TESTSDK_CASES")) - testsdk.context_executor_class.default_cases = [cases_path,] - - testsdk.run(d) -} -addtask testsdk -do_testsdk[nostamp] = "1" -do_testsdk[network] = "1" -do_testsdk[depends] += "xz-native:do_populate_sysroot" +# Directory that contains testcases +TESTSDK_CASE_DIRS = "sdkbuildtools" \ No newline at end of file