diff mbox series

[V2,2/2] core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE

Message ID 20251205083202.330982-2-Qi.Chen@windriver.com
State New
Headers show
Series [V2,1/2] oeqa/runtime/cases/ptest.py: use to_boolean for PTEST_EXPECT_FAILURE | expand

Commit Message

ChenQi Dec. 5, 2025, 8:32 a.m. UTC 
From: Chen Qi <Qi.Chen@windriver.com>

We need to allow users to easily override this value in local.conf.
This has the benefit that 'bitbake core-image-ptest-xxx:do_testimage'
fails when expected.

As an example, I used 'bitbake core-image-ptest-util-linux:do_testimage'.
It succeeded with warning message. I didn't notice the warning message.
I saw the command succeeded and I tought util-linux ptest is OK. But
in actual fact, the ptest failed. It's the PTEST_EXPECT_FAILURE setting
in this core-image-ptest.bb that is not giving me error.

With this change, I can put PTEST_EXPECT_FAILURE = "0" in my local.conf,
and I can easily see if 'core-image-ptest-xxx:do_testimage' succeeds or
not.

The PTEST_EXPECT_FAILURE change should result in testdata regenerated
because ptest.py test case's behavior is affected by it.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/images/core-image-ptest.bb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Comments

Alexander Kanavin Dec. 5, 2025, 9:43 a.m. UTC | #1 
On Fri, 5 Dec 2025 at 09:32, Chen Qi via lists.openembedded.org
<Qi.Chen=windriver.com@lists.openembedded.org> wrote:
>  # Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
> -PTEST_EXPECT_FAILURE = "1"
> +PTEST_EXPECT_FAILURE ?= "1"

I'd say let's rather drop this assignment and the comment. This was
the case years ago when ptests were indeed flaky, but they've come a
long way since. And even though because of this assignment ptest fails
become AB warnings, de facto such warnings are still treated as fails
that block the patch that caused them.

You'd also need to tweak the regexp in
yocto-autobuilder2/steps/observer.py that Mathieu has recently
introduced.

Alex
Alexander Kanavin Dec. 5, 2025, 10:33 a.m. UTC | #2 
On Fri, 5 Dec 2025 at 10:43, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
> You'd also need to tweak the regexp in
> yocto-autobuilder2/steps/observer.py that Mathieu has recently
> introduced.

Actually this is not needed. The warning the regexp matches on is
still printed, the error is a separate print.

Alex
diff mbox series

Patch

diff --git a/meta/recipes-core/images/core-image-ptest.bb b/meta/recipes-core/images/core-image-ptest.bb
index 017f05f81b..b554964d20 100644
--- a/meta/recipes-core/images/core-image-ptest.bb
+++ b/meta/recipes-core/images/core-image-ptest.bb
@@ -42,7 +42,8 @@  QB_MEM:virtclass-mcextend-tcl = "-m 5100"
 TEST_SUITES = "ping ssh parselogs ptest"
 
 # Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
-PTEST_EXPECT_FAILURE = "1"
+PTEST_EXPECT_FAILURE ?= "1"
+write_image_test_data[vardeps] += "PTEST_EXPECT_FAILURE"
 
 python () {
     if not d.getVar("MCNAME"):