diff mbox series

[master,v2] libsdl2: Add tests package and ptest support

Message ID 20250904000724.1575300-1-a-christidis@ti.com
State New
Headers show
Series [master,v2] libsdl2: Add tests package and ptest support | expand

Commit Message

Antonios Christidis Sept. 4, 2025, 12:07 a.m. UTC 
From: Antonios Christidis <a-christidis@ti.com>

The SDL2 software comes with its own set of tests. Following that
introduce support for ptest.

Signed-off-by: Antonios Christidis <a-christidis@ti.com>
---
v2:
- Updated run-ptest to be posix compliant 
- Use SDL_INSTALL_TESTS cmake config parameter

 meta/conf/distro/include/ptest-packagelists.inc |  1 +
 meta/recipes-graphics/libsdl2/libsdl2/run-ptest | 16 ++++++++++++++++
 meta/recipes-graphics/libsdl2/libsdl2_2.32.8.bb | 13 +++++++++++--
 3 files changed, 28 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-graphics/libsdl2/libsdl2/run-ptest

Comments

Mathieu Dubois-Briand Sept. 4, 2025, 8:41 a.m. UTC | #1 
On Thu Sep 4, 2025 at 2:07 AM CEST, a-christidis wrote:
> From: Antonios Christidis <a-christidis@ti.com>
>
> The SDL2 software comes with its own set of tests. Following that
> introduce support for ptest.
>
> Signed-off-by: Antonios Christidis <a-christidis@ti.com>
> ---

Hi Antonios,

Thanks for the new version.

It looks like we still have a QA issue:

ERROR: nativesdk-libsdl2-2.32.8-r0 do_package: QA Issue: nativesdk-libsdl2: Files/directories were installed but not shipped in any package:
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testsurround.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testerror.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testtimer.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testatomic.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testfilesystem.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testthread.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testaudioinfo.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testlocale.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testevdev.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testpower.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testqsort.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testplatform.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testautomation.test
  /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testver.test
Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.

https://autobuilder.yoctoproject.org/valkyrie/#/builders/19/builds/2327

Can you fix it please?

Thanks,
Mathieu
Joao Marcos Costa Sept. 4, 2025, 8:52 a.m. UTC | #2 
Hello,

I hope this email finds you well.

On 9/4/25 10:41, Mathieu Dubois-Briand via lists.openembedded.org wrote:
> On Thu Sep 4, 2025 at 2:07 AM CEST, a-christidis wrote:
>> From: Antonios Christidis <a-christidis@ti.com>
>>
>> The SDL2 software comes with its own set of tests. Following that
>> introduce support for ptest.
>>
>> Signed-off-by: Antonios Christidis <a-christidis@ti.com>
>> ---
> 
> Hi Antonios,
> 
> Thanks for the new version.
> 
> It looks like we still have a QA issue:
> 
> ERROR: nativesdk-libsdl2-2.32.8-r0 do_package: QA Issue: nativesdk-libsdl2: Files/directories were installed but not shipped in any package:
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testsurround.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testerror.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testtimer.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testatomic.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testfilesystem.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testthread.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testaudioinfo.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testlocale.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testevdev.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testpower.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testqsort.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testplatform.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testautomation.test
>    /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/share/installed-tests/SDL2/testver.test
> Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.
> 
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/19/builds/2327
> 
> Can you fix it please?
> 
> Thanks,
> Mathieu

Just in case, there are some ptest failures as well:

Failed ptests:
{'libsdl2': ['testdraw2', 'testgeometry', 'testsprite2', 
'testgles2', 'testoffscreen']}

https://autobuilder.yoctoproject.org/valkyrie/#/builders/73/builds/2220/steps/12/logs/stdio
Alexander Kanavin Sept. 4, 2025, 11:27 a.m. UTC | #3 
On Thu, 4 Sept 2025 at 02:07, <a-christidis@ti.com> wrote:
> +test_array="testdraw2 testgeometry testsprite2 testgles2 testoffscreen"
> +test_path="/usr/libexec/installed-tests/SDL2/"
> +
> +for test in ${test_array}
> +do
> +    timeout --preserve-status 10 "$test_path""$test"
> +    if [ $? -eq 0 ]; then
> +        echo "PASS: $test"
> +    else
> +        echo "FAIL: $test"
> +    fi
> +done

Hardcoding a list of tests is prone to regressions and incomplete coverage.

I applied the patch and checked what actually gets installed. SDL
conforms to gnome's installed tests spec, which means you can inherit
ptest-gnome, and run-ptest should be as simple as:

#! /bin/sh

gnome-desktop-testing-runner SDL2

There are examples in oe-core if you need them for reference.
Obviously you should run it too and check that it passes.

meta-oe has a recipe for libsdl3, whether you want to forward-port the
test support to that recipe is at your discretion, but at some point
libsdl2 will be obsoleted and replaced by that in core.

Alex
Antonios Christidis Sept. 12, 2025, 8:54 p.m. UTC | #4 
Dear Alexander,


I ran the tests using the gnome-desktop-testing-runner. Here is what I 
found:

libsdl2 has 70 tests. Currently, following the gnome standard and being 
packaged under /usr/libexec/installed-tests .

18 out of the 70 tests are executed with the 
gnome-desktop-testing-runner. The standard only covers tests that run 
for a finite amount of time, not infinitely, and requires ".test" files 
under /usr/share.

However, when running the 18 tests on qemux86-64 qemuarm, the following 
tests fail:

testaudioinfo

testautomation

testlocale

testsurround

Most of the test failures are due to the audio stack, which I do not 
have experience with in order to debug.

Here's my questions, what should I do here ?

It is acceptable to create a oe-core only patch for libsdl2, that would 
block the generation of the ".test" files for the above tests, ".test" 
files are what dictate which tests are run?

Or, I can take a look into updating the version of gnome-desktop-testing 
? More recent versions of the software, have a CLI option to exclude 
tests. The current packages version within oe-core does not.

I'm leaning on the 2nd option, but want to get your opinion on this 
since you have more experience on ptest and related topics.

Kind Regards,

Antonios

On 9/4/25 6:27 AM, Alexander Kanavin wrote:
> On Thu, 4 Sept 2025 at 02: 07, <a-christidis@ ti. com> wrote: > 
> +test_array="testdraw2 testgeometry testsprite2 testgles2 
> testoffscreen" > +test_path="/usr/libexec/installed-tests/SDL2/" > + > 
> +for test in ${test_array} > +do
> ZjQcmQRYFpfptBannerStart
> This message was sent from outside of Texas Instruments.
> Do not click links or open attachments unless you recognize the source 
> of this email and know the content is safe.
> Report Suspicious
> <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/G3vK!tDdkczhAu0BQaazEFn6sssEtcdxUhEvtTan3pBnTy9YsgLjvWwOA7YJejpfF1TdW-I8bxkDedD2ZVkPZUvwfaMUJ9SDMz1bIjJfXyt52mx6YGg_dzNkYWXvJTDg$> 
>
> ZjQcmQRYFpfptBannerEnd
> On Thu, 4 Sept 2025 at 02:07, <a-christidis@ti.com> wrote:
> > +test_array="testdraw2 testgeometry testsprite2 testgles2 testoffscreen"
> > +test_path="/usr/libexec/installed-tests/SDL2/"
> > +
> > +for test in ${test_array}
> > +do
> > +    timeout --preserve-status 10 "$test_path""$test"
> > +    if [ $? -eq 0 ]; then
> > +        echo "PASS: $test"
> > +    else
> > +        echo "FAIL: $test"
> > +    fi
> > +done
>
> Hardcoding a list of tests is prone to regressions and incomplete coverage.
>
> I applied the patch and checked what actually gets installed. SDL
> conforms to gnome's installed tests spec, which means you can inherit
> ptest-gnome, and run-ptest should be as simple as:
>
> #! /bin/sh
>
> gnome-desktop-testing-runner SDL2
>
> There are examples in oe-core if you need them for reference.
> Obviously you should run it too and check that it passes.
>
> meta-oe has a recipe for libsdl3, whether you want to forward-port the
> test support to that recipe is at your discretion, but at some point
> libsdl2 will be obsoleted and replaced by that in core.
>
> Alex
Alexander Kanavin Sept. 16, 2025, 9:45 a.m. UTC | #5 
On Fri, 12 Sept 2025 at 22:54, Antonios Christidis <a-christidis@ti.com> wrote:
> However, when running the 18 tests on qemux86-64 qemuarm, the following
> tests fail:
>
> testaudioinfo
>
> testautomation
>
> testlocale
>
> testsurround
>
> Most of the test failures are due to the audio stack, which I do not
> have experience with in order to debug.
>
> Here's my questions, what should I do here ?
>
> It is acceptable to create a oe-core only patch for libsdl2, that would
> block the generation of the ".test" files for the above tests, ".test"
> files are what dictate which tests are run?
>
> Or, I can take a look into updating the version of gnome-desktop-testing
> ? More recent versions of the software, have a CLI option to exclude
> tests. The current packages version within oe-core does not.
>
> I'm leaning on the 2nd option, but want to get your opinion on this
> since you have more experience on ptest and related topics.

If these tests are failing, then it is not right to just go and
disable them without investigating why.

Maybe the failures are due to real bugs, maybe something is
misconfigured, or a test dependency is missing, we should try and get
to the bottom of it. Otherwise what's the point of running tests in
the first place, if any test failure results in adding the test to the
skip list?

Basically, show what the failures are, provide the patch and the steps
to reproduce them, and we'll try to find a fix. I've fixed many a
ptest over the years, and have a bag of tricks in my head.

Alex
diff mbox series

Patch

diff --git a/meta/conf/distro/include/ptest-packagelists.inc b/meta/conf/distro/include/ptest-packagelists.inc
index 9a7b25a916..a5476927b5 100644
--- a/meta/conf/distro/include/ptest-packagelists.inc
+++ b/meta/conf/distro/include/ptest-packagelists.inc
@@ -32,6 +32,7 @@  PTESTS_FAST = "\
     libgpg-error\
     libnl \
     libpcre \
+    libsdl2 \
     libssh2 \
     libtest-fatal-perl \
     libtest-needs-perl \
diff --git a/meta/recipes-graphics/libsdl2/libsdl2/run-ptest b/meta/recipes-graphics/libsdl2/libsdl2/run-ptest
new file mode 100644
index 0000000000..acdfa951e9
--- /dev/null
+++ b/meta/recipes-graphics/libsdl2/libsdl2/run-ptest
@@ -0,0 +1,16 @@ 
+#!/bin/sh
+
+test_array="testdraw2 testgeometry testsprite2 testgles2 testoffscreen"
+
+
+test_path="/usr/libexec/installed-tests/SDL2/"
+
+for test in ${test_array}
+do
+    timeout --preserve-status 10 "$test_path""$test"
+    if [ $? -eq 0 ]; then
+        echo "PASS: $test"
+    else
+        echo "FAIL: $test"
+    fi
+done
diff --git a/meta/recipes-graphics/libsdl2/libsdl2_2.32.8.bb b/meta/recipes-graphics/libsdl2/libsdl2_2.32.8.bb
index 98291e0f80..9134a408c6 100644
--- a/meta/recipes-graphics/libsdl2/libsdl2_2.32.8.bb
+++ b/meta/recipes-graphics/libsdl2/libsdl2_2.32.8.bb
@@ -21,13 +21,15 @@  LIC_FILES_CHKSUM:append = " ${@bb.utils.contains('PACKAGECONFIG', 'arm-neon', 'f
 
 PROVIDES = "virtual/libsdl2"
 
-SRC_URI = "https://www.libsdl.org/release/SDL2-${PV}.tar.gz"
+SRC_URI = "https://www.libsdl.org/release/SDL2-${PV}.tar.gz \
+           file://run-ptest \
+	   "
 
 S = "${UNPACKDIR}/SDL2-${PV}"
 
 SRC_URI[sha256sum] = "0ca83e9c9b31e18288c7ec811108e58bac1f1bb5ec6577ad386830eac51c787e"
 
-inherit cmake lib_package binconfig-disabled pkgconfig upstream-version-is-even
+inherit cmake lib_package binconfig-disabled pkgconfig upstream-version-is-even ptest
 UPSTREAM_CHECK_REGEX = "SDL2-(?P<pver>\d+\.(\d*[02468])+(\.\d+)+)\.tar"
 
 BINCONFIG = "${bindir}/sdl2-config"
@@ -48,6 +50,8 @@  EXTRA_OECMAKE = "-DSDL_OSS=OFF -DSDL_ESD=OFF -DSDL_ARTS=OFF \
                  -DSDL_X11_XRANDR=OFF \
                  -DSDL_X11_XSCRNSAVER=OFF \
                  -DSDL_X11_XSHAPE=OFF \
+                 -DSDL_TESTS=ON \
+                 -DSDL_INSTALL_TESTS=ON \
 "
 
 # opengl packageconfig factored out to make it easy for distros
@@ -82,6 +86,11 @@  PACKAGECONFIG[x11]        = "-DSDL_X11=ON,-DSDL_X11=OFF,virtual/libx11 libxext l
 
 CFLAGS:append:class-native = " -DNO_SHARED_MEMORY"
 
+PACKAGE_BEFORE_PN = "${PN}-tests"
+
 FILES:${PN} += "${datadir}/licenses/SDL2/LICENSE.txt"
+FILES:${PN}-tests += "${libexecdir} /usr/share/installed-tests/SDL2"
+
+RDEPENDS:${PN}-ptest = "${PN}-tests"
 
 BBCLASSEXTEND = "native nativesdk"