diff mbox series

[v9,5/7] oeqa/selftest: Add tests for source download enrichment

Message ID 20260312153845.164369-6-stondo@gmail.com
State Under Review
Headers show
Series SPDX 3.0 SBOM enrichment and compliance improvements | expand

Commit Message

Stefano Tondo March 12, 2026, 3:38 p.m. UTC 
From: Stefano Tondo <stefano.tondo.ext@siemens.com>

Add two new SPDX 3.0 selftest cases:

test_download_location_defensive_handling:
  Verifies SPDX generation succeeds for recipes with tarball sources
  and that external references are properly structured (ExternalRef
  locator is a list of strings per SPDX 3.0 spec).

test_version_extraction_patterns:
  Verifies that version extraction works correctly and all source
  packages have proper version strings containing digits.

These tests validate the source download enrichment added in the
previous commit.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/lib/oeqa/selftest/cases/spdx.py | 71 +++++++++++++++++++++++++++-
 1 file changed, 70 insertions(+), 1 deletion(-)

Comments

Mathieu Dubois-Briand March 13, 2026, 6:14 a.m. UTC | #1 
On Thu Mar 12, 2026 at 4:38 PM CET, Stefano Tondo via lists.openembedded.org wrote:
> From: Stefano Tondo <stefano.tondo.ext@siemens.com>
>
> Add two new SPDX 3.0 selftest cases:
>
> test_download_location_defensive_handling:
>   Verifies SPDX generation succeeds for recipes with tarball sources
>   and that external references are properly structured (ExternalRef
>   locator is a list of strings per SPDX 3.0 spec).
>
> test_version_extraction_patterns:
>   Verifies that version extraction works correctly and all source
>   packages have proper version strings containing digits.
>
> These tests validate the source download enrichment added in the
> previous commit.
>
> Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
> ---

Hi Stefano,

Thanks for the new version. Builds look correct so far, except for
these 3 selftest errors:

2026-03-12 22:29:04,908 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
2026-03-12 22:29:04,909 - oe-selftest - INFO -  ... FAIL
...
2026-03-12 22:29:04,910 - oe-selftest - INFO - 6: 39/53 444/679 (18.85s) (0 failed) (spdx.SPDX30Check.test_download_location_defensive_handling)
2026-03-12 22:29:04,911 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 424, in test_download_location_defensive_handling
    objset = self.check_recipe_spdx(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
    return self.check_spdx_file(filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
    self.assertExists(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-3170645/tmp/deploy/spdx/3.0.1/cortexa57/recipes/build-m4.spdx.json' does not exist
...
2026-03-12 23:32:02,849 - oe-selftest - INFO - spdx.SPDX30Check.test_packageconfig_spdx (subunit.RemotedTestCase)
2026-03-12 23:32:02,849 - oe-selftest - INFO -  ... FAIL
...
2026-03-12 23:32:02,850 - oe-selftest - INFO - 6: 43/53 634/679 (70.33s) (2 failed) (spdx.SPDX30Check.test_packageconfig_spdx)
2026-03-12 23:32:02,850 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 393, in test_packageconfig_spdx
    objset = self.check_recipe_spdx(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
    return self.check_spdx_file(filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
    self.assertExists(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-3170645/tmp/deploy/spdx/3.0.1/cortexa57/recipes/build-tar.spdx.json' does not exist
...
2026-03-12 23:32:16,627 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
2026-03-12 23:32:16,628 - oe-selftest - INFO -  ... FAIL
...
2026-03-12 23:32:16,628 - oe-selftest - INFO - 6: 44/53 635/679 (13.78s) (4 failed) (spdx.SPDX30Check.test_version_extraction_patterns)
2026-03-12 23:32:16,628 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 452, in test_version_extraction_patterns
    objset = self.check_recipe_spdx(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
    return self.check_spdx_file(filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
    self.assertExists(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-3170645/tmp/deploy/spdx/3.0.1/cortexa57/recipes/build-tar.spdx.json' does not exist

https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3513
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3395
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3286

Looking at the error, I suspect this is to address changes from the
Joshua series, but I didn't had this series in my branch. Is that right?

I will keep these changes in my branch, so we can go further, but please
confirm everything is correct.

Thanks,
Mathieu
Tondo, Stefano March 13, 2026, 8:30 a.m. UTC | #2 
Hi Mathieu,

Yes, that's correct, the build-m4/build-tar naming depends on
Joshua's series which renames recipe-* to build-* in the SPDX
output. My series is intended to be applied on top of his.

Once his series lands, these tests should pass.

Thanks,
Stefano
diff mbox series

Patch

diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py
index 41ef52fce1..859667dd6b 100644
--- a/meta/lib/oeqa/selftest/cases/spdx.py
+++ b/meta/lib/oeqa/selftest/cases/spdx.py
@@ -392,7 +392,7 @@  class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     def test_packageconfig_spdx(self):
         objset = self.check_recipe_spdx(
             "tar",
-            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/build-tar.spdx.json",
             extraconf="""\
                 SPDX_INCLUDE_PACKAGECONFIG = "1"
                 """,
@@ -414,3 +414,72 @@  class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
                 value, ["enabled", "disabled"],
                 f"Unexpected PACKAGECONFIG value '{value}' for {key}"
             )
+
+    def test_download_location_defensive_handling(self):
+        """Test that download_location handling is defensive.
+
+        Verifies SPDX generation succeeds and external references are
+        properly structured when download_location retrieval works.
+        """
+        objset = self.check_recipe_spdx(
+            "m4",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/build-m4.spdx.json",
+        )
+
+        found_external_refs = False
+        for pkg in objset.foreach_type(oe.spdx30.software_Package):
+            if pkg.externalRef:
+                found_external_refs = True
+                for ref in pkg.externalRef:
+                    self.assertIsNotNone(ref.externalRefType)
+                    self.assertIsNotNone(ref.locator)
+                    self.assertGreater(len(ref.locator), 0, "Locator should have at least one entry")
+                    for loc in ref.locator:
+                        self.assertIsInstance(loc, str)
+                break
+
+        self.logger.info(
+            f"External references {'found' if found_external_refs else 'not found'} "
+            f"in SPDX output (defensive handling verified)"
+        )
+
+    def test_version_extraction_patterns(self):
+        """Test that version extraction works for various package formats.
+
+        Verifies that version patterns correctly extract versions from
+        tarball sources and that all packages have proper version strings.
+        """
+        objset = self.check_recipe_spdx(
+            "tar",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/build-tar.spdx.json",
+        )
+
+        # Collect all packages with versions
+        packages_with_versions = []
+        for pkg in objset.foreach_type(oe.spdx30.software_Package):
+            if pkg.software_packageVersion:
+                packages_with_versions.append((pkg.name, pkg.software_packageVersion))
+
+        self.assertGreater(
+            len(packages_with_versions), 0,
+            "Should find packages with extracted versions"
+        )
+
+        self.logger.info(f"Found {len(packages_with_versions)} packages with versions")
+
+        # Log some examples for debugging
+        for name, version in packages_with_versions[:5]:
+            self.logger.info(f"  {name}: {version}")
+
+        # Verify that versions follow expected patterns
+        for name, version in packages_with_versions:
+            # Version should not be empty
+            self.assertIsNotNone(version)
+            self.assertNotEqual(version, "")
+
+            # Version should contain digits
+            self.assertRegex(
+                version,
+                r'\d',
+                f"Version '{version}' for package '{name}' should contain digits"
+            )