mbox series

[v6,00/15] Add SPDX 3 Recipe Information

Message ID 20260310184058.533343-1-JPEWhacker@gmail.com
Headers show
Series Add SPDX 3 Recipe Information | expand

Message

Joshua Watt March 10, 2026, 6:38 p.m. UTC 
Changes the SPDX 3 output to include a "recipe" package that describe
static information available at parse time (without building). This is
primarily useful for gathering SPDX 3 VEX information about some or all
recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and
vex.bbclass.

Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for
helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not
be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is
incorrect (this is already handled by bitbake in the taskgraph, and
doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular
dependency. meta-world-recipe-sbom also no longer runs in world builds,
as there's no reason to this. Finally, fixes a bug where
NO_GENERIC_LICENSE files would fail to be found in do_create_spdx
(because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX
information is linked to binary packages, or just recipes. Defaults to
"0" to significantly reduce the size of the SPDX output.

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it
does funny things with its arch which prevents it from rebuilding SPDX
data properly, and no SPDX data is needed for it anyway

V6: Fixes a bug where SPDX task would not correctly re-run when they
change, which would cause errors about missing SPDX document. Also
updates to the latest version of the SPDX bindings which improves
performance

Joshua Watt (15):
  llvm-project-source: Use allarch.bbclass
  gcc-source: Use allarch.bbclass
  spdx3: Add recipe SPDX data
  spdx3: Add recipe SBoM task
  spdx3: Add is-native property
  spdx30: Include patch file information in VEX
  spdx: De-duplicate CreationInfo
  spdx_common: Check for dependent task in task flags
  spdx30: Skip install package CVE information
  dummy-sdk-package: Disable SPDX
  spdx: Remove fatal errors for missing providers
  spdx3: Use common variable for vardeps
  glibc-testsuite: Do not generate SPDX
  spdx: Remove do_collect_spdx_deps task
  spdx: Update to latest bindings

 meta/classes-global/sstate.bbclass            |    4 +-
 .../create-spdx-image-3.0.bbclass             |    4 +-
 .../create-spdx-sdk-3.0.bbclass               |    4 +-
 meta/classes-recipe/kernel.bbclass            |    2 +-
 meta/classes-recipe/nospdx.bbclass            |    2 +-
 meta/classes/create-spdx-2.2.bbclass          |   33 +-
 meta/classes/create-spdx-3.0.bbclass          |   92 +-
 meta/classes/spdx-common.bbclass              |   34 +-
 meta/conf/distro/include/maintainers.inc      |    1 +
 meta/lib/oe/sbom30.py                         |  239 +-
 meta/lib/oe/spdx30/__init__.py                |    8 +
 meta/lib/oe/spdx30/__main__.py                |   12 +
 meta/lib/oe/spdx30/cmd.py                     |   75 +
 meta/lib/oe/{spdx30.py => spdx30/model.py}    | 5935 ++++++++++-------
 meta/lib/oe/spdx30/stub.pyi                   | 2544 +++++++
 meta/lib/oe/spdx30_tasks.py                   |  512 +-
 meta/lib/oe/spdx_common.py                    |   78 +-
 meta/lib/oeqa/selftest/cases/spdx.py          |   41 +-
 .../glibc/glibc-testsuite_2.42.bb             |    1 +
 meta/recipes-core/meta/dummy-sdk-package.inc  |    1 +
 .../meta/meta-world-recipe-sbom.bb            |   29 +
 .../clang/llvm-project-source.inc             |    8 +-
 meta/recipes-devtools/gcc/gcc-source.inc      |   16 +-
 scripts/contrib/make-spdx-bindings.sh         |    3 +-
 24 files changed, 6922 insertions(+), 2756 deletions(-)
 create mode 100644 meta/lib/oe/spdx30/__init__.py
 create mode 100644 meta/lib/oe/spdx30/__main__.py
 create mode 100644 meta/lib/oe/spdx30/cmd.py
 rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%)
 create mode 100644 meta/lib/oe/spdx30/stub.pyi
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb

Comments

Mathieu Dubois-Briand March 11, 2026, 1:55 p.m. UTC | #1 
On Tue Mar 10, 2026 at 7:38 PM CET, Joshua Watt via lists.openembedded.org wrote:
> Changes the SPDX 3 output to include a "recipe" package that describe
> static information available at parse time (without building). This is
> primarily useful for gathering SPDX 3 VEX information about some or all
> recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and
> vex.bbclass.
>
> Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for
> helping work through this.
>
> V2: Fixes a bug where do_populate_sysroot was running when it should not
> be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is
> incorrect (this is already handled by bitbake in the taskgraph, and
> doesn't need to be manually removed).
>
> V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular
> dependency. meta-world-recipe-sbom also no longer runs in world builds,
> as there's no reason to this. Finally, fixes a bug where
> NO_GENERIC_LICENSE files would fail to be found in do_create_spdx
> (because do_unpack was not run).
>
> V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX
> information is linked to binary packages, or just recipes. Defaults to
> "0" to significantly reduce the size of the SPDX output.
>
> V5: Fixes dummy-sdk-packages to not generate SPDX output, since it
> does funny things with its arch which prevents it from rebuilding SPDX
> data properly, and no SPDX data is needed for it anyway
>
> V6: Fixes a bug where SPDX task would not correctly re-run when they
> change, which would cause errors about missing SPDX document. Also
> updates to the latest version of the SPDX bindings which improves
> performance
>

Hi Joshua,

Ok, we are almost there!

I suspect it would work fine on master, but we have a fail on two tests
that were recently added by Stefano, and were not merged so far.

As both series might still evolve or get reviews, I will probably keep
both in my branch, but some changes are needed if we want to merge both
series.

2026-03-11 11:31:27,495 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
2026-03-11 11:31:27,495 - oe-selftest - INFO -  ... FAIL
...
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 451, in test_download_location_defensive_handling
    objset = self.check_recipe_spdx(
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
    return self.check_spdx_file(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
    self.assertExists(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-m4.spdx.json' does not exist
...
2026-03-11 12:39:25,602 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
2026-03-11 12:39:25,603 - oe-selftest - INFO -  ... FAIL
...
2026-03-11 12:39:25,611 - oe-selftest - INFO - 6: 45/55 656/681 (14.27s) (2 failed) (spdx.SPDX30Check.test_version_extraction_patterns)
2026-03-11 12:39:25,611 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 479, in test_version_extraction_patterns
    objset = self.check_recipe_spdx(
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
    return self.check_spdx_file(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
    self.assertExists(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-tar.spdx.json' does not exist

https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3499
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3380
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3270

For reference, this oe-core branch was used during the build:
https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3385

Thanks,
Mathieu
Joshua Watt March 11, 2026, 4:39 p.m. UTC | #2 
On Wed, Mar 11, 2026 at 7:55 AM Mathieu Dubois-Briand
<mathieu.dubois-briand@bootlin.com> wrote:
>
> On Tue Mar 10, 2026 at 7:38 PM CET, Joshua Watt via lists.openembedded.org wrote:
> > Changes the SPDX 3 output to include a "recipe" package that describe
> > static information available at parse time (without building). This is
> > primarily useful for gathering SPDX 3 VEX information about some or all
> > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and
> > vex.bbclass.
> >
> > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for
> > helping work through this.
> >
> > V2: Fixes a bug where do_populate_sysroot was running when it should not
> > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is
> > incorrect (this is already handled by bitbake in the taskgraph, and
> > doesn't need to be manually removed).
> >
> > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular
> > dependency. meta-world-recipe-sbom also no longer runs in world builds,
> > as there's no reason to this. Finally, fixes a bug where
> > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx
> > (because do_unpack was not run).
> >
> > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX
> > information is linked to binary packages, or just recipes. Defaults to
> > "0" to significantly reduce the size of the SPDX output.
> >
> > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it
> > does funny things with its arch which prevents it from rebuilding SPDX
> > data properly, and no SPDX data is needed for it anyway
> >
> > V6: Fixes a bug where SPDX task would not correctly re-run when they
> > change, which would cause errors about missing SPDX document. Also
> > updates to the latest version of the SPDX bindings which improves
> > performance
> >
>
> Hi Joshua,
>
> Ok, we are almost there!
>
> I suspect it would work fine on master, but we have a fail on two tests
> that were recently added by Stefano, and were not merged so far.
>
> As both series might still evolve or get reviews, I will probably keep
> both in my branch, but some changes are needed if we want to merge both
> series.

This is actually semi-intentional. I renamed the "recipe-" SPDX files
to "build-". The fix is simple, but it either needs to be applied to
my changes or Stephanos, depending on the order. Do you have a
preference?

>
> 2026-03-11 11:31:27,495 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
> 2026-03-11 11:31:27,495 - oe-selftest - INFO -  ... FAIL
> ...
>   File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 451, in test_download_location_defensive_handling
>     objset = self.check_recipe_spdx(
>   File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
>     return self.check_spdx_file(filename)
>   File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
>     self.assertExists(filename)
>   File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
>     raise self.failureException(msg)
> AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-m4.spdx.json' does not exist
> ...
> 2026-03-11 12:39:25,602 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
> 2026-03-11 12:39:25,603 - oe-selftest - INFO -  ... FAIL
> ...
> 2026-03-11 12:39:25,611 - oe-selftest - INFO - 6: 45/55 656/681 (14.27s) (2 failed) (spdx.SPDX30Check.test_version_extraction_patterns)
> 2026-03-11 12:39:25,611 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
>   File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 479, in test_version_extraction_patterns
>     objset = self.check_recipe_spdx(
>   File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
>     return self.check_spdx_file(filename)
>   File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
>     self.assertExists(filename)
>   File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
>     raise self.failureException(msg)
> AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-tar.spdx.json' does not exist
>
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3499
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3380
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3270
>
> For reference, this oe-core branch was used during the build:
> https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3385
>
> Thanks,
> Mathieu
>
> --
> Mathieu Dubois-Briand, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
Mathieu Dubois-Briand March 11, 2026, 7:33 p.m. UTC | #3 
On Wed Mar 11, 2026 at 5:39 PM CET, Joshua Watt wrote:
> On Wed, Mar 11, 2026 at 7:55 AM Mathieu Dubois-Briand
> <mathieu.dubois-briand@bootlin.com> wrote:
>>
>> On Tue Mar 10, 2026 at 7:38 PM CET, Joshua Watt via lists.openembedded.org wrote:
>> > Changes the SPDX 3 output to include a "recipe" package that describe
>> > static information available at parse time (without building). This is
>> > primarily useful for gathering SPDX 3 VEX information about some or all
>> > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and
>> > vex.bbclass.
>> >
>> > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for
>> > helping work through this.
>> >
>> > V2: Fixes a bug where do_populate_sysroot was running when it should not
>> > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is
>> > incorrect (this is already handled by bitbake in the taskgraph, and
>> > doesn't need to be manually removed).
>> >
>> > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular
>> > dependency. meta-world-recipe-sbom also no longer runs in world builds,
>> > as there's no reason to this. Finally, fixes a bug where
>> > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx
>> > (because do_unpack was not run).
>> >
>> > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX
>> > information is linked to binary packages, or just recipes. Defaults to
>> > "0" to significantly reduce the size of the SPDX output.
>> >
>> > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it
>> > does funny things with its arch which prevents it from rebuilding SPDX
>> > data properly, and no SPDX data is needed for it anyway
>> >
>> > V6: Fixes a bug where SPDX task would not correctly re-run when they
>> > change, which would cause errors about missing SPDX document. Also
>> > updates to the latest version of the SPDX bindings which improves
>> > performance
>> >
>>
>> Hi Joshua,
>>
>> Ok, we are almost there!
>>
>> I suspect it would work fine on master, but we have a fail on two tests
>> that were recently added by Stefano, and were not merged so far.
>>
>> As both series might still evolve or get reviews, I will probably keep
>> both in my branch, but some changes are needed if we want to merge both
>> series.
>
> This is actually semi-intentional. I renamed the "recipe-" SPDX files
> to "build-". The fix is simple, but it either needs to be applied to
> my changes or Stephanos, depending on the order. Do you have a
> preference?
>

I do not have any particular preference, so please do as is easier for
you.
Joshua Watt March 11, 2026, 10:56 p.m. UTC | #4 
On Wed, Mar 11, 2026 at 1:33 PM Mathieu Dubois-Briand
<mathieu.dubois-briand@bootlin.com> wrote:
>
> On Wed Mar 11, 2026 at 5:39 PM CET, Joshua Watt wrote:
> > On Wed, Mar 11, 2026 at 7:55 AM Mathieu Dubois-Briand
> > <mathieu.dubois-briand@bootlin.com> wrote:
> >>
> >> On Tue Mar 10, 2026 at 7:38 PM CET, Joshua Watt via lists.openembedded.org wrote:
> >> > Changes the SPDX 3 output to include a "recipe" package that describe
> >> > static information available at parse time (without building). This is
> >> > primarily useful for gathering SPDX 3 VEX information about some or all
> >> > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and
> >> > vex.bbclass.
> >> >
> >> > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for
> >> > helping work through this.
> >> >
> >> > V2: Fixes a bug where do_populate_sysroot was running when it should not
> >> > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is
> >> > incorrect (this is already handled by bitbake in the taskgraph, and
> >> > doesn't need to be manually removed).
> >> >
> >> > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular
> >> > dependency. meta-world-recipe-sbom also no longer runs in world builds,
> >> > as there's no reason to this. Finally, fixes a bug where
> >> > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx
> >> > (because do_unpack was not run).
> >> >
> >> > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX
> >> > information is linked to binary packages, or just recipes. Defaults to
> >> > "0" to significantly reduce the size of the SPDX output.
> >> >
> >> > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it
> >> > does funny things with its arch which prevents it from rebuilding SPDX
> >> > data properly, and no SPDX data is needed for it anyway
> >> >
> >> > V6: Fixes a bug where SPDX task would not correctly re-run when they
> >> > change, which would cause errors about missing SPDX document. Also
> >> > updates to the latest version of the SPDX bindings which improves
> >> > performance
> >> >
> >>
> >> Hi Joshua,
> >>
> >> Ok, we are almost there!
> >>
> >> I suspect it would work fine on master, but we have a fail on two tests
> >> that were recently added by Stefano, and were not merged so far.
> >>
> >> As both series might still evolve or get reviews, I will probably keep
> >> both in my branch, but some changes are needed if we want to merge both
> >> series.
> >
> > This is actually semi-intentional. I renamed the "recipe-" SPDX files
> > to "build-". The fix is simple, but it either needs to be applied to
> > my changes or Stephanos, depending on the order. Do you have a
> > preference?
> >
>
> I do not have any particular preference, so please do as is easier for
> you.

Stefano has some feedback to address, so please take my patch series
first and Stefano can fix it in the next revision.

Thanks

>
> --
> Mathieu Dubois-Briand, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>