| Message ID | 20260318134655.953233-1-JPEWhacker@gmail.com |
|---|---|
| Headers | show |
| Series | Add SPDX 3 Recipe Information | expand |
Mathieu, Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK Thank you, Joshua On Wed, Mar 18, 2026 at 7:46 AM Joshua Watt <jpewhacker@gmail.com> wrote: > > Changes the SPDX 3 output to include a "recipe" package that describe > static information available at parse time (without building). This is > primarily useful for gathering SPDX 3 VEX information about some or all > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > vex.bbclass. > > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for > helping work through this. > > V2: Fixes a bug where do_populate_sysroot was running when it should not > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is > incorrect (this is already handled by bitbake in the taskgraph, and > doesn't need to be manually removed). > > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular > dependency. meta-world-recipe-sbom also no longer runs in world builds, > as there's no reason to this. Finally, fixes a bug where > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx > (because do_unpack was not run). > > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX > information is linked to binary packages, or just recipes. Defaults to > "0" to significantly reduce the size of the SPDX output. > > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it > does funny things with its arch which prevents it from rebuilding SPDX > data properly, and no SPDX data is needed for it anyway > > V6: Fixes a bug where SPDX task would not correctly re-run when they > change, which would cause errors about missing SPDX document. Also > updates to the latest version of the SPDX bindings which improves > performance > > V7: Makes meta-world-recipe-sbom create the world SBoM when run with a > task (e.g. as part of do_build). Drops the removal of SPDX from > dummy-sdk-packages as these have been fixed to work properly. > > Joshua Watt (12): > spdx3: Add recipe SPDX data > spdx3: Add recipe SBoM task > spdx3: Add is-native property > spdx30: Include patch file information in VEX > spdx: De-duplicate CreationInfo > spdx_common: Check for dependent task in task flags > spdx30: Remove package VEX > spdx: Remove fatal errors for missing providers > spdx3: Use common variable for vardeps > glibc-testsuite: Do not generate SPDX > spdx: Remove do_collect_spdx_deps task > spdx: Update to latest bindings > > meta/classes-global/sstate.bbclass | 4 +- > .../create-spdx-image-3.0.bbclass | 4 +- > .../create-spdx-sdk-3.0.bbclass | 4 +- > meta/classes-recipe/kernel.bbclass | 2 +- > meta/classes-recipe/nospdx.bbclass | 2 +- > meta/classes/create-spdx-2.2.bbclass | 33 +- > meta/classes/create-spdx-3.0.bbclass | 81 +- > meta/classes/spdx-common.bbclass | 34 +- > meta/conf/distro/include/maintainers.inc | 1 + > meta/lib/oe/sbom30.py | 239 +- > meta/lib/oe/spdx30/__init__.py | 8 + > meta/lib/oe/spdx30/__main__.py | 12 + > meta/lib/oe/spdx30/cmd.py | 75 + > meta/lib/oe/{spdx30.py => spdx30/model.py} | 5935 ++++++++++------- > meta/lib/oe/spdx30/stub.pyi | 2544 +++++++ > meta/lib/oe/spdx30_tasks.py | 459 +- > meta/lib/oe/spdx_common.py | 78 +- > meta/lib/oeqa/selftest/cases/spdx.py | 28 +- > .../glibc/glibc-testsuite_2.43.bb | 1 + > .../meta/meta-world-recipe-sbom.bb | 38 + > scripts/contrib/make-spdx-bindings.sh | 3 +- > 21 files changed, 6837 insertions(+), 2748 deletions(-) > create mode 100644 meta/lib/oe/spdx30/__init__.py > create mode 100644 meta/lib/oe/spdx30/__main__.py > create mode 100644 meta/lib/oe/spdx30/cmd.py > rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%) > create mode 100644 meta/lib/oe/spdx30/stub.pyi > create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb > > -- > 2.53.0 >
On Wed Mar 18, 2026 at 2:49 PM CET, Joshua Watt wrote: > Mathieu, > > Here is my latest SPDX patch set (finally!). I'm sorry for the delay > and the mix up with Stefano's patches. I believe that you can now > stack Stefanos patches on top of mine and everything should be OK > > Thank you, > Joshua > Thanks! I did apply this series, but got some conflicts. I did try to solve them, but I might have solved them wrong. For reference, the result is here: https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3456 So with this, I've got a few issues. First an error appearing in various builds: ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405 https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 And still some selftest failures: 2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase) 2026-03-18 21:56:34,083 - oe-selftest - INFO - ... FAIL ... 2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) 2026-03-18 22:25:15,422 - oe-selftest - INFO - ... FAIL ... 2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) 2026-03-18 23:06:54,297 - oe-selftest - INFO - ... FAIL https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568 I will probably launch a build with this series alone a bit later today, to distinguish what comes from this series and what comes from the combination. Thank, Mathieu
On Thu Mar 19, 2026 at 8:07 AM CET, Mathieu Dubois-Briand wrote: > On Wed Mar 18, 2026 at 2:49 PM CET, Joshua Watt wrote: >> Mathieu, >> >> Here is my latest SPDX patch set (finally!). I'm sorry for the delay >> and the mix up with Stefano's patches. I believe that you can now >> stack Stefanos patches on top of mine and everything should be OK >> >> Thank you, >> Joshua >> > > Thanks! > > I did apply this series, but got some conflicts. I did try to solve > them, but I might have solved them wrong. > > For reference, the result is here: > > https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3456 > > So with this, I've got a few issues. First an error appearing in various > builds: > > ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 > > And still some selftest failures: > > 2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase) > 2026-03-18 21:56:34,083 - oe-selftest - INFO - ... FAIL > ... > 2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) > 2026-03-18 22:25:15,422 - oe-selftest - INFO - ... FAIL > ... > 2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) > 2026-03-18 23:06:54,297 - oe-selftest - INFO - ... FAIL > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568 > > I will probably launch a build with this series alone a bit later today, > to distinguish what comes from this series and what comes from the > combination. > > Thank, > Mathieu So I did launch some build with this series applied on top of master, and we basically have the same errors: https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3410 https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3389 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3467 The last one is still building right now, but we do see some errors already. Thanks, Mathieu
Richard,
These errors are caused because those recipes do:
PACKAGE_ARCH = "${SDK_ARCH}_${SDK_OS}"
Which is not in SSTATE_ARCHS, and therefore the required SPDX files
cannot be found. It appears this was done circa 2016: 1dbc6ec4ca
("buildtools/uninative-tarball: Fix deployment overlap issues") , but
I'm not sure it was correct before that.
IDK what the fix should be; Maybe SSTATE_PKGARCH should always be part
of SSTATE_ARCHS? Or maybe ${SDK_ARCH}_${SDK_OS} needs to be manually
added?
On Thu, Mar 19, 2026 at 1:07 AM Mathieu Dubois-Briand
<mathieu.dubois-briand@bootlin.com> wrote:
>
> On Wed Mar 18, 2026 at 2:49 PM CET, Joshua Watt wrote:
> > Mathieu,
> >
> > Here is my latest SPDX patch set (finally!). I'm sorry for the delay
> > and the mix up with Stefano's patches. I believe that you can now
> > stack Stefanos patches on top of mine and everything should be OK
> >
> > Thank you,
> > Joshua
> >
>
> Thanks!
>
> I did apply this series, but got some conflicts. I did try to solve
> them, but I might have solved them wrong.
>
> For reference, the result is here:
>
> https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3456
>
> So with this, I've got a few issues. First an error appearing in various
> builds:
>
> ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball
>
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
>
> And still some selftest failures:
>
> 2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase)
> 2026-03-18 21:56:34,083 - oe-selftest - INFO - ... FAIL
> ...
> 2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
> 2026-03-18 22:25:15,422 - oe-selftest - INFO - ... FAIL
> ...
> 2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
> 2026-03-18 23:06:54,297 - oe-selftest - INFO - ... FAIL
>
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568
>
> I will probably launch a build with this series alone a bit later today,
> to distinguish what comes from this series and what comes from the
> combination.
>
> Thank,
> Mathieu
>
> --
> Mathieu Dubois-Briand, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
Hi, On Thu, 2026-03-19 at 15:55 -0600, Joshua Watt wrote: > These errors are caused because those recipes do: > > PACKAGE_ARCH = "${SDK_ARCH}_${SDK_OS}" > > Which is not in SSTATE_ARCHS, and therefore the required SPDX files > cannot be found. It appears this was done circa 2016: 1dbc6ec4ca > ("buildtools/uninative-tarball: Fix deployment overlap issues") , but > I'm not sure it was correct before that. > > IDK what the fix should be; Maybe SSTATE_PKGARCH should always be > part of SSTATE_ARCHS? Or maybe ${SDK_ARCH}_${SDK_OS} needs to be > manually added? Sorry, this was my fault then, broken by https://git.openembedded.org/openembedded-core/commit/?id=834efe5eeaa7edae27d54a382ab864ef8f924b2d I think they need changing to ${SDK_ARCH}-${SDKPKGSUFFIX}. I'll try a patch. Cheers, Richard
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass. Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for helping work through this. V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed). V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run). V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output. V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance V7: Makes meta-world-recipe-sbom create the world SBoM when run with a task (e.g. as part of do_build). Drops the removal of SPDX from dummy-sdk-packages as these have been fixed to work properly. Joshua Watt (12): spdx3: Add recipe SPDX data spdx3: Add recipe SBoM task spdx3: Add is-native property spdx30: Include patch file information in VEX spdx: De-duplicate CreationInfo spdx_common: Check for dependent task in task flags spdx30: Remove package VEX spdx: Remove fatal errors for missing providers spdx3: Use common variable for vardeps glibc-testsuite: Do not generate SPDX spdx: Remove do_collect_spdx_deps task spdx: Update to latest bindings meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 2 +- meta/classes/create-spdx-2.2.bbclass | 33 +- meta/classes/create-spdx-3.0.bbclass | 81 +- meta/classes/spdx-common.bbclass | 34 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 239 +- meta/lib/oe/spdx30/__init__.py | 8 + meta/lib/oe/spdx30/__main__.py | 12 + meta/lib/oe/spdx30/cmd.py | 75 + meta/lib/oe/{spdx30.py => spdx30/model.py} | 5935 ++++++++++------- meta/lib/oe/spdx30/stub.pyi | 2544 +++++++ meta/lib/oe/spdx30_tasks.py | 459 +- meta/lib/oe/spdx_common.py | 78 +- meta/lib/oeqa/selftest/cases/spdx.py | 28 +- .../glibc/glibc-testsuite_2.43.bb | 1 + .../meta/meta-world-recipe-sbom.bb | 38 + scripts/contrib/make-spdx-bindings.sh | 3 +- 21 files changed, 6837 insertions(+), 2748 deletions(-) create mode 100644 meta/lib/oe/spdx30/__init__.py create mode 100644 meta/lib/oe/spdx30/__main__.py create mode 100644 meta/lib/oe/spdx30/cmd.py rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%) create mode 100644 meta/lib/oe/spdx30/stub.pyi create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb