| Message ID | 20240703140059.4096394-1-JPEWhacker@gmail.com |
|---|---|
| Headers | show |
| Series | Add SPDX 3.0 support | expand |
On Wed, 2024-07-03 at 07:59 -0600, Joshua Watt via lists.openembedded.org wrote: > This patch series add support for SPDX 3.0 and sets it as the > default. > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > the same time > > v2: Added tests and addressed feedback > v3: Fixed several oe-selftest and build failures > v4: Fixed silly typo mistake in staging.bbclass > v5: Reworked to make SPDX 3 output reproducible by default. Variables > that introduce non-reproducible output are documented as such. Thanks for working on this. The patches generally seem to working well in testing but this does look related: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio Particularly these bits: AssertionError: 10 != 0 : Missing objects in the cache: /srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst Cheers, Richard
On Fri, 2024-07-05 at 08:17 +0100, Richard Purdie via lists.openembedded.org wrote: > > On Wed, 2024-07-03 at 07:59 -0600, Joshua Watt via > > lists.openembedded.org wrote: > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > default. > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > > > > the same time > > > > > > > > v2: Added tests and addressed feedback > > > > v3: Fixed several oe-selftest and build failures > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > v5: Reworked to make SPDX 3 output reproducible by default. Variables > > > > that introduce non-reproducible output are documented as such. > > > > Thanks for working on this. The patches generally seem to working well > > in testing but this does look related: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio > > > > Particularly these bits: > > > > AssertionError: 10 != 0 : Missing objects in the cache: > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst I think I understand what might have happened here. I suspect we've had test builds on the new test cluster which can write to the hash equivalence server but not the sstate. I think this means entries were added for artefacts which don't exist on sstate. In theory those should be created on new build runs but in practise I suspect they're shadowed by other artefacts so newer builds probably don't create them. That would at least explain how we end up where we are. If I'm correct, if there are changes to the code (as we discussed on yesterdays tech call), everything will rerun and assuming we run this only on the main cluster, things should work out ok. I would be curious to understand how these things aren't being recreated when missing though? I did also notice an interesting issue in that if you run a build with PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and rerun. This isn't ideal as it effectively breaks our ability to turn package backends on/off without rebuilds :(. Can we avoid that or at least minimise things a bit more? Cheers, Richard
On Wed, Jul 10, 2024 at 6:18 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Fri, 2024-07-05 at 08:17 +0100, Richard Purdie via lists.openembedded.org wrote: > > > On Wed, 2024-07-03 at 07:59 -0600, Joshua Watt via > > > lists.openembedded.org wrote: > > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > > default. > > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > > > > > the same time > > > > > > > > > > v2: Added tests and addressed feedback > > > > > v3: Fixed several oe-selftest and build failures > > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > > v5: Reworked to make SPDX 3 output reproducible by default. Variables > > > > > that introduce non-reproducible output are documented as such. > > > > > > Thanks for working on this. The patches generally seem to working well > > > in testing but this does look related: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio > > > > > > Particularly these bits: > > > > > > AssertionError: 10 != 0 : Missing objects in the cache: > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst > > > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst > > I think I understand what might have happened here. I suspect we've had > test builds on the new test cluster which can write to the hash > equivalence server but not the sstate. I think this means entries were > added for artefacts which don't exist on sstate. > > In theory those should be created on new build runs but in practise I > suspect they're shadowed by other artefacts so newer builds probably > don't create them. That would at least explain how we end up where we > are. There is also https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n933 which I beleive is still accurate and probably why the images themselves show up; we'll need to adjust the exclusions to match the new SPDX tasks I added. I think '.*:do_create.*_spdx' would be OK? > > If I'm correct, if there are changes to the code (as we discussed on > yesterdays tech call), everything will rerun and assuming we run this > only on the main cluster, things should work out ok. > > I would be curious to understand how these things aren't being > recreated when missing though? I'm not sure. I can't really think of any reason in the SPDX code that it wouldn't recreate a missing sstate object; I'm not doing anything particularly strange with sstate > > I did also notice an interesting issue in that if you run a build with > PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the > do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and > rerun. This isn't ideal as it effectively breaks our ability to turn > package backends on/off without rebuilds :(. Can we avoid that or at > least minimise things a bit more? Ya, that's a leftover from when I was trying to tie the SPDX packages to the produced package files. I had to give up thought because it's actually really difficult to figure out what .rpm and .deb files correspond to a PACKAGE (at least, I couldn't see an easy way to do it). We can just remove the dependency on do_package_write_* for now and figure out how to do it later. > > Cheers, > > Richard > > >
On Wed, 2024-07-10 at 08:03 -0600, Joshua Watt wrote: > On Wed, Jul 10, 2024 at 6:18 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Fri, 2024-07-05 at 08:17 +0100, Richard Purdie via lists.openembedded.org wrote: > > > > On Wed, 2024-07-03 at 07:59 -0600, Joshua Watt via > > > > lists.openembedded.org wrote: > > > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > > > default. > > > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > > > > > > the same time > > > > > > > > > > > > v2: Added tests and addressed feedback > > > > > > v3: Fixed several oe-selftest and build failures > > > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > > > v5: Reworked to make SPDX 3 output reproducible by default. Variables > > > > > > that introduce non-reproducible output are documented as such. > > > > > > > > Thanks for working on this. The patches generally seem to working well > > > > in testing but this does look related: > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio > > > > > > > > Particularly these bits: > > > > > > > > AssertionError: 10 != 0 : Missing objects in the cache: > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst > > > > > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst > > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst > > > > I think I understand what might have happened here. I suspect we've had > > test builds on the new test cluster which can write to the hash > > equivalence server but not the sstate. I think this means entries were > > added for artefacts which don't exist on sstate. > > > > In theory those should be created on new build runs but in practise I > > suspect they're shadowed by other artefacts so newer builds probably > > don't create them. That would at least explain how we end up where we > > are. > > There is also https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n933 > which I beleive is still accurate and probably why the images > themselves show up; we'll need to adjust the exclusions to match the > new SPDX tasks I added. I think '.*:do_create.*_spdx' would be OK? If that restriction was just for images I'd be ok with it. It is saying we'd ignore any missing spdx sstate artefacts anywhere though which seems wrong, assuming I understand it correctly. > > If I'm correct, if there are changes to the code (as we discussed on > > yesterdays tech call), everything will rerun and assuming we run this > > only on the main cluster, things should work out ok. > > > > I would be curious to understand how these things aren't being > > recreated when missing though? > > I'm not sure. I can't really think of any reason in the SPDX code that > it wouldn't recreate a missing sstate object; I'm not doing anything > particularly strange with sstate Imagine you have some tasks which depend on do_package like do_package_write_ipk and friends. If you only ever build the ipk task, the package task would never run again. I suspect spdx may have similar tasks. > > I did also notice an interesting issue in that if you run a build with > > PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the > > do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and > > rerun. This isn't ideal as it effectively breaks our ability to turn > > package backends on/off without rebuilds :(. Can we avoid that or at > > least minimise things a bit more? > > Ya, that's a leftover from when I was trying to tie the SPDX packages > to the produced package files. I had to give up thought because it's > actually really difficult to figure out what .rpm and .deb files > correspond to a PACKAGE (at least, I couldn't see an easy way to do > it). We can just remove the dependency on do_package_write_* for now > and figure out how to do it later. Ok, that makes sense. We should be able to map package files to PACKAGES but we can discuss that one later! Cheers, Richard
This patch series add support for SPDX 3.0 and sets it as the default. Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time v2: Added tests and addressed feedback v3: Fixed several oe-selftest and build failures v4: Fixed silly typo mistake in staging.bbclass v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such. Joshua Watt (8): classes-recipe/image: Add image file manifest classes-recipe/baremetal-image: Add image file manifest classes/create-spdx-3.0: Add classes classes-global/staging: Exclude do_create_spdx from automatic sysroot extension classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images selftest: spdx: Add SPDX 3.0 test cases classes-recipe: nospdx: Add class Switch default spdx version to 3.0 meta/classes-global/staging.bbclass | 9 +- meta/classes-recipe/baremetal-image.bbclass | 32 +- meta/classes-recipe/image.bbclass | 58 + meta/classes-recipe/image_types.bbclass | 2 + meta/classes-recipe/image_types_wic.bbclass | 1 + meta/classes-recipe/nospdx.bbclass | 13 + meta/classes-recipe/packagegroup.bbclass | 2 + meta/classes/create-spdx-3.0.bbclass | 1231 ++++ meta/classes/create-spdx-image-3.0.bbclass | 224 + meta/classes/create-spdx.bbclass | 2 +- meta/classes/spdx-common.bbclass | 6 +- meta/lib/oe/sbom30.py | 1138 ++++ meta/lib/oe/spdx30.py | 5996 ++++++++++++++++++ meta/lib/oeqa/selftest/cases/spdx.py | 118 +- meta/recipes-core/meta/build-sysroots.bb | 5 +- meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +- 16 files changed, 8819 insertions(+), 21 deletions(-) create mode 100644 meta/classes-recipe/nospdx.bbclass create mode 100644 meta/classes/create-spdx-3.0.bbclass create mode 100644 meta/classes/create-spdx-image-3.0.bbclass create mode 100644 meta/lib/oe/sbom30.py create mode 100644 meta/lib/oe/spdx30.py