| Message ID | 20240619221805.3038528-1-JPEWhacker@gmail.com |
|---|---|
| Headers | show |
| Series | Add SPDX 3.0 support | expand |
Hey Joshua https://errors.yoctoproject.org/Errors/Build/187139/ does this seem related to this changeset On Wed, Jun 19, 2024 at 3:18 PM Joshua Watt via lists.openembedded.org <JPEWhacker=gmail.com@lists.openembedded.org> wrote: > > This patch series add support for SPDX 3.0 and sets it as the default. > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > the same time; users may choose to specifically use SPDX 3.0 instead of > 2.2 by doing INHERIT += "create-spdx-3.0" > > v2: Added tests and addressed feedback > > > Joshua Watt (7): > classes-recipe/image: Add image file manifest > classes/create-spdx-3.0: Add class > classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images > selftest: spdx: Add SPDX 3.0 test cases > classes-recipe: nospdx: Add class > classes/spdx-common: Move SPDX_SUPPLIER > Switch default spdx version to 3.0 > > meta/classes-recipe/image.bbclass | 58 + > meta/classes-recipe/image_types.bbclass | 2 + > meta/classes-recipe/image_types_wic.bbclass | 1 + > meta/classes-recipe/nospdx.bbclass | 13 + > meta/classes-recipe/packagegroup.bbclass | 2 + > meta/classes/create-spdx-2.2.bbclass | 8 + > meta/classes/create-spdx-3.0.bbclass | 1116 ++++ > meta/classes/create-spdx-image-3.0.bbclass | 203 + > meta/classes/create-spdx.bbclass | 2 +- > meta/classes/spdx-common.bbclass | 32 +- > meta/lib/oe/sbom30.py | 1118 ++++ > meta/lib/oe/spdx30.py | 5996 ++++++++++++++++++ > meta/lib/oeqa/selftest/cases/spdx.py | 119 +- > meta/recipes-core/meta/build-sysroots.bb | 5 +- > meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +- > 15 files changed, 8654 insertions(+), 24 deletions(-) > create mode 100644 meta/classes-recipe/nospdx.bbclass > create mode 100644 meta/classes/create-spdx-3.0.bbclass > create mode 100644 meta/classes/create-spdx-image-3.0.bbclass > create mode 100644 meta/lib/oe/sbom30.py > create mode 100644 meta/lib/oe/spdx30.py > > -- > 2.43.2 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#200929): https://lists.openembedded.org/g/openembedded-core/message/200929 > Mute This Topic: https://lists.openembedded.org/mt/106770352/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Unfortunately, this still causes failures: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7060/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/9481/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/9574/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/9149/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/6555/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4969/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/9220/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/166/builds/518/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/165/builds/514/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/48/builds/9186/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/9166/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/104/builds/8044/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/103/builds/8038/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/6112/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/9159/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/9145/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/6733/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/52/builds/8994/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/6122/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/6122/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/58/builds/8923/steps/15/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6837/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6881/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6906/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6892/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3501/steps/15/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/9099/steps/23/logs/stdio On 19/06/2024 16:13:22-0600, Joshua Watt wrote: > This patch series add support for SPDX 3.0 and sets it as the default. > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > the same time; users may choose to specifically use SPDX 3.0 instead of > 2.2 by doing INHERIT += "create-spdx-3.0" > > v2: Added tests and addressed feedback > > > Joshua Watt (7): > classes-recipe/image: Add image file manifest > classes/create-spdx-3.0: Add class > classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images > selftest: spdx: Add SPDX 3.0 test cases > classes-recipe: nospdx: Add class > classes/spdx-common: Move SPDX_SUPPLIER > Switch default spdx version to 3.0 > > meta/classes-recipe/image.bbclass | 58 + > meta/classes-recipe/image_types.bbclass | 2 + > meta/classes-recipe/image_types_wic.bbclass | 1 + > meta/classes-recipe/nospdx.bbclass | 13 + > meta/classes-recipe/packagegroup.bbclass | 2 + > meta/classes/create-spdx-2.2.bbclass | 8 + > meta/classes/create-spdx-3.0.bbclass | 1116 ++++ > meta/classes/create-spdx-image-3.0.bbclass | 203 + > meta/classes/create-spdx.bbclass | 2 +- > meta/classes/spdx-common.bbclass | 32 +- > meta/lib/oe/sbom30.py | 1118 ++++ > meta/lib/oe/spdx30.py | 5996 ++++++++++++++++++ > meta/lib/oeqa/selftest/cases/spdx.py | 119 +- > meta/recipes-core/meta/build-sysroots.bb | 5 +- > meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +- > 15 files changed, 8654 insertions(+), 24 deletions(-) > create mode 100644 meta/classes-recipe/nospdx.bbclass > create mode 100644 meta/classes/create-spdx-3.0.bbclass > create mode 100644 m: failures:eta/classes/create-spdx-image-3.0.bbclass > create mode 100644 meta/lib/oe/sbom30.py > create mode 100644 meta/lib/oe/spdx30.py > > -- > 2.43.2 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#200929): https://lists.openembedded.org/g/openembedded-core/message/200929 > Mute This Topic: https://lists.openembedded.org/mt/106770352/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Fri, Jun 21, 2024 at 12:24 AM Alexandre Belloni <alexandre.belloni@bootlin.com> wrote: > > Unfortunately, this still causes failures: Thanks, I'll take a look > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7060/steps/23/logs/stdio > > https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/9481/steps/14/logs/stdio > > https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/9574/steps/11/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/9149/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/6555/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4969/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/9220/steps/11/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/166/builds/518/steps/11/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/165/builds/514/steps/11/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/48/builds/9186/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/9166/steps/11/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/104/builds/8044/steps/11/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/103/builds/8038/steps/11/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/6112/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/9159/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/9145/steps/11/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/6733/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/52/builds/8994/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/6122/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/6122/steps/12/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/58/builds/8923/steps/15/logs/stdio > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6837/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6881/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6906/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6892/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3501/steps/15/logs/stdio > > https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/9099/steps/23/logs/stdio > > > On 19/06/2024 16:13:22-0600, Joshua Watt wrote: > > This patch series add support for SPDX 3.0 and sets it as the default. > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > > the same time; users may choose to specifically use SPDX 3.0 instead of > > 2.2 by doing INHERIT += "create-spdx-3.0" > > > > v2: Added tests and addressed feedback > > > > > > Joshua Watt (7): > > classes-recipe/image: Add image file manifest > > classes/create-spdx-3.0: Add class > > classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images > > selftest: spdx: Add SPDX 3.0 test cases > > classes-recipe: nospdx: Add class > > classes/spdx-common: Move SPDX_SUPPLIER > > Switch default spdx version to 3.0 > > > > meta/classes-recipe/image.bbclass | 58 + > > meta/classes-recipe/image_types.bbclass | 2 + > > meta/classes-recipe/image_types_wic.bbclass | 1 + > > meta/classes-recipe/nospdx.bbclass | 13 + > > meta/classes-recipe/packagegroup.bbclass | 2 + > > meta/classes/create-spdx-2.2.bbclass | 8 + > > meta/classes/create-spdx-3.0.bbclass | 1116 ++++ > > meta/classes/create-spdx-image-3.0.bbclass | 203 + > > meta/classes/create-spdx.bbclass | 2 +- > > meta/classes/spdx-common.bbclass | 32 +- > > meta/lib/oe/sbom30.py | 1118 ++++ > > meta/lib/oe/spdx30.py | 5996 ++++++++++++++++++ > > meta/lib/oeqa/selftest/cases/spdx.py | 119 +- > > meta/recipes-core/meta/build-sysroots.bb | 5 +- > > meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +- > > 15 files changed, 8654 insertions(+), 24 deletions(-) > > create mode 100644 meta/classes-recipe/nospdx.bbclass > > create mode 100644 meta/classes/create-spdx-3.0.bbclass > > create mode 100644 m: failures:eta/classes/create-spdx-image-3.0.bbclass > > create mode 100644 meta/lib/oe/sbom30.py > > create mode 100644 meta/lib/oe/spdx30.py > > > > -- > > 2.43.2 > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#200929): https://lists.openembedded.org/g/openembedded-core/message/200929 > > Mute This Topic: https://lists.openembedded.org/mt/106770352/3617179 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
This jogged my memory on why I added the patch to allow tasks to opt
out of running `extend_recipe_sysroot`. I'm sending this to get it
written down before I forget, we can talk about it in the tech meeting
on Tuesday if necessary. There are two build failure modes that having
do_create_spdx run extend_recipe_sysroot causes.
The first failure mode is when a recipe explicitly does
`do_populate_sysroot[noexec] = "1"` (an example here is
glib-y2038-tests). In this case, when do_create_spdx tries to run
extend_recipe_sysroot, it sees that the task depends on
do_populate_sysroot, and this looks for the
glibc-y2038-tests:populate_sysroot manifest file, which doesn't exist
because the task was not run which results in a failure. Example here:
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/6733/steps/12/logs/stdio
The second failure mode is (I think) a bug in
binutils-cross-testsuite. This recipe is odd because it doesn't
actually inherit cross.bbclass, and thus isn't _actually_ a cross
recipe, despite the name. This confuses find_sstate_manifest(), which
assumes that if the recipe has "-cross-" in the name, the package arch
must be ${BUILD_ARCH}, which is not the case here because it doesn't
inherit cross.bbclass. When do_create_spdx tries to run
extend_recipe_sysroot, it is looking for the
binutils-cross-testsuite:populate_sysroot manifest file using the
wrong architecture, and thus errors out. I _think_ this might be fixed
by renaming the recipe so that it no longer contains the string
"-cross-". Example here:
https://errors.yoctoproject.org/Errors/Details/787278/
do_create_spdx doesn't actually need the sysroot extended; it only
actually depends on do_populate_sysroot do that it can inspect the
sysroot produced by the current recipe to add the files to the SPDX
output. IIUC, the automatic call to extend_recipe_sysroot when a task
depends on do_populate_ssyroot is mostly a heuristic designed to
ensure that tasks get the full recipe sysroot correctly without having
to manually invoke the call everywhere. However, in the case of
do_create_spdx() it is actually not necessary, thus I added the flag
to disable that behavior on a per-task basis, since I thought that
might be better than hard coding an exception for do_create_spdx.
The other solution would be to identify each individual recipe that is
doing something weird and disable SPDX generation in that specific
recipe (I've made a new class called nospdx.bbclass that makes this
easy). I'm not sure how many recipes this would be, probably only a
few, but it might be confusing to other layers why failures occur and
what to do about it.... and again since the heuristic for when to call
extend_recipe_sysroot is incorrect for do_create_spdx, I'd prefer to
explore that route.
On Fri, Jun 21, 2024 at 12:24 AM Alexandre Belloni
<alexandre.belloni@bootlin.com> wrote:
>
> Unfortunately, this still causes failures:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7060/steps/23/logs/stdio
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/9481/steps/14/logs/stdio
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/9574/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/9149/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/6555/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4969/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/9220/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/166/builds/518/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/165/builds/514/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/48/builds/9186/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/9166/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/104/builds/8044/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/103/builds/8038/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/6112/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/9159/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/9145/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/6733/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/52/builds/8994/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/6122/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/6122/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/58/builds/8923/steps/15/logs/stdio
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6837/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6881/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6906/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6892/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3501/steps/15/logs/stdio
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/9099/steps/23/logs/stdio
>
>
> On 19/06/2024 16:13:22-0600, Joshua Watt wrote:
> > This patch series add support for SPDX 3.0 and sets it as the default.
> > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
> > the same time; users may choose to specifically use SPDX 3.0 instead of
> > 2.2 by doing INHERIT += "create-spdx-3.0"
> >
> > v2: Added tests and addressed feedback
> >
> >
> > Joshua Watt (7):
> > classes-recipe/image: Add image file manifest
> > classes/create-spdx-3.0: Add class
> > classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
> > selftest: spdx: Add SPDX 3.0 test cases
> > classes-recipe: nospdx: Add class
> > classes/spdx-common: Move SPDX_SUPPLIER
> > Switch default spdx version to 3.0
> >
> > meta/classes-recipe/image.bbclass | 58 +
> > meta/classes-recipe/image_types.bbclass | 2 +
> > meta/classes-recipe/image_types_wic.bbclass | 1 +
> > meta/classes-recipe/nospdx.bbclass | 13 +
> > meta/classes-recipe/packagegroup.bbclass | 2 +
> > meta/classes/create-spdx-2.2.bbclass | 8 +
> > meta/classes/create-spdx-3.0.bbclass | 1116 ++++
> > meta/classes/create-spdx-image-3.0.bbclass | 203 +
> > meta/classes/create-spdx.bbclass | 2 +-
> > meta/classes/spdx-common.bbclass | 32 +-
> > meta/lib/oe/sbom30.py | 1118 ++++
> > meta/lib/oe/spdx30.py | 5996 ++++++++++++++++++
> > meta/lib/oeqa/selftest/cases/spdx.py | 119 +-
> > meta/recipes-core/meta/build-sysroots.bb | 5 +-
> > meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +-
> > 15 files changed, 8654 insertions(+), 24 deletions(-)
> > create mode 100644 meta/classes-recipe/nospdx.bbclass
> > create mode 100644 meta/classes/create-spdx-3.0.bbclass
> > create mode 100644 m: failures:eta/classes/create-spdx-image-3.0.bbclass
> > create mode 100644 meta/lib/oe/sbom30.py
> > create mode 100644 meta/lib/oe/spdx30.py
> >
> > --
> > 2.43.2
> >
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#200929): https://lists.openembedded.org/g/openembedded-core/message/200929
> > Mute This Topic: https://lists.openembedded.org/mt/106770352/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com