Message ID | 20240712160304.3514496-1-JPEWhacker@gmail.com |
---|---|
Headers | show |
Series | Add SPDX 3.0 support | expand |
On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via lists.openembedded.org wrote: > This patch series add support for SPDX 3.0 and sets it as the > default. > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > the same time > > v2: Added tests and addressed feedback > v3: Fixed several oe-selftest and build failures > v4: Fixed silly typo mistake in staging.bbclass > v5: Reworked to make SPDX 3 output reproducible by default. Variables > that introduce non-reproducible output are documented as such. > v6: Many changes: > * Fixed bug where building baremetal images would break SPDX 2.2 > * Most SPDX code is now in python library files instead of tasks > * Removed dependency on pacakge_write_* tasks > * Fixed sstate selftest cases to account for SPDX 3.0 task names This had a lot of failures in testing I'm afraid: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 Cheers, Richard
On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > lists.openembedded.org wrote: > > This patch series add support for SPDX 3.0 and sets it as the > > default. > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at > > the same time > > > > v2: Added tests and addressed feedback > > v3: Fixed several oe-selftest and build failures > > v4: Fixed silly typo mistake in staging.bbclass > > v5: Reworked to make SPDX 3 output reproducible by default. Variables > > that introduce non-reproducible output are documented as such. > > v6: Many changes: > > * Fixed bug where building baremetal images would break SPDX 2.2 > > * Most SPDX code is now in python library files instead of tasks > > * Removed dependency on pacakge_write_* tasks > > * Fixed sstate selftest cases to account for SPDX 3.0 task names > > This had a lot of failures in testing I'm afraid: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 This appears to be caused because the SPDX tasks are not running for some recipes (e.g. automake). It looks like this like: do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" is not actually strong enough to make sure the SPDX tasks for automake run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let me know > > Cheers, > > Richard
On Mon, 2024-07-15 at 14:40 -0600, Joshua Watt wrote: > On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > > lists.openembedded.org wrote: > > > This patch series add support for SPDX 3.0 and sets it as the > > > default. > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 > > > enabled at > > > the same time > > > > > > v2: Added tests and addressed feedback > > > v3: Fixed several oe-selftest and build failures > > > v4: Fixed silly typo mistake in staging.bbclass > > > v5: Reworked to make SPDX 3 output reproducible by default. > > > Variables > > > that introduce non-reproducible output are documented as > > > such. > > > v6: Many changes: > > > * Fixed bug where building baremetal images would break SPDX > > > 2.2 > > > * Most SPDX code is now in python library files instead of > > > tasks > > > * Removed dependency on pacakge_write_* tasks > > > * Fixed sstate selftest cases to account for SPDX 3.0 task > > > names > > > > This had a lot of failures in testing I'm afraid: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 > > This appears to be caused because the SPDX tasks are not running for > some recipes (e.g. automake). It looks like this like: > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > do_create_package_spdx" > > is not actually strong enough to make sure the SPDX tasks for > automake > run for e.g. core-image-sato-sdk, but I don't know why. I'll keep > looking, but if anyone happens to know off the top of their head let > me know Can you be specific about which tasks you mean when you say "make sure the SPDX tasks for automake run"? Do you mean do_create_spdx, do_create_package_spdx or a different one? Cheers, Richard
On Mon, Jul 15, 2024 at 3:07 PM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Mon, 2024-07-15 at 14:40 -0600, Joshua Watt wrote: > > On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > > > lists.openembedded.org wrote: > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > default. > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 > > > > enabled at > > > > the same time > > > > > > > > v2: Added tests and addressed feedback > > > > v3: Fixed several oe-selftest and build failures > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > v5: Reworked to make SPDX 3 output reproducible by default. > > > > Variables > > > > that introduce non-reproducible output are documented as > > > > such. > > > > v6: Many changes: > > > > * Fixed bug where building baremetal images would break SPDX > > > > 2.2 > > > > * Most SPDX code is now in python library files instead of > > > > tasks > > > > * Removed dependency on pacakge_write_* tasks > > > > * Fixed sstate selftest cases to account for SPDX 3.0 task > > > > names > > > > > > This had a lot of failures in testing I'm afraid: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 > > > > This appears to be caused because the SPDX tasks are not running for > > some recipes (e.g. automake). It looks like this like: > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > do_create_package_spdx" > > > > is not actually strong enough to make sure the SPDX tasks for > > automake > > run for e.g. core-image-sato-sdk, but I don't know why. I'll keep > > looking, but if anyone happens to know off the top of their head let > > me know > > Can you be specific about which tasks you mean when you say "make sure > the SPDX tasks for automake run"? Do you mean do_create_spdx, > do_create_package_spdx or a different one? Specifically, do_create_package_spdx must be run for each package installed in the rootfs before do_create_rootfs_spdx runs. I thought that do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" would do this (the do_create_spdx is probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something? > > Cheers, > > Richard
On Mon, 2024-07-15 at 15:26 -0600, Joshua Watt wrote: > On Mon, Jul 15, 2024 at 3:07 PM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Mon, 2024-07-15 at 14:40 -0600, Joshua Watt wrote: > > > On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > > > > lists.openembedded.org wrote: > > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > > default. > > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 > > > > > enabled at > > > > > the same time > > > > > > > > > > v2: Added tests and addressed feedback > > > > > v3: Fixed several oe-selftest and build failures > > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > > v5: Reworked to make SPDX 3 output reproducible by default. > > > > > Variables > > > > > that introduce non-reproducible output are documented as > > > > > such. > > > > > v6: Many changes: > > > > > * Fixed bug where building baremetal images would break > > > > > SPDX > > > > > 2.2 > > > > > * Most SPDX code is now in python library files instead of > > > > > tasks > > > > > * Removed dependency on pacakge_write_* tasks > > > > > * Fixed sstate selftest cases to account for SPDX 3.0 task > > > > > names > > > > > > > > This had a lot of failures in testing I'm afraid: > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 > > > > > > This appears to be caused because the SPDX tasks are not running > > > for > > > some recipes (e.g. automake). It looks like this like: > > > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > > do_create_package_spdx" > > > > > > is not actually strong enough to make sure the SPDX tasks for > > > automake > > > run for e.g. core-image-sato-sdk, but I don't know why. I'll keep > > > looking, but if anyone happens to know off the top of their head > > > let > > > me know > > > > Can you be specific about which tasks you mean when you say "make > > sure > > the SPDX tasks for automake run"? Do you mean do_create_spdx, > > do_create_package_spdx or a different one? > > Specifically, do_create_package_spdx must be run for each package > installed in the rootfs before do_create_rootfs_spdx runs. I thought > that > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > do_create_package_spdx" > > would do this (the do_create_spdx is probably not necessary), since > AFIACT, this is also how the packages get generated before being > installed in the root file system via manipulation of > do_rootfs[recrdeptask], but I think I'm missing something? In packagegroups.bbclass there is this being added: do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = "" and bitbake core-image-sato-sdk -g -c create_rootfs_spdx lists "core-image-sato-sdk.do_create_rootfs_spdx" -> "automake.do_create_package_spdx" in tasks-depends.dot if I disable it. I'm not 100% sure what is going on and should sleep but wanted to share that before I did. Cheers, Richard
On Tue, 2024-07-16 at 00:00 +0100, Richard Purdie via lists.openembedded.org wrote: > On Mon, 2024-07-15 at 15:26 -0600, Joshua Watt wrote: > > On Mon, Jul 15, 2024 at 3:07 PM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > On Mon, 2024-07-15 at 14:40 -0600, Joshua Watt wrote: > > > > On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie > > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > > > > > lists.openembedded.org wrote: > > > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > > > default. > > > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 > > > > > > enabled at > > > > > > the same time > > > > > > > > > > > > v2: Added tests and addressed feedback > > > > > > v3: Fixed several oe-selftest and build failures > > > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > > > v5: Reworked to make SPDX 3 output reproducible by default. > > > > > > Variables > > > > > > that introduce non-reproducible output are documented as > > > > > > such. > > > > > > v6: Many changes: > > > > > > * Fixed bug where building baremetal images would break > > > > > > SPDX > > > > > > 2.2 > > > > > > * Most SPDX code is now in python library files instead of > > > > > > tasks > > > > > > * Removed dependency on pacakge_write_* tasks > > > > > > * Fixed sstate selftest cases to account for SPDX 3.0 task > > > > > > names > > > > > > > > > > This had a lot of failures in testing I'm afraid: > > > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 > > > > > > > > This appears to be caused because the SPDX tasks are not running > > > > for > > > > some recipes (e.g. automake). It looks like this like: > > > > > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > > > do_create_package_spdx" > > > > > > > > is not actually strong enough to make sure the SPDX tasks for > > > > automake > > > > run for e.g. core-image-sato-sdk, but I don't know why. I'll keep > > > > looking, but if anyone happens to know off the top of their head > > > > let > > > > me know > > > > > > Can you be specific about which tasks you mean when you say "make > > > sure > > > the SPDX tasks for automake run"? Do you mean do_create_spdx, > > > do_create_package_spdx or a different one? > > > > Specifically, do_create_package_spdx must be run for each package > > installed in the rootfs before do_create_rootfs_spdx runs. I thought > > that > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > do_create_package_spdx" > > > > would do this (the do_create_spdx is probably not necessary), since > > AFIACT, this is also how the packages get generated before being > > installed in the root file system via manipulation of > > do_rootfs[recrdeptask], but I think I'm missing something? > > > In packagegroups.bbclass there is this being added: > > do_create_package_spdx[deptask] = "do_create_spdx" > do_create_package_spdx[rdeptask] = "" > > and > > bitbake core-image-sato-sdk -g -c create_rootfs_spdx > > lists > > "core-image-sato-sdk.do_create_rootfs_spdx" -> "automake.do_create_package_spdx" > > in tasks-depends.dot if I disable it. > > I'm not 100% sure what is going on and should sleep but wanted to share > that before I did. I deleted those two lines and ran a build which passed testing. I don't think we need them with spdx 3 since it doesn't have the "hash changing" issue that we had that caused us to add that for spdx2? The commit was: https://git.yoctoproject.org/poky/commit/meta/classes-recipe/packagegroup.bbclass?id=06b5f249ced23b6bc442758131832b8640164b44 Cheers, Richard
On Tue, Jul 16, 2024, 7:18 AM Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > On Tue, 2024-07-16 at 00:00 +0100, Richard Purdie via > lists.openembedded.org wrote: > > On Mon, 2024-07-15 at 15:26 -0600, Joshua Watt wrote: > > > On Mon, Jul 15, 2024 at 3:07 PM Richard Purdie > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > On Mon, 2024-07-15 at 14:40 -0600, Joshua Watt wrote: > > > > > On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie > > > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > > > > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > > > > > > lists.openembedded.org wrote: > > > > > > > This patch series add support for SPDX 3.0 and sets it as the > > > > > > > default. > > > > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 > > > > > > > enabled at > > > > > > > the same time > > > > > > > > > > > > > > v2: Added tests and addressed feedback > > > > > > > v3: Fixed several oe-selftest and build failures > > > > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > > > > v5: Reworked to make SPDX 3 output reproducible by default. > > > > > > > Variables > > > > > > > that introduce non-reproducible output are documented as > > > > > > > such. > > > > > > > v6: Many changes: > > > > > > > * Fixed bug where building baremetal images would break > > > > > > > SPDX > > > > > > > 2.2 > > > > > > > * Most SPDX code is now in python library files instead of > > > > > > > tasks > > > > > > > * Removed dependency on pacakge_write_* tasks > > > > > > > * Fixed sstate selftest cases to account for SPDX 3.0 task > > > > > > > names > > > > > > > > > > > > This had a lot of failures in testing I'm afraid: > > > > > > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 > > > > > > > > > > This appears to be caused because the SPDX tasks are not running > > > > > for > > > > > some recipes (e.g. automake). It looks like this like: > > > > > > > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > > > > do_create_package_spdx" > > > > > > > > > > is not actually strong enough to make sure the SPDX tasks for > > > > > automake > > > > > run for e.g. core-image-sato-sdk, but I don't know why. I'll keep > > > > > looking, but if anyone happens to know off the top of their head > > > > > let > > > > > me know > > > > > > > > Can you be specific about which tasks you mean when you say "make > > > > sure > > > > the SPDX tasks for automake run"? Do you mean do_create_spdx, > > > > do_create_package_spdx or a different one? > > > > > > Specifically, do_create_package_spdx must be run for each package > > > installed in the rootfs before do_create_rootfs_spdx runs. I thought > > > that > > > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > > do_create_package_spdx" > > > > > > would do this (the do_create_spdx is probably not necessary), since > > > AFIACT, this is also how the packages get generated before being > > > installed in the root file system via manipulation of > > > do_rootfs[recrdeptask], but I think I'm missing something? > > > > > > In packagegroups.bbclass there is this being added: > > > > do_create_package_spdx[deptask] = "do_create_spdx" > > do_create_package_spdx[rdeptask] = "" > > > > and > > > > bitbake core-image-sato-sdk -g -c create_rootfs_spdx > > > > lists > > > > "core-image-sato-sdk.do_create_rootfs_spdx" -> > "automake.do_create_package_spdx" > > > > in tasks-depends.dot if I disable it. > > > > I'm not 100% sure what is going on and should sleep but wanted to share > > that before I did. > > I deleted those two lines and ran a build which passed testing. > > I don't think we need them with spdx 3 since it doesn't have the "hash > changing" issue that we had that caused us to add that for spdx2? > Good catch, I missed that was related to packagegroups. No, we don't need that for SPDX 3 (still need it for 2.2 though) > The commit was: > > https://git.yoctoproject.org/poky/commit/meta/classes-recipe/packagegroup.bbclass?id=06b5f249ced23b6bc442758131832b8640164b44 > > Cheers, > > Richard > >
On Tue, 2024-07-16 at 14:18 +0100, Richard Purdie via lists.openembedded.org wrote: > On Tue, 2024-07-16 at 00:00 +0100, Richard Purdie via > lists.openembedded.org wrote: > > On Mon, 2024-07-15 at 15:26 -0600, Joshua Watt wrote: > > > On Mon, Jul 15, 2024 at 3:07 PM Richard Purdie > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > On Mon, 2024-07-15 at 14:40 -0600, Joshua Watt wrote: > > > > > On Sat, Jul 13, 2024 at 12:44 AM Richard Purdie > > > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > > > > > On Fri, 2024-07-12 at 09:58 -0600, Joshua Watt via > > > > > > lists.openembedded.org wrote: > > > > > > > This patch series add support for SPDX 3.0 and sets it as > > > > > > > the > > > > > > > default. > > > > > > > Currently it is not possible to have SPDX 2.2 and SPDX > > > > > > > 3.0 > > > > > > > enabled at > > > > > > > the same time > > > > > > > > > > > > > > v2: Added tests and addressed feedback > > > > > > > v3: Fixed several oe-selftest and build failures > > > > > > > v4: Fixed silly typo mistake in staging.bbclass > > > > > > > v5: Reworked to make SPDX 3 output reproducible by > > > > > > > default. > > > > > > > Variables > > > > > > > that introduce non-reproducible output are documented > > > > > > > as > > > > > > > such. > > > > > > > v6: Many changes: > > > > > > > * Fixed bug where building baremetal images would break > > > > > > > SPDX > > > > > > > 2.2 > > > > > > > * Most SPDX code is now in python library files instead > > > > > > > of > > > > > > > tasks > > > > > > > * Removed dependency on pacakge_write_* tasks > > > > > > > * Fixed sstate selftest cases to account for SPDX 3.0 > > > > > > > task > > > > > > > names > > > > > > > > > > > > This had a lot of failures in testing I'm afraid: > > > > > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134 > > > > > > > > > > This appears to be caused because the SPDX tasks are not > > > > > running > > > > > for > > > > > some recipes (e.g. automake). It looks like this like: > > > > > > > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > > > > do_create_package_spdx" > > > > > > > > > > is not actually strong enough to make sure the SPDX tasks for > > > > > automake > > > > > run for e.g. core-image-sato-sdk, but I don't know why. I'll > > > > > keep > > > > > looking, but if anyone happens to know off the top of their > > > > > head > > > > > let > > > > > me know > > > > > > > > Can you be specific about which tasks you mean when you say > > > > "make > > > > sure > > > > the SPDX tasks for automake run"? Do you mean do_create_spdx, > > > > do_create_package_spdx or a different one? > > > > > > Specifically, do_create_package_spdx must be run for each package > > > installed in the rootfs before do_create_rootfs_spdx runs. I > > > thought > > > that > > > > > > do_create_rootfs_spdx[recrdeptask] += "do_create_spdx > > > do_create_package_spdx" > > > > > > would do this (the do_create_spdx is probably not necessary), > > > since > > > AFIACT, this is also how the packages get generated before being > > > installed in the root file system via manipulation of > > > do_rootfs[recrdeptask], but I think I'm missing something? > > > > > > In packagegroups.bbclass there is this being added: > > > > do_create_package_spdx[deptask] = "do_create_spdx" > > do_create_package_spdx[rdeptask] = "" > > > > and > > > > bitbake core-image-sato-sdk -g -c create_rootfs_spdx > > > > lists > > > > "core-image-sato-sdk.do_create_rootfs_spdx" -> > > "automake.do_create_package_spdx" > > > > in tasks-depends.dot if I disable it. > > > > I'm not 100% sure what is going on and should sleep but wanted to > > share > > that before I did. > > I deleted those two lines and ran a build which passed testing. > > I don't think we need them with spdx 3 since it doesn't have the > "hash > changing" issue that we had that caused us to add that for spdx2? > > The commit was: > https://git.yoctoproject.org/poky/commit/meta/classes-recipe/packagegroup.bbclass?id=06b5f249ced23b6bc442758131832b8640164b44 For the purposes of the archives, Joshua and I agreed we don't need these lines for SPDX 3.0. I squashed in that fix and merged SPDX 3.0 support (as default). Thanks Joshua, Marta and others for work on this. Cheers, Richard