| Message ID | 20241120055036.1002075-1-hongxu.jia@windriver.com |
|---|---|
| Headers | show |
| Series | SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host | expand |
On Tue, Nov 19, 2024 at 09:50:33PM -0800, hongxu via lists.openembedded.org wrote: > Summary: SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host > Hi, Thanks for the v2, but I believe this is causing some issues on the autobuilder: ERROR: python3-dbusmock-0.32.2-r0 do_create_package_spdx: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: [...] Exception: FileExistsError: [Errno 17] File exists: '../packages/python3-dbusmock-dev.spdx.json' -> '/srv/pokybuild/yocto-worker/qemuppc-alt/build/build/tmp/work/ppc7400-poky-linux/python3-dbusmock/0.32.2/spdx/3.0.1/runtime-deploy/ppc7400/by-spdxid-link/python3-dbusmock.spdx.json' And same kind of error for several packages, on most of the builders. https://valkyrie.yoctoproject.org/#/builders/64/builds/408/steps/12/logs/stdio Can you have a look at this, please ?
Hi, From the error log, it seems you do not apply patch '[PATCH V2 1/3] sbom30/spdx30: add link prefix and name to namespace of spdxId and alias' There are three patches for this review: [PATCH V2 1/3] sbom30/spdx30: add link prefix and name to namespace of spdxId and alias [PATCH V2 2/3] sbom30.py: reduce redundant spdxid symlinks to save inode on host [PATCH 3/3] oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal build [...] Exception: FileExistsError: [Errno 17] File exists: '../packages/python3-dbusmock-dev.spdx.json' -> '/srv/pokybuild/yocto-worker/qemuppc-alt/build/build/tmp/work/ppc7400-poky-linux/python3-dbusmock/0.32.2/spdx/3.0.1/runtime-deploy/ppc7400/by-spdxid-link/python3-dbusmock.spdx.json' [...] As comparing, after applying the patches I my build, the link should have "package" prefix and "-dev" suffix, runtime-deploy/core2-64/by-spdxid-link/package-python3-dbusmock-dev.spdx.json -> ../packages/python3-dbusmock-dev.spdx.json //Hongxu
Ping Joshua //Hongxu
Summary: SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host * Git logs [oe-core] commit 5ebf6f058259bc073e83bd2fa2b3b116e6156d60 Author: Hongxu Jia <hongxu.jia@windriver.com> Date: Sat Nov 9 17:16:31 2024 +0800 oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal build $ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source 2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries: 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-yocto-bsp/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-selftest/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/meta-openembedded/meta-oe/lib 2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source) 2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message 2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok 2024-11-20 05:21:12,798 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s 2024-11-20 05:21:12,798 - oe-selftest - INFO - OK 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS: 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s) 2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY: 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> commit 3753b084cffcbef374093a442d4e692821d1a29b Author: Hongxu Jia <hongxu.jia@windriver.com> Date: Sat Nov 9 14:18:26 2024 +0800 sbom30.py: reduce redundant spdxid symlinks to save inode on host In order to support all in-scope SPDX data within a single JSON-LD file for SPDX 3.0.1, Yocto's SBOM: - In native/target/nativesdk recipe, created spdxid-hash symlink for each element to point to the JSON-LD file that contains element details; - In image recipe, use spdxid-hash symlink to collect element details from varies of JSON-LD files While SPDX_INCLUDE_SOURCES = "1", it adds sources to JSON-LD file and create 2N+ spdxid-hash symlinks for N source files. (N for software_File, N for hasDeclaredLicense's Relationship) For large numbers of source files, adding an extra symlink -> real file will occupy one more inode (per file), which will need a slot in the OS's inode cache. In this situation, disk performance is slow and inode is used up quickly After commit [sbom30/spdx30: add link prefix and name to namespace of spdxId and alias] applied, the namespace of spdxId and alias in recipe and package jsonld differs. Use it to create symlink to jsonld, take recipe shadow, package shadow and package shadow-src for example: For recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/... alias: recipe-shadow/UNIHASH/... symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/recipe-shadow.spdx.json -> ../recipes/shadow.spdx.json For package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/... alias: package-shadow/UNIHASH/... symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow.spdx.json -> ../packages/shadow.spdx.json In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/... alias: package-shadow-src/UNIHASH/... symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow-src.spdx.json -> ../packages/shadow-src.spdx.json Build core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043 echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf Without this commit: $ time bitbake core-image-minimal real 100m17.769s user 0m24.516s sys 0m4.334s $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l 7281824 With this commit: $ time bitbake core-image-minimal real 85m12.994s user 0m20.423s sys 0m4.228s $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l 6043 Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> commit 0ea8bcaa9a8fe63f1935e02968fbaa541734024b Author: Hongxu Jia <hongxu.jia@windriver.com> Date: Tue Nov 19 20:12:37 2024 +0800 sbom30/spdx30: add link prefix and name to namespace of spdxId and alias In order to simple reference the SPDX ID to instead of making jsonld hash path for each element, only creating one symlink for one file and referencing it multiple times, add link prefix and name to the namespace of spdxId and alias to replace ${PN} to avoid namespace conflict between recipe, packages and images. Take recipe shadow, package shadow and package shadow-src for example: Without this commit, spdxId and alias in recipe and package jsonld have the same namespace spdxId: http://spdx.org/spdxdocs/shadow-xxx/... alias: shadow/UNIHASH/... After apply this commit, the namespace of spdxId in recipe and package jsonld differs: In recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/... alias: recipe-shadow/UNIHASH/... In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/... alias: package-shadow/UNIHASH/... In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/... alias: package-shadow-src/UNIHASH/... Then will use namespace of spdxId and alias to create link for jsonld file, one symlink for one jsonld file, referenced by elements multiple times Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> ====== Testing ====== * Commands Build core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043 echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf Without this commit: $ time bitbake core-image-minimal real 100m17.769s user 0m24.516s sys 0m4.334s $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l 7281824 With this commit: $ time bitbake core-image-minimal real 85m12.994s user 0m20.423s sys 0m4.228s $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l 6043 $ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source 2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries: 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-yocto-bsp/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-selftest/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/meta-openembedded/meta-oe/lib 2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source) 2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message 2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok 2024-11-20 05:21:12,798 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s 2024-11-20 05:21:12,798 - oe-selftest - INFO - OK 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS: 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s) 2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY: 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)