diff mbox series

ref-manual: Document SPDX 3.0.1 variables

Message ID 20251114145220.71823-1-stondo@gmail.com
State Superseded
Headers show
Series ref-manual: Document SPDX 3.0.1 variables | expand

Commit Message

Stefano Tondo Nov. 14, 2025, 2:52 p.m. UTC
From: Stefano Tondo <stefano.tondo.ext@siemens.com>

Add comprehensive documentation for SPDX-related variables in the
Yocto reference manual. This includes documenting previously
undocumented variables and updating existing documentation with
SPDX 3.0.1 specific information.

Variables documented:
- SPDX_INCLUDE_SOURCES: Control source file inclusion in SBOM
  Updated with concrete SPDX 3.0.1 size data (~260 MB without sources,
  ~2.5-2.6 GB with sources)
- SPDX_INCLUDE_COMPILED_SOURCES: Control compiled source inclusion
- SPDX_UUID_NAMESPACE: Namespace for UUID generation in SPDX documents
- SPDX_NAMESPACE_PREFIX: URI prefix for SPDX documents
- SPDX_PRETTY: JSON output formatting control
- SPDX_LICENSES: Path to SPDX license identifier mapping file
- SPDX_CUSTOM_ANNOTATION_VARS: Custom annotation variables
- SPDX_MULTILIB_SSTATE_ARCHS: Architecture list for dependency collection

These variables are defined in meta/classes/spdx-common.bbclass and
are now properly documented in the reference manual with detailed
descriptions and practical usage information.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 documentation/ref-manual/variables.rst | 186 ++++---------------------
 1 file changed, 30 insertions(+), 156 deletions(-)

Comments

Stefano Tondo Nov. 24, 2025, 9:42 p.m. UTC | #1
Hi Antonin,

Thank you very much for the review.

I've addressed all your points in v3:

1. Langdale testing claim - Moved to SPDX 2.2 context only, removed
untested claims for SPDX 3.0.1
2. JSON-LD terminology - Changed to "JSON" throughout for clarity
3. SPDX_INCLUDE_COMPILED_SOURCES size data - Removed untested claims, added
explanatory note
4. RST formatting - Converted all paragraphs to proper `.. note::` blocks
5. Hardcoded values - Replaced with generic examples and added link to
upstream SPDX license-list-data

I also researched compression and added verified instructions with zstd.
Testing showed 94-97% compression ratios (e.g., 262 MB SBOM → 8.9 MB),
which should be very helpful for users dealing with large SPDX 3.0.1 files.

I'll send v3 shortly.

Best regards,
Stefano

On Fri, Nov 14, 2025 at 4:21 PM Stefano Tondo via lists.yoctoproject.org
<stondo=gmail.com@lists.yoctoproject.org> wrote:

> From: Stefano Tondo <stefano.tondo.ext@siemens.com>
>
> Add comprehensive documentation for SPDX-related variables in the
> Yocto reference manual. This includes documenting previously
> undocumented variables and updating existing documentation with
> SPDX 3.0.1 specific information.
>
> New variables documented:
> - SPDX_LICENSES: Path to SPDX license identifier mapping file
> - SPDX_MULTILIB_SSTATE_ARCHS: Architecture list for dependency collection
> - SPDX_UUID_NAMESPACE: Namespace for UUID generation in SPDX documents
>
> Updated existing variables with SPDX 3.0.1 data:
> - SPDX_INCLUDE_SOURCES: Added concrete SPDX 3.0.1 size data
>   (~260 MB without sources, ~2.5-2.6 GB with sources)
> - SPDX_INCLUDE_COMPILED_SOURCES: Updated with SPDX 3.0.1 format information
>
> The size information is based on building core-image-minimal for
> qemux86-64 with SPDX 3.0.1 JSON-LD format. The uncompressed SBOM
> file is approximately 260 MB without sources and increases to
> ~2.5-2.6 GB when sources are included (approximately 10x increase).
>
> Note: SPDX 3.0.1 uses uncompressed JSON-LD format by default,
> unlike the tar.zst compression used in SPDX 2.2.
>
> These variables are defined in meta/classes/spdx-common.bbclass.
>
> Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
> ---
>
> Changes in v2:
> - Fixed unintended removal of non-SPDX variables (CCACHE_DISABLE,
>   IMAGE_EXTRA_PARTITION_FILES, REQUIRED_*_FEATURES)
> - Fixed unintended change to LAYERDEPENDS description
> - Only SPDX-related variables are now modified
>
>  documentation/ref-manual/variables.rst | 72 +++++++++++++++++++++++---
>  1 file changed, 66 insertions(+), 6 deletions(-)
>
> diff --git a/documentation/ref-manual/variables.rst
> b/documentation/ref-manual/variables.rst
> index a80ef364e..c098227f2 100644
> --- a/documentation/ref-manual/variables.rst
> +++ b/documentation/ref-manual/variables.rst
> @@ -8962,10 +8962,21 @@ system and gives an overview of their function and
> contents.
>
>           SPDX_INCLUDE_COMPILED_SOURCES = "1"
>
> -      According to our tests, building ``core-image-minimal`` for the
> -      ``qemux86-64`` machine, enabling this option compared with the
> -      :term:`SPDX_INCLUDE_SOURCES` reduces the size of the
> ``tmp/deploy/spdx``
> -      directory from 2GB to 1.6GB.
> +      According to our tests on release 4.1 "langdale" (SPDX 2.2 format),
> building
> +      ``core-image-minimal`` for the ``qemux86-64`` machine, enabling this
> +      option compared with the :term:`SPDX_INCLUDE_SOURCES` reduced the
> size
> +      of the ``tmp/deploy/spdx`` directory from 2GB to 1.6GB.
> +
> +      With SPDX 3.0.1 JSON-LD format, the uncompressed image SBOM file
> +      (``core-image-minimal-qemux86-64.rootfs.spdx.json``) is
> approximately
> +      **260 MB without sources** (``SPDX_INCLUDE_SOURCES = "0"``), and
> increases to
> +      **~2.5-2.6 GB when sources are included** (either
> ``SPDX_INCLUDE_SOURCES = "1"``
> +      or ``SPDX_INCLUDE_COMPILED_SOURCES = "1"``). This represents
> approximately
> +      a **10x size increase** when including source files.
> +
> +      Note: SPDX 3.0.1 JSON-LD files are not compressed by default,
> unlike the
> +      tar.zst format used in SPDX 2.2. Compression (e.g. zstd) can
> significantly
> +      reduce the file size.
>
>     :term:`SPDX_INCLUDE_SOURCES`
>        This option allows to add a description of the source files used to
> build
> @@ -8979,8 +8990,8 @@ system and gives an overview of their function and
> contents.
>
>           SPDX_INCLUDE_SOURCES = "1"
>
> -      According to our tests on release 4.1 "langdale", building
> -      ``core-image-minimal`` for the ``qemux86-64`` machine, enabling
> +      According to our tests on release 4.1 "langdale" (SPDX 2.2 format),
> +      building ``core-image-minimal`` for the ``qemux86-64`` machine,
> enabling
>        this option multiplied the total size of the ``tmp/deploy/spdx``
>        directory by a factor of 3  (+291 MiB for this image),
>        and the size of the ``IMAGE-MACHINE.spdx.tar.zst`` in
> @@ -8988,6 +8999,42 @@ system and gives an overview of their function and
> contents.
>        image), compared to just using the :ref:`ref-classes-create-spdx`
> class
>        with no option.
>
> +      With SPDX 3.0.1 JSON-LD format, the uncompressed image SBOM file
> +      (``core-image-minimal-qemux86-64.rootfs.spdx.json``) is
> approximately
> +      **260 MB without sources** (default: ``SPDX_INCLUDE_SOURCES =
> "0"``),
> +      and increases to **~2.5-2.6 GB when sources are included**
> +      (``SPDX_INCLUDE_SOURCES = "1"``). This represents approximately a
> +      **10x size increase** when including source files.
> +
> +      Note: SPDX 3.0.1 JSON-LD files are not compressed by default,
> unlike the
> +      tar.zst format used in SPDX 2.2. Compression (e.g. zstd) can
> significantly
> +      reduce the file size.
> +
> +   :term:`SPDX_LICENSES`
> +      Path to the JSON file containing SPDX license identifier mappings.
> This
> +      file maps common license names to official SPDX license identifiers
> used
> +      during SBOM generation.
> +
> +      The default value is::
> +
> +         SPDX_LICENSES = "${COREBASE}/meta/files/spdx-licenses.json"
> +
> +      You can override this variable to use a custom license mapping file
> if
> +      your organization uses different license naming conventions.
> +
> +   :term:`SPDX_MULTILIB_SSTATE_ARCHS`
> +      The list of sstate architectures to consider when collecting SPDX
> +      dependencies. This includes multilib architectures when multilib is
> +      enabled.
> +
> +      The default value is::
> +
> +         SPDX_MULTILIB_SSTATE_ARCHS = "${SSTATE_ARCHS}"
> +
> +      This variable is used internally by the SPDX generation classes to
> +      ensure all relevant dependencies are included in the SBOM,
> regardless
> +      of whether multilib is enabled or not.
> +
>     :term:`SPDX_NAMESPACE_PREFIX`
>        This option could be used in order to change the prefix of
> ``spdxDocument``
>        and the prefix of ``documentNamespace``. It is set by default to
> @@ -9015,6 +9062,19 @@ system and gives an overview of their function and
> contents.
>        this option is recommended if you want to inspect the SPDX
>        output files with a text editor.
>
> +   :term:`SPDX_UUID_NAMESPACE`
> +      The namespace used for generating UUIDs in SPDX documents. This
> should
> +      be a domain name or unique identifier for your organization to
> ensure
> +      globally unique SPDX IDs across different builds and organizations.
> +
> +      The default value is::
> +
> +         SPDX_UUID_NAMESPACE = "sbom.openembedded.org"
> +
> +      You can override this variable to use your organization's domain::
> +
> +         SPDX_UUID_NAMESPACE = "sbom.example.com"
> +
>     :term:`SPDXLICENSEMAP`
>        Maps commonly used license names to their SPDX counterparts found in
>        ``meta/files/common-licenses/``. For the default
> :term:`SPDXLICENSEMAP`
> --
> 2.51.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#8044):
> https://lists.yoctoproject.org/g/docs/message/8044
> Mute This Topic: https://lists.yoctoproject.org/mt/116292422/10120103
> Group Owner: docs+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [stondo@gmail.com
> ]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
diff mbox series

Patch

diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index a80ef364e..6a4472531 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -1526,11 +1526,6 @@  system and gives an overview of their function and contents.
    :term:`CC`
       The minimal command and arguments used to run the C compiler.
 
-   :term:`CCACHE_DISABLE`
-      When inheriting the :ref:`ref-classes-ccache` class, the
-      :term:`CCACHE_DISABLE` variable can be set to "1" in a recipe to disable
-      `Ccache` support. This is useful when the recipe is known to not support it.
-
    :term:`CCLD`
       The minimal command and arguments used to run the linker when the C
       compiler is being used as the linker.
@@ -3961,70 +3956,6 @@  system and gives an overview of their function and contents.
       material for Wic is located in the
       ":doc:`/ref-manual/kickstart`" chapter.
 
-   :term:`IMAGE_EXTRA_PARTITION_FILES`
-      A space-separated list of files installed into the extra partition(s)
-      when preparing an image using the Wic tool with the
-      ``extra_partition`` source plugin. By default,
-      the files are
-      installed under the same name as the source files. To change the
-      installed name, separate it from the original name with a semi-colon
-      (;). Source files need to be located in
-      :term:`DEPLOY_DIR_IMAGE`. Here is an
-      example::
-
-         IMAGE_EXTRA_PARTITION_FILES = "foobar file.conf;config"
-
-      In the above example, the file ``foobar`` is installed with its original name
-      ``foobar``, while the file ``file.conf`` is installed and renamed to ``config``.
-
-      Alternatively, source files can be picked up using a glob pattern.
-      However, hidden files are ignored, and the pattern is non-recursive
-      (subdirectories are ignored).
-      The destination file will have the same name as the base
-      name of the source file path. To install files into a renamed directory
-      within the target location, pass its name after a semi-colon (;).
-      Here are two examples::
-
-         IMAGE_EXTRA_PARTITION_FILES = "foo/*"
-         IMAGE_EXTRA_PARTITION_FILES = "foo/*;bar/"
-
-      The first line in this example
-      installs all files from ``foo`` directory
-      into the root of the target partition. The second line in this example installs
-      the same files into a ``bar`` directory within the target partition.
-      The ``bar/`` directory is automatically created if it does not exist.
-
-      You can also specify the target by label, UUID or partition name if multiple
-      extra partitions coexist. Let's take the following example. This would be
-      the WKS file for the image currently being built::
-
-         part --source extra_partition --fstype=ext4 --label foo
-         part --source extra_partition --fstype=ext4 --uuid e7d0824e-cda3-4bed-9f54-9ef5312d105d
-         part --source extra_partition --fstype=ext4 --part-name config
-
-      And the following configuration::
-
-         IMAGE_EXTRA_PARTITION_FILES_label-foo = "foo/*"
-         IMAGE_EXTRA_PARTITION_FILES_uuid-e7d0824e-cda3-4bed-9f54-9ef5312d105d = "foo/*;bar/"
-         IMAGE_EXTRA_PARTITION_FILES_part-name-config = "config"
-
-      Then:
-
-      -  The partition labeled "foo" would get all files from the ``foo``
-         directory.
-
-      -  The partition whose UUID is "e7d0824e-cda3-4bed-9f54-9ef5312d105d"
-         would get all files from the ``foo`` directory, installed into a
-         ``bar`` directory.
-
-      -  The partition named "config" would get the file ``config``.
-
-      You can find information on how to use the Wic tool in the
-      ":ref:`dev-manual/wic:creating partitioned images using wic`"
-      section of the Yocto Project Development Tasks Manual. Reference
-      material for Wic is located in the
-      ":doc:`/ref-manual/kickstart`" chapter.
-
    :term:`IMAGE_FEATURES`
       The primary list of features to include in an image. Typically, you
       configure this variable in an image recipe. Although you can use this
@@ -5523,7 +5454,7 @@  system and gives an overview of their function and contents.
       information on how this variable is used.
 
    :term:`LAYERDEPENDS`
-      Lists the layers, separated by spaces, on which this layer depends.
+      Lists the layers, separated by spaces, on which this recipe depends.
       Optionally, you can specify a specific layer version for a dependency
       by adding it to the end of the layer name. Here is an example::
 
@@ -7940,16 +7871,6 @@  system and gives an overview of their function and contents.
    :term:`REPODIR`
       See :term:`bitbake:REPODIR` in the BitBake manual.
 
-   :term:`REQUIRED_COMBINED_FEATURES`
-      When inheriting the :ref:`ref-classes-features_check` class, this variable
-      identifies combined features (the intersection of :term:`MACHINE_FEATURES`
-      and :term:`DISTRO_FEATURES`) that must exist in the current configuration
-      in order for the :term:`OpenEmbedded Build System` to build the recipe. In
-      other words, if the :term:`REQUIRED_COMBINED_FEATURES` variable lists a
-      feature that does not appear in :term:`COMBINED_FEATURES` within the
-      current configuration, then the recipe will be skipped, and if the build
-      system attempts to build the recipe then an error will be triggered.
-
    :term:`REQUIRED_DISTRO_FEATURES`
       When inheriting the :ref:`ref-classes-features_check`
       class, this variable identifies distribution features that must exist
@@ -7960,41 +7881,6 @@  system and gives an overview of their function and contents.
       the recipe will be skipped, and if the build system attempts to build
       the recipe then an error will be triggered.
 
-   :term:`REQUIRED_IMAGE_FEATURES`
-      When inheriting the :ref:`ref-classes-features_check` class, this variable
-      identifies image features that must exist in the current
-      configuration in order for the :term:`OpenEmbedded Build System` to build
-      the recipe. In other words, if the :term:`REQUIRED_IMAGE_FEATURES` variable
-      lists a feature that does not appear in :term:`IMAGE_FEATURES` within the
-      current configuration, then the recipe will be skipped, and if the build
-      system attempts to build the recipe then an error will be triggered.
-
-      Compared to other ``REQUIRED_*_FEATURES`` variables, the
-      :term:`REQUIRED_IMAGE_FEATURES` varible only targets image recipes, as the
-      :term:`IMAGE_FEATURES` variable is handled by the :ref:`ref-classes-core-image`
-      class). However, the :term:`REQUIRED_IMAGE_FEATURES` varible can also be
-      set from a :term:`Configuration File`, such as a distro
-      configuration file, if the list of required image features should apply to
-      all images using this :term:`DISTRO`.
-
-   :term:`REQUIRED_MACHINE_FEATURES`
-      When inheriting the :ref:`ref-classes-features_check` class, this variable
-      identifies :term:`MACHINE_FEATURES` that must exist in the current
-      configuration in order for the :term:`OpenEmbedded Build System` to build
-      the recipe. In other words, if the :term:`REQUIRED_MACHINE_FEATURES` variable
-      lists a feature that does not appear in :term:`MACHINE_FEATURES` within the
-      current configuration, then the recipe will be skipped, and if the build
-      system attempts to build the recipe then an error will be triggered.
-
-   :term:`REQUIRED_TUNE_FEATURES`
-      When inheriting the :ref:`ref-classes-features_check` class, this variable
-      identifies tune features that must exist in the current configuration in
-      order for the :term:`OpenEmbedded Build System` to build the recipe. In
-      other words, if the :term:`REQUIRED_TUNE_FEATURES` variable lists a
-      feature that does not appear in :term:`TUNE_FEATURES` within the current
-      configuration, then the recipe will be skipped, and if the build system
-      attempts to build the recipe then an error will be triggered.
-
    :term:`REQUIRED_VERSION`
       If there are multiple versions of a recipe available, this variable
       determines which version should be given preference.
@@ -8962,10 +8848,21 @@  system and gives an overview of their function and contents.
 
          SPDX_INCLUDE_COMPILED_SOURCES = "1"
 
-      According to our tests, building ``core-image-minimal`` for the
-      ``qemux86-64`` machine, enabling this option compared with the
-      :term:`SPDX_INCLUDE_SOURCES` reduces the size of the  ``tmp/deploy/spdx``
-      directory from 2GB to 1.6GB.
+      According to our tests on release 4.1 "langdale" (SPDX 2.2 format), building
+      ``core-image-minimal`` for the ``qemux86-64`` machine, enabling this
+      option compared with the :term:`SPDX_INCLUDE_SOURCES` reduced the size
+      of the ``tmp/deploy/spdx`` directory from 2GB to 1.6GB.
+
+      With SPDX 3.0.1 JSON-LD format, the uncompressed image SBOM file
+      (``core-image-minimal-qemux86-64.rootfs.spdx.json``) is approximately
+      **260 MB without sources** (``SPDX_INCLUDE_SOURCES = "0"``), and increases to
+      **~2.5-2.6 GB when sources are included** (either ``SPDX_INCLUDE_SOURCES = "1"``
+      or ``SPDX_INCLUDE_COMPILED_SOURCES = "1"``). This represents approximately
+      a **10x size increase** when including source files.
+
+      Note: SPDX 3.0.1 JSON-LD files are not compressed by default, unlike the
+      tar.zst format used in SPDX 2.2. Compression (e.g. zstd) can significantly
+      reduce the file size.
 
    :term:`SPDX_INCLUDE_SOURCES`
       This option allows to add a description of the source files used to build
@@ -8979,8 +8876,8 @@  system and gives an overview of their function and contents.
 
          SPDX_INCLUDE_SOURCES = "1"
 
-      According to our tests on release 4.1 "langdale", building
-      ``core-image-minimal`` for the ``qemux86-64`` machine, enabling
+      According to our tests on release 4.1 "langdale" (SPDX 2.2 format),
+      building ``core-image-minimal`` for the ``qemux86-64`` machine, enabling
       this option multiplied the total size of the ``tmp/deploy/spdx``
       directory by a factor of 3  (+291 MiB for this image),
       and the size of the ``IMAGE-MACHINE.spdx.tar.zst`` in
@@ -8988,6 +8885,17 @@  system and gives an overview of their function and contents.
       image), compared to just using the :ref:`ref-classes-create-spdx` class
       with no option.
 
+      With SPDX 3.0.1 JSON-LD format, the uncompressed image SBOM file
+      (``core-image-minimal-qemux86-64.rootfs.spdx.json``) is approximately
+      **260 MB without sources** (default: ``SPDX_INCLUDE_SOURCES = "0"``),
+      and increases to **~2.5-2.6 GB when sources are included**
+      (``SPDX_INCLUDE_SOURCES = "1"``). This represents approximately a
+      **10x size increase** when including source files.
+
+      Note: SPDX 3.0.1 JSON-LD files are not compressed by default, unlike the
+      tar.zst format used in SPDX 2.2. Compression (e.g. zstd) can significantly
+      reduce the file size.
+
    :term:`SPDX_NAMESPACE_PREFIX`
       This option could be used in order to change the prefix of ``spdxDocument``
       and the prefix of ``documentNamespace``. It is set by default to
@@ -10906,12 +10814,6 @@  system and gives an overview of their function and contents.
       Please see the "Selection of Processor Architecture and Board Type"
       section in the U-Boot README for valid values for this variable.
 
-   :term:`UBOOT_MAKE_OPTS`
-      The :term:`UBOOT_MAKE_OPTS` variable can be used to pass extra options to
-      ``make`` when U-Boot is configured and compiled.
-
-      See the :ref:`ref-classes-uboot-config` class for more information.
-
    :term:`UBOOT_MAKE_TARGET`
       Specifies the target called in the ``Makefile``. The default target
       is "all".
@@ -11034,22 +10936,6 @@  system and gives an overview of their function and contents.
       `Unified Kernel Image (UKI) <https://uapi-group.org/specifications/specs/unified_kernel_image/>`__.
       Defaults to ``ukify build``.
 
-   :term:`UNINATIVE_CHECKSUM`
-      When inheriting the :ref:`ref-classes-uninative` class, the
-      :term:`UNINATIVE_CHECKSUM` variable flags contain the checksums of the
-      uninative tarball as specified by the :term:`UNINATIVE_URL` variable.
-      There should be one checksum per tarballs published at
-      :term:`UNINATIVE_URL`, which match architectures. For example::
-
-         UNINATIVE_CHECKSUM[aarch64] ?= "812045d826b7fda88944055e8526b95a5a9440bfef608d5b53fd52faab49bf85"
-         UNINATIVE_CHECKSUM[i686] ?= "5cc28efd0c15a75de4bcb147c6cce65f1c1c9d442173a220f08427f40a3ffa09"
-         UNINATIVE_CHECKSUM[x86_64] ?= "4c03d1ed2b7b4e823aca4a1a23d8f2e322f1770fc10e859adcede5777aff4f3a"
-
-   :term:`UNINATIVE_URL`
-      When inheriting the :ref:`ref-classes-uninative` class, the
-      :term:`UNINATIVE_URL` variable contains the URL where the uninative
-      tarballs are published.
-
    :term:`UNKNOWN_CONFIGURE_OPT_IGNORE`
       Specifies a list of options that, if reported by the configure script
       as being invalid, should not generate a warning during the
@@ -11069,7 +10955,7 @@  system and gives an overview of their function and contents.
 
    :term:`UNPACKDIR`
       This variable, used by the :ref:`ref-classes-base` class,
-      specifies where fetched sources should be unpacked by the
+      specifies where fetches sources should be unpacked by the
       :ref:`ref-tasks-unpack` task.
 
    :term:`UPDATERCPN`
@@ -11145,18 +11031,6 @@  system and gives an overview of their function and contents.
       the Yocto Project Development Tasks Manual for information on how to
       use this variable.
 
-   :term:`USE_NLS`
-      Determine if language translations should be built for recipes that can
-      build them. This variable can be equal to:
-
-      -  ``yes``: translations are enabled.
-      -  ``no``: translation are disabled.
-
-      Recipes can use the value of this variable to enable language
-      translations in their build. Classes such as :ref:`ref-classes-gettext`
-      use the value of this variable to enable :wikipedia:`Gettext <Gettext>`
-      support.
-
    :term:`USE_VT`
       When using
       :ref:`SysVinit <dev-manual/new-recipe:enabling system services>`,