new file mode 100644
@@ -0,0 +1,672 @@
+.. SPDX-License-Identifier: CC-BY-2.5
+
+=============================================
+Setting Up The Environment With bitbake-setup
+=============================================
+
+|
+
+Setting up layers and configuring builds can be done through the
+``bitbake-setup`` tool. This tool acts as a top-level utility which can perform
+the following tasks:
+
+- Parse a JSON configuration file that describes what layers and which snippets
+ of configuration to use.
+
+- Clone the layers onto the versions specified in the configuration file.
+
+- Create and setup a build directory ready for building what is specified in
+ the configuration file.
+
+- Behave according to global or per-project settings.
+
+- Synchronize with upstream configuration changes.
+
+Quick Start
+===========
+
+#. ``bitbake-setup`` is part of the BitBake repository under
+ ``./bin/bitbake-setup``. To start, run:
+
+ .. code-block:: shell
+
+ $ ./bin/bitbake-setup init
+
+ This command will ask you to choose which configurations to use available as
+ part of the default BitBake :term:`generic configurations <generic
+ Configuration>`.
+
+ .. note::
+
+ These default configurations are located in ``./bitbake/default-registry/``.
+ See the :ref:`ref-bbsetup-section-config-reference` section to learn more
+ about ``bitbake-setup`` input configuration files.
+
+#. With the default choices, the ``bitbake-setup init`` command creates the
+ following directories::
+
+ ~/bitbake-builds/
+ └── poky-master-poky-distro_poky-machine_qemux86-64/
+ ├── build/
+ ├── config/
+ └── layers/
+
+ With:
+
+ - ``~/bitbake-build``: the :term:`Top Directory`, where ``bitbake-setup``
+ configures everything (and never writes outside of). This directory can be
+ configured with the :ref:`ref-bbsetup-setting-top-dir-prefix` and
+ :ref:`ref-bbsetup-setting-top-dir-name` settings.
+
+ - ``poky-master-poky-distro_poky-machine_qemux86-64``: a
+ :term:`Setup`, which is the result of the choices made during
+ the ``bitbake-setup init`` execution. The name of the directory will vary
+ depending on the choices.
+
+ - ``build/``: the :term:`BitBake Build` directory, where BitBake stores
+ its own configuration and outputs for the builds.
+
+ - ``config/``: a copy of the upstream :term:`generic configuration` used.
+
+ - ``layers/``: the directory where :ref:`:ref:`layers
+ <bitbake-user-manual/bitbake-user-manual-intro:Layers>` are stored.
+
+#. Source the ``init-build-env`` file present in the :term:`BitBake Build`
+ directory:
+
+ .. code-block:: shell
+
+ $ source ./poky-master-poky-distro_poky-machine_qemux86-64/build/init-build-env
+
+ This command will prepare your current shell with the BitBake environment.
+
+#. You can then start running ``bitbake`` in the current shell. For more information
+ on how to use ``bitbake``, see the :doc:`/bitbake-user-manual/bitbake-user-manual-execution`
+ section of this manual.
+
+Terminology
+===========
+
+The ``bitbake-setup`` tool revolves around some common terms we define in this
+section.
+
+The following terms correspond to the hierarchy of directories that
+``bitbake-setup`` creates:
+
+.. glossary::
+ :term:`Top Directory`
+ The top directory is the place under which ``bitbake-setup`` reads and
+ writes everything. ``bitbake-setup`` makes a promise to not touch anything
+ outside of that directory. The location of this directory can be changed
+ with the :ref:`ref-bbsetup-setting-top-dir-prefix` and
+ :ref:`ref-bbsetup-setting-top-dir-name` settings.
+
+ The top directory contains one or more :term:`Setup`.
+
+ :term:`Setup`
+ A Setup is the result of the :ref:`ref-bbsetup-command-init`
+ command. It maps one to one to the JSON data it was constructed from,
+ and is the combination of a :term:`Generic Configuration` and choices made
+ during the ``init`` command.
+
+ It contains at least a :ref:`layer
+ <bitbake-user-manual/bitbake-user-manual-intro:Layers>` checkout, a
+ :term:`BitBake Build`, and a ``config/`` directory holding the upstream
+ :term:`Generic Configuration` it originates from.
+
+ :term:`BitBake Build`
+ A BitBake Build is a sub-tree inside a :term:`Setup` that BitBake itself
+ operates on. The files in the ``conf/`` directory of a :term:`BitBake
+ Build` constitute the :ref:`BitBake configuration
+ <bitbake-user-manual/bitbake-user-manual-intro:Configuration Files>`.
+
+To illustrate, this will translate to the following hierarchy::
+
+ Top Directory
+ ├── Setup 1
+ │ ├── build
+ │ └── layers
+ ├── Setup 2
+ │ ├── build
+ │ └── layers
+ ...
+
+The following are involved to create these directories and their contents:
+
+.. glossary::
+ :term:`Generic Configuration`
+ A Generic Configuration is a file in JSON format containing a template to
+ create a :term:`Setup`. These files are used during the :ref:`ref-bbsetup-command-init`
+ command as a starting point to configure the :term:`Setup`. When the
+ command runs, the user may be prompted with choices to further specify the
+ :term:`Setup` to create.
+
+ :term:`Generic Configuration` files are stored in :term:`registries
+ <Registry>`, and can be listed with the :ref:`ref-bbsetup-command-init`
+ command.
+
+ .. note::
+
+ The default :term:`Generic Configurations <Generic Configuration>` are
+ located in the BitBake repository in a local registry. the
+ ``default-registry/`` directory. This can be modified with the
+ :ref:`ref-bbsetup-setting-registry` setting.
+
+ :ref:`ref-bbsetup-command-status` will tell if a :term:`Setup`
+ is in sync with the :term:`Generic Configuration` it was constructed from
+ (typically: layer updates).
+
+ :ref:`ref-bbsetup-command-update` will bring a :term:`Setup`
+ in sync with its :term:`Generic Configuration`.
+
+ :term:`Registry`
+ A configuration registry is a place where one or more :term:`Generic
+ Configurations <Generic Configuration>` are stored.
+
+ A registry should have the following structure::
+
+ registry/
+ └── configurations/
+ ├── configuration1.conf.json
+ └── configuration2.conf.json
+
+ The registry location is configured through the
+ :ref:`ref-bbsetup-setting-registry` setting. This location can the URL to
+ a Git repository, a local directory, or any URI supported by the BitBake
+ fetcher (see the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching`
+ section for more information on fetchers).
+
+ :term:`Settings`
+ Settings are operational parameters that are global to all builds under a
+ :term:`Top Directory`, stored in a ``settings.conf`` file. For example,
+ this could the location of the configuration registry, or where the
+ BitBake fetcher should store the downloads.
+
+ There are also global settings, common to all top directories that are
+ stored in ``~/.config/bitbake-setup/settings.conf``.
+
+ See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings`
+ to see the supported settings and where to store them.
+
+ :term:`Source Override`
+ A source override is a JSON file that can be used to modify revisions and
+ origins of layers that need to be checked out into a :term:`Setup` (for
+ example when the master branches need to be changed to master-next for
+ the purpose of testing). Such a file is specified with a command-line
+ option to :ref:`ref-bbsetup-command-init`.
+
+ See the :ref:`ref-bbsetup-source-override` section for more information.
+
+The ``bitbake-setup`` command
+=============================
+
+The ``bitbake-setup`` program has general options and sub-commands. These can be
+obtained using ``bitbake-setup --help``.
+
+The general options, common to all commands, are:
+
+- ``-h`` or ``--help``: Show the help message and exit.
+- ``-d`` or ``--debug``: Enable debug outputs.
+- ``-q`` or ``--quiet``: Print only errors.
+- ``--color``: Colorize output (where COLOR is auto, always, never).
+- ``--no-network``: Do not check whether configuration repositories and layer
+ repositories have been updated; use only the local cache.
+- ``--global-settings``: Path to the global settings file.
+- ``--setting``: Modify a setting (for this bitbake-setup invocation only)
+ For example ``--setting default top-dir-prefix /path/to/top/dir``.
+
+.. _ref-bbsetup-command-init:
+
+``bitbake-setup init``
+----------------------
+
+The ``bitbake-setup init`` sub-command helps initializing a :term:`Setup`.
+
+This command can be run without any arguments to prompt the user with
+configuration options to choose from. These configuration options are taken from
+the input :term:`Generic Configuration` files in the :term:`registry`.
+
+.. note::
+
+ The registry location can be set with the :ref:`ref-bbsetup-setting-registry`
+ setting and the :ref:`ref-bbsetup-command-settings` command.
+
+The choices made during the bare ``bitbake-setup init`` command can also be
+passed directly on the command-line, for example::
+
+ bitbake-setup init poky-master poky distro/poky-tiny
+
+``bitbake-setup`` will stop and ask to make a choice if the above command does
+not contain all of the required configurations to complete the sequence of
+choices.
+
+In addition, the command can take the following arguments:
+
+- ``--non-interactive``: can be used to create :term:`Setups <Setup>`
+ without interactions from the user. The command will fail if not all the
+ required choices are provided in the command.
+
+- ``--source-overrides``: can be used to pass one or more
+ :ref:`source override <ref-bbsetup-source-override>`.
+
+- ``--build-dir-name``: can be used to configure the name of the :term:`Setup`
+ directory.
+
+- ``--skip-selection``: can be used to skip some of the choices
+ (which may result in an incomplete :term:`Setup`!)
+
+.. _ref-bbsetup-command-list:
+
+``bitbake-setup list``
+----------------------
+
+The ``bitbake-setup list`` sub-command lists the available :term:`generic
+configurations <generic configuration>` in the current :term:`registry`.
+
+In addition, the command can take the following arguments:
+
+- ``--with-expired``: list the expired configuration.
+
+- ``--write-json``: write the configurations into a JSON file so they can be
+ programmatically processed.
+
+.. _ref-bbsetup-command-status:
+
+``bitbake-setup status``
+------------------------
+
+The ``bitbake-setup status`` sub-command shows the status of a
+:term:`Setup`. Any differences between the local copy of the :term:`generic
+configuration` and the upstream one are printed on the console.
+
+If the BitBake environment is sourced and ready to build, the ``bitbake-setup
+status`` command (without any arguments) will show the status of the current
+:term:`Setup`.
+
+In addition, the command can take the following arguments:
+
+- ``--build-dir``: path to the :term:`Setup` to check to status for. Not
+ required if :term:`BBPATH` is already configured.
+
+.. _ref-bbsetup-command-update:
+
+``bitbake-setup update``
+------------------------
+
+The ``bitbake-setup update`` sub-command updates a :term:`Setup` to sync with
+the latest changes from a :term:`generic configuration` it was constructed from.
+The :ref:`ref-bbsetup-command-status` command can be used to show the current
+status of the :term:`Setup`.
+
+In addition, the command can take the following arguments:
+
+- ``--build-dir``: path to the :term:`Setup` to check to status for. Not
+ required if :term:`BBPATH` is already configured.
+
+.. _ref-bbsetup-command-install-buildtools:
+
+``bitbake-setup install-buildtools``
+------------------------------------
+
+The ``bitbake-setup install-buildtools`` sub-command installs and extracts a
+buildtools tarball into the specified :term:`Setup`. If the BitBake environment
+is sourced and ready to build, the ``bitbake-setup status`` command (without any
+arguments) will show the status of the current :term:`Setup`.
+
+After completion, help is printed to help the user on how to use the extracted
+tarball.
+
+In addition, the command can take the following arguments:
+
+- ``--force``: force the re-installation of the tarball.
+
+- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
+ required if :term:`BBPATH` is already configured.
+
+.. _ref-bbsetup-command-settings:
+
+``bitbake-setup settings``
+--------------------------
+
+The ``bitbake-setup settings`` sub-command helps modifying the settings of
+``bitbake-setup``. This sub-command has sub-commands itself:
+
+- ``bitbake-setup settings list`` lists the current settings and their value.
+- ``bitbake-setup settings set`` sets a setting.
+- ``bitbake-setup settings unset`` removes a setting.
+
+Settings must be set with a section and a value, for example::
+
+ bitbake-setup setting set default top-dir-name bitbake-builds
+
+Will set the value of ``top-dir-name`` in the ``default`` section to
+"bitbake-builds".
+
+In addition, the command can take the following arguments:
+
+- ``--global``: write to the global settings
+ (``~/.config/bitbake-setup/settings.conf``) instead of the :term:`top
+ directory` settings.
+
+See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings`
+to see the supported settings.
+
+.. note::
+
+ You can set custom new section to ``bitbake-setup`` settings, but the
+ supported setting listed in the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings`
+ section are only affected when set in the ``default`` section.
+
+Settings
+========
+
+The :term:`settings` allow configuring ``bitbake-setup``. Settings are stored in
+a file named ``settings.conf``, in :wikipedia:`INI <INI_file>` format.
+
+There are multiple locations for storing settings. Settings in different
+locations can override each other, but the final value of a setting is computed
+from reading the files in this order:
+
+#. Global settings file: ``~/.config/bitbake-setup/settings.conf``.
+
+#. Local settings file, taken from a ``settings.conf`` file in the :term:`Top
+ Directory`.
+
+#. Command-line settings, depending on the command and settings name.
+
+A valid settings file would for example be:
+
+.. code-block:: ini
+
+ [default]
+ top-dir-prefix = /path/to/workspace
+ top-dir-name = bitbake-builds
+ registry = /path/to/bitbake/default-registry
+ dl-dir = /path/to/bitbake-setup-downloads
+
+Settings and their values can be listed and modified with the ``bitbake-setup
+settings`` command. See the :ref:`ref-bbsetup-command-settings` section for
+more information.
+
+Below are the available settings.
+
+.. _ref-bbsetup-setting-top-dir-prefix:
+
+``top-dir-prefix``
+------------------
+
+The :ref:`ref-bbsetup-setting-top-dir-prefix` setting helps configuring the
+leftmost part of the path to the :term:`Top Directory`.
+
+For example, with:
+
+.. code-block:: ini
+
+ [default]
+ top-dir-prefix = /path/to/workspace
+
+The :term:`top directory` would be ``/path/to/workspace/<top-dir-name>`` with
+the ``<top-dir-name>`` corresponding to the :ref:`ref-bbsetup-setting-top-dir-name`
+setting.
+
+.. _ref-bbsetup-setting-top-dir-name:
+
+``top-dir-name``
+----------------
+
+The :ref:`ref-bbsetup-setting-top-dir-name` setting helps configuring the
+rightmost part of the path to the :term:`Top Directory`.
+
+For example, with:
+
+.. code-block:: ini
+
+ [default]
+ top-dir-name = builds
+
+The :term:`top directory` would be ``<top-dir-prefix>/builds`` with
+the ``<top-dir-prefix>`` corresponding to the :ref:`ref-bbsetup-setting-top-dir-prefix`
+setting.
+
+.. _ref-bbsetup-setting-registry:
+
+``registry``
+------------
+
+The :ref:`ref-bbsetup-setting-registry` setting sets the URI location of the
+registry. This URI can be any URI supported by the BitBake fetcher.
+
+A local registry would be configured as follows:
+
+.. code-block:: ini
+
+ [default]
+ registry = /path/to/registry
+
+When using another fetcher, it must be specified in the URI scheme. For example:
+
+.. code-block:: ini
+
+ [default]
+ registry = git://example.com/bitbake-setup-configurations;protocol=https;branch=master;rev=master
+
+This would fetch the remote configurations from a remote Git remote repository,
+on the ``master`` branch.
+
+See the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` section for more
+information on BitBake fetchers.
+
+.. _ref-bbsetup-setting-dl-dir:
+
+``dl-dir``
+----------
+
+The :ref:`ref-bbsetup-setting-dl-dir` setting set the location of sources
+``bitbake-setup`` will download when configuring a :term:`Setup`.
+
+This includes the registry (if the registry is stored remotely) or layers.
+
+.. _ref-bbsetup-section-config-reference:
+
+Generic Configuration Files Reference
+=====================================
+
+:term:`Generic Configurations <Generic Configuration>` are the input files given
+to ``bitbake-setup`` to configure :term:`Setups <Setup>`.
+
+These files are written in the JSON file format and are stored in a
+:term:`Registry`. They contains three main sections:
+
+- ``version`` (**required**): version of the configuration file.
+
+ Example:
+
+ .. code-block:: json
+ :force:
+
+ {
+ "version": "1.0"
+ }
+
+- ``description`` (**required**): the description of the configuration.
+
+ Example:
+
+ .. code-block:: json
+ :force:
+
+ {
+ "description": "OpenEmbedded - 'nodistro' basic configuration"
+ }
+
+- ``sources``: one or more Git repository to fetch.
+
+ Example:
+
+ .. code-block:: json
+ :force:
+
+ {
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "git://git.openembedded.org/bitbake;protocol=https"
+ }
+ },
+ "branch": "master",
+ "rev": "master"
+ },
+ "path": "bitbake"
+ }
+ }
+ }
+
+ Sources can be specified with the following options:
+
+ - ``uri`` (**required**): a URI that follows the BitBake Git fetcher syntax.
+ See the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` section
+ for more information on the Git fetcher.
+
+ - ``rev`` (**required**): the revision to checkout. Can be the name of the
+ branch to checkout on the latest revision of the specified ``branch``.
+
+ - ``branch`` (**required**): the Git branch, used to check that the
+ specified ``rev`` is indeed on that branch.
+
+ - ``path`` is where the source is extracted.
+
+- ``bitbake-setup`` (**required**): contains a list of configurations.
+
+ Example:
+
+ .. code-block:: json
+
+ {
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"],
+ "bb-env-passthrough-additions": ["DL_DIR","SSTATE_DIR"],
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64", "machine/genericarm64", "machine/genericx86-64"]
+ },
+ "distro": {
+ "description": "Distribution configuration variants",
+ "options" : ["distro/poky", "distro/poky-altcfg", "distro/poky-tiny"]
+ }
+ },
+ "configurations": [
+ {
+ "name": "poky",
+ "description": "Poky - The Yocto Project testing distribution"
+ },
+ {
+ "name": "poky-with-sstate",
+ "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.",
+ "oe-fragments": ["core/yocto/sstate-mirror-cdn"]
+ }
+ ]
+ }
+ ]
+ }
+ }
+
+ Configurations can be specified with the following options:
+
+ - ``name`` (**required**): the name of this configuration snippet. This is
+ what is prompted during the :ref:`ref-bbsetup-command-init` command
+ execution.
+
+ - ``description`` (**required**): the description of this configuration
+ snippet. This is what is prompted during the
+ :ref:`ref-bbsetup-command-init` command execution.
+
+ - ``configurations``: Configurations can contain as many sub-configurations
+ as needed. This will create more choices when running the
+ :ref:`ref-bbsetup-command-init` command.
+
+ - ``bb-env-passthrough-additions`` (*optional*): List of environment
+ variables to include in :term:`BB_ENV_PASSTHROUGH_ADDITIONS`.
+
+ - ``bb-layers`` (*optional*): Layers to add to the ``bblayers.conf`` file.
+ This cannot be used in conjunction with the ``oe-template`` option.
+
+ - ``oe-template`` (*optional*, OpenEmbedded specific): OpenEmbedded template
+ to use. This cannot be used in conjunction with the ``bb-layers`` option.
+
+ See https://docs.yoctoproject.org/dev-manual/custom-template-configuration-directory.html
+ for more information of OpenEmbedded templates.
+
+ - ``oe-fragments-one-of`` (*optional*, OpenEmbedded specific): the OpenEmbedded
+ fragments to select as part of the build.
+
+ This will trigger choices to make during the
+ :ref:`ref-bbsetup-command-init` command execution.
+
+ See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for
+ more information of OpenEmbedded configuration fragments.
+
+ - ``oe-fragments`` (*optional*, OpenEmbedded specific): fragments to select
+ as part of the build.
+
+ See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for
+ more information of OpenEmbedded configuration fragments.
+
+Examples
+--------
+
+OpenEmbedded "nodistro" configuration for master branches:
+
+.. literalinclude:: ../../default-registry/configurations/oe-nodistro.conf.json
+ :language: json
+
+Poky distribution configuration for master branches:
+
+.. literalinclude:: ../../default-registry/configurations/poky-master.conf.json
+ :language: json
+
+.. _ref-bbsetup-source-override:
+
+Source Overrides
+================
+
+A source override is a file that can be used to override some of the
+sources definition of a :term:`Generic Configuration` file.
+
+These files are written in the JSON file format and are optionally passed to the
+``--source-override`` argument of the :ref:`ref-bbsetup-command-init` command.
+The ``--source-override`` option can be passed multiple times, in which case the
+overrides are applied in the order specified in the command-line.
+
+Here is an example source override:
+
+.. code-block:: json
+
+ {
+ "description": "Source override file",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "branch": "master-next",
+ "remotes": {
+ "origin": {
+ "uri": "ssh://git@push.openembedded.org/bitbake"
+ }
+ },
+ "rev": "acf5b02c5aaae1116d33b4e08b2ad7c27d9b94ab"
+ }
+ }
+ },
+ "version": "1.0"
+ }
+
+- The ``version`` parameter contains the version of the used configuration, and
+ should match the one of the :term:`Generic Configuration` file in use.
+
+- The ``sources`` section contains the same options as the ``sources`` option
+ of a :term:`Generic Configuration` file. See the
+ :ref:`ref-bbsetup-section-config-reference` section for more information.
@@ -12,6 +12,7 @@ BitBake User Manual
bitbake-user-manual/bitbake-user-manual-intro
bitbake-user-manual/bitbake-user-manual-execution
+ bitbake-user-manual/bitbake-user-manual-environment-setup
bitbake-user-manual/bitbake-user-manual-metadata
bitbake-user-manual/bitbake-user-manual-ref-variables-context
bitbake-user-manual/bitbake-user-manual-fetching
Add documentation for bitbake-setup. This is split in 6 sections, including a quick start guide, terminology and reference. Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> --- .../bitbake-user-manual-environment-setup.rst | 672 +++++++++++++++++++++ doc/index.rst | 1 + 2 files changed, 673 insertions(+)