| Message ID | 20250819213530.3616042-1-tim.orling@konsulko.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [RFC,v2] brief-yoctoprojectqs: add bitbake-setup info | expand |
Thanks Tim, this is basically fine, but has to be held until bitbake-setup actually lands (and then adjusted to match that version). I also wonder if the documentation needs to show exactly what the tool outputs, as that is prone to becoming out of sync, and the users will see it for themselves anyway. --top-dir value can also be set from an environment variable, so you don't have to pass it in every time (and end up creating bogus top dir if forgetting to do so). Alex On Tue, 19 Aug 2025 at 23:35, Tim Orling via lists.yoctoproject.org <tim.orling=konsulko.com@lists.yoctoproject.org> wrote: > > From: Tim Orling <tim.orling@konsulko.com> > > Largely replace the original Poky instructions with > example instructions to use bitbake-setup. > > Introduce the concept of OE_FRAGMENTS and how it is > used by bitbake-setup to setup SSTATE_MIRRORS, etc. > > Signed-off-by: Tim Orling <tim.orling@konsulko.com> > --- > Changes in v2: > - fix typos/grammar errors > - use commas instead of en dash > > documentation/brief-yoctoprojectqs/index.rst | 221 ++++++++++--------- > 1 file changed, 121 insertions(+), 100 deletions(-) > > diff --git a/documentation/brief-yoctoprojectqs/index.rst b/documentation/brief-yoctoprojectqs/index.rst > index 4a6f13d7e..8ef1f5bf5 100644 > --- a/documentation/brief-yoctoprojectqs/index.rst > +++ b/documentation/brief-yoctoprojectqs/index.rst > @@ -87,75 +87,112 @@ distribution: > see the :ref:`ref-manual/system-requirements:required packages for the build host` > section in the Yocto Project Reference Manual. > > -Use Git to Clone Poky > -===================== > +Use Git to Clone bitbake-setup > +============================== > > Once you complete the setup instructions for your machine, you need to > -get a copy of the Poky repository on your build host. Use the following > -commands to clone the Poky repository. > +get a copy of the bitbake-setup tool to setup the Poky reference > +distribution on your build host. Use the following commands to clone > +the bitbake repository. > > .. code-block:: shell > > - $ git clone git://git.yoctoproject.org/poky > - Cloning into 'poky'... > - remote: Counting > - objects: 432160, done. remote: Compressing objects: 100% > - (102056/102056), done. remote: Total 432160 (delta 323116), reused > - 432037 (delta 323000) Receiving objects: 100% (432160/432160), 153.81 MiB | 8.54 MiB/s, done. > - Resolving deltas: 100% (323116/323116), done. > - Checking connectivity... done. > + $ git clone git://git.openembedded.org/bitbake bitbake-setup > + Cloning into 'bitbake-setup'... > + remote: Enumerating objects: 68454, done. > + remote: Counting objects: 100% (49/49), done. > + remote: Compressing objects: 100% (44/44), done. > + remote: Total 68454 (delta 31), reused 5 (delta 5), pack-reused 68405 (from 1) > + Receiving objects: 100% (68454/68454), 14.17 MiB | 10.56 MiB/s, done. > + Resolving deltas: 100% (51776/51776), done. > > -Go to :yocto_wiki:`Releases wiki page </Releases>`, and choose a release > -codename (such as ``&DISTRO_NAME_NO_CAP;``), corresponding to either the > -latest stable release or a Long Term Support release. > +Setup a build environment with the following command: > > -Then move to the ``poky`` directory and take a look at existing branches: > +.. code-block:: shell > + > + $ ./bitbake-setup/bin/bitbake-setup init > + > +By default, this will setup in $HOME/bitbake-builds. If you prefer to > +setup your builds in a different directory, for example the current > +directory, you can instead use the ``--top-dir`` argument: > > .. code-block:: shell > > - $ cd poky > - $ git branch -a > - . > - . > - . > - remotes/origin/HEAD -> origin/master > - remotes/origin/dunfell > - remotes/origin/dunfell-next > - . > - . > - . > - remotes/origin/gatesgarth > - remotes/origin/gatesgarth-next > - . > - . > - . > - remotes/origin/master > - remotes/origin/master-next > - . > - . > - . > - > - > -For this example, check out the ``&DISTRO_NAME_NO_CAP;`` branch based on the > -``&DISTRO_NAME;`` release: > + $ ./bitbake-setup/bin/bitbake-setup init --top-dir $PWD > + > +`bitbake-setup init` is an interactive program by default and will ask > +you to make some decisions. Depending on your answers, the choices may > +differ from the examples below. > + > +#. Choose a configuration (for example, ``yocto-master-options``): > + > + .. code-block:: shell > + > + Available configurations: > + 0. yocto-master-nested-configs (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 (defined with nested configurations) > + 1. yocto-master-options (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined with options) > + > + Please select one of the above configurations by its number: > + 1 > + > +#. Choose a target machine (for example, ``qemux86-64``): > + > + .. code-block:: shell > + > + Selecting the only available bitbake configuration poky > + Available target machines: > + 0. machine/qemux86-64 > + 1. machine/qemuarm64 > + 2. machine/qemuriscv64 > + Please select one of the above options by its number: > + 0 > + > +#. Choose a distribution (for example, ``poky``): > + > + .. code-block:: shell > + > + Available distributions: > + 0. distro/poky > + 1. distro/poky-altcfg > + 2. distro/poky-tiny > + Please select one of the above options by its number: > + 0 > + Run 'bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64' to select this configuration non-interactively. > + Initializing a build in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64 > + Fetching layer/tool repository bitbake into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/bitbake > + Fetching layer/tool repository openembedded-core into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/openembedded-core > + Fetching layer/tool repository meta-yocto into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/meta-yocto > + Fetching layer/tool repository yocto-docs into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/yocto-docs > + ============================== > + Setting up bitbake configuration in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build > + This bitbake configuration provides: Poky reference distro build > + Usage instructions and additional information are in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README > + > +If you prefer to run non-interactively, you could use a command > +like the following: > > .. code-block:: shell > > - $ git checkout -t origin/&DISTRO_NAME_NO_CAP; -b my-&DISTRO_NAME_NO_CAP; > - Branch 'my-&DISTRO_NAME_NO_CAP;' set up to track remote branch '&DISTRO_NAME_NO_CAP;' from 'origin'. > - Switched to a new branch 'my-&DISTRO_NAME_NO_CAP;' > + $ bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64 > > -The previous Git checkout command creates a local branch named > -``my-&DISTRO_NAME_NO_CAP;``. The files available to you in that branch > -exactly match the repository's files in the ``&DISTRO_NAME_NO_CAP;`` > -release branch. > +Among other things, the script creates the :term:`Build Directory`, which is > +``build`` in this case and is located in the "top-dir" directory. The script > +also clones the layers needed to build the Poky reference distribution, in the > +``layers`` subdirectory. After the script runs, your current working directory > +is set to the :term:`Build Directory`. Later, when the build completes, the > +:term:`Build Directory` contains all the files created during the build. > > -Note that you can regularly type the following command in the same directory > -to keep your local files in sync with the release branch: > + > +Information about the build environment can be found in ``/path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README`` which should look something like the following: > > .. code-block:: shell > > - $ git pull > + Poky reference distro build > + > + Additional information is in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-summary.txt and /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-notes.txt > + > + Source the environment using '. /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env' to run builds from the command line. > + The bitbake configuration files (local.conf, bblayers.conf and more) can be found in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf > > For more options and information about accessing Yocto Project related > repositories, see the > @@ -182,61 +219,47 @@ an entire Linux distribution, including the toolchain, from source. > ":yocto_wiki:`Working Behind a Network Proxy </Working_Behind_a_Network_Proxy>`" > page of the Yocto Project Wiki. > > -#. **Initialize the Build Environment:** From within the ``poky`` > - directory, run the :ref:`ref-manual/structure:``oe-init-build-env``` > - environment > - setup script to define Yocto Project's build environment on your > - build host. > +#. **Initialize the Build Environment:** > + Run the ``init-build-env`` environment setup script within the build directory > + to define Yocto Project's build environment on your build host. > > .. code-block:: shell > > - $ cd poky > - $ source oe-init-build-env > - You had no conf/local.conf file. This configuration file has therefore been > - created for you with some default values. You may wish to edit it to, for > - example, select a different MACHINE (target hardware). See conf/local.conf > - for more information as common configuration options are commented. > - > - You had no conf/bblayers.conf file. This configuration file has therefore > - been created for you with some default values. To add additional metadata > - layers into your configuration please add entries to conf/bblayers.conf. > - > - The Yocto Project has extensive documentation about OE including a reference > - manual which can be found at: > - https://docs.yoctoproject.org > + $ source /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env > + Poky reference distro build > > - For more information about OpenEmbedded see their website: > - https://www.openembedded.org/ > +#. **Examine Your Auto Configuration File:** When you set up the build > + environment, an auto configuration file named ``auto.conf`` becomes > + available in aa ``conf`` subdirectory of the :term:`Build Directory`. For this > + example, the defaults are set to include three ``OE_FRAGMENTS``: > + ``core/yocto/sstate-mirror-cdn``, ``machine/qemux86-64`` and ``distro/poky``. > + These set up the environment similar to what was previously in the local > + configuration file named ``local.conf`` which is now largely empty. > + For more information on fragments, see :ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` directive` in the Bitbake User Manual. > > - ### Shell environment set up for builds. ### > + ``core/yocto/sstate-mirror-cdn`` sets up :term:`BB_HASHSERVE_UPSTREAM` and > + :term:`SSTATE_MIRRORS`. The definition can be found in > + ``../layers/openembedded-core/meta/conf/fragments/yocto/sstate-mirror-cdn.conf``. > > - You can now run 'bitbake <target>' > - > - Common targets are: > - core-image-minimal > - core-image-full-cmdline > - core-image-sato > - core-image-weston > - meta-toolchain > - meta-ide-support > - > - You can also run generated QEMU images with a command like 'runqemu qemux86-64' > + .. code-block:: shell > > - Other commonly useful commands are: > - - 'devtool' and 'recipetool' handle common recipe tasks > - - 'bitbake-layers' handles common layer tasks > - - 'oe-pkgdata-util' handles common target package tasks > + BB_CONF_FRAGMENT_SUMMARY = "Use prebuilt sstate artifacts for standard Yocto build configurations." > + BB_CONF_FRAGMENT_DESCRIPTION = "The Yocto Project has prebuilt artefacts available for standard build configurations. \ > + This fragment enables their use. This will mean the build will query the \ > + the network to check for artefacts at the start of builds, which does slow it down \ > + initially but it will then speed up the builds by not having to build things if they are \ > + present in the cache. It assumes you can download something faster than you can build it \ > + which will depend on your network. \ > + Note: For this to work you also need hash-equivalence passthrough to the matching server \ > + " > > - Among other things, the script creates the :term:`Build Directory`, which is > - ``build`` in this case and is located in the :term:`Source Directory`. After > - the script runs, your current working directory is set to the > - :term:`Build Directory`. Later, when the build completes, the > - :term:`Build Directory` contains all the files created during the build. > + BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws' > + SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > > #. **Examine Your Local Configuration File:** When you set up the build > environment, a local configuration file named ``local.conf`` becomes > available in a ``conf`` subdirectory of the :term:`Build Directory`. For this > - example, the defaults are set to build for a ``qemux86`` target, > + example, the defaults are set to build for a ``qemux86-64`` target, > which is suitable for emulation. The package manager used is set to > the RPM package manager. > > @@ -249,13 +272,11 @@ an entire Linux distribution, including the toolchain, from source. > This is relevant only when your network and the server that you use > can download these artifacts faster than you would be able to build them. > > - To use such mirrors, uncomment the below lines in your ``conf/local.conf`` > - file in the :term:`Build Directory`:: > + To use such mirrors, ``bitbake-setup`` includes the following in your > + ``conf/auto.conf`` file (or you can add it to ``conf/local.conf``) in > + the :term:`Build Directory`:: > > - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" > - SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > - BB_HASHSERVE = "auto" > - BB_SIGNATURE_HANDLER = "OEEquivHash" > + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" > > The hash equivalence server needs the websockets python module version 9.1 > or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream > -- > 2.43.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#7457): https://lists.yoctoproject.org/g/docs/message/7457 > Mute This Topic: https://lists.yoctoproject.org/mt/114789217/1686489 > Group Owner: docs+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Wed, Aug 20, 2025 at 12:46 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > Thanks Tim, this is basically fine, but has to be held until > bitbake-setup actually lands (and then adjusted to match that > version). > Absolutely, this is why I put it out as an RFC. Plus I think (see the next paragraph) there needs to be some more discussion/thought about the documentation. We need to add more documentation about OE fragments and more about bitbake-setup configuration, etc. > > I also wonder if the documentation needs to show exactly what the tool > outputs, as that is prone to becoming out of sync, and the users will > see it for themselves anyway. > I definitely agree and was on the fence about that. I was mostly following the pattern that was already in the quick start guide. Plus links to metadata don't exist yet, so that wasn't an option either. We should continue to discuss what will work best for the quick start guide and what else is needed elsewhere in the reference manual, etc. > > --top-dir value can also be set from an environment variable, so you > don't have to pass it in every time (and end up creating bogus top dir > if forgetting to do so). > Good point. Perhaps I can document that in v3 instead of showing the command line. I look forward to more discussion about all of this. > Alex > > On Tue, 19 Aug 2025 at 23:35, Tim Orling via lists.yoctoproject.org > <tim.orling=konsulko.com@lists.yoctoproject.org> wrote: > > > > From: Tim Orling <tim.orling@konsulko.com> > > > > Largely replace the original Poky instructions with > > example instructions to use bitbake-setup. > > > > Introduce the concept of OE_FRAGMENTS and how it is > > used by bitbake-setup to setup SSTATE_MIRRORS, etc. > > > > Signed-off-by: Tim Orling <tim.orling@konsulko.com> > > --- > > Changes in v2: > > - fix typos/grammar errors > > - use commas instead of en dash > > > > documentation/brief-yoctoprojectqs/index.rst | 221 ++++++++++--------- > > 1 file changed, 121 insertions(+), 100 deletions(-) > > > > diff --git a/documentation/brief-yoctoprojectqs/index.rst > b/documentation/brief-yoctoprojectqs/index.rst > > index 4a6f13d7e..8ef1f5bf5 100644 > > --- a/documentation/brief-yoctoprojectqs/index.rst > > +++ b/documentation/brief-yoctoprojectqs/index.rst > > @@ -87,75 +87,112 @@ distribution: > > see the :ref:`ref-manual/system-requirements:required packages for > the build host` > > section in the Yocto Project Reference Manual. > > > > -Use Git to Clone Poky > > -===================== > > +Use Git to Clone bitbake-setup > > +============================== > > > > Once you complete the setup instructions for your machine, you need to > > -get a copy of the Poky repository on your build host. Use the following > > -commands to clone the Poky repository. > > +get a copy of the bitbake-setup tool to setup the Poky reference > > +distribution on your build host. Use the following commands to clone > > +the bitbake repository. > > > > .. code-block:: shell > > > > - $ git clone git://git.yoctoproject.org/poky > > - Cloning into 'poky'... > > - remote: Counting > > - objects: 432160, done. remote: Compressing objects: 100% > > - (102056/102056), done. remote: Total 432160 (delta 323116), reused > > - 432037 (delta 323000) Receiving objects: 100% (432160/432160), > 153.81 MiB | 8.54 MiB/s, done. > > - Resolving deltas: 100% (323116/323116), done. > > - Checking connectivity... done. > > + $ git clone git://git.openembedded.org/bitbake bitbake-setup > > + Cloning into 'bitbake-setup'... > > + remote: Enumerating objects: 68454, done. > > + remote: Counting objects: 100% (49/49), done. > > + remote: Compressing objects: 100% (44/44), done. > > + remote: Total 68454 (delta 31), reused 5 (delta 5), pack-reused > 68405 (from 1) > > + Receiving objects: 100% (68454/68454), 14.17 MiB | 10.56 MiB/s, done. > > + Resolving deltas: 100% (51776/51776), done. > > > > -Go to :yocto_wiki:`Releases wiki page </Releases>`, and choose a release > > -codename (such as ``&DISTRO_NAME_NO_CAP;``), corresponding to either the > > -latest stable release or a Long Term Support release. > > +Setup a build environment with the following command: > > > > -Then move to the ``poky`` directory and take a look at existing > branches: > > +.. code-block:: shell > > + > > + $ ./bitbake-setup/bin/bitbake-setup init > > + > > +By default, this will setup in $HOME/bitbake-builds. If you prefer to > > +setup your builds in a different directory, for example the current > > +directory, you can instead use the ``--top-dir`` argument: > > > > .. code-block:: shell > > > > - $ cd poky > > - $ git branch -a > > - . > > - . > > - . > > - remotes/origin/HEAD -> origin/master > > - remotes/origin/dunfell > > - remotes/origin/dunfell-next > > - . > > - . > > - . > > - remotes/origin/gatesgarth > > - remotes/origin/gatesgarth-next > > - . > > - . > > - . > > - remotes/origin/master > > - remotes/origin/master-next > > - . > > - . > > - . > > - > > - > > -For this example, check out the ``&DISTRO_NAME_NO_CAP;`` branch based > on the > > -``&DISTRO_NAME;`` release: > > + $ ./bitbake-setup/bin/bitbake-setup init --top-dir $PWD > > + > > +`bitbake-setup init` is an interactive program by default and will ask > > +you to make some decisions. Depending on your answers, the choices may > > +differ from the examples below. > > + > > +#. Choose a configuration (for example, ``yocto-master-options``): > > + > > + .. code-block:: shell > > + > > + Available configurations: > > + 0. yocto-master-nested-configs (future) Official Yocto > configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 > (defined with nested configurations) > > + 1. yocto-master-options (future) Official Yocto configurations: > poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined > with options) > > + > > + Please select one of the above configurations by its number: > > + 1 > > + > > +#. Choose a target machine (for example, ``qemux86-64``): > > + > > + .. code-block:: shell > > + > > + Selecting the only available bitbake configuration poky > > + Available target machines: > > + 0. machine/qemux86-64 > > + 1. machine/qemuarm64 > > + 2. machine/qemuriscv64 > > + Please select one of the above options by its number: > > + 0 > > + > > +#. Choose a distribution (for example, ``poky``): > > + > > + .. code-block:: shell > > + > > + Available distributions: > > + 0. distro/poky > > + 1. distro/poky-altcfg > > + 2. distro/poky-tiny > > + Please select one of the above options by its number: > > + 0 > > + Run 'bitbake-setup yocto-master-options poky distro/poky > machine/qemux86-64' to select this configuration non-interactively. > > + Initializing a build in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64 > > + Fetching layer/tool repository bitbake into > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/bitbake > > + Fetching layer/tool repository openembedded-core into > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/openembedded-core > > + Fetching layer/tool repository meta-yocto into > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/meta-yocto > > + Fetching layer/tool repository yocto-docs into > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/yocto-docs > > + ============================== > > + Setting up bitbake configuration in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build > > + This bitbake configuration provides: Poky reference distro build > > + Usage instructions and additional information are in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README > > + > > +If you prefer to run non-interactively, you could use a command > > +like the following: > > > > .. code-block:: shell > > > > - $ git checkout -t origin/&DISTRO_NAME_NO_CAP; -b > my-&DISTRO_NAME_NO_CAP; > > - Branch 'my-&DISTRO_NAME_NO_CAP;' set up to track remote branch > '&DISTRO_NAME_NO_CAP;' from 'origin'. > > - Switched to a new branch 'my-&DISTRO_NAME_NO_CAP;' > > + $ bitbake-setup yocto-master-options poky distro/poky > machine/qemux86-64 > > > > -The previous Git checkout command creates a local branch named > > -``my-&DISTRO_NAME_NO_CAP;``. The files available to you in that branch > > -exactly match the repository's files in the ``&DISTRO_NAME_NO_CAP;`` > > -release branch. > > +Among other things, the script creates the :term:`Build Directory`, > which is > > +``build`` in this case and is located in the "top-dir" directory. The > script > > +also clones the layers needed to build the Poky reference distribution, > in the > > +``layers`` subdirectory. After the script runs, your current working > directory > > +is set to the :term:`Build Directory`. Later, when the build completes, > the > > +:term:`Build Directory` contains all the files created during the build. > > > > -Note that you can regularly type the following command in the same > directory > > -to keep your local files in sync with the release branch: > > + > > +Information about the build environment can be found in > ``/path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README`` > which should look something like the following: > > > > .. code-block:: shell > > > > - $ git pull > > + Poky reference distro build > > + > > + Additional information is in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-summary.txt > and > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-notes.txt > > + > > + Source the environment using '. > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env' > to run builds from the command line. > > + The bitbake configuration files (local.conf, bblayers.conf and more) > can be found in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf > > > > For more options and information about accessing Yocto Project related > > repositories, see the > > @@ -182,61 +219,47 @@ an entire Linux distribution, including the > toolchain, from source. > > ":yocto_wiki:`Working Behind a Network Proxy > </Working_Behind_a_Network_Proxy>`" > > page of the Yocto Project Wiki. > > > > -#. **Initialize the Build Environment:** From within the ``poky`` > > - directory, run the :ref:`ref-manual/structure:``oe-init-build-env``` > > - environment > > - setup script to define Yocto Project's build environment on your > > - build host. > > +#. **Initialize the Build Environment:** > > + Run the ``init-build-env`` environment setup script within the build > directory > > + to define Yocto Project's build environment on your build host. > > > > .. code-block:: shell > > > > - $ cd poky > > - $ source oe-init-build-env > > - You had no conf/local.conf file. This configuration file has > therefore been > > - created for you with some default values. You may wish to edit it > to, for > > - example, select a different MACHINE (target hardware). See > conf/local.conf > > - for more information as common configuration options are > commented. > > - > > - You had no conf/bblayers.conf file. This configuration file has > therefore > > - been created for you with some default values. To add additional > metadata > > - layers into your configuration please add entries to > conf/bblayers.conf. > > - > > - The Yocto Project has extensive documentation about OE including > a reference > > - manual which can be found at: > > - https://docs.yoctoproject.org > > + $ source > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env > > + Poky reference distro build > > > > - For more information about OpenEmbedded see their website: > > - https://www.openembedded.org/ > > +#. **Examine Your Auto Configuration File:** When you set up the build > > + environment, an auto configuration file named ``auto.conf`` becomes > > + available in aa ``conf`` subdirectory of the :term:`Build > Directory`. For this > > + example, the defaults are set to include three ``OE_FRAGMENTS``: > > + ``core/yocto/sstate-mirror-cdn``, ``machine/qemux86-64`` and > ``distro/poky``. > > + These set up the environment similar to what was previously in the > local > > + configuration file named ``local.conf`` which is now largely empty. > > + For more information on fragments, see > :ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` > directive` in the Bitbake User Manual. > > > > - ### Shell environment set up for builds. ### > > + ``core/yocto/sstate-mirror-cdn`` sets up > :term:`BB_HASHSERVE_UPSTREAM` and > > + :term:`SSTATE_MIRRORS`. The definition can be found in > > + > ``../layers/openembedded-core/meta/conf/fragments/yocto/sstate-mirror-cdn.conf``. > > > > - You can now run 'bitbake <target>' > > - > > - Common targets are: > > - core-image-minimal > > - core-image-full-cmdline > > - core-image-sato > > - core-image-weston > > - meta-toolchain > > - meta-ide-support > > - > > - You can also run generated QEMU images with a command like > 'runqemu qemux86-64' > > + .. code-block:: shell > > > > - Other commonly useful commands are: > > - - 'devtool' and 'recipetool' handle common recipe tasks > > - - 'bitbake-layers' handles common layer tasks > > - - 'oe-pkgdata-util' handles common target package tasks > > + BB_CONF_FRAGMENT_SUMMARY = "Use prebuilt sstate artifacts for > standard Yocto build configurations." > > + BB_CONF_FRAGMENT_DESCRIPTION = "The Yocto Project has prebuilt > artefacts available for standard build configurations. \ > > + This fragment enables their use. This will mean the build will > query the \ > > + the network to check for artefacts at the start of builds, which > does slow it down \ > > + initially but it will then speed up the builds by not having to > build things if they are \ > > + present in the cache. It assumes you can download something > faster than you can build it \ > > + which will depend on your network. \ > > + Note: For this to work you also need hash-equivalence passthrough > to the matching server \ > > + " > > > > - Among other things, the script creates the :term:`Build Directory`, > which is > > - ``build`` in this case and is located in the :term:`Source > Directory`. After > > - the script runs, your current working directory is set to the > > - :term:`Build Directory`. Later, when the build completes, the > > - :term:`Build Directory` contains all the files created during the > build. > > + BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws' > > + SSTATE_MIRRORS ?= "file://.* > http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > > > > #. **Examine Your Local Configuration File:** When you set up the build > > environment, a local configuration file named ``local.conf`` becomes > > available in a ``conf`` subdirectory of the :term:`Build Directory`. > For this > > - example, the defaults are set to build for a ``qemux86`` target, > > + example, the defaults are set to build for a ``qemux86-64`` target, > > which is suitable for emulation. The package manager used is set to > > the RPM package manager. > > > > @@ -249,13 +272,11 @@ an entire Linux distribution, including the > toolchain, from source. > > This is relevant only when your network and the server that you > use > > can download these artifacts faster than you would be able to > build them. > > > > - To use such mirrors, uncomment the below lines in your > ``conf/local.conf`` > > - file in the :term:`Build Directory`:: > > + To use such mirrors, ``bitbake-setup`` includes the following in > your > > + ``conf/auto.conf`` file (or you can add it to > ``conf/local.conf``) in > > + the :term:`Build Directory`:: > > > > - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" > > - SSTATE_MIRRORS ?= "file://.* > http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > > - BB_HASHSERVE = "auto" > > - BB_SIGNATURE_HANDLER = "OEEquivHash" > > + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" > > > > The hash equivalence server needs the websockets python module > version 9.1 > > or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, > CentOS Stream > > -- > > 2.43.0 > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#7457): > https://lists.yoctoproject.org/g/docs/message/7457 > > Mute This Topic: https://lists.yoctoproject.org/mt/114789217/1686489 > > Group Owner: docs+owner@lists.yoctoproject.org > > Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [ > alex.kanavin@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > >
On 8/19/25 23:34, Tim Orling via lists.yoctoproject.org wrote: > + To use such mirrors, ``bitbake-setup`` includes the following in your > + ``conf/auto.conf`` file (or you can add it to ``conf/local.conf``) in > + the :term:`Build Directory`:: > > - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" > - SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > - BB_HASHSERVE = "auto" > - BB_SIGNATURE_HANDLER = "OEEquivHash" > + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" > > The hash equivalence server needs the websockets python module version 9.1 > or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream > Random note: today was the first time I tried the official Yocto SSTATE_MIRRORS, since it is enabled by default with bitbake-setup. Until now I only used SSTATE_MIRRORS only on my local network, using direct IP addresses. One thing I noticed was that it absolutely hammers the DNS: it sends hundreds of requests/second over and over again, resolving the SSTATE_MIRRORS url for each request. My PiHole kept stopping resolving queries just after a few seconds when I started the build - it turns out that by default it rate-limits clients with more than 1000 requests/minute, didn't even know that it does that. I don't know if this is something that should be documented here (as long as it exists), or somewhere else, or maybe nowhere. But anyway, I thought I share while the memories are fresh.
Hi Tim, As said in the call earlier, here are some reviews. I know the tool has changed quite a bit since then, but some of the comments probably apply. Some comments are just formatting related. From our earlier discussion, we want to document an alternate way of settings things up, where we instruct to clone each repository manually and source the oe-init-build-env script instead. Do you want to work on that too? If not that's completely fine - this patch can be limited on bitbake-setup only and I will work on the other "legacy" part. On Tue Aug 19, 2025 at 11:34 PM CEST, Tim Orling via lists.yoctoproject.org wrote: > From: Tim Orling <tim.orling@konsulko.com> > > Largely replace the original Poky instructions with > example instructions to use bitbake-setup. > > Introduce the concept of OE_FRAGMENTS and how it is > used by bitbake-setup to setup SSTATE_MIRRORS, etc. > > Signed-off-by: Tim Orling <tim.orling@konsulko.com> > --- > Changes in v2: > - fix typos/grammar errors > - use commas instead of en dash > > documentation/brief-yoctoprojectqs/index.rst | 221 ++++++++++--------- > 1 file changed, 121 insertions(+), 100 deletions(-) > > diff --git a/documentation/brief-yoctoprojectqs/index.rst b/documentation/brief-yoctoprojectqs/index.rst > index 4a6f13d7e..8ef1f5bf5 100644 > --- a/documentation/brief-yoctoprojectqs/index.rst > +++ b/documentation/brief-yoctoprojectqs/index.rst > @@ -87,75 +87,112 @@ distribution: > see the :ref:`ref-manual/system-requirements:required packages for the build host` > section in the Yocto Project Reference Manual. > > -Use Git to Clone Poky > -===================== > +Use Git to Clone bitbake-setup > +============================== > > Once you complete the setup instructions for your machine, you need to > -get a copy of the Poky repository on your build host. Use the following > -commands to clone the Poky repository. > +get a copy of the bitbake-setup tool to setup the Poky reference s/bitbake-setup/``bitbake-setup``/ > +distribution on your build host. Use the following commands to clone s/commands/command/ > +the bitbake repository. s/repository./repository:/ > > .. code-block:: shell > > - $ git clone git://git.yoctoproject.org/poky > - Cloning into 'poky'... > - remote: Counting > - objects: 432160, done. remote: Compressing objects: 100% > - (102056/102056), done. remote: Total 432160 (delta 323116), reused > - 432037 (delta 323000) Receiving objects: 100% (432160/432160), 153.81 MiB | 8.54 MiB/s, done. > - Resolving deltas: 100% (323116/323116), done. > - Checking connectivity... done. > + $ git clone git://git.openembedded.org/bitbake bitbake-setup > + Cloning into 'bitbake-setup'... > + remote: Enumerating objects: 68454, done. > + remote: Counting objects: 100% (49/49), done. > + remote: Compressing objects: 100% (44/44), done. > + remote: Total 68454 (delta 31), reused 5 (delta 5), pack-reused 68405 (from 1) > + Receiving objects: 100% (68454/68454), 14.17 MiB | 10.56 MiB/s, done. > + Resolving deltas: 100% (51776/51776), done. As Alex said, I also agree on just showing the commands. That's really easier to maintain and makes the docs a bit clearer / less noisy. > > -Go to :yocto_wiki:`Releases wiki page </Releases>`, and choose a release > -codename (such as ``&DISTRO_NAME_NO_CAP;``), corresponding to either the > -latest stable release or a Long Term Support release. > +Setup a build environment with the following command: > > -Then move to the ``poky`` directory and take a look at existing branches: > +.. code-block:: shell > + > + $ ./bitbake-setup/bin/bitbake-setup init > + > +By default, this will setup in $HOME/bitbake-builds. If you prefer to > +setup your builds in a different directory, for example the current > +directory, you can instead use the ``--top-dir`` argument: This has not changed but we do have the ~/.config/bitbake-setup/ directory where to put a default value for this. We should mention it here. > > .. code-block:: shell > > - $ cd poky > - $ git branch -a > - . > - . > - . > - remotes/origin/HEAD -> origin/master > - remotes/origin/dunfell > - remotes/origin/dunfell-next > - . > - . > - . > - remotes/origin/gatesgarth > - remotes/origin/gatesgarth-next > - . > - . > - . > - remotes/origin/master > - remotes/origin/master-next > - . > - . > - . > - > - > -For this example, check out the ``&DISTRO_NAME_NO_CAP;`` branch based on the > -``&DISTRO_NAME;`` release: > + $ ./bitbake-setup/bin/bitbake-setup init --top-dir $PWD > + > +`bitbake-setup init` is an interactive program by default and will ask We need to user double tick here (``bitbake-setup init``). > +you to make some decisions. Depending on your answers, the choices may > +differ from the examples below. > + > +#. Choose a configuration (for example, ``yocto-master-options``): > + > + .. code-block:: shell > + > + Available configurations: > + 0. yocto-master-nested-configs (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 (defined with nested configurations) > + 1. yocto-master-options (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined with options) > + > + Please select one of the above configurations by its number: > + 1 > + > +#. Choose a target machine (for example, ``qemux86-64``): > + > + .. code-block:: shell > + > + Selecting the only available bitbake configuration poky > + Available target machines: > + 0. machine/qemux86-64 > + 1. machine/qemuarm64 > + 2. machine/qemuriscv64 > + Please select one of the above options by its number: > + 0 > + > +#. Choose a distribution (for example, ``poky``): > + > + .. code-block:: shell > + > + Available distributions: > + 0. distro/poky > + 1. distro/poky-altcfg > + 2. distro/poky-tiny > + Please select one of the above options by its number: > + 0 > + Run 'bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64' to select this configuration non-interactively. > + Initializing a build in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64 > + Fetching layer/tool repository bitbake into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/bitbake > + Fetching layer/tool repository openembedded-core into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/openembedded-core > + Fetching layer/tool repository meta-yocto into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/meta-yocto > + Fetching layer/tool repository yocto-docs into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/yocto-docs > + ============================== > + Setting up bitbake configuration in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build > + This bitbake configuration provides: Poky reference distro build > + Usage instructions and additional information are in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README > + > +If you prefer to run non-interactively, you could use a command > +like the following: > > .. code-block:: shell > > - $ git checkout -t origin/&DISTRO_NAME_NO_CAP; -b my-&DISTRO_NAME_NO_CAP; > - Branch 'my-&DISTRO_NAME_NO_CAP;' set up to track remote branch '&DISTRO_NAME_NO_CAP;' from 'origin'. > - Switched to a new branch 'my-&DISTRO_NAME_NO_CAP;' > + $ bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64 Now it's: bitbake-setup init --non-interactive ... > > -The previous Git checkout command creates a local branch named > -``my-&DISTRO_NAME_NO_CAP;``. The files available to you in that branch > -exactly match the repository's files in the ``&DISTRO_NAME_NO_CAP;`` > -release branch. > +Among other things, the script creates the :term:`Build Directory`, which is > +``build`` in this case and is located in the "top-dir" directory. The script > +also clones the layers needed to build the Poky reference distribution, in the > +``layers`` subdirectory. After the script runs, your current working directory > +is set to the :term:`Build Directory`. Later, when the build completes, the > +:term:`Build Directory` contains all the files created during the build. > > -Note that you can regularly type the following command in the same directory > -to keep your local files in sync with the release branch: > + > +Information about the build environment can be found in ``/path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README`` which should look something like the following: > > .. code-block:: shell > > - $ git pull > + Poky reference distro build > + > + Additional information is in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-summary.txt and /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-notes.txt > + > + Source the environment using '. /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env' to run builds from the command line. > + The bitbake configuration files (local.conf, bblayers.conf and more) can be found in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf > > For more options and information about accessing Yocto Project related > repositories, see the > @@ -182,61 +219,47 @@ an entire Linux distribution, including the toolchain, from source. > ":yocto_wiki:`Working Behind a Network Proxy </Working_Behind_a_Network_Proxy>`" > page of the Yocto Project Wiki. > > -#. **Initialize the Build Environment:** From within the ``poky`` > - directory, run the :ref:`ref-manual/structure:``oe-init-build-env``` > - environment > - setup script to define Yocto Project's build environment on your > - build host. > +#. **Initialize the Build Environment:** > + Run the ``init-build-env`` environment setup script within the build directory s/Run/Source/ > + to define Yocto Project's build environment on your build host. > > .. code-block:: shell > > - $ cd poky > - $ source oe-init-build-env > - You had no conf/local.conf file. This configuration file has therefore been > - created for you with some default values. You may wish to edit it to, for > - example, select a different MACHINE (target hardware). See conf/local.conf > - for more information as common configuration options are commented. > - > - You had no conf/bblayers.conf file. This configuration file has therefore > - been created for you with some default values. To add additional metadata > - layers into your configuration please add entries to conf/bblayers.conf. > - > - The Yocto Project has extensive documentation about OE including a reference > - manual which can be found at: > - https://docs.yoctoproject.org > + $ source /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env > + Poky reference distro build > > - For more information about OpenEmbedded see their website: > - https://www.openembedded.org/ > +#. **Examine Your Auto Configuration File:** When you set up the build > + environment, an auto configuration file named ``auto.conf`` becomes > + available in aa ``conf`` subdirectory of the :term:`Build Directory`. For this s/aa/a/ > + example, the defaults are set to include three ``OE_FRAGMENTS``: > + ``core/yocto/sstate-mirror-cdn``, ``machine/qemux86-64`` and ``distro/poky``. This might be subject to change a so a bit of a maintenance burden. Can we rephrase to: """ Fragments of configurations are listed in the :term:`OE_FRAGMENTS` variable. """ OE_FRAGMENTS would need to be documented too ideally > + These set up the environment similar to what was previously in the local > + configuration file named ``local.conf`` which is now largely empty. > + For more information on fragments, see :ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` directive` in the Bitbake User Manual. > > - ### Shell environment set up for builds. ### > + ``core/yocto/sstate-mirror-cdn`` sets up :term:`BB_HASHSERVE_UPSTREAM` and > + :term:`SSTATE_MIRRORS`. The definition can be found in > + ``../layers/openembedded-core/meta/conf/fragments/yocto/sstate-mirror-cdn.conf``. Do we want to list this here? I feel like this could distract from the point of this section, which is to showcase how to use bitbake-setup. I think we need a separate section on fragments that we could link from here. I know this is additional work: we can merge this as-is but probably a task for later. > > - You can now run 'bitbake <target>' > - > - Common targets are: > - core-image-minimal > - core-image-full-cmdline > - core-image-sato > - core-image-weston > - meta-toolchain > - meta-ide-support > - > - You can also run generated QEMU images with a command like 'runqemu qemux86-64' > + .. code-block:: shell > > - Other commonly useful commands are: > - - 'devtool' and 'recipetool' handle common recipe tasks > - - 'bitbake-layers' handles common layer tasks > - - 'oe-pkgdata-util' handles common target package tasks > + BB_CONF_FRAGMENT_SUMMARY = "Use prebuilt sstate artifacts for standard Yocto build configurations." > + BB_CONF_FRAGMENT_DESCRIPTION = "The Yocto Project has prebuilt artefacts available for standard build configurations. \ > + This fragment enables their use. This will mean the build will query the \ > + the network to check for artefacts at the start of builds, which does slow it down \ > + initially but it will then speed up the builds by not having to build things if they are \ > + present in the cache. It assumes you can download something faster than you can build it \ > + which will depend on your network. \ > + Note: For this to work you also need hash-equivalence passthrough to the matching server \ > + " > > - Among other things, the script creates the :term:`Build Directory`, which is > - ``build`` in this case and is located in the :term:`Source Directory`. After > - the script runs, your current working directory is set to the > - :term:`Build Directory`. Later, when the build completes, the > - :term:`Build Directory` contains all the files created during the build. > + BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws' > + SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" This can be printed with `bitbake-config-build show-fragment core/yocto/sstate-mirror-cdn` now. > > #. **Examine Your Local Configuration File:** When you set up the build > environment, a local configuration file named ``local.conf`` becomes > available in a ``conf`` subdirectory of the :term:`Build Directory`. For this > - example, the defaults are set to build for a ``qemux86`` target, > + example, the defaults are set to build for a ``qemux86-64`` target, > which is suitable for emulation. The package manager used is set to > the RPM package manager. > > @@ -249,13 +272,11 @@ an entire Linux distribution, including the toolchain, from source. > This is relevant only when your network and the server that you use > can download these artifacts faster than you would be able to build them. > > - To use such mirrors, uncomment the below lines in your ``conf/local.conf`` > - file in the :term:`Build Directory`:: > + To use such mirrors, ``bitbake-setup`` includes the following in your > + ``conf/auto.conf`` file (or you can add it to ``conf/local.conf``) in > + the :term:`Build Directory`:: > > - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" > - SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > - BB_HASHSERVE = "auto" > - BB_SIGNATURE_HANDLER = "OEEquivHash" > + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" > > The hash equivalence server needs the websockets python module version 9.1 > or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream Thanks! Antonin
On Tue, 23 Sept 2025 at 18:06, Antonin Godard via lists.yoctoproject.org <antonin.godard=bootlin.com@lists.yoctoproject.org> wrote: > Do we want to list this here? I feel like this could distract from the point of > this section, which is to showcase how to use bitbake-setup. I think we need a > separate section on fragments that we could link from here. I know this is > additional work: we can merge this as-is but probably a task for later. I too think properly documenting the oe-core fragment implementation and available fragments should be done first, and only then full bitbake-setup documentation should be based on it. Bitbake manual describes the addfragments keyword, but not what is actually done with that keyword in oe-core, or available tooling (in oe-core) to manage fragments, or available fragments. You can write a top level user guide for 'initial setup/status/update' with bitbake-setup without getting into fragments, but the moment you go into how bitbake-setup really works and what needs to be in a config file for it, they're are a key piece. Alex
On Tue Sep 23, 2025 at 6:38 PM CEST, Alexander Kanavin via lists.yoctoproject.org wrote: > On Tue, 23 Sept 2025 at 18:06, Antonin Godard via > lists.yoctoproject.org > <antonin.godard=bootlin.com@lists.yoctoproject.org> wrote: >> Do we want to list this here? I feel like this could distract from the point of >> this section, which is to showcase how to use bitbake-setup. I think we need a >> separate section on fragments that we could link from here. I know this is >> additional work: we can merge this as-is but probably a task for later. > > I too think properly documenting the oe-core fragment implementation > and available fragments should be done first, and only then full > bitbake-setup documentation should be based on it. Bitbake manual > describes the addfragments keyword, but not what is actually done with > that keyword in oe-core, or available tooling (in oe-core) to manage > fragments, or available fragments. I'm working on adding the docs on the fragments at the moment, it should be out this week. Antonin
On Thu, 2025-08-21 at 22:17 +0200, Gyorgy Sarvari wrote: > On 8/19/25 23:34, Tim Orling via lists.yoctoproject.org wrote: > > + To use such mirrors, ``bitbake-setup`` includes the following in your > > + ``conf/auto.conf`` file (or you can add it to ``conf/local.conf``) in > > + the :term:`Build Directory`:: > > > > - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" > > - SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > > - BB_HASHSERVE = "auto" > > - BB_SIGNATURE_HANDLER = "OEEquivHash" > > + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" > > > > The hash equivalence server needs the websockets python module version 9.1 > > or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream > > > > Random note: today was the first time I tried the official Yocto > SSTATE_MIRRORS, since it is enabled by default with bitbake-setup. Until > now I only used SSTATE_MIRRORS only on my local network, using direct IP > addresses. > > One thing I noticed was that it absolutely hammers the DNS: it sends > hundreds of requests/second over and over again, resolving the > SSTATE_MIRRORS url for each request. My PiHole kept stopping resolving > queries just after a few seconds when I started the build - it turns out > that by default it rate-limits clients with more than 1000 > requests/minute, didn't even know that it does that. > > I don't know if this is something that should be documented here (as > long as it exists), or somewhere else, or maybe nowhere. But anyway, I > thought I share while the memories are fresh. I spotted this email while reading the more recent replies in the thread. If everything is working as intended, DNS responses will be cached by the client for a sensible amount of time and you won't see hundreds of requests per second. I've seen a similar issue before with pi-hole though. Pi-hole uses dnsmasq which gives replies with a TTL of zero when the DNS entry is locally set or automatically added by the DHCP server - this tells the client not to cache the result at all. The dnsmasq docs say that the default local-ttl of zero "is the correct thing to do in almost all situations" but I strongly disagree! To fix this I created a config file at `/etc/dnsmasq.d/99-local.conf` on my pi-hole server with the following contents so that local records will be cached by clients for a minute: local-ttl=60 dhcp-ttl=60 (if you're running pi-hole in docker then setting this inside the docker container is left as an exercise for the reader) I saw this while using an sstate mirror on an internal Minio instance, so the DNS records were local to my pi-hole instance. You may need to look into other dnsmasq TTL settings or the TTL values being set by your upstream DNS server if you're seeing external DNS entries not being cached. Thanks,
On Tue, 2025-09-23 at 18:38 +0200, Alexander Kanavin wrote: > On Tue, 23 Sept 2025 at 18:06, Antonin Godard via > lists.yoctoproject.org > <antonin.godard=bootlin.com@lists.yoctoproject.org> wrote: > > Do we want to list this here? I feel like this could distract from the point of > > this section, which is to showcase how to use bitbake-setup. I think we need a > > separate section on fragments that we could link from here. I know this is > > additional work: we can merge this as-is but probably a task for later. > > I too think properly documenting the oe-core fragment implementation > and available fragments should be done first, and only then full > bitbake-setup documentation should be based on it. Bitbake manual > describes the addfragments keyword, but not what is actually done with > that keyword in oe-core, or available tooling (in oe-core) to manage > fragments, or available fragments. > > You can write a top level user guide for 'initial setup/status/update' > with bitbake-setup without getting into fragments, but the moment you > go into how bitbake-setup really works and what needs to be in a > config file for it, they're are a key piece. Is there also a plan to document how to create the conf.json files for use with bitbake-setup? I tried to use `bitbake-layers create-layers-setup ...` but that doesn't seem to be the right way to create them - the bitbake repository was not included in the generated json file. Apologies if I've missed something! Thanks,
On Wed, 24 Sept 2025 at 18:34, Paul Barker <paul@pbarker.dev> wrote: > Is there also a plan to document how to create the conf.json files for use with > bitbake-setup? > > I tried to use `bitbake-layers create-layers-setup ...` but that doesn't seem > to be the right way to create them - the bitbake repository was not included in > the generated json file. > > Apologies if I've missed something! I don't think anything was missed. There are three still to be done items here: - extending code in bitbake-layers so it can create a full bitbake-setup config from an active build, rather than just the layers bit (what it does now is intended for the oe-setup-layers tool, which is a sort of precursor to bitbake-setup). I believe all the pieces can be programmatically obtained: particularly active fragments and the list of enabled layers. - documenting how to use that; - documenting how to write advanced bitbake-setup configurations by hand and the full range of syntax and features. Alex
On 9/24/25 18:20, Paul Barker wrote: > On Thu, 2025-08-21 at 22:17 +0200, Gyorgy Sarvari wrote: >> On 8/19/25 23:34, Tim Orling via lists.yoctoproject.org wrote: >>> + To use such mirrors, ``bitbake-setup`` includes the following in your >>> + ``conf/auto.conf`` file (or you can add it to ``conf/local.conf``) in >>> + the :term:`Build Directory`:: >>> >>> - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" >>> - SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" >>> - BB_HASHSERVE = "auto" >>> - BB_SIGNATURE_HANDLER = "OEEquivHash" >>> + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" >>> >>> The hash equivalence server needs the websockets python module version 9.1 >>> or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream >>> >> Random note: today was the first time I tried the official Yocto >> SSTATE_MIRRORS, since it is enabled by default with bitbake-setup. Until >> now I only used SSTATE_MIRRORS only on my local network, using direct IP >> addresses. >> >> One thing I noticed was that it absolutely hammers the DNS: it sends >> hundreds of requests/second over and over again, resolving the >> SSTATE_MIRRORS url for each request. My PiHole kept stopping resolving >> queries just after a few seconds when I started the build - it turns out >> that by default it rate-limits clients with more than 1000 >> requests/minute, didn't even know that it does that. >> >> I don't know if this is something that should be documented here (as >> long as it exists), or somewhere else, or maybe nowhere. But anyway, I >> thought I share while the memories are fresh. > I spotted this email while reading the more recent replies in the thread. > > If everything is working as intended, DNS responses will be cached by the > client for a sensible amount of time and you won't see hundreds of requests per > second. > > I've seen a similar issue before with pi-hole though. Pi-hole uses dnsmasq > which gives replies with a TTL of zero when the DNS entry is locally set or > automatically added by the DHCP server - this tells the client not to cache the > result at all. The dnsmasq docs say that the default local-ttl of zero "is the > correct thing to do in almost all situations" but I strongly disagree! > > To fix this I created a config file at `/etc/dnsmasq.d/99-local.conf` on my > pi-hole server with the following contents so that local records will be cached > by clients for a minute: > > local-ttl=60 > dhcp-ttl=60 > > (if you're running pi-hole in docker then setting this inside the docker > container is left as an exercise for the reader) > > I saw this while using an sstate mirror on an internal Minio instance, so the > DNS records were local to my pi-hole instance. You may need to look into other > dnsmasq TTL settings or the TTL values being set by your upstream DNS server if > you're seeing external DNS entries not being cached. > > Thanks a lot! Will definitely set this up and give it a go.
diff --git a/documentation/brief-yoctoprojectqs/index.rst b/documentation/brief-yoctoprojectqs/index.rst index 4a6f13d7e..8ef1f5bf5 100644 --- a/documentation/brief-yoctoprojectqs/index.rst +++ b/documentation/brief-yoctoprojectqs/index.rst @@ -87,75 +87,112 @@ distribution: see the :ref:`ref-manual/system-requirements:required packages for the build host` section in the Yocto Project Reference Manual. -Use Git to Clone Poky -===================== +Use Git to Clone bitbake-setup +============================== Once you complete the setup instructions for your machine, you need to -get a copy of the Poky repository on your build host. Use the following -commands to clone the Poky repository. +get a copy of the bitbake-setup tool to setup the Poky reference +distribution on your build host. Use the following commands to clone +the bitbake repository. .. code-block:: shell - $ git clone git://git.yoctoproject.org/poky - Cloning into 'poky'... - remote: Counting - objects: 432160, done. remote: Compressing objects: 100% - (102056/102056), done. remote: Total 432160 (delta 323116), reused - 432037 (delta 323000) Receiving objects: 100% (432160/432160), 153.81 MiB | 8.54 MiB/s, done. - Resolving deltas: 100% (323116/323116), done. - Checking connectivity... done. + $ git clone git://git.openembedded.org/bitbake bitbake-setup + Cloning into 'bitbake-setup'... + remote: Enumerating objects: 68454, done. + remote: Counting objects: 100% (49/49), done. + remote: Compressing objects: 100% (44/44), done. + remote: Total 68454 (delta 31), reused 5 (delta 5), pack-reused 68405 (from 1) + Receiving objects: 100% (68454/68454), 14.17 MiB | 10.56 MiB/s, done. + Resolving deltas: 100% (51776/51776), done. -Go to :yocto_wiki:`Releases wiki page </Releases>`, and choose a release -codename (such as ``&DISTRO_NAME_NO_CAP;``), corresponding to either the -latest stable release or a Long Term Support release. +Setup a build environment with the following command: -Then move to the ``poky`` directory and take a look at existing branches: +.. code-block:: shell + + $ ./bitbake-setup/bin/bitbake-setup init + +By default, this will setup in $HOME/bitbake-builds. If you prefer to +setup your builds in a different directory, for example the current +directory, you can instead use the ``--top-dir`` argument: .. code-block:: shell - $ cd poky - $ git branch -a - . - . - . - remotes/origin/HEAD -> origin/master - remotes/origin/dunfell - remotes/origin/dunfell-next - . - . - . - remotes/origin/gatesgarth - remotes/origin/gatesgarth-next - . - . - . - remotes/origin/master - remotes/origin/master-next - . - . - . - - -For this example, check out the ``&DISTRO_NAME_NO_CAP;`` branch based on the -``&DISTRO_NAME;`` release: + $ ./bitbake-setup/bin/bitbake-setup init --top-dir $PWD + +`bitbake-setup init` is an interactive program by default and will ask +you to make some decisions. Depending on your answers, the choices may +differ from the examples below. + +#. Choose a configuration (for example, ``yocto-master-options``): + + .. code-block:: shell + + Available configurations: + 0. yocto-master-nested-configs (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 (defined with nested configurations) + 1. yocto-master-options (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined with options) + + Please select one of the above configurations by its number: + 1 + +#. Choose a target machine (for example, ``qemux86-64``): + + .. code-block:: shell + + Selecting the only available bitbake configuration poky + Available target machines: + 0. machine/qemux86-64 + 1. machine/qemuarm64 + 2. machine/qemuriscv64 + Please select one of the above options by its number: + 0 + +#. Choose a distribution (for example, ``poky``): + + .. code-block:: shell + + Available distributions: + 0. distro/poky + 1. distro/poky-altcfg + 2. distro/poky-tiny + Please select one of the above options by its number: + 0 + Run 'bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64' to select this configuration non-interactively. + Initializing a build in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64 + Fetching layer/tool repository bitbake into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/bitbake + Fetching layer/tool repository openembedded-core into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/openembedded-core + Fetching layer/tool repository meta-yocto into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/meta-yocto + Fetching layer/tool repository yocto-docs into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/yocto-docs + ============================== + Setting up bitbake configuration in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build + This bitbake configuration provides: Poky reference distro build + Usage instructions and additional information are in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README + +If you prefer to run non-interactively, you could use a command +like the following: .. code-block:: shell - $ git checkout -t origin/&DISTRO_NAME_NO_CAP; -b my-&DISTRO_NAME_NO_CAP; - Branch 'my-&DISTRO_NAME_NO_CAP;' set up to track remote branch '&DISTRO_NAME_NO_CAP;' from 'origin'. - Switched to a new branch 'my-&DISTRO_NAME_NO_CAP;' + $ bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64 -The previous Git checkout command creates a local branch named -``my-&DISTRO_NAME_NO_CAP;``. The files available to you in that branch -exactly match the repository's files in the ``&DISTRO_NAME_NO_CAP;`` -release branch. +Among other things, the script creates the :term:`Build Directory`, which is +``build`` in this case and is located in the "top-dir" directory. The script +also clones the layers needed to build the Poky reference distribution, in the +``layers`` subdirectory. After the script runs, your current working directory +is set to the :term:`Build Directory`. Later, when the build completes, the +:term:`Build Directory` contains all the files created during the build. -Note that you can regularly type the following command in the same directory -to keep your local files in sync with the release branch: + +Information about the build environment can be found in ``/path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README`` which should look something like the following: .. code-block:: shell - $ git pull + Poky reference distro build + + Additional information is in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-summary.txt and /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-notes.txt + + Source the environment using '. /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env' to run builds from the command line. + The bitbake configuration files (local.conf, bblayers.conf and more) can be found in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf For more options and information about accessing Yocto Project related repositories, see the @@ -182,61 +219,47 @@ an entire Linux distribution, including the toolchain, from source. ":yocto_wiki:`Working Behind a Network Proxy </Working_Behind_a_Network_Proxy>`" page of the Yocto Project Wiki. -#. **Initialize the Build Environment:** From within the ``poky`` - directory, run the :ref:`ref-manual/structure:``oe-init-build-env``` - environment - setup script to define Yocto Project's build environment on your - build host. +#. **Initialize the Build Environment:** + Run the ``init-build-env`` environment setup script within the build directory + to define Yocto Project's build environment on your build host. .. code-block:: shell - $ cd poky - $ source oe-init-build-env - You had no conf/local.conf file. This configuration file has therefore been - created for you with some default values. You may wish to edit it to, for - example, select a different MACHINE (target hardware). See conf/local.conf - for more information as common configuration options are commented. - - You had no conf/bblayers.conf file. This configuration file has therefore - been created for you with some default values. To add additional metadata - layers into your configuration please add entries to conf/bblayers.conf. - - The Yocto Project has extensive documentation about OE including a reference - manual which can be found at: - https://docs.yoctoproject.org + $ source /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env + Poky reference distro build - For more information about OpenEmbedded see their website: - https://www.openembedded.org/ +#. **Examine Your Auto Configuration File:** When you set up the build + environment, an auto configuration file named ``auto.conf`` becomes + available in aa ``conf`` subdirectory of the :term:`Build Directory`. For this + example, the defaults are set to include three ``OE_FRAGMENTS``: + ``core/yocto/sstate-mirror-cdn``, ``machine/qemux86-64`` and ``distro/poky``. + These set up the environment similar to what was previously in the local + configuration file named ``local.conf`` which is now largely empty. + For more information on fragments, see :ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` directive` in the Bitbake User Manual. - ### Shell environment set up for builds. ### + ``core/yocto/sstate-mirror-cdn`` sets up :term:`BB_HASHSERVE_UPSTREAM` and + :term:`SSTATE_MIRRORS`. The definition can be found in + ``../layers/openembedded-core/meta/conf/fragments/yocto/sstate-mirror-cdn.conf``. - You can now run 'bitbake <target>' - - Common targets are: - core-image-minimal - core-image-full-cmdline - core-image-sato - core-image-weston - meta-toolchain - meta-ide-support - - You can also run generated QEMU images with a command like 'runqemu qemux86-64' + .. code-block:: shell - Other commonly useful commands are: - - 'devtool' and 'recipetool' handle common recipe tasks - - 'bitbake-layers' handles common layer tasks - - 'oe-pkgdata-util' handles common target package tasks + BB_CONF_FRAGMENT_SUMMARY = "Use prebuilt sstate artifacts for standard Yocto build configurations." + BB_CONF_FRAGMENT_DESCRIPTION = "The Yocto Project has prebuilt artefacts available for standard build configurations. \ + This fragment enables their use. This will mean the build will query the \ + the network to check for artefacts at the start of builds, which does slow it down \ + initially but it will then speed up the builds by not having to build things if they are \ + present in the cache. It assumes you can download something faster than you can build it \ + which will depend on your network. \ + Note: For this to work you also need hash-equivalence passthrough to the matching server \ + " - Among other things, the script creates the :term:`Build Directory`, which is - ``build`` in this case and is located in the :term:`Source Directory`. After - the script runs, your current working directory is set to the - :term:`Build Directory`. Later, when the build completes, the - :term:`Build Directory` contains all the files created during the build. + BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws' + SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" #. **Examine Your Local Configuration File:** When you set up the build environment, a local configuration file named ``local.conf`` becomes available in a ``conf`` subdirectory of the :term:`Build Directory`. For this - example, the defaults are set to build for a ``qemux86`` target, + example, the defaults are set to build for a ``qemux86-64`` target, which is suitable for emulation. The package manager used is set to the RPM package manager. @@ -249,13 +272,11 @@ an entire Linux distribution, including the toolchain, from source. This is relevant only when your network and the server that you use can download these artifacts faster than you would be able to build them. - To use such mirrors, uncomment the below lines in your ``conf/local.conf`` - file in the :term:`Build Directory`:: + To use such mirrors, ``bitbake-setup`` includes the following in your + ``conf/auto.conf`` file (or you can add it to ``conf/local.conf``) in + the :term:`Build Directory`:: - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" - SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" - BB_HASHSERVE = "auto" - BB_SIGNATURE_HANDLER = "OEEquivHash" + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" The hash equivalence server needs the websockets python module version 9.1 or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream