Message ID | 20250819213530.3616042-1-tim.orling@konsulko.com |
---|---|
State | New |
Headers | show |
Series | [RFC,v2] brief-yoctoprojectqs: add bitbake-setup info | expand |
Thanks Tim, this is basically fine, but has to be held until bitbake-setup actually lands (and then adjusted to match that version). I also wonder if the documentation needs to show exactly what the tool outputs, as that is prone to becoming out of sync, and the users will see it for themselves anyway. --top-dir value can also be set from an environment variable, so you don't have to pass it in every time (and end up creating bogus top dir if forgetting to do so). Alex On Tue, 19 Aug 2025 at 23:35, Tim Orling via lists.yoctoproject.org <tim.orling=konsulko.com@lists.yoctoproject.org> wrote: > > From: Tim Orling <tim.orling@konsulko.com> > > Largely replace the original Poky instructions with > example instructions to use bitbake-setup. > > Introduce the concept of OE_FRAGMENTS and how it is > used by bitbake-setup to setup SSTATE_MIRRORS, etc. > > Signed-off-by: Tim Orling <tim.orling@konsulko.com> > --- > Changes in v2: > - fix typos/grammar errors > - use commas instead of en dash > > documentation/brief-yoctoprojectqs/index.rst | 221 ++++++++++--------- > 1 file changed, 121 insertions(+), 100 deletions(-) > > diff --git a/documentation/brief-yoctoprojectqs/index.rst b/documentation/brief-yoctoprojectqs/index.rst > index 4a6f13d7e..8ef1f5bf5 100644 > --- a/documentation/brief-yoctoprojectqs/index.rst > +++ b/documentation/brief-yoctoprojectqs/index.rst > @@ -87,75 +87,112 @@ distribution: > see the :ref:`ref-manual/system-requirements:required packages for the build host` > section in the Yocto Project Reference Manual. > > -Use Git to Clone Poky > -===================== > +Use Git to Clone bitbake-setup > +============================== > > Once you complete the setup instructions for your machine, you need to > -get a copy of the Poky repository on your build host. Use the following > -commands to clone the Poky repository. > +get a copy of the bitbake-setup tool to setup the Poky reference > +distribution on your build host. Use the following commands to clone > +the bitbake repository. > > .. code-block:: shell > > - $ git clone git://git.yoctoproject.org/poky > - Cloning into 'poky'... > - remote: Counting > - objects: 432160, done. remote: Compressing objects: 100% > - (102056/102056), done. remote: Total 432160 (delta 323116), reused > - 432037 (delta 323000) Receiving objects: 100% (432160/432160), 153.81 MiB | 8.54 MiB/s, done. > - Resolving deltas: 100% (323116/323116), done. > - Checking connectivity... done. > + $ git clone git://git.openembedded.org/bitbake bitbake-setup > + Cloning into 'bitbake-setup'... > + remote: Enumerating objects: 68454, done. > + remote: Counting objects: 100% (49/49), done. > + remote: Compressing objects: 100% (44/44), done. > + remote: Total 68454 (delta 31), reused 5 (delta 5), pack-reused 68405 (from 1) > + Receiving objects: 100% (68454/68454), 14.17 MiB | 10.56 MiB/s, done. > + Resolving deltas: 100% (51776/51776), done. > > -Go to :yocto_wiki:`Releases wiki page </Releases>`, and choose a release > -codename (such as ``&DISTRO_NAME_NO_CAP;``), corresponding to either the > -latest stable release or a Long Term Support release. > +Setup a build environment with the following command: > > -Then move to the ``poky`` directory and take a look at existing branches: > +.. code-block:: shell > + > + $ ./bitbake-setup/bin/bitbake-setup init > + > +By default, this will setup in $HOME/bitbake-builds. If you prefer to > +setup your builds in a different directory, for example the current > +directory, you can instead use the ``--top-dir`` argument: > > .. code-block:: shell > > - $ cd poky > - $ git branch -a > - . > - . > - . > - remotes/origin/HEAD -> origin/master > - remotes/origin/dunfell > - remotes/origin/dunfell-next > - . > - . > - . > - remotes/origin/gatesgarth > - remotes/origin/gatesgarth-next > - . > - . > - . > - remotes/origin/master > - remotes/origin/master-next > - . > - . > - . > - > - > -For this example, check out the ``&DISTRO_NAME_NO_CAP;`` branch based on the > -``&DISTRO_NAME;`` release: > + $ ./bitbake-setup/bin/bitbake-setup init --top-dir $PWD > + > +`bitbake-setup init` is an interactive program by default and will ask > +you to make some decisions. Depending on your answers, the choices may > +differ from the examples below. > + > +#. Choose a configuration (for example, ``yocto-master-options``): > + > + .. code-block:: shell > + > + Available configurations: > + 0. yocto-master-nested-configs (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 (defined with nested configurations) > + 1. yocto-master-options (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined with options) > + > + Please select one of the above configurations by its number: > + 1 > + > +#. Choose a target machine (for example, ``qemux86-64``): > + > + .. code-block:: shell > + > + Selecting the only available bitbake configuration poky > + Available target machines: > + 0. machine/qemux86-64 > + 1. machine/qemuarm64 > + 2. machine/qemuriscv64 > + Please select one of the above options by its number: > + 0 > + > +#. Choose a distribution (for example, ``poky``): > + > + .. code-block:: shell > + > + Available distributions: > + 0. distro/poky > + 1. distro/poky-altcfg > + 2. distro/poky-tiny > + Please select one of the above options by its number: > + 0 > + Run 'bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64' to select this configuration non-interactively. > + Initializing a build in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64 > + Fetching layer/tool repository bitbake into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/bitbake > + Fetching layer/tool repository openembedded-core into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/openembedded-core > + Fetching layer/tool repository meta-yocto into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/meta-yocto > + Fetching layer/tool repository yocto-docs into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/yocto-docs > + ============================== > + Setting up bitbake configuration in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build > + This bitbake configuration provides: Poky reference distro build > + Usage instructions and additional information are in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README > + > +If you prefer to run non-interactively, you could use a command > +like the following: > > .. code-block:: shell > > - $ git checkout -t origin/&DISTRO_NAME_NO_CAP; -b my-&DISTRO_NAME_NO_CAP; > - Branch 'my-&DISTRO_NAME_NO_CAP;' set up to track remote branch '&DISTRO_NAME_NO_CAP;' from 'origin'. > - Switched to a new branch 'my-&DISTRO_NAME_NO_CAP;' > + $ bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64 > > -The previous Git checkout command creates a local branch named > -``my-&DISTRO_NAME_NO_CAP;``. The files available to you in that branch > -exactly match the repository's files in the ``&DISTRO_NAME_NO_CAP;`` > -release branch. > +Among other things, the script creates the :term:`Build Directory`, which is > +``build`` in this case and is located in the "top-dir" directory. The script > +also clones the layers needed to build the Poky reference distribution, in the > +``layers`` subdirectory. After the script runs, your current working directory > +is set to the :term:`Build Directory`. Later, when the build completes, the > +:term:`Build Directory` contains all the files created during the build. > > -Note that you can regularly type the following command in the same directory > -to keep your local files in sync with the release branch: > + > +Information about the build environment can be found in ``/path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README`` which should look something like the following: > > .. code-block:: shell > > - $ git pull > + Poky reference distro build > + > + Additional information is in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-summary.txt and /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-notes.txt > + > + Source the environment using '. /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env' to run builds from the command line. > + The bitbake configuration files (local.conf, bblayers.conf and more) can be found in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf > > For more options and information about accessing Yocto Project related > repositories, see the > @@ -182,61 +219,47 @@ an entire Linux distribution, including the toolchain, from source. > ":yocto_wiki:`Working Behind a Network Proxy </Working_Behind_a_Network_Proxy>`" > page of the Yocto Project Wiki. > > -#. **Initialize the Build Environment:** From within the ``poky`` > - directory, run the :ref:`ref-manual/structure:``oe-init-build-env``` > - environment > - setup script to define Yocto Project's build environment on your > - build host. > +#. **Initialize the Build Environment:** > + Run the ``init-build-env`` environment setup script within the build directory > + to define Yocto Project's build environment on your build host. > > .. code-block:: shell > > - $ cd poky > - $ source oe-init-build-env > - You had no conf/local.conf file. This configuration file has therefore been > - created for you with some default values. You may wish to edit it to, for > - example, select a different MACHINE (target hardware). See conf/local.conf > - for more information as common configuration options are commented. > - > - You had no conf/bblayers.conf file. This configuration file has therefore > - been created for you with some default values. To add additional metadata > - layers into your configuration please add entries to conf/bblayers.conf. > - > - The Yocto Project has extensive documentation about OE including a reference > - manual which can be found at: > - https://docs.yoctoproject.org > + $ source /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env > + Poky reference distro build > > - For more information about OpenEmbedded see their website: > - https://www.openembedded.org/ > +#. **Examine Your Auto Configuration File:** When you set up the build > + environment, an auto configuration file named ``auto.conf`` becomes > + available in aa ``conf`` subdirectory of the :term:`Build Directory`. For this > + example, the defaults are set to include three ``OE_FRAGMENTS``: > + ``core/yocto/sstate-mirror-cdn``, ``machine/qemux86-64`` and ``distro/poky``. > + These set up the environment similar to what was previously in the local > + configuration file named ``local.conf`` which is now largely empty. > + For more information on fragments, see :ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` directive` in the Bitbake User Manual. > > - ### Shell environment set up for builds. ### > + ``core/yocto/sstate-mirror-cdn`` sets up :term:`BB_HASHSERVE_UPSTREAM` and > + :term:`SSTATE_MIRRORS`. The definition can be found in > + ``../layers/openembedded-core/meta/conf/fragments/yocto/sstate-mirror-cdn.conf``. > > - You can now run 'bitbake <target>' > - > - Common targets are: > - core-image-minimal > - core-image-full-cmdline > - core-image-sato > - core-image-weston > - meta-toolchain > - meta-ide-support > - > - You can also run generated QEMU images with a command like 'runqemu qemux86-64' > + .. code-block:: shell > > - Other commonly useful commands are: > - - 'devtool' and 'recipetool' handle common recipe tasks > - - 'bitbake-layers' handles common layer tasks > - - 'oe-pkgdata-util' handles common target package tasks > + BB_CONF_FRAGMENT_SUMMARY = "Use prebuilt sstate artifacts for standard Yocto build configurations." > + BB_CONF_FRAGMENT_DESCRIPTION = "The Yocto Project has prebuilt artefacts available for standard build configurations. \ > + This fragment enables their use. This will mean the build will query the \ > + the network to check for artefacts at the start of builds, which does slow it down \ > + initially but it will then speed up the builds by not having to build things if they are \ > + present in the cache. It assumes you can download something faster than you can build it \ > + which will depend on your network. \ > + Note: For this to work you also need hash-equivalence passthrough to the matching server \ > + " > > - Among other things, the script creates the :term:`Build Directory`, which is > - ``build`` in this case and is located in the :term:`Source Directory`. After > - the script runs, your current working directory is set to the > - :term:`Build Directory`. Later, when the build completes, the > - :term:`Build Directory` contains all the files created during the build. > + BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws' > + SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > > #. **Examine Your Local Configuration File:** When you set up the build > environment, a local configuration file named ``local.conf`` becomes > available in a ``conf`` subdirectory of the :term:`Build Directory`. For this > - example, the defaults are set to build for a ``qemux86`` target, > + example, the defaults are set to build for a ``qemux86-64`` target, > which is suitable for emulation. The package manager used is set to > the RPM package manager. > > @@ -249,13 +272,11 @@ an entire Linux distribution, including the toolchain, from source. > This is relevant only when your network and the server that you use > can download these artifacts faster than you would be able to build them. > > - To use such mirrors, uncomment the below lines in your ``conf/local.conf`` > - file in the :term:`Build Directory`:: > + To use such mirrors, ``bitbake-setup`` includes the following in your > + ``conf/auto.conf`` file (or you can add it to ``conf/local.conf``) in > + the :term:`Build Directory`:: > > - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" > - SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > - BB_HASHSERVE = "auto" > - BB_SIGNATURE_HANDLER = "OEEquivHash" > + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" > > The hash equivalence server needs the websockets python module version 9.1 > or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream > -- > 2.43.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#7457): https://lists.yoctoproject.org/g/docs/message/7457 > Mute This Topic: https://lists.yoctoproject.org/mt/114789217/1686489 > Group Owner: docs+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Wed, Aug 20, 2025 at 12:46 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > Thanks Tim, this is basically fine, but has to be held until > bitbake-setup actually lands (and then adjusted to match that > version). > Absolutely, this is why I put it out as an RFC. Plus I think (see the next paragraph) there needs to be some more discussion/thought about the documentation. We need to add more documentation about OE fragments and more about bitbake-setup configuration, etc. > > I also wonder if the documentation needs to show exactly what the tool > outputs, as that is prone to becoming out of sync, and the users will > see it for themselves anyway. > I definitely agree and was on the fence about that. I was mostly following the pattern that was already in the quick start guide. Plus links to metadata don't exist yet, so that wasn't an option either. We should continue to discuss what will work best for the quick start guide and what else is needed elsewhere in the reference manual, etc. > > --top-dir value can also be set from an environment variable, so you > don't have to pass it in every time (and end up creating bogus top dir > if forgetting to do so). > Good point. Perhaps I can document that in v3 instead of showing the command line. I look forward to more discussion about all of this. > Alex > > On Tue, 19 Aug 2025 at 23:35, Tim Orling via lists.yoctoproject.org > <tim.orling=konsulko.com@lists.yoctoproject.org> wrote: > > > > From: Tim Orling <tim.orling@konsulko.com> > > > > Largely replace the original Poky instructions with > > example instructions to use bitbake-setup. > > > > Introduce the concept of OE_FRAGMENTS and how it is > > used by bitbake-setup to setup SSTATE_MIRRORS, etc. > > > > Signed-off-by: Tim Orling <tim.orling@konsulko.com> > > --- > > Changes in v2: > > - fix typos/grammar errors > > - use commas instead of en dash > > > > documentation/brief-yoctoprojectqs/index.rst | 221 ++++++++++--------- > > 1 file changed, 121 insertions(+), 100 deletions(-) > > > > diff --git a/documentation/brief-yoctoprojectqs/index.rst > b/documentation/brief-yoctoprojectqs/index.rst > > index 4a6f13d7e..8ef1f5bf5 100644 > > --- a/documentation/brief-yoctoprojectqs/index.rst > > +++ b/documentation/brief-yoctoprojectqs/index.rst > > @@ -87,75 +87,112 @@ distribution: > > see the :ref:`ref-manual/system-requirements:required packages for > the build host` > > section in the Yocto Project Reference Manual. > > > > -Use Git to Clone Poky > > -===================== > > +Use Git to Clone bitbake-setup > > +============================== > > > > Once you complete the setup instructions for your machine, you need to > > -get a copy of the Poky repository on your build host. Use the following > > -commands to clone the Poky repository. > > +get a copy of the bitbake-setup tool to setup the Poky reference > > +distribution on your build host. Use the following commands to clone > > +the bitbake repository. > > > > .. code-block:: shell > > > > - $ git clone git://git.yoctoproject.org/poky > > - Cloning into 'poky'... > > - remote: Counting > > - objects: 432160, done. remote: Compressing objects: 100% > > - (102056/102056), done. remote: Total 432160 (delta 323116), reused > > - 432037 (delta 323000) Receiving objects: 100% (432160/432160), > 153.81 MiB | 8.54 MiB/s, done. > > - Resolving deltas: 100% (323116/323116), done. > > - Checking connectivity... done. > > + $ git clone git://git.openembedded.org/bitbake bitbake-setup > > + Cloning into 'bitbake-setup'... > > + remote: Enumerating objects: 68454, done. > > + remote: Counting objects: 100% (49/49), done. > > + remote: Compressing objects: 100% (44/44), done. > > + remote: Total 68454 (delta 31), reused 5 (delta 5), pack-reused > 68405 (from 1) > > + Receiving objects: 100% (68454/68454), 14.17 MiB | 10.56 MiB/s, done. > > + Resolving deltas: 100% (51776/51776), done. > > > > -Go to :yocto_wiki:`Releases wiki page </Releases>`, and choose a release > > -codename (such as ``&DISTRO_NAME_NO_CAP;``), corresponding to either the > > -latest stable release or a Long Term Support release. > > +Setup a build environment with the following command: > > > > -Then move to the ``poky`` directory and take a look at existing > branches: > > +.. code-block:: shell > > + > > + $ ./bitbake-setup/bin/bitbake-setup init > > + > > +By default, this will setup in $HOME/bitbake-builds. If you prefer to > > +setup your builds in a different directory, for example the current > > +directory, you can instead use the ``--top-dir`` argument: > > > > .. code-block:: shell > > > > - $ cd poky > > - $ git branch -a > > - . > > - . > > - . > > - remotes/origin/HEAD -> origin/master > > - remotes/origin/dunfell > > - remotes/origin/dunfell-next > > - . > > - . > > - . > > - remotes/origin/gatesgarth > > - remotes/origin/gatesgarth-next > > - . > > - . > > - . > > - remotes/origin/master > > - remotes/origin/master-next > > - . > > - . > > - . > > - > > - > > -For this example, check out the ``&DISTRO_NAME_NO_CAP;`` branch based > on the > > -``&DISTRO_NAME;`` release: > > + $ ./bitbake-setup/bin/bitbake-setup init --top-dir $PWD > > + > > +`bitbake-setup init` is an interactive program by default and will ask > > +you to make some decisions. Depending on your answers, the choices may > > +differ from the examples below. > > + > > +#. Choose a configuration (for example, ``yocto-master-options``): > > + > > + .. code-block:: shell > > + > > + Available configurations: > > + 0. yocto-master-nested-configs (future) Official Yocto > configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 > (defined with nested configurations) > > + 1. yocto-master-options (future) Official Yocto configurations: > poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined > with options) > > + > > + Please select one of the above configurations by its number: > > + 1 > > + > > +#. Choose a target machine (for example, ``qemux86-64``): > > + > > + .. code-block:: shell > > + > > + Selecting the only available bitbake configuration poky > > + Available target machines: > > + 0. machine/qemux86-64 > > + 1. machine/qemuarm64 > > + 2. machine/qemuriscv64 > > + Please select one of the above options by its number: > > + 0 > > + > > +#. Choose a distribution (for example, ``poky``): > > + > > + .. code-block:: shell > > + > > + Available distributions: > > + 0. distro/poky > > + 1. distro/poky-altcfg > > + 2. distro/poky-tiny > > + Please select one of the above options by its number: > > + 0 > > + Run 'bitbake-setup yocto-master-options poky distro/poky > machine/qemux86-64' to select this configuration non-interactively. > > + Initializing a build in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64 > > + Fetching layer/tool repository bitbake into > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/bitbake > > + Fetching layer/tool repository openembedded-core into > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/openembedded-core > > + Fetching layer/tool repository meta-yocto into > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/meta-yocto > > + Fetching layer/tool repository yocto-docs into > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/yocto-docs > > + ============================== > > + Setting up bitbake configuration in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build > > + This bitbake configuration provides: Poky reference distro build > > + Usage instructions and additional information are in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README > > + > > +If you prefer to run non-interactively, you could use a command > > +like the following: > > > > .. code-block:: shell > > > > - $ git checkout -t origin/&DISTRO_NAME_NO_CAP; -b > my-&DISTRO_NAME_NO_CAP; > > - Branch 'my-&DISTRO_NAME_NO_CAP;' set up to track remote branch > '&DISTRO_NAME_NO_CAP;' from 'origin'. > > - Switched to a new branch 'my-&DISTRO_NAME_NO_CAP;' > > + $ bitbake-setup yocto-master-options poky distro/poky > machine/qemux86-64 > > > > -The previous Git checkout command creates a local branch named > > -``my-&DISTRO_NAME_NO_CAP;``. The files available to you in that branch > > -exactly match the repository's files in the ``&DISTRO_NAME_NO_CAP;`` > > -release branch. > > +Among other things, the script creates the :term:`Build Directory`, > which is > > +``build`` in this case and is located in the "top-dir" directory. The > script > > +also clones the layers needed to build the Poky reference distribution, > in the > > +``layers`` subdirectory. After the script runs, your current working > directory > > +is set to the :term:`Build Directory`. Later, when the build completes, > the > > +:term:`Build Directory` contains all the files created during the build. > > > > -Note that you can regularly type the following command in the same > directory > > -to keep your local files in sync with the release branch: > > + > > +Information about the build environment can be found in > ``/path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README`` > which should look something like the following: > > > > .. code-block:: shell > > > > - $ git pull > > + Poky reference distro build > > + > > + Additional information is in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-summary.txt > and > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-notes.txt > > + > > + Source the environment using '. > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env' > to run builds from the command line. > > + The bitbake configuration files (local.conf, bblayers.conf and more) > can be found in > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf > > > > For more options and information about accessing Yocto Project related > > repositories, see the > > @@ -182,61 +219,47 @@ an entire Linux distribution, including the > toolchain, from source. > > ":yocto_wiki:`Working Behind a Network Proxy > </Working_Behind_a_Network_Proxy>`" > > page of the Yocto Project Wiki. > > > > -#. **Initialize the Build Environment:** From within the ``poky`` > > - directory, run the :ref:`ref-manual/structure:``oe-init-build-env``` > > - environment > > - setup script to define Yocto Project's build environment on your > > - build host. > > +#. **Initialize the Build Environment:** > > + Run the ``init-build-env`` environment setup script within the build > directory > > + to define Yocto Project's build environment on your build host. > > > > .. code-block:: shell > > > > - $ cd poky > > - $ source oe-init-build-env > > - You had no conf/local.conf file. This configuration file has > therefore been > > - created for you with some default values. You may wish to edit it > to, for > > - example, select a different MACHINE (target hardware). See > conf/local.conf > > - for more information as common configuration options are > commented. > > - > > - You had no conf/bblayers.conf file. This configuration file has > therefore > > - been created for you with some default values. To add additional > metadata > > - layers into your configuration please add entries to > conf/bblayers.conf. > > - > > - The Yocto Project has extensive documentation about OE including > a reference > > - manual which can be found at: > > - https://docs.yoctoproject.org > > + $ source > /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env > > + Poky reference distro build > > > > - For more information about OpenEmbedded see their website: > > - https://www.openembedded.org/ > > +#. **Examine Your Auto Configuration File:** When you set up the build > > + environment, an auto configuration file named ``auto.conf`` becomes > > + available in aa ``conf`` subdirectory of the :term:`Build > Directory`. For this > > + example, the defaults are set to include three ``OE_FRAGMENTS``: > > + ``core/yocto/sstate-mirror-cdn``, ``machine/qemux86-64`` and > ``distro/poky``. > > + These set up the environment similar to what was previously in the > local > > + configuration file named ``local.conf`` which is now largely empty. > > + For more information on fragments, see > :ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` > directive` in the Bitbake User Manual. > > > > - ### Shell environment set up for builds. ### > > + ``core/yocto/sstate-mirror-cdn`` sets up > :term:`BB_HASHSERVE_UPSTREAM` and > > + :term:`SSTATE_MIRRORS`. The definition can be found in > > + > ``../layers/openembedded-core/meta/conf/fragments/yocto/sstate-mirror-cdn.conf``. > > > > - You can now run 'bitbake <target>' > > - > > - Common targets are: > > - core-image-minimal > > - core-image-full-cmdline > > - core-image-sato > > - core-image-weston > > - meta-toolchain > > - meta-ide-support > > - > > - You can also run generated QEMU images with a command like > 'runqemu qemux86-64' > > + .. code-block:: shell > > > > - Other commonly useful commands are: > > - - 'devtool' and 'recipetool' handle common recipe tasks > > - - 'bitbake-layers' handles common layer tasks > > - - 'oe-pkgdata-util' handles common target package tasks > > + BB_CONF_FRAGMENT_SUMMARY = "Use prebuilt sstate artifacts for > standard Yocto build configurations." > > + BB_CONF_FRAGMENT_DESCRIPTION = "The Yocto Project has prebuilt > artefacts available for standard build configurations. \ > > + This fragment enables their use. This will mean the build will > query the \ > > + the network to check for artefacts at the start of builds, which > does slow it down \ > > + initially but it will then speed up the builds by not having to > build things if they are \ > > + present in the cache. It assumes you can download something > faster than you can build it \ > > + which will depend on your network. \ > > + Note: For this to work you also need hash-equivalence passthrough > to the matching server \ > > + " > > > > - Among other things, the script creates the :term:`Build Directory`, > which is > > - ``build`` in this case and is located in the :term:`Source > Directory`. After > > - the script runs, your current working directory is set to the > > - :term:`Build Directory`. Later, when the build completes, the > > - :term:`Build Directory` contains all the files created during the > build. > > + BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws' > > + SSTATE_MIRRORS ?= "file://.* > http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > > > > #. **Examine Your Local Configuration File:** When you set up the build > > environment, a local configuration file named ``local.conf`` becomes > > available in a ``conf`` subdirectory of the :term:`Build Directory`. > For this > > - example, the defaults are set to build for a ``qemux86`` target, > > + example, the defaults are set to build for a ``qemux86-64`` target, > > which is suitable for emulation. The package manager used is set to > > the RPM package manager. > > > > @@ -249,13 +272,11 @@ an entire Linux distribution, including the > toolchain, from source. > > This is relevant only when your network and the server that you > use > > can download these artifacts faster than you would be able to > build them. > > > > - To use such mirrors, uncomment the below lines in your > ``conf/local.conf`` > > - file in the :term:`Build Directory`:: > > + To use such mirrors, ``bitbake-setup`` includes the following in > your > > + ``conf/auto.conf`` file (or you can add it to > ``conf/local.conf``) in > > + the :term:`Build Directory`:: > > > > - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" > > - SSTATE_MIRRORS ?= "file://.* > http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > > - BB_HASHSERVE = "auto" > > - BB_SIGNATURE_HANDLER = "OEEquivHash" > > + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" > > > > The hash equivalence server needs the websockets python module > version 9.1 > > or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, > CentOS Stream > > -- > > 2.43.0 > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#7457): > https://lists.yoctoproject.org/g/docs/message/7457 > > Mute This Topic: https://lists.yoctoproject.org/mt/114789217/1686489 > > Group Owner: docs+owner@lists.yoctoproject.org > > Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [ > alex.kanavin@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > >
On 8/19/25 23:34, Tim Orling via lists.yoctoproject.org wrote: > + To use such mirrors, ``bitbake-setup`` includes the following in your > + ``conf/auto.conf`` file (or you can add it to ``conf/local.conf``) in > + the :term:`Build Directory`:: > > - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" > - SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" > - BB_HASHSERVE = "auto" > - BB_SIGNATURE_HANDLER = "OEEquivHash" > + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" > > The hash equivalence server needs the websockets python module version 9.1 > or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream > Random note: today was the first time I tried the official Yocto SSTATE_MIRRORS, since it is enabled by default with bitbake-setup. Until now I only used SSTATE_MIRRORS only on my local network, using direct IP addresses. One thing I noticed was that it absolutely hammers the DNS: it sends hundreds of requests/second over and over again, resolving the SSTATE_MIRRORS url for each request. My PiHole kept stopping resolving queries just after a few seconds when I started the build - it turns out that by default it rate-limits clients with more than 1000 requests/minute, didn't even know that it does that. I don't know if this is something that should be documented here (as long as it exists), or somewhere else, or maybe nowhere. But anyway, I thought I share while the memories are fresh.
diff --git a/documentation/brief-yoctoprojectqs/index.rst b/documentation/brief-yoctoprojectqs/index.rst index 4a6f13d7e..8ef1f5bf5 100644 --- a/documentation/brief-yoctoprojectqs/index.rst +++ b/documentation/brief-yoctoprojectqs/index.rst @@ -87,75 +87,112 @@ distribution: see the :ref:`ref-manual/system-requirements:required packages for the build host` section in the Yocto Project Reference Manual. -Use Git to Clone Poky -===================== +Use Git to Clone bitbake-setup +============================== Once you complete the setup instructions for your machine, you need to -get a copy of the Poky repository on your build host. Use the following -commands to clone the Poky repository. +get a copy of the bitbake-setup tool to setup the Poky reference +distribution on your build host. Use the following commands to clone +the bitbake repository. .. code-block:: shell - $ git clone git://git.yoctoproject.org/poky - Cloning into 'poky'... - remote: Counting - objects: 432160, done. remote: Compressing objects: 100% - (102056/102056), done. remote: Total 432160 (delta 323116), reused - 432037 (delta 323000) Receiving objects: 100% (432160/432160), 153.81 MiB | 8.54 MiB/s, done. - Resolving deltas: 100% (323116/323116), done. - Checking connectivity... done. + $ git clone git://git.openembedded.org/bitbake bitbake-setup + Cloning into 'bitbake-setup'... + remote: Enumerating objects: 68454, done. + remote: Counting objects: 100% (49/49), done. + remote: Compressing objects: 100% (44/44), done. + remote: Total 68454 (delta 31), reused 5 (delta 5), pack-reused 68405 (from 1) + Receiving objects: 100% (68454/68454), 14.17 MiB | 10.56 MiB/s, done. + Resolving deltas: 100% (51776/51776), done. -Go to :yocto_wiki:`Releases wiki page </Releases>`, and choose a release -codename (such as ``&DISTRO_NAME_NO_CAP;``), corresponding to either the -latest stable release or a Long Term Support release. +Setup a build environment with the following command: -Then move to the ``poky`` directory and take a look at existing branches: +.. code-block:: shell + + $ ./bitbake-setup/bin/bitbake-setup init + +By default, this will setup in $HOME/bitbake-builds. If you prefer to +setup your builds in a different directory, for example the current +directory, you can instead use the ``--top-dir`` argument: .. code-block:: shell - $ cd poky - $ git branch -a - . - . - . - remotes/origin/HEAD -> origin/master - remotes/origin/dunfell - remotes/origin/dunfell-next - . - . - . - remotes/origin/gatesgarth - remotes/origin/gatesgarth-next - . - . - . - remotes/origin/master - remotes/origin/master-next - . - . - . - - -For this example, check out the ``&DISTRO_NAME_NO_CAP;`` branch based on the -``&DISTRO_NAME;`` release: + $ ./bitbake-setup/bin/bitbake-setup init --top-dir $PWD + +`bitbake-setup init` is an interactive program by default and will ask +you to make some decisions. Depending on your answers, the choices may +differ from the examples below. + +#. Choose a configuration (for example, ``yocto-master-options``): + + .. code-block:: shell + + Available configurations: + 0. yocto-master-nested-configs (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64 and arm64 (defined with nested configurations) + 1. yocto-master-options (future) Official Yocto configurations: poky, poky-altcfg, poky-tiny, for qemux86-64, riscv64 and arm64 (defined with options) + + Please select one of the above configurations by its number: + 1 + +#. Choose a target machine (for example, ``qemux86-64``): + + .. code-block:: shell + + Selecting the only available bitbake configuration poky + Available target machines: + 0. machine/qemux86-64 + 1. machine/qemuarm64 + 2. machine/qemuriscv64 + Please select one of the above options by its number: + 0 + +#. Choose a distribution (for example, ``poky``): + + .. code-block:: shell + + Available distributions: + 0. distro/poky + 1. distro/poky-altcfg + 2. distro/poky-tiny + Please select one of the above options by its number: + 0 + Run 'bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64' to select this configuration non-interactively. + Initializing a build in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64 + Fetching layer/tool repository bitbake into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/bitbake + Fetching layer/tool repository openembedded-core into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/openembedded-core + Fetching layer/tool repository meta-yocto into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/meta-yocto + Fetching layer/tool repository yocto-docs into /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/layers/yocto-docs + ============================== + Setting up bitbake configuration in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build + This bitbake configuration provides: Poky reference distro build + Usage instructions and additional information are in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README + +If you prefer to run non-interactively, you could use a command +like the following: .. code-block:: shell - $ git checkout -t origin/&DISTRO_NAME_NO_CAP; -b my-&DISTRO_NAME_NO_CAP; - Branch 'my-&DISTRO_NAME_NO_CAP;' set up to track remote branch '&DISTRO_NAME_NO_CAP;' from 'origin'. - Switched to a new branch 'my-&DISTRO_NAME_NO_CAP;' + $ bitbake-setup yocto-master-options poky distro/poky machine/qemux86-64 -The previous Git checkout command creates a local branch named -``my-&DISTRO_NAME_NO_CAP;``. The files available to you in that branch -exactly match the repository's files in the ``&DISTRO_NAME_NO_CAP;`` -release branch. +Among other things, the script creates the :term:`Build Directory`, which is +``build`` in this case and is located in the "top-dir" directory. The script +also clones the layers needed to build the Poky reference distribution, in the +``layers`` subdirectory. After the script runs, your current working directory +is set to the :term:`Build Directory`. Later, when the build completes, the +:term:`Build Directory` contains all the files created during the build. -Note that you can regularly type the following command in the same directory -to keep your local files in sync with the release branch: + +Information about the build environment can be found in ``/path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/README`` which should look something like the following: .. code-block:: shell - $ git pull + Poky reference distro build + + Additional information is in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-summary.txt and /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf/conf-notes.txt + + Source the environment using '. /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env' to run builds from the command line. + The bitbake configuration files (local.conf, bblayers.conf and more) can be found in /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/conf For more options and information about accessing Yocto Project related repositories, see the @@ -182,61 +219,47 @@ an entire Linux distribution, including the toolchain, from source. ":yocto_wiki:`Working Behind a Network Proxy </Working_Behind_a_Network_Proxy>`" page of the Yocto Project Wiki. -#. **Initialize the Build Environment:** From within the ``poky`` - directory, run the :ref:`ref-manual/structure:``oe-init-build-env``` - environment - setup script to define Yocto Project's build environment on your - build host. +#. **Initialize the Build Environment:** + Run the ``init-build-env`` environment setup script within the build directory + to define Yocto Project's build environment on your build host. .. code-block:: shell - $ cd poky - $ source oe-init-build-env - You had no conf/local.conf file. This configuration file has therefore been - created for you with some default values. You may wish to edit it to, for - example, select a different MACHINE (target hardware). See conf/local.conf - for more information as common configuration options are commented. - - You had no conf/bblayers.conf file. This configuration file has therefore - been created for you with some default values. To add additional metadata - layers into your configuration please add entries to conf/bblayers.conf. - - The Yocto Project has extensive documentation about OE including a reference - manual which can be found at: - https://docs.yoctoproject.org + $ source /path/to/top-dir/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env + Poky reference distro build - For more information about OpenEmbedded see their website: - https://www.openembedded.org/ +#. **Examine Your Auto Configuration File:** When you set up the build + environment, an auto configuration file named ``auto.conf`` becomes + available in aa ``conf`` subdirectory of the :term:`Build Directory`. For this + example, the defaults are set to include three ``OE_FRAGMENTS``: + ``core/yocto/sstate-mirror-cdn``, ``machine/qemux86-64`` and ``distro/poky``. + These set up the environment similar to what was previously in the local + configuration file named ``local.conf`` which is now largely empty. + For more information on fragments, see :ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` directive` in the Bitbake User Manual. - ### Shell environment set up for builds. ### + ``core/yocto/sstate-mirror-cdn`` sets up :term:`BB_HASHSERVE_UPSTREAM` and + :term:`SSTATE_MIRRORS`. The definition can be found in + ``../layers/openembedded-core/meta/conf/fragments/yocto/sstate-mirror-cdn.conf``. - You can now run 'bitbake <target>' - - Common targets are: - core-image-minimal - core-image-full-cmdline - core-image-sato - core-image-weston - meta-toolchain - meta-ide-support - - You can also run generated QEMU images with a command like 'runqemu qemux86-64' + .. code-block:: shell - Other commonly useful commands are: - - 'devtool' and 'recipetool' handle common recipe tasks - - 'bitbake-layers' handles common layer tasks - - 'oe-pkgdata-util' handles common target package tasks + BB_CONF_FRAGMENT_SUMMARY = "Use prebuilt sstate artifacts for standard Yocto build configurations." + BB_CONF_FRAGMENT_DESCRIPTION = "The Yocto Project has prebuilt artefacts available for standard build configurations. \ + This fragment enables their use. This will mean the build will query the \ + the network to check for artefacts at the start of builds, which does slow it down \ + initially but it will then speed up the builds by not having to build things if they are \ + present in the cache. It assumes you can download something faster than you can build it \ + which will depend on your network. \ + Note: For this to work you also need hash-equivalence passthrough to the matching server \ + " - Among other things, the script creates the :term:`Build Directory`, which is - ``build`` in this case and is located in the :term:`Source Directory`. After - the script runs, your current working directory is set to the - :term:`Build Directory`. Later, when the build completes, the - :term:`Build Directory` contains all the files created during the build. + BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws' + SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" #. **Examine Your Local Configuration File:** When you set up the build environment, a local configuration file named ``local.conf`` becomes available in a ``conf`` subdirectory of the :term:`Build Directory`. For this - example, the defaults are set to build for a ``qemux86`` target, + example, the defaults are set to build for a ``qemux86-64`` target, which is suitable for emulation. The package manager used is set to the RPM package manager. @@ -249,13 +272,11 @@ an entire Linux distribution, including the toolchain, from source. This is relevant only when your network and the server that you use can download these artifacts faster than you would be able to build them. - To use such mirrors, uncomment the below lines in your ``conf/local.conf`` - file in the :term:`Build Directory`:: + To use such mirrors, ``bitbake-setup`` includes the following in your + ``conf/auto.conf`` file (or you can add it to ``conf/local.conf``) in + the :term:`Build Directory`:: - BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" - SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" - BB_HASHSERVE = "auto" - BB_SIGNATURE_HANDLER = "OEEquivHash" + OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn" The hash equivalence server needs the websockets python module version 9.1 or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream