From patchwork Wed Jul 22 12:58:10 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Antonin Godard X-Patchwork-Id: 93218 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FF83C4453C for ; Wed, 22 Jul 2026 12:58:34 +0000 (UTC) Received: from smtpout-03.galae.net (smtpout-03.galae.net [185.246.85.4]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.47561.1784725107932302708 for ; Wed, 22 Jul 2026 05:58:29 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@bootlin.com header.s=dkim header.b=ROdLnHMM; spf=pass (domain: bootlin.com, ip: 185.246.85.4, mailfrom: antonin.godard@bootlin.com) Received: from smtpout-01.galae.net (smtpout-01.galae.net [212.83.139.233]) by smtpout-03.galae.net (Postfix) with ESMTPS id 4902A4E40ECD for ; Wed, 22 Jul 2026 12:58:26 +0000 (UTC) Received: from mail.galae.net (mail.galae.net [212.83.136.155]) by smtpout-01.galae.net (Postfix) with ESMTPS id 1BBC960388 for ; Wed, 22 Jul 2026 12:58:26 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id 5D1AF11BD3CBF; Wed, 22 Jul 2026 14:58:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=dkim; t=1784725105; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=G7+UPAcJ3jisNCWbv07hESnBPIOxZgiPAy+bf9iNFno=; b=ROdLnHMMRaebwwfjMSV0UqXFhMHXtCV9anqPRrTobkAAf+6Pq6mp2MKyQIfA47VWajHEio aRL6D6RWVrqtNSsreOev/pL6okEej/voxrMb77e3qFkdENtnHoBL4ZnAMJ8Vc7n81URpQO cj74q8dJdhT7kdhqywcOyvQ8bnHVQV4A6FgU1xS/3Mn4A/zPUr2v3tYJZfJNawP0fLmZzU yRpjwrNjgluTOYJdyHNEYSx/4c/CrerIt45KJ7nnENcLiIHAKffqs7pzGwCh2dbpEz+vLw wvSm4RcRy/UTf9waFlXCF/LKEr7nFAcXwTqtJF4VlC7shz57+J7AROVCofccOQ== From: Antonin Godard Date: Wed, 22 Jul 2026 14:58:10 +0200 Subject: [PATCH 1/5] dev-manual: add a shared state mirror setup document MIME-Version: 1.0 Message-Id: <20260722-sstate-mirrors-doc-v1-1-570baeb41c32@bootlin.com> References: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> In-Reply-To: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> To: docs@lists.yoctoproject.org Cc: Thomas Petazzoni , Antonin Godard X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=8903; i=antonin.godard@bootlin.com; h=from:subject:message-id; bh=dY8iGxCo+7Kjz7vJJDupx+I1GE9YvMx+ZpDpv2C+WME=; b=owEBbQKS/ZANAwAKAdGAQUApo6g2AcsmYgBqYL5ud6ccvkDey4F8dd+ZdMrfWIfSWG3vRCy5n OV1GRJELXGJAjMEAAEKAB0WIQSGSHJRiN1AG7mg0//RgEFAKaOoNgUCamC+bgAKCRDRgEFAKaOo NiAfD/9TvwyDT0mAagUXLqFI1tGP1HX0hDlWaShUCcjCrj65qtabluoP9oXZFdoRNnXN/sT/adX 1gdoSxbn6d/LBjHuxrqnKo9AiC+iJsLLgdH8LkDCoDlm2rAKb+vVHWsaJLZ4PRAdBf4jVpaifJ1 y2Yzx1OctcfK/DxE2Cvrz09dfPQv/p1AFkWD0md8WRF2JPFlIGNUFOd7yVHUogPZ+t2TkjYQgcq RmqBKQayMf3s8FObCMCgLmrUA+b+IUB1bSutfQ5Jp2UVEbwyV60wfHIzZlsINqdIxFk8y7aT4jr PdvSecWhIbzgB2UK+Wa9G3X+GPjBEHgDtsZk7qRxqZku8O5tSvBaHHfQf30VtwlUQPQlfJWxEoG tcCctn+RLm59ZPx0OTT8PWLJk0w+craPV9f8p/Ndft4kllS/VPvpF0UCqjP2dVXd/XU/6tHtKAt j8Xtj5MKdtaqhNZCT7tVvxlP4bhC3dy4XhBebPD3v1KUoYwReIMheqUNs8ArHGpqpVP5MssQ0di SSvOTAmvRScwWh1mA6+eufzd/yKCAMVqx5NsIYK84wxkoLNWUj/mMwx4ZBu8UzMZBJY6/AJayM9 EV8eDMbbe1o2hLLZTYlogfLb57QNjeMbHeh2LeRNvM6B24hkN5ee3MegXwAu3tpDnkix9H5A2z+ HsFN1sKroTWTgzw== X-Developer-Key: i=antonin.godard@bootlin.com; a=openpgp; fpr=8648725188DD401BB9A0D3FFD180414029A3A836 X-Last-TLS-Session-Version: TLSv1.3 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 22 Jul 2026 12:58:34 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/10110 The aim of this document is to provide an introduction on how to setup a shared state mirror on a server for use by clients with SSTATE_MIRRORS. Setting it up is relatively straightforward. The document gives some tips which is where the value of the document lies (e.g. by reminding to use a hash equivalence service alongside it). [YOCTO #13589] Signed-off-by: Antonin Godard --- documentation/dev-manual/index.rst | 1 + documentation/dev-manual/sstate-mirrors-setup.rst | 166 ++++++++++++++++++++++ 2 files changed, 167 insertions(+) diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst index e9bf17bdc..0e1aa8958 100644 --- a/documentation/dev-manual/index.rst +++ b/documentation/dev-manual/index.rst @@ -49,6 +49,7 @@ Yocto Project Development Tasks Manual wayland qemu bblock + sstate-mirrors-setup hashequivserver .. include:: /boilerplate.rst diff --git a/documentation/dev-manual/sstate-mirrors-setup.rst b/documentation/dev-manual/sstate-mirrors-setup.rst new file mode 100644 index 000000000..6c369ff4e --- /dev/null +++ b/documentation/dev-manual/sstate-mirrors-setup.rst @@ -0,0 +1,166 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +Setting up a Shared State Cache Mirror +************************************** + +This document explains how to set up a server that hosts shared state artifacts +that can be reused by machines connecting to the server. + +The concept of "shared state" is explained in the +:ref:`overview-manual/concepts:Setscene Tasks and Shared State` section of the +Yocto Project Overview and Concepts Manual. These artifacts are files that are +by default placed in the shared state directory (:term:`SSTATE_DIR`). + +This document explains how to share the content of this directory through the +use of the :term:`SSTATE_MIRRORS` variable. + +Use-case +======== + +The most common use-case for setting up a shared state cache mirror is usually +when there is a dedicated machine in an infrastructure building images at a +regular interval, thus populating a shared state cache directory frequently. +Usually, this machine is part of a :wikipedia:`CI/CD ` type of +environment. + +Since the shared state directory of this machine contains most of the items +needed to accelerate a new build, it can be interesting to share it to remote +clients connecting to the machine and fetching the artifacts over the network: + +.. code-block:: text + + +---------------------+ + | | + +----------+ Build Machine +-------------+ + | | | | + | +--------+------------+ | + | | | + | | | + | | | + | | | + | | | + +-------v------+ +-------v------+ +-------v------+ + | | | | | | + | Client 1 | | Client 2 | . . . | Client N | + | | | | | | + +--------------+ +--------------+ +--------------+ + +With this kind of setup, it is assumed that: + +- Clients connect to the server over some protocol such as HTTP. In practice, + the protocol should be one supported by BitBake (see :doc:`the supported + fetchers `). + +- The shared state directory (:term:`SSTATE_DIR`) of the build machine is + is read-only from the point of view of the clients. + +- The previous points also means that clients hold their own copy of the shared + state artifacts in their own shared state directory. + +Server Setup +============ + +There are many ways of setting up a file hosting server. To illustrate, this +document will use a basic HTTP server started thanks to the ``http.server`` +Python module. + +On our build machine, we assume that the server shares the shared state from the +:term:`Build Directory`: + +.. code-block:: text + + build/ + ├── ... + ├── sstate-cache/ + └── ... + +With the ``http.server`` Python module, the server can be started as follows: + +.. code-block:: console + + $ python3 -m http.server -d .../build/sstate-cache + Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ... + +Client Configuration +==================== + +Configuring clients to this server happens through a :term:`configuration file`, +for example, the :ref:`site.conf ` file. Only +the :term:`SSTATE_MIRRORS` variable is needed to setup the connection:: + + SSTATE_MIRRORS = "file://.* http://127.0.0.1:8000/PATH;downloadfilename=PATH" + +The line above depends on your server configuration. Here, the example uses the +local ``127.0.0.1`` IP address (the client is running on the same machine) and +the files are served on the port 8000. + +With this, the client is configured to download shared state artifacts from the +server. This will happen automatically based on the :ref:`signatures +` of the tasks which are about +to run (so only relevant artifacts are fetched). + +After running a build, the local shared state directory of the client +(configured through its :term:`SSTATE_DIR` variable), will be populated with the +files downloaded from the server. + +Going Further +============= + +- It is recommended to setup a :ref:`overview-manual/concepts:Hash Equivalence` + service on the build machine --- running in parallel of the shared state + mirror --- to further speed up builds. See the + :doc:`/dev-manual/hashequivserver` section of the Yocto Project Development + Tasks Manual for more information. + +- If the :term:`BB_NO_NETWORK` variable is set to "1" on clients as a means to + disable any accesses to the network, the :term:`SSTATE_MIRROR_ALLOW_NETWORK` + variable may be used to allow the clients to fetch from the shared state + mirror as an exception. + +- Multiple shared state sources can be specified in the :term:`SSTATE_MIRRORS` + variable. For example:: + + SSTATE_MIRRORS = "\ + file://.* https://someserver.com/PATH;downloadfilename=PATH \ + file://.* https://someotherserver.com/PATH;downloadfilename=PATH \ + " + + The shared state artifacts will be searched on the remote locations in the + same order as they are specified in this variable. + +- Fetching the shared state artifacts from a local directory, such as an + :wikipedia:`NFS `-mounted directory, is also possible + using the ``file://`` fetcher:: + + SSTATE_MIRRORS = "file://.* file:///path/to/shared-state/PATH;downloadfilename=PATH" + +- The shared state mirror may be used in combination with GPG signatures to + ensure the authenticity of the downloaded files. See the + :doc:`/security-manual/sstate-signing` section of the Yocto Project Security + Manual for more information. + +- The size of the shared state directory on the server may grow over time. The + :oecore_path:`sstate-cache-management.py ` + script may be used to remove duplicate files. Otherwise, removing files + that have not been accessed for a certain period of time can be done with the + standard ``find`` command, e.g. for removing files older than 2 months: + + .. code-block:: console + + $ find sstate-cache/ -type f -atime +60 -delete + +Troubleshooting +=============== + +The best way to check that files are being downloaded from the server is +probably to check the server logs. For example, using the Python ``http.server`` +module, the following logs are shown: + +.. code-block:: text + + 127.0.0.1 - - [20/Jul/2026 14:36:23] "HEAD /d1/75/sstate%3Afile%3Ax86-64-v3-oe-linux%3A5.48%3Ar0%3Ax86-64-v3%3A14%3Ad175b18b8adc9cc3003edfc62f112fd421505bc715d41f36b4d04eef05d226ad_packagedata.tar.zst HTTP/1.1" 200 - + 127.0.0.1 - - [20/Jul/2026 14:36:23] "HEAD /f4/11/sstate%3Aos-release%3Aall-oe-linux%3A1.0%3Ar0%3Aallarch%3A14%3Af411b2f27319283694ab13bf0d8965a7dd26428431418dfb7505f3d1cdd0dbfd_package_write_ipk.tar.zst HTTP/1.1" 404 - + +Some artifacts may be found (returning 200 above). Some others will not be found +because the expected artifacts on the client is not present on the server, in +which case the client will rebuild the task (and its dependencies). From patchwork Wed Jul 22 12:58:11 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonin Godard X-Patchwork-Id: 93220 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2240C44539 for ; Wed, 22 Jul 2026 12:58:34 +0000 (UTC) Received: from smtpout-03.galae.net (smtpout-03.galae.net [185.246.85.4]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.47563.1784725108589844108 for ; Wed, 22 Jul 2026 05:58:29 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=dkim header.b=jwgGhme+; spf=pass (domain: bootlin.com, ip: 185.246.85.4, mailfrom: antonin.godard@bootlin.com) Received: from smtpout-01.galae.net (smtpout-01.galae.net [212.83.139.233]) by smtpout-03.galae.net (Postfix) with ESMTPS id C0DC34E40ECC for ; Wed, 22 Jul 2026 12:58:26 +0000 (UTC) Received: from mail.galae.net (mail.galae.net [212.83.136.155]) by smtpout-01.galae.net (Postfix) with ESMTPS id 9695E60388 for ; Wed, 22 Jul 2026 12:58:26 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id 07B6611BD1E93; Wed, 22 Jul 2026 14:58:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=dkim; t=1784725106; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=7Z13VE1DoJfhjr9Ib2V+1mBLxayxqP2gUh28KIYQJ9U=; b=jwgGhme+zjcvxRRBEn+AryRRQeF2CHCfTbfe0qorU+udW2BDVQNvLUEJDLyHXdeYJdHQRv ZWVlWKQSvGtJUuZhr/5gk8Qo8mumxgLp3H+fKuXBpr04k73TSrD3lC1kg9e6NI5r/mu036 4AHQLxerXz0ic68nHEpqH9Lskh8fc4L1Lbhn+Ci6TQPpVmGB7f2VfDlRszlx6yeyRKwe2Y /y6e3gqzYH2tO5Yn9qex400jVOKEYq58Olg0/UUn2Io+qneNgSPojUWwe3rMOBmsRmzTAV SUKMaRRGvJ2qITmNte5d5TsawGsXfbSl0g3p26pHNvWlm9yEj4s66PbgpLzX/Q== From: Antonin Godard Date: Wed, 22 Jul 2026 14:58:11 +0200 Subject: [PATCH 2/5] ref-manual/variables.rst: SSTATE_MIRRORS: remove uninative mapping note MIME-Version: 1.0 Message-Id: <20260722-sstate-mirrors-doc-v1-2-570baeb41c32@bootlin.com> References: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> In-Reply-To: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> To: docs@lists.yoctoproject.org Cc: Thomas Petazzoni , Antonin Godard X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=1722; i=antonin.godard@bootlin.com; h=from:subject:message-id; bh=iAToj9MmDrT+tX7Zm9C7sqTw0bc7qL1jgAEGonkXlsQ=; b=owEBbQKS/ZANAwAKAdGAQUApo6g2AcsmYgBqYL5v1bJrzX4Gh2r2X/LY2dWqWlBY0QI2gF9MF VXS+jlB8PCJAjMEAAEKAB0WIQSGSHJRiN1AG7mg0//RgEFAKaOoNgUCamC+bwAKCRDRgEFAKaOo NtHGD/0U8nkM/nUEcpHA9+Ukiw3bUrTbU4tLpWBi0xqGSoFeksyY2VXRRHEIrPaMOutVOBgQINr 76FoTi8PmX4CTYePo/ovJ5HIcsfp0GDIe/rf8BregW4+KrTcIzIcyEnnpsH+Yxf4FF9OrMJs4Dp vtmlTxyunT5dw2jSdwV+SvxvBpetNw05rFYvKIA7ZMLo3w6x5vcSe6n+3EaNaXof85LHz9sjYXX 8kPUpPT0Xro793RdOOBufHz/Ih8SuFPAyfbFWDobloie/cYe7f2apoxFx5KEfhu91bR9+N7VFor UzUWAa6qpYCSHK6yooQRaXi3eOmvWC6VHr96otT9B6KolSvHNjWl1NUc7EszTMtYmx7BjRVEhb7 UJeGPOFQST+0uevWfHVg8WFF7wKyIbgEf8veaf66S6RLZwYTCL/QpKFXdYXSNNkgDI56NRR4nhJ OO1QTsfhGdSPDu5rL1B4oyWSHl0U35p3OT3bEEUFXQiuRiocqtS/S930HB0TIhopR/C3QkXelI8 JNTFAOi1It/Cfv8K5OCxMOfTnkkNzS3J/z2OV2x4Y5zau9wreZ8vnN33bTFqLpBxG8c4UAenlVn GCFQWe4CU0Wni4seFj7lroLzojxtZvoSn5XVVxeIFcdwJ/Z7IiT8+k9utdjPv0GchM3//vIVKuv 5bCvQUiESjx1ijA== X-Developer-Key: i=antonin.godard@bootlin.com; a=openpgp; fpr=8648725188DD401BB9A0D3FFD180414029A3A836 X-Last-TLS-Session-Version: TLSv1.3 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 22 Jul 2026 12:58:34 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/10111 Since 552e037bf598 ("uninative/utils: Drop workarounds for gcc 4.8/4.9") in OE-Core the uninative directory created in the shared state directory no longer has "-4.8" or "-4.9" appended to it, and the directory is always named "universal", so there's no need for this mapping anymore. Signed-off-by: Antonin Godard --- documentation/ref-manual/variables.rst | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index 394797ca3..4344b6143 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -10100,16 +10100,6 @@ system and gives an overview of their function and contents. cache (sstate-cache) results from previous builds. The sstate-cache you point to can also be from builds on other machines. - When pointing to sstate build artifacts on another machine that uses - a different GCC version for native builds, you must configure - :term:`SSTATE_MIRRORS` with a regular expression that maps local search - paths to server paths. The paths need to take into account - :term:`NATIVELSBSTRING` set by the :ref:`ref-classes-uninative` class. - For example, the following maps the local search path ``universal-4.9`` - to the server-provided path server_url_sstate_path:: - - SSTATE_MIRRORS ?= "file://universal-4.9/(.*) https://server_url_sstate_path/universal-4.8/\1" - If a mirror uses the same structure as :term:`SSTATE_DIR`, you need to add "PATH" at the end as shown in the examples below. The build system substitutes the From patchwork Wed Jul 22 12:58:12 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonin Godard X-Patchwork-Id: 93217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39F19C44536 for ; Wed, 22 Jul 2026 12:58:34 +0000 (UTC) Received: from smtpout-03.galae.net (smtpout-03.galae.net [185.246.85.4]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.47277.1784725108979580443 for ; Wed, 22 Jul 2026 05:58:29 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=dkim header.b=G9uE14dx; spf=pass (domain: bootlin.com, ip: 185.246.85.4, mailfrom: antonin.godard@bootlin.com) Received: from smtpout-01.galae.net (smtpout-01.galae.net [212.83.139.233]) by smtpout-03.galae.net (Postfix) with ESMTPS id 4AA094E40ECE for ; Wed, 22 Jul 2026 12:58:27 +0000 (UTC) Received: from mail.galae.net (mail.galae.net [212.83.136.155]) by smtpout-01.galae.net (Postfix) with ESMTPS id 19A9A60388 for ; Wed, 22 Jul 2026 12:58:27 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id 7CDC311BD3D01; Wed, 22 Jul 2026 14:58:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=dkim; t=1784725106; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=V7r/yV9m1m4mDYk7tncDnLfhU+uBgjOy5Z/cFcqQuJY=; b=G9uE14dxibO67Q/3FumYCRSOr0C10XpwoJkdyHaZIR4iKtk0MQcugnyWjqp9kM8YqIVAV8 BDGTrCeGQncKPcENynqlfVj+8jmu2kR1hKz+udmn0VhT4Yhho3VC3NOZ+njqIUYIlB+17b LL2xZ7z1HXTuXA7va7psWyFRkjfmeLj64DQj1DKsxEogP9rQt5tBZH6VWwwFZ6/RGBQxiG hh7DDZ1QNauF0VA4HL+MKFF1v6jkrX1Uwv/BF4jRIqR5HxHfK74Op+EgyV93ApCQ8qQ16V KTqFBNwDuQS7nXBT2t++HVoHaRWucqYiWGmx8d8v6+JZJGZlq8mXxaVver420g== From: Antonin Godard Date: Wed, 22 Jul 2026 14:58:12 +0200 Subject: [PATCH 3/5] ref-manual/variables.rst: SSTATE_MIRRORS: remove directory structure note MIME-Version: 1.0 Message-Id: <20260722-sstate-mirrors-doc-v1-3-570baeb41c32@bootlin.com> References: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> In-Reply-To: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> To: docs@lists.yoctoproject.org Cc: Thomas Petazzoni , Antonin Godard X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=1535; i=antonin.godard@bootlin.com; h=from:subject:message-id; bh=hQmvAzoudH63vC0ls9vR69KvZHCqYem6AKRyrnVLo4U=; b=owEBbQKS/ZANAwAKAdGAQUApo6g2AcsmYgBqYL5vmikSI7Eu+gS43T86g1tYW5OM/yWUl7dEu mDkDjhuWfyJAjMEAAEKAB0WIQSGSHJRiN1AG7mg0//RgEFAKaOoNgUCamC+bwAKCRDRgEFAKaOo NtG7D/wOr1xR7KotWwRfBFIvWizIqkQBSJVYg6J9PtNWRDXBOoLQcUTW0iwjycGraIC0FcVU4ZR /3yjPlPufXg7OUKdkxc36kM+F8KzKynkmtMPI+1tQWYahvOFHaoFY66qUaAwTMbzfR144EtfqDx uFzD3w3SgQnuNoC0IoSlcKKiqXiaLwx75uC1BOABKBwniustzCeIVCjs9l4emVC8iF2Hqz4vpfB ir2nGZkiMsC3fWv6GVRWWiLTuztkFWJLJx91TwbuGKDszUD1uw3NY/asokk+WFCFfu2i9ZnY/Vc TG+7eZjsr2Pdym+O7l5DV5gjE1Y+mk0rlnJiVmsC+OGWO93TEzPwy964x6aj/sYoqYWwT0vo7hi ybjWNCgT1tXwg2QPAmi8gJgYGSC3qTD54SLMJYwbek39pliZHGJ8bcZE3RPLOaOOcAzSIhKkRWU 93GwkdlEDS3Msfzns5yYytz4TUTnCBBDZdSNbtlJh40Yki8sDhYZ7CX8P21K+GaHomMaDr250j+ 03h5wFTLyTYXgHxHwASodw43XFsFgzoZ9CBIPgJW7RmucuUuJ/hfRjmEldAUyTutna/4j5+aO5c AdfB/fvxnKWubTkHd+uODg97ZZ0IvlZ+W/9NYkEorYBlVy8QNS0NmzNxwBEawhL001n89g6FX8t e2PkB/zm/YkhQwQ== X-Developer-Key: i=antonin.godard@bootlin.com; a=openpgp; fpr=8648725188DD401BB9A0D3FFD180414029A3A836 X-Last-TLS-Session-Version: TLSv1.3 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 22 Jul 2026 12:58:34 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/10109 Setting up a shared state mirror would most likely always happen by sharing a shared state directory constructed by BitBake, which has a specific layout. Otherwise, the following definition: SSTATE_MIRRORS = "file://.* https://someserver.tld" would require the server to have all the sstate artifacts at the root of the server, in a flat layout, which is probably never the case. Signed-off-by: Antonin Godard --- documentation/ref-manual/variables.rst | 9 --------- 1 file changed, 9 deletions(-) diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index 4344b6143..e650fecda 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -10100,15 +10100,6 @@ system and gives an overview of their function and contents. cache (sstate-cache) results from previous builds. The sstate-cache you point to can also be from builds on other machines. - If a mirror uses the same structure as - :term:`SSTATE_DIR`, you need to add "PATH" at the - end as shown in the examples below. The build system substitutes the - correct path within the directory structure:: - - SSTATE_MIRRORS ?= "\ - file://.* https://someserver.tld/share/sstate/PATH;downloadfilename=PATH \ - file://.* file:///some-local-dir/sstate/PATH" - .. note:: If the mirror is protected behind a username and password, the From patchwork Wed Jul 22 12:58:13 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonin Godard X-Patchwork-Id: 93219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 366CFC531BC for ; Wed, 22 Jul 2026 12:58:35 +0000 (UTC) Received: from smtpout-03.galae.net (smtpout-03.galae.net [185.246.85.4]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.47564.1784725109780995732 for ; Wed, 22 Jul 2026 05:58:30 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=dkim header.b=KMCA4Ldf; spf=pass (domain: bootlin.com, ip: 185.246.85.4, mailfrom: antonin.godard@bootlin.com) Received: from smtpout-01.galae.net (smtpout-01.galae.net [212.83.139.233]) by smtpout-03.galae.net (Postfix) with ESMTPS id 1BE6C4E40ECF for ; Wed, 22 Jul 2026 12:58:28 +0000 (UTC) Received: from mail.galae.net (mail.galae.net [212.83.136.155]) by smtpout-01.galae.net (Postfix) with ESMTPS id E1D8760388 for ; Wed, 22 Jul 2026 12:58:27 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id 02BCC11BD3D0C; Wed, 22 Jul 2026 14:58:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=dkim; t=1784725107; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=0OZysO7RvypX10rqayIycr7HrN11fRitm9YkTxRfaV4=; b=KMCA4LdffD9LU6KgXjASkyN6j918AVWb/AX7wGyjpTLmoxndYDgFhxihcantBNd8xmQQxr VgBfQfgAJRUCyCcKqEMv1Z9VRQTCsprM2vwDQX3r4DD/QDVaE18ukDs7yVuLX95zDp2L8e 3nJAfIUZK2qG1NcvBjbr31c8FmA0jpoesgGjyWW7jW4xs7VeOKe/UTfgMn3DQRmZQHsxPo g3RsEeJFMGl/NDoKlAH8OMOC/tvMjP4SjPXe/aqiC2beZFFglnaEd2+JHJptau0fOh+jcz UQ5KpZ92N5B0GMWJ3w8LD4RSYX19DAbSEoLWFF3HO1pGSaA57FBrNDV0VBrRWw== From: Antonin Godard Date: Wed, 22 Jul 2026 14:58:13 +0200 Subject: [PATCH 4/5] ref-manual/variables.rst: SSTATE_MIRRORS: move password note to shared state mirror document MIME-Version: 1.0 Message-Id: <20260722-sstate-mirrors-doc-v1-4-570baeb41c32@bootlin.com> References: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> In-Reply-To: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> To: docs@lists.yoctoproject.org Cc: Thomas Petazzoni , Antonin Godard X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=3839; i=antonin.godard@bootlin.com; h=from:subject:message-id; bh=K4QnNEAxjqhCrhN4rZJ5q4eLFri+hYmAhekCJmOnuOs=; b=owEBbQKS/ZANAwAKAdGAQUApo6g2AcsmYgBqYL5vCVMQKSHi/Q+o3QHjVFpCx2mKy4CQNR5Xq qe0/4TIbqCJAjMEAAEKAB0WIQSGSHJRiN1AG7mg0//RgEFAKaOoNgUCamC+bwAKCRDRgEFAKaOo NnSwD/9213guXKowX6aUyAShykNDKj2q4UzmbOeLQbjIwopSzeh7iVL2gudECvl5yjwljSRY/CE Ip+2eIOhZD6PgauRTtYkguSDCL6sFallIGKYsCKBFnXPkYwwG0x4EvpM2CHCmk4yJOtw1r/Gfkc g93dQ/iTbq+zOVkYCGJU+kjQ7mHHO8SFnHJJ2ebqBsMsMxRl9xqX1btp+MIq7bv1/wjbcRZDhSt 6ipmSM1JtHUiC4+MvqlhiibWctcb9ePIuSr1+GufNRtYyxTUwe0uaiGvOU2nHYhbLKrQ/NLAXYz J5IAcN33axTgVGcThmbJWCkzFreDTPTSfervmsxmxBhN849DYyT1ZRQlPvK77b6lHMwbxSdbyA/ svYXlD/QfCx+XrCaLPRbz45yUnWDz07XO4kWZ2vXZx7vc47m8bm8XNdtysB5AH8RVs8LURkc4oX 8i08zxyMkli47eWztGtRtLS5soBDsBaxoaeevaRZEA0RVNhN/jkk4F73xdAMLEKrRYDphC9xM2p nFGeQa0sapVgMxebX+FPJuNu2n6zlI3/jKBY0gtCFtwRXqlVmuLdapzlFZSYFDS1WvcWMrPr5YC QcWw9bh+KnHF/inte0th5H7CNuL2ZL+9HwFzTf9BI70b0H5jA8tlCIW738PV7WpuTPd0SctRlbV aK5m631C6nF8RBQ== X-Developer-Key: i=antonin.godard@bootlin.com; a=openpgp; fpr=8648725188DD401BB9A0D3FFD180414029A3A836 X-Last-TLS-Session-Version: TLSv1.3 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 22 Jul 2026 12:58:35 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/10112 Let's gather the different tips on setting up a shared state mirror in the same document, so move the user/password note from the SSTATE_MIRRORS variable definition to the sstate-mirrors-setup.rst doc. Signed-off-by: Antonin Godard --- documentation/dev-manual/sstate-mirrors-setup.rst | 28 +++++++++++++++++++++++ documentation/ref-manual/variables.rst | 28 ----------------------- 2 files changed, 28 insertions(+), 28 deletions(-) diff --git a/documentation/dev-manual/sstate-mirrors-setup.rst b/documentation/dev-manual/sstate-mirrors-setup.rst index 6c369ff4e..e69aa00d1 100644 --- a/documentation/dev-manual/sstate-mirrors-setup.rst +++ b/documentation/dev-manual/sstate-mirrors-setup.rst @@ -112,6 +112,34 @@ Going Further :doc:`/dev-manual/hashequivserver` section of the Yocto Project Development Tasks Manual for more information. +- If the mirror is protected behind a username and password, the + :term:`build host` needs to be configured so the :term:`build system + ` is able to download the shared state cache using + authentication. + + The recommended way to do that is by setting the following parameters + in ``$HOME/.netrc`` (``$HOME`` being the :term:`build host` home + directory):: + + machine someserver.tld + login + password + + This file requires permissions set to ``400`` or ``600`` to prevent + other users from reading the file: + + .. code-block:: console + + $ chmod 600 "$HOME/.netrc" + + Another method to configure the username and password is from the + URL in :term:`SSTATE_MIRRORS` directly, with the ``user`` and ``pswd`` + parameters:: + + SSTATE_MIRRORS ?= "\ + file://.* https://someserver.tld/share/sstate/PATH;user=;pswd=;downloadfilename=PATH \ + " + - If the :term:`BB_NO_NETWORK` variable is set to "1" on clients as a means to disable any accesses to the network, the :term:`SSTATE_MIRROR_ALLOW_NETWORK` variable may be used to allow the clients to fetch from the shared state diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index e650fecda..3f4788648 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -10100,34 +10100,6 @@ system and gives an overview of their function and contents. cache (sstate-cache) results from previous builds. The sstate-cache you point to can also be from builds on other machines. - .. note:: - - If the mirror is protected behind a username and password, the - :term:`build host` needs to be configured so the :term:`build system - ` is able to download the sstate cache using - authentication. - - The recommended way to do that is by setting the following parameters - in ``$HOME/.netrc`` (``$HOME`` being the :term:`build host` home - directory):: - - machine someserver.tld - login - password - - This file requires permissions set to ``400`` or ``600`` to prevent - other users from reading the file:: - - chmod 600 "$HOME/.netrc" - - Another method to configure the username and password is from the - URL in :term:`SSTATE_MIRRORS` directly, with the ``user`` and ``pswd`` - parameters:: - - SSTATE_MIRRORS ?= "\ - file://.* https://someserver.tld/share/sstate/PATH;user=;pswd=;downloadfilename=PATH \ - " - The Yocto Project actually shares the cache data objects built by its autobuilder:: From patchwork Wed Jul 22 12:58:14 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonin Godard X-Patchwork-Id: 93221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89ADBC531C7 for ; Wed, 22 Jul 2026 12:58:35 +0000 (UTC) Received: from smtpout-03.galae.net (smtpout-03.galae.net [185.246.85.4]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.47279.1784725110905802733 for ; Wed, 22 Jul 2026 05:58:31 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=dkim header.b=oS6MQnhL; spf=pass (domain: bootlin.com, ip: 185.246.85.4, mailfrom: antonin.godard@bootlin.com) Received: from smtpout-01.galae.net (smtpout-01.galae.net [212.83.139.233]) by smtpout-03.galae.net (Postfix) with ESMTPS id 887B64E40ED0 for ; Wed, 22 Jul 2026 12:58:28 +0000 (UTC) Received: from mail.galae.net (mail.galae.net [212.83.136.155]) by smtpout-01.galae.net (Postfix) with ESMTPS id 5DFC160388 for ; Wed, 22 Jul 2026 12:58:28 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id BD7CF11BD3BAA; Wed, 22 Jul 2026 14:58:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=dkim; t=1784725108; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=/gTmnCpZaRXq55CRlfDNIDiqAN45Uk/GMTLNFeKgeD8=; b=oS6MQnhLKVbPp8zY/bHlpfueI7YvHGxfGEhW9eeg8AvwG+R+aN8GmbcRKVRRlFTLxFwoVX 8TJGmLv50TJZJELADTd7nKnSUUb79i0bDcjOMUuiIvBg3ddn1p6tGKHPzv9nxBW5kBmiDo 58kX6JtqW/Okjkv6mS2Wd7wgQ9gk3HMZ8IzQAzLPfikH9PVGN0qq4vbrmunq7pIdXlGDPA 1n/b7DnDBqdhKoF5nf3GjGg0uvvAbrDBIQEKr751kZKv5ZPLpOy+XUCdDVKz6o/2a06RZ6 x4POE+B9izeU6Y9tNd2BOnWVM1JIW5BUdmZshtnKpL8VQBgEcVaKfwj0Yp8pCA== From: Antonin Godard Date: Wed, 22 Jul 2026 14:58:14 +0200 Subject: [PATCH 5/5] ref-manual/variables.rst: SSTATE_MIRRORS: link to the new sstate-mirrors-setup doc MIME-Version: 1.0 Message-Id: <20260722-sstate-mirrors-doc-v1-5-570baeb41c32@bootlin.com> References: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> In-Reply-To: <20260722-sstate-mirrors-doc-v1-0-570baeb41c32@bootlin.com> To: docs@lists.yoctoproject.org Cc: Thomas Petazzoni , Antonin Godard X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=1120; i=antonin.godard@bootlin.com; h=from:subject:message-id; bh=3hBvOqt1rJo2ZKsAkDCnI2N9St2APiBz/42C4VTdf8I=; b=owEBbQKS/ZANAwAKAdGAQUApo6g2AcsmYgBqYL5vNvH9BhpJLEERm74E43pVRzzKE7XhgB0Uu AYyxeKRPxiJAjMEAAEKAB0WIQSGSHJRiN1AG7mg0//RgEFAKaOoNgUCamC+bwAKCRDRgEFAKaOo NoD9EACMCzmUPTK/kBuGlS91Qe6pIbZAbXQW71218AmeulduPMCo6BLeT9JKGucxX7uE3+ju217 UjneEkFLcLP8l5/z5ImRRR6BN4KEt2CHGRw07hrTTNwn7SMKafuKWeE+7ECUpSy7NhQ/2BPhwc5 AZfDsBYtncreUDPeQ/y4P4dh/hpHEn0SOCtHLtk9kxMi3oQcf0wXZev+fFW6DS0CvGIToWUvd1c nkTb7rWCwER6qPcqBDTpOgkjKTmk1paCslJgJXINoyE43imJPGcGeX2w3BHdYiW2y6/iSMV5cdn QXDY2zEr3N1bHCYg09+NwvqrPfm93aJ/FiqkUErFwEQoa3GnJZddyCFd9qCrS+PJDDARKP8/4V1 zDsjxwan6GHdO15SLOlyT3CSSLplz4MhFl+rryynb0Xfff8wBZAQz1XjNwHNozFUa7X7q2/0eYT fD7Ikf2S2+KBchFT0Jm/ar6OSVhE38g+uS/wJpYvNzF6gc2LKWh1HtgHIScc8GQCTRpu+XsGnrM fA3ciN6dpoIkMcoTWqB6DOF6ZlPVQVH7CEp70pHXcL4spBxIkSHuh/5VdAo9pInvPic6PkOVORi HjkALrV/ISi+vnNtnkXutW22KZjwsHisT/I7UMprP+mV7mV5Wfzcm7IJoPgl4xMjyudWWmBRM/J trJV5+2kRMomhSA== X-Developer-Key: i=antonin.godard@bootlin.com; a=openpgp; fpr=8648725188DD401BB9A0D3FFD180414029A3A836 X-Last-TLS-Session-Version: TLSv1.3 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 22 Jul 2026 12:58:35 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/10113 Add a link from the SSTATE_MIRRORS variable definition to our new sstate-mirrors-setup document, which instructs how to set a basic server along with tips on hash equivalence, etc. Signed-off-by: Antonin Godard --- documentation/ref-manual/variables.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index 3f4788648..2c1d77ec9 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -10110,6 +10110,10 @@ system and gives an overview of their function and contents. reusable in real projects building binaries optimized for a specific CPU family. + See the :doc:`/dev-manual/sstate-mirrors-setup` section of the Yocto + Project Development Tasks Manual for details on how to setup a shared + state mirror. + :term:`SSTATE_SCAN_FILES` Controls the list of files the OpenEmbedded build system scans for hardcoded installation paths. The variable uses a space-separated