diff mbox series

[1/2] cooker: use BB_HASHSERVE_DB_DIR for hash server database location

Message ID 20260112130418.3166002-1-alex.kanavin@gmail.com
State New
Headers show
Series [1/2] cooker: use BB_HASHSERVE_DB_DIR for hash server database location | expand

Commit Message

Alexander Kanavin Jan. 12, 2026, 1:04 p.m. UTC 
From: Alexander Kanavin <alex@linutronix.de>

If unset, the existing behavior is preseved.

The use case is sharing the database in bitbake-setup's driven builds
without having to set up/start/stop a common single server shared
between them (this is added to bitbake-setup in the next commit).

Also create the specified directory if it doesn't yet exist.

Add a check that the directory is not on a NFS path, and error out then,
guiding the user to set up a standalone hash equivalency server.

Note: the check runs 'stat' executable with a format string parameter,
ensuring it prints only the filesystem type and nothing else.
Python's standard library does not have a statfs() wrapper,
and using ctypes to call into it would've required delicate,
crash-prone data type definitions, full of magic numbers.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 .../bitbake-user-manual-ref-variables.rst     | 23 +++++++++++++++++++
 lib/bb/cooker.py                              | 11 ++++++++-
 lib/bb/utils.py                               |  8 +++++++
 3 files changed, 41 insertions(+), 1 deletion(-)
diff mbox series

Patch

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
index af911f306..4b3b10d46 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
@@ -475,6 +475,29 @@  overview of their function and contents.
       equivalences that correspond to Share State caches that are
       only available on specific clients.
 
+   :term:`BB_HASHSERVE_DB_DIR`
+      When :term:`BB_HASHSERVE` is set to ``auto``, bitbake will use
+      a private location inside a particular build directory for the sqlite
+      database file that holds hash equivalency data.
+
+      This variable allows using a different path, which can be shared
+      between multiple build directories and bitbake instances
+      that operate on them. This enables using a common ``${SSTATE_DIR}``
+      together with common hash equivalency data for local builds, without having to
+      separately manage a hash equivalency server.
+
+      .. note::
+
+         This variable cannot be set to a NFS mount and bitbake will error out then.
+         The reason is that NFS implementations can have file locking issues, which
+         can cause data loss and corruption when there are multiple writers operating
+         on a file at the same time as explained in https://sqlite.org/faq.html#q5
+
+         If you'd like to share hash equivalency data between multiple computers, you
+         need to set up a hash equivalency server separately and point :term:`BB_HASHSERVE`
+         to it. See https://docs.yoctoproject.org/dev-manual/hashequivserver.html for
+         additional information.
+
    :term:`BB_HASHSERVE_UPSTREAM`
       Specifies an upstream Hash Equivalence server.
 
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index fa7747d64..4b6ba3196 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -328,7 +328,16 @@  class BBCooker:
         if self.data.getVar("BB_HASHSERVE") == "auto":
             # Create a new hash server bound to a unix domain socket
             if not self.hashserv:
-                dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db"
+                bb_hashserve_db_dir = self.data.getVar("BB_HASHSERVE_DB_DIR")
+                if bb_hashserve_db_dir and utils.is_path_on_nfs(bb_hashserve_db_dir):
+                    bb.fatal("""Hash equivalency database location (set via BB_HASHSERVE_DB_DIR to {})
+cannot be on a NFS mount due to potential NFS locking issues between sqlite clients, per https://sqlite.org/faq.html#q5
+
+If you need to share the database between several computers, set up a permanently running hash equivalency server
+according to https://docs.yoctoproject.org/dev-manual/hashequivserver.html""".format(bb_hashserve_db_dir))
+                dbdir = bb_hashserve_db_dir or self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")
+                os.makedirs(dbdir, exist_ok=True)
+                dbfile = dbdir + "/hashserv.db"
                 upstream = self.data.getVar("BB_HASHSERVE_UPSTREAM") or None
                 if upstream:
                     try:
diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 366836bfc..974611bc7 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -2264,3 +2264,11 @@  def lock_timeout_nocheck(lock):
         if l:
             lock.release()
         sigmask(signal.SIG_SETMASK, s)
+
+def is_path_on_nfs(path):
+    """
+    Returns True if ``path`` argument is on a NFS mount.
+    """
+    import bb.process
+    fstype = bb.process.run("stat -f -c %T {}".format(path))[0].strip()
+    return fstype == "nfs"