diff mbox series

cache: Reuse os.path.exists() results during dependency validation

Message ID 20260923170633.1313487-1-ecordonnier@snap.com
State New
Headers show
Series cache: Reuse os.path.exists() results during dependency validation | expand

Commit Message

Etienne Cordonnier Sept. 23, 2026, 5:06 p.m. UTC 
From: Etienne Cordonnier <ecordonnier@snap.com>

cacheValidUpdate() checks each recipe's file_checksums entries with
os.path.exists(). Many recipes reference the same optional class or
checksum candidate paths, so the same path is checked repeatedly within
one validation pass. In a warm parse of 5,112 recipes (poky, meta-yocto,
meta-oe, meta-python), the real cache data shows 585,348 such calls
against only 48,757 unique paths, with the hottest path checked 16,894
times: 536,591 calls are redundant. At 3.5us per os.path.exists() call on
a nonexistent path on this machine, that is about 1.9s of avoidable
syscall time per validation pass. Cache the result per Cache instance,
which is discarded and recreated at the start of each validation pass, so
a filesystem change between passes is still observed.

AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
 lib/bb/cache.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
diff mbox series

Patch

diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index 2361c5684..01020f844 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -401,6 +401,8 @@  class Cache(object):
         self.clean = set()
         self.checked = set()
         self.depends_cache = {}
+        # os.path.exists() results for this validation pass (see file_exists())
+        self.exists_cache = {}
         self.data_fn = None
         self.cacheclean = True
         self.data_hash = data_hash
@@ -619,7 +621,8 @@  class Cache(object):
                     if not f:
                         continue
                     f, exist = f.rsplit(":", 1)
-                    if (exist == "True" and not os.path.exists(f)) or (exist == "False" and os.path.exists(f)):
+                    file_exists = self.file_exists(f)
+                    if (exist == "True" and not file_exists) or (exist == "False" and file_exists):
                         self.logger.debug2("%s's file checksum list file %s changed",
                                              fn, f)
                         self.remove(fn)
@@ -657,6 +660,11 @@  class Cache(object):
         self.clean.add(fn)
         return True
 
+    def file_exists(self, fn):
+        if fn not in self.exists_cache:
+            self.exists_cache[fn] = os.path.exists(fn)
+        return self.exists_cache[fn]
+
     def remove(self, fn):
         """
         Remove a fn from the cache