diff mbox series

[1/3] sstate-cache-management: find work-shared stamps too

Message ID 20261005065322.30522-2-alex.kiernan@gmail.com
State New
Headers show
Series [1/3] sstate-cache-management: find work-shared stamps too | expand

Commit Message

Alex Kiernan Oct. 5, 2026, 6:53 a.m. UTC 
remove_by_stamps() collects the hashes still in use by globbing the stamps
directory at a fixed depth, "*/*/*.do_*", matching <arch>/<pn>/<file>; the
fixed depth is deliberate, so that the walk does not stat() the whole tree,
which over NFS costs a round trip per entry. But the shared-source recipes -
gcc-source, llvm-project-source and rust-source - set STAMP to
${STAMPS_DIR}/work-shared/<pn>-<pv>-<pr>, one level shallower, so their
hashes are never collected and --stamps-dir plans their sstate objects for
removal while a build still uses them. (externalsrc's work-shared/<pn>/ is
at the usual depth and was already found.)

Glob that layout too, at its own fixed depth, rather than recursively.

Against a cache shared by eight build directories, the stock script
planned 740 of 91486 files for removal; 43 of them, all gcc-source,
llvm-project-source and rust-source objects, had hashes in live
work-shared stamps. With this change it plans 697, the same list as a
recursive "**" glob produces. Filesystem calls during the run (strace):
353084 before, 353196 after, against 377568 for the recursive glob.

The shell script this replaced used "find $stamps -maxdepth 3", which did
include them.

Fixes: 2fa1b25d7485 ("sstate-cache-management: Rewrite in python")
AI-Generated: Claude Opus 5.5
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---
 scripts/sstate-cache-management.py | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
diff mbox series

Patch

diff --git a/scripts/sstate-cache-management.py b/scripts/sstate-cache-management.py
index f5737004c544..d088878a92df 100755
--- a/scripts/sstate-cache-management.py
+++ b/scripts/sstate-cache-management.py
@@ -142,6 +142,15 @@  def collect_sstate_paths(args):
     return paths
 
 
+# Where stamps live under STAMPS_DIR, each at a fixed depth: globbing those
+# depths exactly, rather than recursively, keeps the walk from stat()ing
+# everything in the tree, which over NFS is slow with a round trip per entry.
+STAMP_LAYOUTS = (
+    "*/*/",  # <arch>/<pn>/<pv>... (bitbake.conf); also externalsrc's work-shared/<pn>/
+    "work-shared/",  # work-shared/<pn>-<pv>-<pr>... (gcc-source, llvm-project-source, rust-source)
+)
+
+
 def remove_by_stamps(args, paths):
     all_sums = set()
     for stamps_dir in args.stamps_dir:
@@ -151,14 +160,16 @@  def remove_by_stamps(args, paths):
         all_sums |= set(
             [
                 re_sigdata.search(x.parts[-1]).group(1)
-                for x in stamps_path.glob("*/*/*.do_*.sigdata.*")
+                for layout in STAMP_LAYOUTS
+                for x in stamps_path.glob(layout + "*.do_*.sigdata.*")
             ]
         )
         re_setscene = re.compile(r"do_.*_setscene\.([^.]*)")
         all_sums |= set(
             [
                 re_setscene.search(x.parts[-1]).group(1)
-                for x in stamps_path.glob("*/*/*.do_*_setscene.*")
+                for layout in STAMP_LAYOUTS
+                for x in stamps_path.glob(layout + "*.do_*_setscene.*")
             ]
         )
     return [p for p in paths if p.bb_unihash not in all_sums]