mbox series

[RFC,0/1] siggen: Support non-compressed sigdata files

Message ID 20250515144455.2799533-1-philip.lorenz@bmw.de
Headers show
Series siggen: Support non-compressed sigdata files | expand

Message

Philip Lorenz May 15, 2025, 2:44 p.m. UTC 
To gain a better understanding of sstate invalidations in our CI systems, we
are planning to collect all sigdata files produced by all of our builds.
However, our typical builds produce 200 - 300 MB of sigdata files which would
significantly increase the log data we store for each build.

Evaluations showed that using zstd to compress uncompressed sigdata files
yielded significant compression rate improvements (in the order of 6-16x) which
lead to the creation of this patch. The patch adds support for creating
uncompressed sigdata files, and more importantly, transparent support for
consuming them. In turn, utilities such as bitbake-dumpsig can operate both
on compressed and uncompressed files, which allows mixing of compressed and
uncompressed files (e.g. compressed .siginfo files from sstate and uncompressed
.sigdata files produced by a build).

We've also tried other approaches (e.g. recompressing the files using zstd with
a dictionary trained on the decompressed files) and these also yielded somewhat
improved compression ratios. However, the results were still 10 times larger
than compressing the uncompressed files which finally lead to the
implementation of the patch.

Let me know if this is something that would be appropriate for inclusion and
I'll add some unit tests and resubmit for inclusion.

Signed-off-by: Philip Lorenz philip.lorenz@bmw.de

Philip Lorenz (1):
  siggen: Support non-compressed sigdata files

 lib/bb/siggen.py | 84 +++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 76 insertions(+), 8 deletions(-)