diff mbox series

[kirkstone,2.0,1/1] data_smart: Improve performance for VariableHistory

Message ID ec2a99a077da9aa0e99e8b05e0c65dcbd45864b1.1723554034.git.steve@sakoman.com
State Accepted, archived
Commit ec2a99a077da9aa0e99e8b05e0c65dcbd45864b1
Headers show
Series [kirkstone,2.0,1/1] data_smart: Improve performance for VariableHistory | expand

Commit Message

Steve Sakoman Aug. 13, 2024, 1:09 p.m. UTC 
From: Robert Yang <liezhi.yang@windriver.com>

Fixed:
- BBMULTICONFIG = "qemux86-64 qemuarm64" and more than 70 layers in BBLAYERS
$ bitbake -p -P
Check profile.log.processed, the record() cost more than 20 seconds, it is less
than 1 second when multiconfig is not enabled, and there would be the following
error when more muticonfigs are enabled:

Timeout while waiting for a reply from the bitbake server

Don't change the type of loginfo['detail'] or re-assign it can make record()
back to less than 1 second, this won't affect COW since loginfo is a mutable
type.

The time mainly affected by two factors:
1) The number of enabled layers, nearly 1 second added per layer when the
   number is larger than 50.

2) The global var such as USER_CLASSES, about 1 ~ 2 seconds added per layer
   when the layers number is larger than 50.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0596aa0d5b0e4ed3db11b5bd560f1d3439963a41)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/data_smart.py | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
diff mbox series

Patch

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index dd20ca557..111377a75 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -261,12 +261,9 @@  class VariableHistory(object):
             return
         if 'op' not in loginfo or not loginfo['op']:
             loginfo['op'] = 'set'
-        if 'detail' in loginfo:
-            loginfo['detail'] = str(loginfo['detail'])
         if 'variable' not in loginfo or 'file' not in loginfo:
             raise ValueError("record() missing variable or file.")
         var = loginfo['variable']
-
         if var not in self.variables:
             self.variables[var] = []
         if not isinstance(self.variables[var], list):
@@ -325,7 +322,8 @@  class VariableHistory(object):
                     flag = '[%s] ' % (event['flag'])
                 else:
                     flag = ''
-                o.write("#   %s %s:%s%s\n#     %s\"%s\"\n" % (event['op'], event['file'], event['line'], display_func, flag, re.sub('\n', '\n#     ', event['detail'])))
+                o.write("#   %s %s:%s%s\n#     %s\"%s\"\n" % \
+                    (event['op'], event['file'], event['line'], display_func, flag, re.sub('\n', '\n#     ', str(event['detail']))))
             if len(history) > 1:
                 o.write("# pre-expansion value:\n")
                 o.write('#   "%s"\n' % (commentVal))
@@ -379,7 +377,7 @@  class VariableHistory(object):
             if isset and event['op'] == 'set?':
                 continue
             isset = True
-            items = d.expand(event['detail']).split()
+            items = d.expand(str(event['detail'])).split()
             for item in items:
                 # This is a little crude but is belt-and-braces to avoid us
                 # having to handle every possible operation type specifically