| Message ID | 20241106202531.1701148-1-yoann.congal@smile.fr |
|---|---|
| State | New |
| Headers | show |
| Series | [RFC,v2] bitbake-getvar: Expand keys and run anonymous functions to mirror "bitbake -e" | expand |
On Wed, 2024-11-06 at 21:25 +0100, Yoann Congal via lists.openembedded.org wrote: > From: Yoann Congal <yoann.congal@smile.fr> > > Create a new tinfoil API "finalizeData" which does key expantion and > runs the anonymous function to allow bitbake-getvar to have the same > output as "bitbake -e" > > Fixes [YOCTO #15638] > > Signed-off-by: Yoann Congal <yoann.congal@smile.fr> > --- > RFC: > * Return type of the API is currently a bb.data_smart.DataSmart instead > of the expected bb.tinfoil.TinfoilDataStoreConnector because it is > faster in my tests. > * Tinfoil API adding and bb-getvar usage should be split in separate > commits. > * That bug most likely need a non-regression test. I was a little surprised this worked. At some point we clearly fixed the datastore to be serialisable and it can be sent over IPC. I did some tests as I was curious how much data this involved and for a single base configuration environment it is around single 1MB lump of data. I'm a bit torn but I think the correct way to do this is to keep the data store on the server side and use the connector to access it as in the v3 patch. This isn't a huge deal in this case and I'd prefer to have an example of that in the codebase. I'm guessing that whilst it is slower, it won't make much difference in the context of getVar. I did check and the data transferred is a couple of thousand bytes for the connector approach. The "slowness" is that there ends up being multiple trips over the IPC to get the expanded/unexpanded forms, varflags etc. but that is the code functioning as it should. I did also notice one more thing which doesn't block this patch but it is something someone will notice and complain about at some point so maybe a secondary bug we need to file/track. If you read bitbake-cookerdaemon.log for "bitbake-getvar -r bash" you will see disableDataTracking() call before finalizeData(). This means the variable history for anything run at finalize time will be missing. That may mean the history output for something like DISTRO_FEATURES will differ between bitbake -e and bitbake-getvar when a recipe is specified. The reason for the difference is that the parse_recipe_file turns history on and off which ends up disabling it. That would be a bug as the tinfoil session already has tracking. This is a separate bug though. Thanks for working on this, I think the v3 looks like the right code. Cheers, Richard
diff --git a/bin/bitbake-getvar b/bin/bitbake-getvar index 8901f99ae..bc0b989c8 100755 --- a/bin/bitbake-getvar +++ b/bin/bitbake-getvar @@ -45,6 +45,10 @@ if __name__ == "__main__": tinfoil.prepare(quiet=2, config_only=True) d = tinfoil.config_data + # Expand keys and run anonymous functions to get identical result to + # "bitbake -e" + d = tinfoil.finalizeData() + value = None if args.flag: value = d.getVarFlag(args.variable, args.flag, expand=not args.unexpand) diff --git a/lib/bb/command.py b/lib/bb/command.py index 1fcb9bf14..8d01a9e5f 100644 --- a/lib/bb/command.py +++ b/lib/bb/command.py @@ -24,6 +24,7 @@ import io import bb.event import bb.cooker import bb.remotedata +import bb.parse class DataStoreConnectionHandle(object): def __init__(self, dsindex=0): @@ -582,6 +583,12 @@ class CommandsSync: return DataStoreConnectionHandle(idx) parseRecipeFile.readonly = True + def finalizeData(self, command, params): + newdata = command.cooker.data.createCopy() + bb.data.expandKeys(newdata) + bb.parse.ast.runAnonFuncs(newdata) + return newdata + class CommandsAsync: """ A class of asynchronous commands diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py index dcd3910cc..35c65cb88 100644 --- a/lib/bb/tinfoil.py +++ b/lib/bb/tinfoil.py @@ -25,6 +25,7 @@ import bb.command import bb.remotedata from bb.main import setup_bitbake, BitBakeConfigParameters import bb.fetch2 +import bb.parse # We need this in order to shut down the connection to the bitbake server, @@ -633,6 +634,12 @@ class Tinfoil: fn = self.get_recipe_file(pn) return self.parse_recipe_file(fn) + def finalizeData(self): + """ + Run anonymous functions and expand keys + """ + return self.run_command('finalizeData') + def parse_recipe_file(self, fn, appends=True, appendlist=None, config_data=None): """ Parse the specified recipe file (with or without bbappends)