[RFC] bitbake-getvar: Expand keys and run anonymous functions to get identical result to "bitbake -e"

From: Yoann Congal <yoann.congal@smile.fr>

With this path, outputs of
  bitbake-getvar DISTRO_FEATURES
and
  bitbake -e |grep '^DISTRO_FEATURES='
are the same.

Fixes [YOCTO #15638]

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>

---
RFC:
* I don't know if a "sync command" is the right choice here.
* Maybe the runAnonFunc call needs to be more integrated in the default
  flow?
---
 bin/bitbake-getvar | 5 +++++
 lib/bb/command.py  | 4 ++++
 lib/bb/tinfoil.py  | 6 ++++++
 3 files changed, 15 insertions(+)

On Sat, 2024-11-02 at 17:21 +0100, Yoann Congal via
lists.openembedded.org wrote:
> From: Yoann Congal <yoann.congal@smile.fr>
> 
> With this path, outputs of
>   bitbake-getvar DISTRO_FEATURES
> and
>   bitbake -e |grep '^DISTRO_FEATURES='
> are the same.
> 
> Fixes [YOCTO #15638]
> 
> Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
> 
> ---
> RFC:
> * I don't know if a "sync command" is the right choice here.
> * Maybe the runAnonFunc call needs to be more integrated in the
> default
>   flow?

This looks like the right direction.

I'm a bit concerned this modifies cooker.data on the server side
though. That is not meant to be expanded like this and may cause some
interesting other issues if left like this.

We therefore need to work on a copy of the datastore, so the sequence
is:

newdata = d.createCopy()
bb.data.expandKeys(newdata)
bb.parse.ast.runAnonFuncs(newdata)

and we just need to make that happen over the IPC. I think tinfoil's
datastore connector does already support this.

Also, it may be more efficient and a lot faster to run the expandKeys
on the server side. We might want to combine both operations into a
single piece of new tinfoil API?

A sync command should fine.

Cheers,

Richard

> ---
>  bin/bitbake-getvar | 5 +++++
>  lib/bb/command.py  | 4 ++++
>  lib/bb/tinfoil.py  | 6 ++++++
>  3 files changed, 15 insertions(+)
> 
> diff --git a/bin/bitbake-getvar b/bin/bitbake-getvar
> index 8901f99ae..1002d827b 100755
> --- a/bin/bitbake-getvar
> +++ b/bin/bitbake-getvar
> @@ -45,6 +45,11 @@ if __name__ == "__main__":
>              tinfoil.prepare(quiet=2, config_only=True)
>              d = tinfoil.config_data
>  
> +        # Expand keys and run anonymous functions to get identical
> result to
> +        # "bitbake -e"
> +        bb.data.expandKeys(d)
> +        tinfoil.runAnonFuncs()
> +
>          value = None
>          if args.flag:
>              value = d.getVarFlag(args.variable, args.flag,
> expand=not args.unexpand)
> diff --git a/lib/bb/command.py b/lib/bb/command.py
> index 1fcb9bf14..27e99a584 100644
> --- a/lib/bb/command.py
> +++ b/lib/bb/command.py
> @@ -24,6 +24,7 @@ import io
>  import bb.event
>  import bb.cooker
>  import bb.remotedata
> +import bb.parse
>  
>  class DataStoreConnectionHandle(object):
>      def __init__(self, dsindex=0):
> @@ -582,6 +583,9 @@ class CommandsSync:
>          return DataStoreConnectionHandle(idx)
>      parseRecipeFile.readonly = True
>  
> +    def runAnonFuncs(self, command, params):
> +        bb.parse.ast.runAnonFuncs(command.cooker.data)
> +
>  class CommandsAsync:
>      """
>      A class of asynchronous commands
> diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
> index dcd3910cc..4d6d07539 100644
> --- a/lib/bb/tinfoil.py
> +++ b/lib/bb/tinfoil.py
> @@ -633,6 +633,12 @@ class Tinfoil:
>          fn = self.get_recipe_file(pn)
>          return self.parse_recipe_file(fn)
>  
> +    def runAnonFuncs(self):
> +        """
> +        Run anonymous functions
> +        """
> +        self.run_command('runAnonFuncs')
> +
>      def parse_recipe_file(self, fn, appends=True, appendlist=None,
> config_data=None):
>          """
>          Parse the specified recipe file (with or without bbappends)
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16771):
> https://lists.openembedded.org/g/bitbake-devel/message/16771
> Mute This Topic: https://lists.openembedded.org/mt/109354058/1686473
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe:
> https://lists.openembedded.org/g/bitbake-devel/unsub [richard.purdie@linuxfoundation.org
> ]
> -=-=-=-=-=-=-=-=-=-=-=-
>