codeparser: Skipping typing when inspecting Python modules

From: Pedro Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>

If a custom python module is added thru BBIMPORTS and it
uses typing(Any,Tuple,Union...), codeparser will fail because
inspect.py raises TypeError exception if the object is a
built-in module, class, or function.

Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
---
 lib/bb/codeparser.py | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

On Thu, 2025-03-06 at 15:06 +0000, Pedro Ferreira via lists.openembedded.org wrote:
> From: Pedro Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
> 
> If a custom python module is added thru BBIMPORTS and it
> uses typing(Any,Tuple,Union...), codeparser will fail because
> inspect.py raises TypeError exception if the object is a
> built-in module, class, or function.
> 
> Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
> ---
>  lib/bb/codeparser.py | 22 ++++++++++++++--------
>  1 file changed, 14 insertions(+), 8 deletions(-)
> 
> diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py
> index d249af326..7f6adaf36 100644
> --- a/lib/bb/codeparser.py
> +++ b/lib/bb/codeparser.py
> @@ -72,15 +72,21 @@ def add_module_functions(fn, functions, namespace):
>              parser.parse_python(None, filename=fn, lineno=1, fixedhash=fixedhash+f)
>              #bb.warn("Cached %s" % f)
>          except KeyError:
> -            targetfn = inspect.getsourcefile(functions[f])
> -            if fn != targetfn:
> -                # Skip references to other modules outside this file
> -                #bb.warn("Skipping %s" % name)
> +            try:
> +                targetfn = inspect.getsourcefile(functions[f])
> +                if fn != targetfn:
> +                    # Skip references to other modules outside this file
> +                    #bb.warn("Skipping %s" % name)
> +                    continue
> +                lines, lineno = inspect.getsourcelines(functions[f])
> +                src = "".join(lines)
> +                parser.parse_python(src, filename=fn, lineno=lineno, fixedhash=fixedhash+f)
> +                #bb.warn("Not cached %s" % f)
> +            except TypeError:
> +                # Since Python inspect wont support special types
> +                # Skip references of typing special forms, aliases and tuples
> +                bb.warn("Skipping refs to %s : %s" % (f, type(functions[f]).__name__))
>                  continue
> -            lines, lineno = inspect.getsourcelines(functions[f])
> -            src = "".join(lines)
> -            parser.parse_python(src, filename=fn, lineno=lineno, fixedhash=fixedhash+f)
> -            #bb.warn("Not cached %s" % f)
>          execs = parser.execs.copy()
>          # Expand internal module exec references
>          for e in parser.execs:
> 


This may be a good reason not to use typing in python module code we're
importing. Catching and ignoring all TypeErrors like this sounds like
it will cause us other problems in the future by hiding real errors :(

Cheers,

Richard

On Thu, Mar 6, 2025 at 3:05 PM Richard Purdie via
lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>
> On Thu, 2025-03-06 at 15:06 +0000, Pedro Ferreira via lists.openembedded.org wrote:
> > From: Pedro Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
> >
> > If a custom python module is added thru BBIMPORTS and it
> > uses typing(Any,Tuple,Union...), codeparser will fail because
> > inspect.py raises TypeError exception if the object is a
> > built-in module, class, or function.
> >
> > Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
> > ---
> >  lib/bb/codeparser.py | 22 ++++++++++++++--------
> >  1 file changed, 14 insertions(+), 8 deletions(-)
> >
> > diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py
> > index d249af326..7f6adaf36 100644
> > --- a/lib/bb/codeparser.py
> > +++ b/lib/bb/codeparser.py
> > @@ -72,15 +72,21 @@ def add_module_functions(fn, functions, namespace):
> >              parser.parse_python(None, filename=fn, lineno=1, fixedhash=fixedhash+f)
> >              #bb.warn("Cached %s" % f)
> >          except KeyError:
> > -            targetfn = inspect.getsourcefile(functions[f])
> > -            if fn != targetfn:
> > -                # Skip references to other modules outside this file
> > -                #bb.warn("Skipping %s" % name)
> > +            try:
> > +                targetfn = inspect.getsourcefile(functions[f])
> > +                if fn != targetfn:
> > +                    # Skip references to other modules outside this file
> > +                    #bb.warn("Skipping %s" % name)
> > +                    continue
> > +                lines, lineno = inspect.getsourcelines(functions[f])
> > +                src = "".join(lines)
> > +                parser.parse_python(src, filename=fn, lineno=lineno, fixedhash=fixedhash+f)
> > +                #bb.warn("Not cached %s" % f)
> > +            except TypeError:
> > +                # Since Python inspect wont support special types
> > +                # Skip references of typing special forms, aliases and tuples
> > +                bb.warn("Skipping refs to %s : %s" % (f, type(functions[f]).__name__))
> >                  continue
> > -            lines, lineno = inspect.getsourcelines(functions[f])
> > -            src = "".join(lines)
> > -            parser.parse_python(src, filename=fn, lineno=lineno, fixedhash=fixedhash+f)
> > -            #bb.warn("Not cached %s" % f)
> >          execs = parser.execs.copy()
> >          # Expand internal module exec references
> >          for e in parser.execs:
> >
>
>
> This may be a good reason not to use typing in python module code we're
> importing. Catching and ignoring all TypeErrors like this sounds like
> it will cause us other problems in the future by hiding real errors :(

This could be OK. The API definition of getsourcefile() clearly states
it rasies TypeError on a builtin, and None if the file is not found,
so we can tell the difference. That said, it would be better to scope
it as tightly as possible to prevent hiding other errors in the code

try:
    targetfn = inspect.getsourcefile(functions[f])
except TypeError:
    # Builtin
    continue

>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#17396): https://lists.openembedded.org/g/bitbake-devel/message/17396
> Mute This Topic: https://lists.openembedded.org/mt/111549472/3616693
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [JPEWhacker@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>