[RFC] base: Switch to use addpylib directive

Since bitbake now supports an official method to inject python modules,
switch to it.

Anyone using OE_EXTRA_IMPORTS will need to adjust their code accordingly,
probably switching to their own module namespace.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
[This patch is an RFC as part of a series on the bitbake-devel list]

 meta/classes-global/base.bbclass | 23 -----------------------
 meta/conf/layer.conf             |  2 ++
 meta/lib/oe/__init__.py          |  4 ++++
 3 files changed, 6 insertions(+), 23 deletions(-)

Hi,

I get the below backtrace with this patch on master-next. The same patchset
works fine on debian-11. Are there any specific versions of packages that
need to be installed as  dependency ?

Tested on host os: ubuntu-18.04/ubuntu-20.04

tuxbake@b8b6586bfb2c:/source/source/poky/build$ bitbake -e
ERROR: Error parsing configuration files
Traceback (most recent call last):
  File "/source/source/poky/bitbake/lib/bb/cookerdata.py", line 187, in
parse_config_file(fn='/source/source/poky/meta/conf/layer.conf',
data=<bb.data_smart.DataSmart object at 0x7f020867e250>, include=True):
     def parse_config_file(fn, data, include=True):
    >    return bb.parse.handle(fn, data, include)

  File "/source/source/poky/bitbake/lib/bb/parse/__init__.py", line 107, in
handle(fn='/source/source/poky/meta/conf/layer.conf',
data=<bb.data_smart.DataSmart object at 0x7f020867e250>, include=True):
                 with data.inchistory.include(fn):
    >                return h['handle'](fn, data, include)
         raise ParseError("not a BitBake file", fn)
  File "/source/source/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py",
line 152, in handle(fn='/source/source/poky/meta/conf/layer.conf',
data=<bb.data_smart.DataSmart object at 0x7f020867e250>, include=True):
         data.setVar('FILE', abs_fn)
    >    statements.eval(data)
         if oldfile:
  File "/source/source/poky/bitbake/lib/bb/parse/ast.py", line 21, in
StatementGroup.eval(data=<bb.data_smart.DataSmart object at
0x7f020867e250>):
             for statement in self:
    >            statement.eval(data)

  File "/source/source/poky/bitbake/lib/bb/parse/ast.py", line 324, in
PyLibNode.eval(data=<bb.data_smart.DataSmart object at 0x7f020867e250>):
                             bb.codeparser.modulecode_deps[entry][2] |=
bb.codeparser.modulecode_deps[dup][2]
    >                        bb.codeparser.modulecode_deps[entry][3] |=
bb.codeparser.modulecode_deps[dup][3]

TypeError: unsupported operand type(s) for |=: 'dict' and 'dict'



On Mon, 28 Nov 2022 at 03:06, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Since bitbake now supports an official method to inject python modules,
> switch to it.
>
> Anyone using OE_EXTRA_IMPORTS will need to adjust their code accordingly,
> probably switching to their own module namespace.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> [This patch is an RFC as part of a series on the bitbake-devel list]
>
>  meta/classes-global/base.bbclass | 23 -----------------------
>  meta/conf/layer.conf             |  2 ++
>  meta/lib/oe/__init__.py          |  4 ++++
>  3 files changed, 6 insertions(+), 23 deletions(-)
>
> diff --git a/meta/classes-global/base.bbclass
> b/meta/classes-global/base.bbclass
> index c81aa51ef3f..c4ac43c5696 100644
> --- a/meta/classes-global/base.bbclass
> +++ b/meta/classes-global/base.bbclass
> @@ -15,31 +15,8 @@ inherit utils
>  inherit utility-tasks
>  inherit logging
>
> -OE_EXTRA_IMPORTS ?= ""
> -
> -OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package
> oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license oe.qa
> oe.reproducible oe.rust oe.buildcfg ${OE_EXTRA_IMPORTS}"
> -OE_IMPORTS[type] = "list"
> -
>  PACKAGECONFIG_CONFARGS ??= ""
>
> -def oe_import(d):
> -    import sys
> -
> -    bbpath = [os.path.join(dir, "lib") for dir in
> d.getVar("BBPATH").split(":")]
> -    sys.path[0:0] = [dir for dir in bbpath if dir not in sys.path]
> -
> -    import oe.data
> -    for toimport in oe.data.typed_value("OE_IMPORTS", d):
> -        try:
> -            # Make a python object accessible from the metadata
> -            bb.utils._context[toimport.split(".", 1)[0]] =
> __import__(toimport)
> -        except AttributeError as e:
> -            bb.error("Error importing OE modules: %s" % str(e))
> -    return ""
> -
> -# We need the oe module name space early (before INHERITs get added)
> -OE_IMPORTED := "${@oe_import(d)}"
> -
>  inherit metadata_scm
>
>  def lsb_distro_identifier(d):
> diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf
> index 0ce90355baf..0fe05beebf0 100644
> --- a/meta/conf/layer.conf
> +++ b/meta/conf/layer.conf
> @@ -125,3 +125,5 @@ SSTATE_EXCLUDEDEPS_SYSROOT +=
> ".*->autoconf-archive-native"
>  # Avoid empty path entries
>  BITBAKEPATH :=
> "${@os.path.dirname(bb.utils.which(d.getVar('PATH'),'bitbake'))}"
>  PATH := "${@'${BITBAKEPATH}:' if '${BITBAKEPATH}' != '' else
> ''}${HOSTTOOLS_DIR}"
> +
> +addpylib ${LAYERDIR}/lib oe
> diff --git a/meta/lib/oe/__init__.py b/meta/lib/oe/__init__.py
> index 92f002d8726..9166420b00e 100644
> --- a/meta/lib/oe/__init__.py
> +++ b/meta/lib/oe/__init__.py
> @@ -6,3 +6,7 @@
>
>  from pkgutil import extend_path
>  __path__ = extend_path(__path__, __name__)
> +
> +BBIMPORTS = ["os", "sys", "time", "oe.data", "oe.path", "oe.utils",
> "oe.types", "oe.package", \
> +             "oe.packagegroup", "oe.sstatesig", "oe.lsb",
> "oe.cachedpath", "oe.license", \
> +             "oe.qa", "oe.reproducible", "oe.rust", "oe.buildcfg"]
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#173885):
> https://lists.openembedded.org/g/openembedded-core/message/173885
> Mute This Topic: https://lists.openembedded.org/mt/95297771/3616781
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> vishal.bhoj@linaro.org]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

On Mon, 2022-11-28 at 19:29 +0530, Vishal Bhoj wrote:
> I get the below backtrace with this patch on master-next. The same
> patchset works fine on debian-11. Are there any specific versions of
> packages that need to be installed as  dependency ?
> 
> Tested on host os: ubuntu-18.04/ubuntu-20.04

Which python versions are those? It looks like I perhaps assumed a
piece of newer python syntax accidentally. I suspect this code only
works with python 3.9+:

https://peps.python.org/pep-0584/

I was wondering whether to remove this piece entirely anyway, I need to
check the performance implications.

Cheers,

Richard

On Mon, 28 Nov 2022 at 20:14, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2022-11-28 at 19:29 +0530, Vishal Bhoj wrote:
> > I get the below backtrace with this patch on master-next. The same
> > patchset works fine on debian-11. Are there any specific versions of
> > packages that need to be installed as  dependency ?
> >
> > Tested on host os: ubuntu-18.04/ubuntu-20.04


> Which python versions are those? It looks like I perhaps assumed a
> piece of newer python syntax accidentally. I suspect this code only
> works with python 3.9+:
>

Looks like it is the case:
ubuntu-18.04: python 3.6
ubuntu-20.04: python 3.8

>
> https://peps.python.org/pep-0584/
>
> I was wondering whether to remove this piece entirely anyway, I need to
> check the performance implications.
>
> Cheers,
>
> Richard
>
>

On Mon, 2022-11-28 at 21:00 +0530, Vishal Bhoj wrote:
> 
> 
> On Mon, 28 Nov 2022 at 20:14, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Mon, 2022-11-28 at 19:29 +0530, Vishal Bhoj wrote:
> > > I get the below backtrace with this patch on master-next. The
> > > same
> > > patchset works fine on debian-11. Are there any specific versions
> > > of
> > > packages that need to be installed as  dependency ?
> > > 
> > > Tested on host os: ubuntu-18.04/ubuntu-20.04 
> > 
> > 
> > Which python versions are those? It looks like I perhaps assumed a
> > piece of newer python syntax accidentally. I suspect this code only
> > works with python 3.9+:
> > 
> 
> 
> Looks like it is the case:
> ubuntu-18.04: python 3.6
> ubuntu-20.04: python 3.8
> 

Thanks. I've tested and performance is the same without this chunk of
code so I've sent an updated version.

Cheers,

Richard