diff mbox series

command.py: cleanup bb.cache.parse_recipe

Message ID 20221124091045.12018-1-Qi.Chen@windriver.com
State Accepted, archived
Commit d386fa81848247a3d407debf889db8cbcce03359
Headers show
Series command.py: cleanup bb.cache.parse_recipe | expand

Commit Message

ChenQi Nov. 24, 2022, 9:10 a.m. UTC 
The parse_recipe has been removed from bb.cache, replace it with
the databuilder._parse_recipe.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 lib/bb/command.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

Richard Purdie Nov. 24, 2022, 12:01 p.m. UTC | #1 
On Thu, 2022-11-24 at 01:10 -0800, Chen Qi wrote:
> The parse_recipe has been removed from bb.cache, replace it with
> the databuilder._parse_recipe.
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  lib/bb/command.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/lib/bb/command.py b/lib/bb/command.py
> index fa9fd054c..c9571f63d 100644
> --- a/lib/bb/command.py
> +++ b/lib/bb/command.py
> @@ -564,7 +564,7 @@ class CommandsSync:
>          if config_data:
>              # We have to use a different function here if we're passing in a datastore
>              # NOTE: we took a copy above, so we don't do it here again
> -            envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
> +            envdata = command.cooker.databuilder._parse_recipe(config_data, fn, appendfiles, mc)['']
>          else:
>              # Use the standard path
>              envdata = command.cooker.databuilder.parseRecipe(fn, appendfiles)

How did you run into that? I'm a bit worried we're missing some test
coverage here?

Cheers,

Richard
ChenQi Nov. 24, 2022, 12:17 p.m. UTC | #2 
Hi Richard,

I met the error on our layerindex.

Error message is as below.
NOTE: Starting bitbake server...
ERROR: Unable to read /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted
-firmware-m/trusted-firmware-m_1.6.0.bb: Traceback (most recent call last):
  File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 90, in runCommand
    result = command_method(self, commandline)
  File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 567, in parseRecipeFile
    envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
AttributeError: module 'bb.cache' has no attribute 'parse_recipe'

Regards,
Qi

-----Original Message-----
From: Richard Purdie <richard.purdie@linuxfoundation.org> 
Sent: Thursday, November 24, 2022 8:02 PM
To: Chen, Qi <Qi.Chen@windriver.com>; bitbake-devel@lists.openembedded.org
Subject: Re: [bitbake-devel] [PATCH] command.py: cleanup bb.cache.parse_recipe

On Thu, 2022-11-24 at 01:10 -0800, Chen Qi wrote:
> The parse_recipe has been removed from bb.cache, replace it with the 
> databuilder._parse_recipe.
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  lib/bb/command.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/lib/bb/command.py b/lib/bb/command.py index 
> fa9fd054c..c9571f63d 100644
> --- a/lib/bb/command.py
> +++ b/lib/bb/command.py
> @@ -564,7 +564,7 @@ class CommandsSync:
>          if config_data:
>              # We have to use a different function here if we're passing in a datastore
>              # NOTE: we took a copy above, so we don't do it here again
> -            envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
> +            envdata = 
> + command.cooker.databuilder._parse_recipe(config_data, fn, 
> + appendfiles, mc)['']
>          else:
>              # Use the standard path
>              envdata = command.cooker.databuilder.parseRecipe(fn, 
> appendfiles)

How did you run into that? I'm a bit worried we're missing some test coverage here?

Cheers,

Richard
Richard Purdie Nov. 24, 2022, 3:13 p.m. UTC | #3 
Hi Qi,

On Thu, 2022-11-24 at 12:17 +0000, Chen, Qi wrote:
> I met the error on our layerindex.
> 
> Error message is as below.
> NOTE: Starting bitbake server...
> ERROR: Unable to read /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted
> -firmware-m/trusted-firmware-m_1.6.0.bb: Traceback (most recent call last):
>   File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 90, in runCommand
>     result = command_method(self, commandline)
>   File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 567, in parseRecipeFile
>     envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
> AttributeError: module 'bb.cache' has no attribute 'parse_recipe'

Thanks.

I had a look at what could call parseRecipeFile(). I can see some
commented out code in layerindex but it doesn't pass a config_data. The
only other calls are from tinfoil's parse_recipe_file() function and
none of those I can see pass config_data either.

Can you see where the config_data is coming from?

It looks like we've broken this at least once before:

https://git.yoctoproject.org/poky/commit/bitbake/lib/bb/command.py?id=70ee928f8f4152f198f4db61517c958d54f1c0a7

so we really need to figure out what is using it and add a test to
ensure coverage. Is this some local customisation you have to
layerindex?

Cheers,

Richard
Martin Jansa Nov. 29, 2022, 6:20 p.m. UTC | #4 
On Thu, Nov 24, 2022 at 1:17 PM Chen Qi <Qi.Chen@windriver.com> wrote:

> Hi Richard,
>
> I met the error on our layerindex.
>

"our" is
https://layers.openembedded.org

or some other instance in windriver?

I'm asking because parsing on https://layers.openembedded.org seems to be
broken for a while. I've checked randomly selected 3 layers and all show
the same issue as "Updates" tab in:
https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/

ERROR: fatal: Unable to create
'/opt/workdir/git___git_openembedded_org_bitbake/.git/index.lock': File
exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
ERROR: Traceback (most recent call last):
  File "update_layer.py", line 384, in main
    (tinfoil, tempdir) = recipeparse.init_parser(settings, branch,
bitbakepath, nocheckout=options.nocheckout, logger=logger)
  File "/opt/layerindex/layerindex/recipeparse.py", line 36, in init_parser
    utils.checkout_repo(bitbakepath, bitbake_ref, logger=logger)
  File "/opt/layerindex/layerindex/utils.py", line 282, in checkout_repo
    runcmd(['git', 'reset', '--hard'], repodir, logger=logger)
  File "/opt/layerindex/layerindex/utils.py", line 369, in runcmd
    raise e
subprocess.CalledProcessError: Command '['git', 'reset', '--hard']'
returned non-zero exit status 128.



> Error message is as below.
> NOTE: Starting bitbake server...
> ERROR: Unable to read
> /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted
> -firmware-m/trusted-firmware-m_1.6.0.bb: Traceback (most recent call
> last):
>   File
> "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py",
> line 90, in runCommand
>     result = command_method(self, commandline)
>   File
> "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py",
> line 567, in parseRecipeFile
>     envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
> AttributeError: module 'bb.cache' has no attribute 'parse_recipe'
>
> Regards,
> Qi
>
> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: Thursday, November 24, 2022 8:02 PM
> To: Chen, Qi <Qi.Chen@windriver.com>; bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH] command.py: cleanup
> bb.cache.parse_recipe
>
> On Thu, 2022-11-24 at 01:10 -0800, Chen Qi wrote:
> > The parse_recipe has been removed from bb.cache, replace it with the
> > databuilder._parse_recipe.
> >
> > Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> > ---
> >  lib/bb/command.py | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/lib/bb/command.py b/lib/bb/command.py index
> > fa9fd054c..c9571f63d 100644
> > --- a/lib/bb/command.py
> > +++ b/lib/bb/command.py
> > @@ -564,7 +564,7 @@ class CommandsSync:
> >          if config_data:
> >              # We have to use a different function here if we're passing
> in a datastore
> >              # NOTE: we took a copy above, so we don't do it here again
> > -            envdata = bb.cache.parse_recipe(config_data, fn,
> appendfiles, mc)['']
> > +            envdata =
> > + command.cooker.databuilder._parse_recipe(config_data, fn,
> > + appendfiles, mc)['']
> >          else:
> >              # Use the standard path
> >              envdata = command.cooker.databuilder.parseRecipe(fn,
> > appendfiles)
>
> How did you run into that? I'm a bit worried we're missing some test
> coverage here?
>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14107):
> https://lists.openembedded.org/g/bitbake-devel/message/14107
> Mute This Topic: https://lists.openembedded.org/mt/95234638/3617156
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> Martin.Jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
Robert Yang Dec. 1, 2022, 12:10 p.m. UTC | #5 
Hi Richard and Martin,

I've setup a pure oe-core layerindex (No Wind River specific settings) locally,
it works well.

For Richard's question, where is the config_data from, it is from 
layerindex-web/layerindex/update_layer.py:

envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data)

The error will be gone if remove "config_date=data", so we can confirm that it 
is from the line.

For Martin, maybe a git process had crashed so that index.lock was not cleaned
up, you can try to remove the lock file (Or ask layerindex maintainer to do that):

$ rm -f /opt/workdir/git___git_openembedded_org_bitbake/.git/index.lock

// Robert


On 11/30/22 02:20, Martin Jansa wrote:
> On Thu, Nov 24, 2022 at 1:17 PM Chen Qi <Qi.Chen@windriver.com 
> <mailto:Qi.Chen@windriver.com>> wrote:
> 
>     Hi Richard,
> 
>     I met the error on our layerindex.
> 
> 
> "our" is
> https://layers.openembedded.org 
> <https://urldefense.com/v3/__https://layers.openembedded.org__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcmIy7OIU$>
> 
> or some other instance in windriver?
> 
> I'm asking because parsing on https://layers.openembedded.org 
> <https://urldefense.com/v3/__https://layers.openembedded.org__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcmIy7OIU$> seems to be broken for a while. I've checked randomly selected 3 layers and all show the same issue as "Updates" tab in:
> https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/ <https://urldefense.com/v3/__https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcM7oxI8A$>
> 
> ERROR: fatal: Unable to create 
> '/opt/workdir/git___git_openembedded_org_bitbake/.git/index.lock': File exists.
> 
> Another git process seems to be running in this repository, e.g.
> an editor opened by 'git commit'. Please make sure all processes
> are terminated then try again. If it still fails, a git process
> may have crashed in this repository earlier:
> remove the file manually to continue.
> ERROR: Traceback (most recent call last):
>    File "update_layer.py", line 384, in main
>      (tinfoil, tempdir) = recipeparse.init_parser(settings, branch, bitbakepath, 
> nocheckout=options.nocheckout, logger=logger)
>    File "/opt/layerindex/layerindex/recipeparse.py", line 36, in init_parser
>      utils.checkout_repo(bitbakepath, bitbake_ref, logger=logger)
>    File "/opt/layerindex/layerindex/utils.py", line 282, in checkout_repo
>      runcmd(['git', 'reset', '--hard'], repodir, logger=logger)
>    File "/opt/layerindex/layerindex/utils.py", line 369, in runcmd
>      raise e
> subprocess.CalledProcessError: Command '['git', 'reset', '--hard']' returned 
> non-zero exit status 128.
> 
> 
> 
>     Error message is as below.
>     NOTE: Starting bitbake server...
>     ERROR: Unable to read
>     /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted
>     -firmware-m/trusted-firmware-m_1.6.0.bb
>     <https://urldefense.com/v3/__http://trusted-firmware-m_1.6.0.bb__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcR8tYvdQ$>: Traceback (most recent call last):
>        File
>     "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py",
>     line 90, in runCommand
>          result = command_method(self, commandline)
>        File
>     "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py",
>     line 567, in parseRecipeFile
>          envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
>     AttributeError: module 'bb.cache' has no attribute 'parse_recipe'
> 
>     Regards,
>     Qi
> 
>     -----Original Message-----
>     From: Richard Purdie <richard.purdie@linuxfoundation.org
>     <mailto:richard.purdie@linuxfoundation.org>>
>     Sent: Thursday, November 24, 2022 8:02 PM
>     To: Chen, Qi <Qi.Chen@windriver.com <mailto:Qi.Chen@windriver.com>>;
>     bitbake-devel@lists.openembedded.org
>     <mailto:bitbake-devel@lists.openembedded.org>
>     Subject: Re: [bitbake-devel] [PATCH] command.py
>     <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$>: cleanup bb.cache.parse_recipe
> 
>     On Thu, 2022-11-24 at 01:10 -0800, Chen Qi wrote:
>      > The parse_recipe has been removed from bb.cache, replace it with the
>      > databuilder._parse_recipe.
>      >
>      > Signed-off-by: Chen Qi <Qi.Chen@windriver.com <mailto:Qi.Chen@windriver.com>>
>      > ---
>      >  lib/bb/command.py
>     <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> | 2 +-
>      >  1 file changed, 1 insertion(+), 1 deletion(-)
>      >
>      > diff --git a/lib/bb/command.py
>     <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> b/lib/bb/command.py <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> index
>      > fa9fd054c..c9571f63d 100644
>      > --- a/lib/bb/command.py
>     <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$>
>      > +++ b/lib/bb/command.py
>     <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$>
>      > @@ -564,7 +564,7 @@ class CommandsSync:
>      >          if config_data:
>      >              # We have to use a different function here if we're passing
>     in a datastore
>      >              # NOTE: we took a copy above, so we don't do it here again
>      > -            envdata = bb.cache.parse_recipe(config_data, fn,
>     appendfiles, mc)['']
>      > +            envdata =
>      > + command.cooker.databuilder._parse_recipe(config_data, fn,
>      > + appendfiles, mc)['']
>      >          else:
>      >              # Use the standard path
>      >              envdata = command.cooker.databuilder.parseRecipe(fn,
>      > appendfiles)
> 
>     How did you run into that? I'm a bit worried we're missing some test
>     coverage here?
> 
>     Cheers,
> 
>     Richard
> 
>     -=-=-=-=-=-=-=-=-=-=-=-
>     Links: You receive all messages sent to this group.
>     View/Reply Online (#14107):
>     https://lists.openembedded.org/g/bitbake-devel/message/14107
>     <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/message/14107__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcFi8osuw$>
>     Mute This Topic: https://lists.openembedded.org/mt/95234638/3617156
>     <https://urldefense.com/v3/__https://lists.openembedded.org/mt/95234638/3617156__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcb2VAskM$>
>     Group Owner: bitbake-devel+owner@lists.openembedded.org
>     <mailto:bitbake-devel%2Bowner@lists.openembedded.org>
>     Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub
>     <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/unsub__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYc339AjuA$> [Martin.Jansa@gmail.com <mailto:Martin.Jansa@gmail.com>]
>     -=-=-=-=-=-=-=-=-=-=-=-
>
Richard Purdie Dec. 1, 2022, 5:42 p.m. UTC | #6 
On Thu, 2022-12-01 at 20:10 +0800, Robert Yang wrote:
> Hi Richard and Martin,
> 
> I've setup a pure oe-core layerindex (No Wind River specific settings) locally,
> it works well.
> 
> For Richard's question, where is the config_data from, it is from 
> layerindex-web/layerindex/update_layer.py:
> 
> envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data)
> 
> The error will be gone if remove "config_date=data", so we can confirm that it 
> is from the line.
> 

Thanks, that helps. I've sent a patch to the tinfoil selftests in OE-
Core so we will test this piece of API in future and try and prevent
regressions.

Cheers,

Richard
diff mbox series

Patch

diff --git a/lib/bb/command.py b/lib/bb/command.py
index fa9fd054c..c9571f63d 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -564,7 +564,7 @@  class CommandsSync:
         if config_data:
             # We have to use a different function here if we're passing in a datastore
             # NOTE: we took a copy above, so we don't do it here again
-            envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
+            envdata = command.cooker.databuilder._parse_recipe(config_data, fn, appendfiles, mc)['']
         else:
             # Use the standard path
             envdata = command.cooker.databuilder.parseRecipe(fn, appendfiles)