[layerindexlib] Prevent loading unnecessary resources from layerindex-web

When I was using `bitbake-layers layerindex-fetch ...` I noticed it being quite
slow. Upon further investigation, I found that the call to `load_layerindex()`
inside layerindexlib was called with the deafult `load` argument.

The deafault was set to ['layerDependencies', 'recipes', 'machines', 'distros']
where the `layerindex-fetch` effectively requires only ['layerDependencies'].

So whenever a `layerindex-fetch` is used, we pull for example 'recipes', which
is about 18-20MB in size - it takes time to produce that response by the
layerindex-web and ship over the internet, just to be discarded.

This change addresses the issue by setting a "sane" default (empty `load` list)
and sets a correct, explicit `load` list whenever it's needed (e.g. toaster)
by the caller.

Lastly, there seems to be an error when the cooker index is loaded in the
process with the `load='layerDependencies'`, that's incorrect, as the
implementation expects also a list, so that's corrected as well.

Signed-off-by: Piotr Buliński <piotr@qbee.io>
---
 lib/bblayers/layerindex.py                       | 4 ++--
 lib/layerindexlib/__init__.py                    | 6 +++---
 lib/toaster/orm/management/commands/lsupdates.py | 3 ++-
 3 files changed, 7 insertions(+), 6 deletions(-)

I think probably you meant to send this to yocto-patches@ ML?


On 3/30/26 15:51, Piotr Buliński via lists.openembedded.org wrote:
> When I was using `bitbake-layers layerindex-fetch ...` I noticed it being quite
> slow. Upon further investigation, I found that the call to `load_layerindex()`
> inside layerindexlib was called with the deafult `load` argument.
> 
> The deafault was set to ['layerDependencies', 'recipes', 'machines', 'distros']
> where the `layerindex-fetch` effectively requires only ['layerDependencies'].
> 
> So whenever a `layerindex-fetch` is used, we pull for example 'recipes', which
> is about 18-20MB in size - it takes time to produce that response by the
> layerindex-web and ship over the internet, just to be discarded.
> 
> This change addresses the issue by setting a "sane" default (empty `load` list)
> and sets a correct, explicit `load` list whenever it's needed (e.g. toaster)
> by the caller.
> 
> Lastly, there seems to be an error when the cooker index is loaded in the
> process with the `load='layerDependencies'`, that's incorrect, as the
> implementation expects also a list, so that's corrected as well.
> 
> Signed-off-by: Piotr Buliński <piotr@qbee.io>
> ---
>  lib/bblayers/layerindex.py                       | 4 ++--
>  lib/layerindexlib/__init__.py                    | 6 +++---
>  lib/toaster/orm/management/commands/lsupdates.py | 3 ++-
>  3 files changed, 7 insertions(+), 6 deletions(-)
> 
> diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
> index 308a5532d..2ba1103a3 100644
> --- a/lib/bblayers/layerindex.py
> +++ b/lib/bblayers/layerindex.py
> @@ -116,7 +116,7 @@ class LayerIndexPlugin(ActionPlugin):
>  
>          # Load the cooker DB
>          cookerIndex = layerindexlib.LayerIndex(self.tinfoil.config_data)
> -        cookerIndex.load_layerindex('cooker://', load='layerDependencies')
> +        cookerIndex.load_layerindex('cooker://', load=['layerDependencies'])
>  
>          # Fast path, check if we already have what has been requested!
>          (dependencies, invalidnames) = cookerIndex.find_dependencies(names=args.layername, ignores=ignore_layers)
> @@ -137,7 +137,7 @@ class LayerIndexPlugin(ActionPlugin):
>  
>              for remoteurl in _construct_url(apiurl, branches):
>                  logger.plain("Loading %s..." % remoteurl)
> -                remoteIndex.load_layerindex(remoteurl)
> +                remoteIndex.load_layerindex(remoteurl, load=['layerDependencies'])
>  
>              if remoteIndex.is_empty():
>                  logger.error("Remote layer index %s is empty for branches %s" % (apiurl, branches))
> diff --git a/lib/layerindexlib/__init__.py b/lib/layerindexlib/__init__.py
> index c3265ddaa..e693e5c11 100644
> --- a/lib/layerindexlib/__init__.py
> +++ b/lib/layerindexlib/__init__.py
> @@ -174,15 +174,15 @@ class LayerIndex():
>          return res
>  
>  
> -    def load_layerindex(self, indexURI, load=['layerDependencies', 'recipes', 'machines', 'distros'], reload=False):
> +    def load_layerindex(self, indexURI, load=[], reload=False):
>          '''Load the layerindex.
>  
>             indexURI - An index to load.  (Use multiple calls to load multiple indexes)
>  
>             reload - If reload is True, then any previously loaded indexes will be forgotten.
>  
> -           load - List of elements to load.  Default loads all items.
> -                  Note: plugs may ignore this.
> +           load - List of elements to load. By default, an empty list is used to keep things lean.
> +                  Callers need to specify a minimal set of elements to load, such as ['layerDependencies'] for dependency resolution.
>  
>  The format of the indexURI:
>  
> diff --git a/lib/toaster/orm/management/commands/lsupdates.py b/lib/toaster/orm/management/commands/lsupdates.py
> index 0ee00aa15..153b26f60 100644
> --- a/lib/toaster/orm/management/commands/lsupdates.py
> +++ b/lib/toaster/orm/management/commands/lsupdates.py
> @@ -105,7 +105,8 @@ class Command(BaseCommand):
>              url_branches = ";branch=%s" % ','.join(allowed_branch_names)
>          else:
>              url_branches = ""
> -        layerindex.load_layerindex("%s%s" % (self.apiurl, url_branches))
> +        layerindex.load_layerindex("%s%s" % (self.apiurl, url_branches),
> +                                   load=['layerDependencies', 'recipes', 'machines', 'distros'])
>  
>          http_progress.stop()
>  
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#125866): https://lists.openembedded.org/g/openembedded-devel/message/125866
> Mute This Topic: https://lists.openembedded.org/mt/118578936/6084445
> Group Owner: openembedded-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [skandigraun@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Sorry, that was supposed to go to bitbake-devel@lists.openembedded.org


On Mon, Mar 30, 2026 at 4:24 PM Gyorgy Sarvari <skandigraun@gmail.com>
wrote:

> I think probably you meant to send this to yocto-patches@ ML?
>
>
> On 3/30/26 15:51, Piotr Buliński via lists.openembedded.org wrote:
> > When I was using `bitbake-layers layerindex-fetch ...` I noticed it
> being quite
> > slow. Upon further investigation, I found that the call to
> `load_layerindex()`
> > inside layerindexlib was called with the deafult `load` argument.
> >
> > The deafault was set to ['layerDependencies', 'recipes', 'machines',
> 'distros']
> > where the `layerindex-fetch` effectively requires only
> ['layerDependencies'].
> >
> > So whenever a `layerindex-fetch` is used, we pull for example 'recipes',
> which
> > is about 18-20MB in size - it takes time to produce that response by the
> > layerindex-web and ship over the internet, just to be discarded.
> >
> > This change addresses the issue by setting a "sane" default (empty
> `load` list)
> > and sets a correct, explicit `load` list whenever it's needed (e.g.
> toaster)
> > by the caller.
> >
> > Lastly, there seems to be an error when the cooker index is loaded in the
> > process with the `load='layerDependencies'`, that's incorrect, as the
> > implementation expects also a list, so that's corrected as well.
> >
> > Signed-off-by: Piotr Buliński <piotr@qbee.io>
> > ---
> >  lib/bblayers/layerindex.py                       | 4 ++--
> >  lib/layerindexlib/__init__.py                    | 6 +++---
> >  lib/toaster/orm/management/commands/lsupdates.py | 3 ++-
> >  3 files changed, 7 insertions(+), 6 deletions(-)
> >
> > diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
> > index 308a5532d..2ba1103a3 100644
> > --- a/lib/bblayers/layerindex.py
> > +++ b/lib/bblayers/layerindex.py
> > @@ -116,7 +116,7 @@ class LayerIndexPlugin(ActionPlugin):
> >
> >          # Load the cooker DB
> >          cookerIndex = layerindexlib.LayerIndex(self.tinfoil.config_data)
> > -        cookerIndex.load_layerindex('cooker://',
> load='layerDependencies')
> > +        cookerIndex.load_layerindex('cooker://',
> load=['layerDependencies'])
> >
> >          # Fast path, check if we already have what has been requested!
> >          (dependencies, invalidnames) =
> cookerIndex.find_dependencies(names=args.layername, ignores=ignore_layers)
> > @@ -137,7 +137,7 @@ class LayerIndexPlugin(ActionPlugin):
> >
> >              for remoteurl in _construct_url(apiurl, branches):
> >                  logger.plain("Loading %s..." % remoteurl)
> > -                remoteIndex.load_layerindex(remoteurl)
> > +                remoteIndex.load_layerindex(remoteurl,
> load=['layerDependencies'])
> >
> >              if remoteIndex.is_empty():
> >                  logger.error("Remote layer index %s is empty for
> branches %s" % (apiurl, branches))
> > diff --git a/lib/layerindexlib/__init__.py
> b/lib/layerindexlib/__init__.py
> > index c3265ddaa..e693e5c11 100644
> > --- a/lib/layerindexlib/__init__.py
> > +++ b/lib/layerindexlib/__init__.py
> > @@ -174,15 +174,15 @@ class LayerIndex():
> >          return res
> >
> >
> > -    def load_layerindex(self, indexURI, load=['layerDependencies',
> 'recipes', 'machines', 'distros'], reload=False):
> > +    def load_layerindex(self, indexURI, load=[], reload=False):
> >          '''Load the layerindex.
> >
> >             indexURI - An index to load.  (Use multiple calls to load
> multiple indexes)
> >
> >             reload - If reload is True, then any previously loaded
> indexes will be forgotten.
> >
> > -           load - List of elements to load.  Default loads all items.
> > -                  Note: plugs may ignore this.
> > +           load - List of elements to load. By default, an empty list
> is used to keep things lean.
> > +                  Callers need to specify a minimal set of elements to
> load, such as ['layerDependencies'] for dependency resolution.
> >
> >  The format of the indexURI:
> >
> > diff --git a/lib/toaster/orm/management/commands/lsupdates.py
> b/lib/toaster/orm/management/commands/lsupdates.py
> > index 0ee00aa15..153b26f60 100644
> > --- a/lib/toaster/orm/management/commands/lsupdates.py
> > +++ b/lib/toaster/orm/management/commands/lsupdates.py
> > @@ -105,7 +105,8 @@ class Command(BaseCommand):
> >              url_branches = ";branch=%s" % ','.join(allowed_branch_names)
> >          else:
> >              url_branches = ""
> > -        layerindex.load_layerindex("%s%s" % (self.apiurl, url_branches))
> > +        layerindex.load_layerindex("%s%s" % (self.apiurl, url_branches),
> > +                                   load=['layerDependencies',
> 'recipes', 'machines', 'distros'])
> >
> >          http_progress.stop()
> >
> >
> >
> >
> > 
> >
>
>