[1/2] sstate: don't limit the thread pool size when checking mirrors

This improves the time needed to check the mirrors,
reducing it from 50s to 8s when building the core-image-minimal
using the yocto upstream sstate and hashequivlance servers
on a machine with 8 cpu cores.

Tested with:

 SSTATE_MIRRORS = "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
 BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
 bitbake core-image-minimal

Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

On Thu, 2021-12-02 at 22:05 +0000, Jose Quaresma wrote:
> This improves the time needed to check the mirrors,
> reducing it from 50s to 8s when building the core-image-minimal
> using the yocto upstream sstate and hashequivlance servers
> on a machine with 8 cpu cores.
> 
> Tested with:
> 
>  SSTATE_MIRRORS = "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
>  BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
>  bitbake core-image-minimal
> 
> Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
> ---
>  meta/classes/sstate.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 0326d27c74..5e404d7cd8 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -1005,7 +1005,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
>              tasklist.append((tid, sstatefile))
>  
>          if tasklist:
> -            nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist))
> +            nproc = len(tasklist)
>  
>              progress = len(tasklist) >= 100
>              if progress:

No, we're not doing that. That opens a ton (thousands) of parallel connections
to the servers and isn't fair on the systems hosting the data. In many cases
that would look like a DoS attack.

Cheers,

Richard

Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia quinta,
2/12/2021 à(s) 23:16:

> On Thu, 2021-12-02 at 22:05 +0000, Jose Quaresma wrote:
> > This improves the time needed to check the mirrors,
> > reducing it from 50s to 8s when building the core-image-minimal
> > using the yocto upstream sstate and hashequivlance servers
> > on a machine with 8 cpu cores.
> >
> > Tested with:
> >
> >  SSTATE_MIRRORS = "file://.*
> http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
> >  BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
> >  bitbake core-image-minimal
> >
> > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
> > ---
> >  meta/classes/sstate.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> > index 0326d27c74..5e404d7cd8 100644
> > --- a/meta/classes/sstate.bbclass
> > +++ b/meta/classes/sstate.bbclass
> > @@ -1005,7 +1005,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False,
> currentcount=0, summary=True,
> >              tasklist.append((tid, sstatefile))
> >
> >          if tasklist:
> > -            nproc = min(int(d.getVar("BB_NUMBER_THREADS")),
> len(tasklist))
> > +            nproc = len(tasklist)
> >
> >              progress = len(tasklist) >= 100
> >              if progress:
>
> No, we're not doing that. That opens a ton (thousands) of parallel
> connections
> to the servers and isn't fair on the systems hosting the data. In many
> cases
> that would look like a DoS attack.
>
> Cheers,
>
> Richard
>
>
>
>
Got it and I have to agree that it will launch a DoS to the server.
However there are some cases around that have the mirror hosted locally on
the file system.
I'll try to find a solution that doesn't affect the http mirror.