runqueue: Fix performance of multiconfigs with large overlap

There have been complaints about the performance of large multiconfig builds
for a while. The key missing data point was that the builds needed to have large
overlaps in sstate objects. This can be simulated by building the same things with
just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of
deferred tasks.

The issue is that the expensive checks in the setscene loop were hit every time
through runqueue's execute function before the check on deferred tasks. This leads
to task execution starvation as that only happens once per iteration.

Move the skip check earlier in the function which speeds things up enormously
and should improve performance of such builds for users.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/runqueue.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

I noticed that particularly 'bitbake core-image-ptest-all' tends to
sit for significant amount of time without spawning tasks. I
cherry-picked this patch and it doesn't seem to be helping :(

Alex

On Fri, 29 Nov 2024 at 13:55, Richard Purdie via
lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>
> There have been complaints about the performance of large multiconfig builds
> for a while. The key missing data point was that the builds needed to have large
> overlaps in sstate objects. This can be simulated by building the same things with
> just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of
> deferred tasks.
>
> The issue is that the expensive checks in the setscene loop were hit every time
> through runqueue's execute function before the check on deferred tasks. This leads
> to task execution starvation as that only happens once per iteration.
>
> Move the skip check earlier in the function which speeds things up enormously
> and should improve performance of such builds for users.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/runqueue.py | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> index 61608ac603..2179ee1302 100644
> --- a/lib/bb/runqueue.py
> +++ b/lib/bb/runqueue.py
> @@ -2210,6 +2210,9 @@ class RunQueueExecute:
>              # Find the next setscene to run
>              for nexttask in self.sorted_setscene_tids:
>                  if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values() and nexttask not in self.sq_harddep_deferred:
> +                    if nexttask in self.sq_deferred and self.sq_deferred[nexttask] not in self.runq_complete:
> +                        # Skip deferred tasks quickly before the 'expensive' tests below - this is key to performant multiconfig builds
> +                        continue
>                      if nexttask not in self.sqdata.unskippable and self.sqdata.sq_revdeps[nexttask] and \
>                              nexttask not in self.sq_needed_harddeps and \
>                              self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and \
> @@ -2239,8 +2242,7 @@ class RunQueueExecute:
>                          if t in self.runq_running and t not in self.runq_complete:
>                              continue
>                      if nexttask in self.sq_deferred:
> -                        if self.sq_deferred[nexttask] not in self.runq_complete:
> -                            continue
> +                        # Deferred tasks that were still deferred were skipped above so we now need to process
>                          logger.debug("Task %s no longer deferred" % nexttask)
>                          del self.sq_deferred[nexttask]
>                          valid = self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False, summary=False)
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16848): https://lists.openembedded.org/g/bitbake-devel/message/16848
> Mute This Topic: https://lists.openembedded.org/mt/109833404/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On Fri, 2024-11-29 at 14:12 +0100, Alexander Kanavin wrote:
> I noticed that particularly 'bitbake core-image-ptest-all' tends to
> sit for significant amount of time without spawning tasks. I
> cherry-picked this patch and it doesn't seem to be helping :(

Is that with an already populated sstate cache or empty? Hash
equivalence enabled locally or remote?

Does:

https://git.yoctoproject.org/poky/commit/?h=master-next&id=d8620795a7800801f52b098fe3f9508e9f3f26f2

help?

I'd like to try and reproduce if not...

Cheers,

Richard

On Fri, 2024-11-29 at 14:12 +0100, Alexander Kanavin wrote:
> I noticed that particularly 'bitbake core-image-ptest-all' tends to
> sit for significant amount of time without spawning tasks. I
> cherry-picked this patch and it doesn't seem to be helping :(

Try:

https://git.yoctoproject.org/poky/commit/?h=master-next&id=6522505769cea6bf96739469e64ec3a0a99be2b4

since the issue there appears to be a different performance bottleneck.

Cheers,

Richard

Thanks for looking into this. I’m trying to keep weekends relatively yocto
free (at least not start bitbake), so I’ll check this on Monday.

Alex

On Fri 29. Nov 2024 at 19.54, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2024-11-29 at 14:12 +0100, Alexander Kanavin wrote:
> > I noticed that particularly 'bitbake core-image-ptest-all' tends to
> > sit for significant amount of time without spawning tasks. I
> > cherry-picked this patch and it doesn't seem to be helping :(
>
> Try:
>
>
> https://git.yoctoproject.org/poky/commit/?h=master-next&id=6522505769cea6bf96739469e64ec3a0a99be2b4
>
> since the issue there appears to be a different performance bottleneck.
>
> Cheers,
>
> Richard
>

On Fri, 29 Nov 2024 at 19:54, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > I noticed that particularly 'bitbake core-image-ptest-all' tends to
> > sit for significant amount of time without spawning tasks. I
> > cherry-picked this patch and it doesn't seem to be helping :(
>
> Try:
>
> https://git.yoctoproject.org/poky/commit/?h=master-next&id=6522505769cea6bf96739469e64ec3a0a99be2b4
>
> since the issue there appears to be a different performance bottleneck.

Thanks, this does seem to improve things. The typical pattern was
pulling latest changes from master, and rebasing whatever I've been
working on (typically some version update batch). This results in
sstate being only half-populated and bitbake seemingly having trouble
figuring out what task to run next. Hash equivalence is always local
for me.

Alex