| Message ID | 20211123150331.2431516-1-ross.burton@arm.com |
|---|---|
| State | New |
| Headers | show |
| Series | [yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS | expand |
Is there a particular reason? Alex On Tue, 23 Nov 2021 at 16:03, Ross Burton <ross@burtonini.com> wrote: > Signed-off-by: Ross Burton <ross.burton@arm.com> > --- > config.json | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/config.json b/config.json > index 7203783..c0543d9 100644 > --- a/config.json > +++ b/config.json > @@ -47,6 +47,7 @@ > "PARALLEL_MAKE = '-j 16 -l 52'", > "XZ_MEMLIMIT = '5%'", > "XZ_THREADS = '8'", > + "ZSTD_THREADS = '8'", > "BB_TASK_NICE_LEVEL = '5'", > "BB_TASK_NICE_LEVEL:task-testimage = '0'", > "BB_TASK_IONICE_LEVEL = '2.7'", > -- > 2.25.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#55396): > https://lists.yoctoproject.org/g/yocto/message/55396 > Mute This Topic: https://lists.yoctoproject.org/mt/87260143/1686489 > Group Owner: yocto+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
My personal build machine has been crashing and dying due to overload with the default ZSTD_THREADS, and the AB has had problems with xz due to overload so I was being proactive and ensuring zstd is limited on the AB the same way xz is. Ross On Tue, 23 Nov 2021 at 17:11, Alexander Kanavin <alex.kanavin@gmail.com> wrote: > > Is there a particular reason? > > Alex > > On Tue, 23 Nov 2021 at 16:03, Ross Burton <ross@burtonini.com> wrote: >> >> Signed-off-by: Ross Burton <ross.burton@arm.com> >> --- >> config.json | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/config.json b/config.json >> index 7203783..c0543d9 100644 >> --- a/config.json >> +++ b/config.json >> @@ -47,6 +47,7 @@ >> "PARALLEL_MAKE = '-j 16 -l 52'", >> "XZ_MEMLIMIT = '5%'", >> "XZ_THREADS = '8'", >> + "ZSTD_THREADS = '8'", >> "BB_TASK_NICE_LEVEL = '5'", >> "BB_TASK_NICE_LEVEL:task-testimage = '0'", >> "BB_TASK_IONICE_LEVEL = '2.7'", >> -- >> 2.25.1 >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#55396): https://lists.yoctoproject.org/g/yocto/message/55396 >> Mute This Topic: https://lists.yoctoproject.org/mt/87260143/1686489 >> Group Owner: yocto+owner@lists.yoctoproject.org >> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >>
There was a long thread about this on oe-core recently: https://lists.openembedded.org/g/openembedded-core/message/157999 //Ernst
But the AB has not been exhibiting any problems with zstd, and this will degrade performance. Let's only fix what is broken. Alex On Tue, 23 Nov 2021 at 20:53, Ross Burton <ross@burtonini.com> wrote: > My personal build machine has been crashing and dying due to overload > with the default ZSTD_THREADS, and the AB has had problems with xz due > to overload so I was being proactive and ensuring zstd is limited on > the AB the same way xz is. > > Ross > > On Tue, 23 Nov 2021 at 17:11, Alexander Kanavin <alex.kanavin@gmail.com> > wrote: > > > > Is there a particular reason? > > > > Alex > > > > On Tue, 23 Nov 2021 at 16:03, Ross Burton <ross@burtonini.com> wrote: > >> > >> Signed-off-by: Ross Burton <ross.burton@arm.com> > >> --- > >> config.json | 1 + > >> 1 file changed, 1 insertion(+) > >> > >> diff --git a/config.json b/config.json > >> index 7203783..c0543d9 100644 > >> --- a/config.json > >> +++ b/config.json > >> @@ -47,6 +47,7 @@ > >> "PARALLEL_MAKE = '-j 16 -l 52'", > >> "XZ_MEMLIMIT = '5%'", > >> "XZ_THREADS = '8'", > >> + "ZSTD_THREADS = '8'", > >> "BB_TASK_NICE_LEVEL = '5'", > >> "BB_TASK_NICE_LEVEL:task-testimage = '0'", > >> "BB_TASK_IONICE_LEVEL = '2.7'", > >> -- > >> 2.25.1 > >> > >> > >> -=-=-=-=-=-=-=-=-=-=-=- > >> Links: You receive all messages sent to this group. > >> View/Reply Online (#55396): > https://lists.yoctoproject.org/g/yocto/message/55396 > >> Mute This Topic: https://lists.yoctoproject.org/mt/87260143/1686489 > >> Group Owner: yocto+owner@lists.yoctoproject.org > >> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [ > alex.kanavin@gmail.com] > >> -=-=-=-=-=-=-=-=-=-=-=- > >> >
On Wed, Nov 24, 2021 at 2:18 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Wed, 2021-11-24 at 09:00 +0100, Alexander Kanavin wrote: > > But the AB has not been exhibiting any problems with zstd, and this will > > degrade performance. Let's only fix what is broken. > > I'm not sure I agree with that. > > We have 60+ "intermittent" bugs and some of us are in weekly meetings trying to > do something about working out why these are failing. It feels like we're not > really getting too far with some subset of them and it is using up a lot of the > SWAT and bug triage time. > > We've made a few changes to try and reduce the load spikes on the systems and > this fits with the other changes we've made. > From a different data point, we have clipped the parallelism for XZ and ZSTD internally to very low ( 2 and 4 ) and it has in fact reduced unexpected failures and seen no impact on build performance. Since parallelism settings are myopic for these tools, they greedily take every CPU which is not best for overall build thats doing many things in parallel, so its best to curtail them to a conservative value. > Cheers, > > Richard > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > You automatically follow any topics you start or reply to. > View/Reply Online (#55404): https://lists.yoctoproject.org/g/yocto/message/55404 > Mute This Topic: https://lists.yoctoproject.org/mt/87260143/1997914 > Group Owner: yocto+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/config.json b/config.json index 7203783..c0543d9 100644 --- a/config.json +++ b/config.json @@ -47,6 +47,7 @@ "PARALLEL_MAKE = '-j 16 -l 52'", "XZ_MEMLIMIT = '5%'", "XZ_THREADS = '8'", + "ZSTD_THREADS = '8'", "BB_TASK_NICE_LEVEL = '5'", "BB_TASK_NICE_LEVEL:task-testimage = '0'", "BB_TASK_IONICE_LEVEL = '2.7'",
Signed-off-by: Ross Burton <ross.burton@arm.com> --- config.json | 1 + 1 file changed, 1 insertion(+)