| Message ID | fde984bfa988c26cf7ef4cacf4ac830d974129b0.1650591341.git.paul.eggleton@linux.microsoft.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/9] migration-3.4: add missing entry on EXTRA_USERS_PARAMS | expand |
Hi Paul, On 4/22/22 03:40, Paul Eggleton wrote: > ZSTD_THREADS is new for kirkstone, XZ_THREADS was introduced in dunfell. > Separate patch please so XZ_THREADS can be backported to all branches up to dunfell. > Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com> > --- > documentation/ref-manual/variables.rst | 13 ++++++++++++- > 1 file changed, 12 insertions(+), 1 deletion(-) > > diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst > index b9448b8..4803578 100644 > --- a/documentation/ref-manual/variables.rst > +++ b/documentation/ref-manual/variables.rst > @@ -8737,4 +8737,15 @@ system and gives an overview of their function and contents. > > The default value of :term:`XSERVER`, if not specified in the machine > configuration, is "xserver-xorg xf86-video-fbdev xf86-input-evdev". > - > + > + :term:`XZ_THREADS` > + Specifies the number of parallel threads that should be used when > + using xz compression. By default this scales with core count, but > + is never set less than 2 to ensure that multi-threaded mode is > + always used (so that the output file contents are deterministic). > + I don't see how having it set to 1 breaks deterministic builds, do you mind explaining? Also, this feels like explanation of the implementation more than what the user should do with this variable. If the variable should never be <2, then it needs to be explicitly stated here (and eventually the reason). Maybe it'd make sense to explain why would anyone want to modify this variable (which IIRC, is for when you have a nproc/ram too high and it OOMs when creating packages). > + :term:`ZSTD_THREADS` > + Specifies the number of parallel threads that should be used when > + using ZStandard compression. By default this scales with core count, > + but is never set less than 2 to ensure that multi-threaded mode is > + always used (so that the output file contents are deterministic). > Ditto. Cheers, Quentin
On Fri, 2022-04-22 at 12:40 +0200, Quentin Schulz wrote: > Hi Paul, > > On 4/22/22 03:40, Paul Eggleton wrote: > > ZSTD_THREADS is new for kirkstone, XZ_THREADS was introduced in dunfell. > > > > Separate patch please so XZ_THREADS can be backported to all branches up > to dunfell. > > > Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com> > > --- > > documentation/ref-manual/variables.rst | 13 ++++++++++++- > > 1 file changed, 12 insertions(+), 1 deletion(-) > > > > diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst > > index b9448b8..4803578 100644 > > --- a/documentation/ref-manual/variables.rst > > +++ b/documentation/ref-manual/variables.rst > > @@ -8737,4 +8737,15 @@ system and gives an overview of their function and contents. > > > > The default value of :term:`XSERVER`, if not specified in the machine > > configuration, is "xserver-xorg xf86-video-fbdev xf86-input-evdev". > > - > > + > > + :term:`XZ_THREADS` > > + Specifies the number of parallel threads that should be used when > > + using xz compression. By default this scales with core count, but > > + is never set less than 2 to ensure that multi-threaded mode is > > + always used (so that the output file contents are deterministic). > > + > > I don't see how having it set to 1 breaks deterministic builds, do you > mind explaining? > > Also, this feels like explanation of the implementation more than what > the user should do with this variable. I think having a cautionary note here about reproducible builds is important. > If the variable should never be <2, then it needs to be explicitly > stated here (and eventually the reason). You can set it, you just won't have reproducible builds. > Maybe it'd make sense to explain why would anyone want to modify this > variable (which IIRC, is for when you have a nproc/ram too high and it > OOMs when creating packages). That would be nice, yes. I've gone ahead and tried a new version of this patch split into two and adding XZ_MEMLIMIT too (ZSTD doesn't have that). Cheers, Richard
Hi Richard, On 4/22/22 13:46, Richard Purdie wrote: > On Fri, 2022-04-22 at 12:40 +0200, Quentin Schulz wrote: >> Hi Paul, >> >> On 4/22/22 03:40, Paul Eggleton wrote: >>> ZSTD_THREADS is new for kirkstone, XZ_THREADS was introduced in dunfell. >>> >> >> Separate patch please so XZ_THREADS can be backported to all branches up >> to dunfell. >> >>> Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com> >>> --- >>> documentation/ref-manual/variables.rst | 13 ++++++++++++- >>> 1 file changed, 12 insertions(+), 1 deletion(-) >>> >>> diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst >>> index b9448b8..4803578 100644 >>> --- a/documentation/ref-manual/variables.rst >>> +++ b/documentation/ref-manual/variables.rst >>> @@ -8737,4 +8737,15 @@ system and gives an overview of their function and contents. >>> >>> The default value of :term:`XSERVER`, if not specified in the machine >>> configuration, is "xserver-xorg xf86-video-fbdev xf86-input-evdev". >>> - >>> + >>> + :term:`XZ_THREADS` >>> + Specifies the number of parallel threads that should be used when >>> + using xz compression. By default this scales with core count, but >>> + is never set less than 2 to ensure that multi-threaded mode is >>> + always used (so that the output file contents are deterministic). >>> + >> >> I don't see how having it set to 1 breaks deterministic builds, do you >> mind explaining? >> >> Also, this feels like explanation of the implementation more than what >> the user should do with this variable. > > I think having a cautionary note here about reproducible builds is important. > >> If the variable should never be <2, then it needs to be explicitly >> stated here (and eventually the reason). > > You can set it, you just won't have reproducible builds. > My understanding right now is that builds with XZ_THREADS=1 are reproducible, and XZ_THREADS>2 are reproducible too. But if you switch between them, the builds aren't reproducible. Is that correct? Cheers, Quentin
On Fri, 2022-04-22 at 14:47 +0200, Quentin Schulz wrote: > Hi Richard, > > On 4/22/22 13:46, Richard Purdie wrote: > > On Fri, 2022-04-22 at 12:40 +0200, Quentin Schulz wrote: > > > Hi Paul, > > > > > > On 4/22/22 03:40, Paul Eggleton wrote: > > > > ZSTD_THREADS is new for kirkstone, XZ_THREADS was introduced in dunfell. > > > > > > > > > > Separate patch please so XZ_THREADS can be backported to all branches up > > > to dunfell. > > > > > > > Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com> > > > > --- > > > > documentation/ref-manual/variables.rst | 13 ++++++++++++- > > > > 1 file changed, 12 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst > > > > index b9448b8..4803578 100644 > > > > --- a/documentation/ref-manual/variables.rst > > > > +++ b/documentation/ref-manual/variables.rst > > > > @@ -8737,4 +8737,15 @@ system and gives an overview of their function and contents. > > > > > > > > The default value of :term:`XSERVER`, if not specified in the machine > > > > configuration, is "xserver-xorg xf86-video-fbdev xf86-input-evdev". > > > > - > > > > + > > > > + :term:`XZ_THREADS` > > > > + Specifies the number of parallel threads that should be used when > > > > + using xz compression. By default this scales with core count, but > > > > + is never set less than 2 to ensure that multi-threaded mode is > > > > + always used (so that the output file contents are deterministic). > > > > + > > > > > > I don't see how having it set to 1 breaks deterministic builds, do you > > > mind explaining? > > > > > > Also, this feels like explanation of the implementation more than what > > > the user should do with this variable. > > > > I think having a cautionary note here about reproducible builds is important. > > > > > If the variable should never be <2, then it needs to be explicitly > > > stated here (and eventually the reason). > > > > You can set it, you just won't have reproducible builds. > > > > My understanding right now is that builds with XZ_THREADS=1 are > reproducible, and XZ_THREADS>2 are reproducible too. But if you switch > between them, the builds aren't reproducible. > > Is that correct? Correct. If you have a pool of builders, they all need to do one or the other too. Cheers, Richard
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index b9448b8..4803578 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -8737,4 +8737,15 @@ system and gives an overview of their function and contents. The default value of :term:`XSERVER`, if not specified in the machine configuration, is "xserver-xorg xf86-video-fbdev xf86-input-evdev". - + + :term:`XZ_THREADS` + Specifies the number of parallel threads that should be used when + using xz compression. By default this scales with core count, but + is never set less than 2 to ensure that multi-threaded mode is + always used (so that the output file contents are deterministic). + + :term:`ZSTD_THREADS` + Specifies the number of parallel threads that should be used when + using ZStandard compression. By default this scales with core count, + but is never set less than 2 to ensure that multi-threaded mode is + always used (so that the output file contents are deterministic).