| Message ID | 20220822045919.1262270-1-wesley.lindauer@gmail.com |
|---|---|
| State | Under Review |
| Headers | show |
| Series | bitbake.conf: Add resource vars to hash ignore list | expand |
If you have a compressed initrd for example this could give you a surprise, so I'm not 100% sure about this. /Regards Den mån 22 aug. 2022 kl 07:00 skrev Wes Lindauer <wesley.lindauer@gmail.com >: > It should not matter what xz/zstd parallelism and resource usage is set to > when determining hash signatures. This allows sharing more sstate across > host machines with different CPU configurations. > > Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com> > --- > meta/conf/bitbake.conf | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index dd2df8a552..d9eb1831ea 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -944,7 +944,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD > BB_TASKHASH BBPATH BBSERVER DL_DI > BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot > DEPLOY_DIR \ > SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ > SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE > GIT_CEILING_DIRECTORIES \ > - OMP_NUM_THREADS BB_CURRENTTASK" > + XZ_MEMLIMIT XZ_THREADS XZ_DEFAULTS ZSTD_THREADS OMP_NUM_THREADS > BB_CURRENTTASK" > BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS > BUILDHISTORY_DIR \ > SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS > RUST_TARGET_SYS" > BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME > SSH_AGENT_PID \ > -- > 2.25.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#169658): > https://lists.openembedded.org/g/openembedded-core/message/169658 > Mute This Topic: https://lists.openembedded.org/mt/93175519/4947266 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > ernstp@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Hi Wes, On 8/22/22 06:59, Wes Lindauer wrote: > It should not matter what xz/zstd parallelism and resource usage is set to > when determining hash signatures. This allows sharing more sstate across > host machines with different CPU configurations. > It seems like it does matter in some ways, c.f.: https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab and https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d Cheers, Quentin
On Mon, 2022-08-22 at 11:28 +0200, Quentin Schulz wrote: > Hi Wes, > > On 8/22/22 06:59, Wes Lindauer wrote: > > It should not matter what xz/zstd parallelism and resource usage is set to > > when determining hash signatures. This allows sharing more sstate across > > host machines with different CPU configurations. > > > > It seems like it does matter in some ways, c.f.: > https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab > and > https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d As Quentin mentions, the value does matter. In particular, those compiled with a single thread differ to those with multiple threads. As such the proposed patch can't be accepted. Cheers, Richard
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded- > core@lists.openembedded.org> On Behalf Of Richard Purdie > Sent: den 22 augusti 2022 13:19 > To: Quentin Schulz <quentin.schulz@theobroma-systems.com>; Wes Lindauer > <wesley.lindauer@gmail.com>; openembedded-core@lists.openembedded.org > Subject: Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash > ignore list > > On Mon, 2022-08-22 at 11:28 +0200, Quentin Schulz wrote: > > Hi Wes, > > > > On 8/22/22 06:59, Wes Lindauer wrote: > > > It should not matter what xz/zstd parallelism and resource usage is set to > > > when determining hash signatures. This allows sharing more sstate across > > > host machines with different CPU configurations. > > > > > > > It seems like it does matter in some ways, c.f.: > > https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab > > and > > https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d > > As Quentin mentions, the value does matter. In particular, those > compiled with a single thread differ to those with multiple threads. As > such the proposed patch can't be accepted. > > Cheers, > > Richard Isn't the problem of varying XZ_THREAD/ZSTD_THREAD vs taskhashes already taken care of by having: XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" XZ_THREADS[vardepvalue] = "1" XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" ZSTD_THREADS[vardepvalue] = "1" in bitbake.conf. //Peter
I think I understand. I am sorry for the FAD. I was experiencing this sstate mismatch on the dunfell branch that does not contain the 'XZ_THREADS[vardepvalue] = "1"' change. This is probably why I attempted my change in the hash exclude list and then tried to "port it forward to master". I appreciate everyone taking the time to explain. I am still a little confused why calculating every rpm task signature with a value of 1 is acceptable, but ignoring the variable in the signature is not. Are these not different implementations of the same thing? Thanks, Wes L On Mon, Aug 22, 2022 at 9:17 AM Peter Kjellerstedt < peter.kjellerstedt@axis.com> wrote: > > -----Original Message----- > > From: openembedded-core@lists.openembedded.org <openembedded- > > core@lists.openembedded.org> On Behalf Of Richard Purdie > > Sent: den 22 augusti 2022 13:19 > > To: Quentin Schulz <quentin.schulz@theobroma-systems.com>; Wes Lindauer > > <wesley.lindauer@gmail.com>; openembedded-core@lists.openembedded.org > > Subject: Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash > > ignore list > > > > On Mon, 2022-08-22 at 11:28 +0200, Quentin Schulz wrote: > > > Hi Wes, > > > > > > On 8/22/22 06:59, Wes Lindauer wrote: > > > > It should not matter what xz/zstd parallelism and resource usage is > set to > > > > when determining hash signatures. This allows sharing more sstate > across > > > > host machines with different CPU configurations. > > > > > > > > > > It seems like it does matter in some ways, c.f.: > > > > https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab > > > and > > > > https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d > > > > As Quentin mentions, the value does matter. In particular, those > > compiled with a single thread differ to those with multiple threads. As > > such the proposed patch can't be accepted. > > > > Cheers, > > > > Richard > > Isn't the problem of varying XZ_THREAD/ZSTD_THREAD vs taskhashes > already taken care of by having: > > XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > XZ_THREADS[vardepvalue] = "1" > XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > ZSTD_THREADS[vardepvalue] = "1" > > in bitbake.conf. > > //Peter > >
On Tue, 2022-08-30 at 15:27 -0400, Wes Lindauer wrote: > I think I understand. I am sorry for the FAD. I was experiencing this > sstate mismatch on the dunfell branch that does not contain the > 'XZ_THREADS[vardepvalue] = "1"' change. This is probably why I > attempted my change in the hash exclude list and then tried to "port > it forward to master". I appreciate everyone taking the time to > explain. I am still a little confused why calculating every rpm task > signature with a value of 1 is acceptable, but ignoring the variable > in the signature is not. Are these not different implementations of > the same thing? > > XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > > XZ_THREADS[vardepvalue] = "1" Note we do *two* things, we set it to at least a count of 2 and give it a known consistent value. It is the two things together than work out ok. If the user changes XZ_THREADS, as long as it is set to a value or 2 or more it is still ok. If the user sets it to a value of 1, builds are not reproducible. You're right that we could have just ignored it, but this at least gives a hint that a value of 2 or more should be used. Ideally we'd have put a warning around a value of 1 but there are a lot of things we'd do ideally and limited time... Cheers, Richard
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index dd2df8a552..d9eb1831ea 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -944,7 +944,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \ - OMP_NUM_THREADS BB_CURRENTTASK" + XZ_MEMLIMIT XZ_THREADS XZ_DEFAULTS ZSTD_THREADS OMP_NUM_THREADS BB_CURRENTTASK" BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \ SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS RUST_TARGET_SYS" BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
It should not matter what xz/zstd parallelism and resource usage is set to when determining hash signatures. This allows sharing more sstate across host machines with different CPU configurations. Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com> --- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)