| Message ID | 20220228084250.3874136-1-rasmus.villemoes@prevas.dk |
|---|---|
| State | New |
| Headers | show |
| Series | bitbake.conf: add BB_NUMBER_THREADS to BB_HASHEXCLUDE_COMMON | expand |
On Mon, 2022-02-28 at 09:42 +0100, Rasmus Villemoes wrote: > The imx-gpu-sdk recipe in the meta-imx layer references > ${BB_NUMBER_THREADS} in its do_compile function. Changing > BB_NUMBER_THREADS between bitbake invocations leads to the well-known > > When reparsing ...meta-imx/meta-sdk/recipes-graphics/imx-gpu-sdk/imx-gpu-sdk_5.8.0.bb:do_compile, the basehash value changed from 69be88cf220840ff2203e11cfe65681880b0bf9b88db67d50c1ba772b883bd18 to 5e6d5029fac8d7856ada4c2eca359568298f82cdb64567d7dd4deda503d9f83a. The metadata is not deterministic and this needs to be fixed. > > And I'm not the first to hit this problem with that recipe: > https://community.nxp.com/t5/i-MX-Processors/imx-gpu-sdk-compile-error-IMX8MP-IMX8MM-IMX8MQ/td-p/1217864 > > This happens because BB_NUMBER_THREADS is in BB_HASHCONFIG_IGNORE_VARS, > so changing it does not cause the recipe to be reparsed, but it is not > included in BB_HASHEXCLUDE_COMMON and thus > BB_BASEHASH_IGNORE_VARS. This is inconsistent with and in contrast to > both PARALLEL_MAKE and OMP_NUM_THREADS, the latter of which even has > ${BB_NUMBER_THREADS} as default value. Technically imx-gpu-sdk is incorrect. BB_NUMBER_THREADS is the number of tasks bitbake should run. PARALLEL_MAKE is what is used for parallelism in do_compile. I appreciate that has -j in but you can use: ${@oe.utils.parallel_make(d)} to obtain the value that recipe needs. Cheers, Richard
On 28/02/2022 14.41, Richard Purdie wrote: > On Mon, 2022-02-28 at 09:42 +0100, Rasmus Villemoes wrote: >> The imx-gpu-sdk recipe in the meta-imx layer references >> ${BB_NUMBER_THREADS} in its do_compile function. Changing >> BB_NUMBER_THREADS between bitbake invocations leads to the well-known >> >> When reparsing ...meta-imx/meta-sdk/recipes-graphics/imx-gpu-sdk/imx-gpu-sdk_5.8.0.bb:do_compile, the basehash value changed from 69be88cf220840ff2203e11cfe65681880b0bf9b88db67d50c1ba772b883bd18 to 5e6d5029fac8d7856ada4c2eca359568298f82cdb64567d7dd4deda503d9f83a. The metadata is not deterministic and this needs to be fixed. >> >> And I'm not the first to hit this problem with that recipe: >> https://community.nxp.com/t5/i-MX-Processors/imx-gpu-sdk-compile-error-IMX8MP-IMX8MM-IMX8MQ/td-p/1217864 >> >> This happens because BB_NUMBER_THREADS is in BB_HASHCONFIG_IGNORE_VARS, >> so changing it does not cause the recipe to be reparsed, but it is not >> included in BB_HASHEXCLUDE_COMMON and thus >> BB_BASEHASH_IGNORE_VARS. This is inconsistent with and in contrast to >> both PARALLEL_MAKE and OMP_NUM_THREADS, the latter of which even has >> ${BB_NUMBER_THREADS} as default value. > > Technically imx-gpu-sdk is incorrect. BB_NUMBER_THREADS is the number of tasks > bitbake should run. PARALLEL_MAKE is what is used for parallelism in do_compile. > > I appreciate that has -j in but you can use: ${@oe.utils.parallel_make(d)} to > obtain the value that recipe needs. Oh, I'm not saying that that recipe isn't buggy, but is still seems awfully user-unfriendly and inconsistent to treat BB_NUMBER_THREADS different from PARALLEL_MAKE wrt. hash computations. But now I've raised the issue, and I don't care much what happens next; I now know the root cause of the error I saw and rarely build using meta-imx in the first place. Rasmus
On Mon, 2022-02-28 at 15:06 +0100, Rasmus Villemoes wrote: > On 28/02/2022 14.41, Richard Purdie wrote: > > On Mon, 2022-02-28 at 09:42 +0100, Rasmus Villemoes wrote: > > > The imx-gpu-sdk recipe in the meta-imx layer references > > > ${BB_NUMBER_THREADS} in its do_compile function. Changing > > > BB_NUMBER_THREADS between bitbake invocations leads to the well-known > > > > > > When reparsing ...meta-imx/meta-sdk/recipes-graphics/imx-gpu-sdk/imx-gpu-sdk_5.8.0.bb:do_compile, the basehash value changed from 69be88cf220840ff2203e11cfe65681880b0bf9b88db67d50c1ba772b883bd18 to 5e6d5029fac8d7856ada4c2eca359568298f82cdb64567d7dd4deda503d9f83a. The metadata is not deterministic and this needs to be fixed. > > > > > > And I'm not the first to hit this problem with that recipe: > > > https://community.nxp.com/t5/i-MX-Processors/imx-gpu-sdk-compile-error-IMX8MP-IMX8MM-IMX8MQ/td-p/1217864 > > > > > > This happens because BB_NUMBER_THREADS is in BB_HASHCONFIG_IGNORE_VARS, > > > so changing it does not cause the recipe to be reparsed, but it is not > > > included in BB_HASHEXCLUDE_COMMON and thus > > > BB_BASEHASH_IGNORE_VARS. This is inconsistent with and in contrast to > > > both PARALLEL_MAKE and OMP_NUM_THREADS, the latter of which even has > > > ${BB_NUMBER_THREADS} as default value. > > > > Technically imx-gpu-sdk is incorrect. BB_NUMBER_THREADS is the number of tasks > > bitbake should run. PARALLEL_MAKE is what is used for parallelism in do_compile. > > > > I appreciate that has -j in but you can use: ${@oe.utils.parallel_make(d)} to > > obtain the value that recipe needs. > > Oh, I'm not saying that that recipe isn't buggy, but is still seems > awfully user-unfriendly and inconsistent to treat BB_NUMBER_THREADS > different from PARALLEL_MAKE wrt. hash computations. The reasoning is that nothing should be using BB_NUMBER_THREADS like that. Your patch would just mask it even more :/. Ideally we'd have something to tell people they were misusing it but that is harder to do. > But now I've raised the issue, and I don't care much what happens next; > I now know the root cause of the error I saw and rarely build using > meta-imx in the first place. Just to check, are the meta-imx people aware of it and able to fix the real problem? Cheers, Richard
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded- > core@lists.openembedded.org> On Behalf Of Richard Purdie via > lists.openembedded.org > Sent: Monday, February 28, 2022 8:57 AM > To: Rasmus Villemoes <rasmus.villemoes@prevas.dk>; openembedded- > core@lists.openembedded.org > Subject: Re: [OE-core] [PATCH] bitbake.conf: add BB_NUMBER_THREADS to > BB_HASHEXCLUDE_COMMON > > > Just to check, are the meta-imx people aware of it and able to fix the real > problem? Yes, I will fix it. Thanks! Tom
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 6fb7bfeb23c..6bc18016056 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -942,7 +942,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \ - OMP_NUM_THREADS BB_CURRENTTASK" + OMP_NUM_THREADS BB_CURRENTTASK BB_NUMBER_THREADS" BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \ SSTATE_DIR SOURCE_DATE_EPOCH" BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
The imx-gpu-sdk recipe in the meta-imx layer references ${BB_NUMBER_THREADS} in its do_compile function. Changing BB_NUMBER_THREADS between bitbake invocations leads to the well-known When reparsing ...meta-imx/meta-sdk/recipes-graphics/imx-gpu-sdk/imx-gpu-sdk_5.8.0.bb:do_compile, the basehash value changed from 69be88cf220840ff2203e11cfe65681880b0bf9b88db67d50c1ba772b883bd18 to 5e6d5029fac8d7856ada4c2eca359568298f82cdb64567d7dd4deda503d9f83a. The metadata is not deterministic and this needs to be fixed. And I'm not the first to hit this problem with that recipe: https://community.nxp.com/t5/i-MX-Processors/imx-gpu-sdk-compile-error-IMX8MP-IMX8MM-IMX8MQ/td-p/1217864 This happens because BB_NUMBER_THREADS is in BB_HASHCONFIG_IGNORE_VARS, so changing it does not cause the recipe to be reparsed, but it is not included in BB_HASHEXCLUDE_COMMON and thus BB_BASEHASH_IGNORE_VARS. This is inconsistent with and in contrast to both PARALLEL_MAKE and OMP_NUM_THREADS, the latter of which even has ${BB_NUMBER_THREADS} as default value. Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk> --- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)