| Message ID | 20251208161219.41335-1-corentin.guillevic@smile.fr |
|---|---|
| State | New |
| Headers | show |
| Series | fetch2: Skip AUTOREV management if no SCM URI | expand |
On Mon, 2025-12-08 at 17:12 +0100, Corentin Guillevic via lists.openembedded.org wrote: > AUTOREV allows to fetch the latest revision. If no SCM URI is used (e.g. git://, ...), > AUTOREV has no effect. However there is one case in which a recipe with AUTOREV > but no SCM URI will lead to a parsing error: > > ERROR: /path/to/yocto/builds/build-qemux86-64/workspace/recipes/i2c-tools/i2c-tools_3.1.2.bb: AUTOREV/SRCPV set too late for the fetcher to work properly, please set the variables earlier in parsing. Erroring instead of later obtuse build failures. > > This error is raised when AUTOREV occurs (__BBAUTOREV_SEEN) but is not acted > upon (__BBAUTOREV_ACTED_UPON). A recipe with one SCM URI causes the > srcrev_internal_helper() function to run, and read the value of SRCREV. If > SRCREV = "${AUTOREV}", then AUTOREV will be acted upon. > > If the recipe does not have an SCM URI, SRCREV is not parsed and AUTOREV remains > unchanged. However, if SRCREV is expanded during the parsing time in Python code > (either pure or evaluated from BitBake code), then AUTOREV will be expanded too > and resulting in the setting of __BBAUTOREV_SEEN. However, as there is no SCM > URI, the code flow that would act upon AUTOREV is never run. > > The fix slightly changes the way AUTOREV is handled: if no SCM URI is provided, > AUTOREV will be considered to have never occurred (__BBAUTOREV_SEEN will not be > defined). It makes more sense to me to throw an error if AUTOREV is seen but no SCM URI is provided, rather than silently ignoring AUTOREV. Perhaps we just need to improve the error that is raised. Is there any valid case for `SRCREV = "${AUTOREV}` when no SCM URI is given? Thanks,
> Is there any valid case for `SRCREV = "${AUTOREV}` when no SCM URI is given? No, however this case occurs indirectly with the recipetool command (from the self-test devtool.DevtoolUpgradeTests). It generates a recipe in which SRCREV is equal to "${AUTOREV}", even though the URI is not a SCM one. The issue arose because I used SRCREV in a new "insane" test, see: https://lists.openembedded.org/g/openembedded-core/topic/116341789 The issue can be reproduced by applying my patches and running the following command: $ recipetool --color=auto create --devtool -o /tmp/devtool4819z5wk ' http://downloads.yoctoproject.org/mirror/sources/i2c-tools-3.1.2.tar.bz2' -x ${BUILDDIR}/workspace/sources/devtoolsrcdcviuvn9 Le lun. 8 déc. 2025 à 17:59, Paul Barker <paul@pbarker.dev> a écrit : > On Mon, 2025-12-08 at 17:12 +0100, Corentin Guillevic via > lists.openembedded.org wrote: > > AUTOREV allows to fetch the latest revision. If no SCM URI is used (e.g. > git://, ...), > > AUTOREV has no effect. However there is one case in which a recipe with > AUTOREV > > but no SCM URI will lead to a parsing error: > > > > ERROR: > /path/to/yocto/builds/build-qemux86-64/workspace/recipes/i2c-tools/ > i2c-tools_3.1.2.bb: AUTOREV/SRCPV set too late for the fetcher to work > properly, please set the variables earlier in parsing. Erroring instead of > later obtuse build failures. > > > > This error is raised when AUTOREV occurs (__BBAUTOREV_SEEN) but is not > acted > > upon (__BBAUTOREV_ACTED_UPON). A recipe with one SCM URI causes the > > srcrev_internal_helper() function to run, and read the value of SRCREV. > If > > SRCREV = "${AUTOREV}", then AUTOREV will be acted upon. > > > > If the recipe does not have an SCM URI, SRCREV is not parsed and AUTOREV > remains > > unchanged. However, if SRCREV is expanded during the parsing time in > Python code > > (either pure or evaluated from BitBake code), then AUTOREV will be > expanded too > > and resulting in the setting of __BBAUTOREV_SEEN. However, as there is > no SCM > > URI, the code flow that would act upon AUTOREV is never run. > > > > The fix slightly changes the way AUTOREV is handled: if no SCM URI is > provided, > > AUTOREV will be considered to have never occurred (__BBAUTOREV_SEEN will > not be > > defined). > > It makes more sense to me to throw an error if AUTOREV is seen but no > SCM URI is provided, rather than silently ignoring AUTOREV. Perhaps we > just need to improve the error that is raised. > > Is there any valid case for `SRCREV = "${AUTOREV}` when no SCM URI is > given? > > Thanks, > > -- > Paul Barker > > >
On Tue, 2025-12-09 at 11:43 +0100, Corentin Guillevic via lists.openembedded.org wrote: > > Is there any valid case for `SRCREV = "${AUTOREV}` when no SCM URI is > given? > > No, however this case occurs indirectly with the recipetool command (from > the self-test devtool.DevtoolUpgradeTests). It generates a recipe in which > SRCREV is equal to "${AUTOREV}", even though the URI is not a SCM one. The > issue arose because I used SRCREV in a new "insane" test, see: > https://lists.openembedded.org/g/openembedded-core/topic/116341789 > > The issue can be reproduced by applying my patches and running the > following command: > > $ recipetool --color=auto create --devtool -o /tmp/devtool4819z5wk ' > http://downloads.yoctoproject.org/mirror/sources/i2c-tools-3.1.2.tar.bz2' > -x ${BUILDDIR}/workspace/sources/devtoolsrcdcviuvn9 Which patches are you referring to here? And, does the issue occur on the current master branch without applying any additional patches? > > Le lun. 8 déc. 2025 à 17:59, Paul Barker <paul@pbarker.dev> a écrit : > > > On Mon, 2025-12-08 at 17:12 +0100, Corentin Guillevic via > > lists.openembedded.org wrote: > > > AUTOREV allows to fetch the latest revision. If no SCM URI is used (e.g. > > git://, ...), > > > AUTOREV has no effect. However there is one case in which a recipe with > > AUTOREV > > > but no SCM URI will lead to a parsing error: > > > > > > ERROR: > > /path/to/yocto/builds/build-qemux86-64/workspace/recipes/i2c-tools/ > > i2c-tools_3.1.2.bb: AUTOREV/SRCPV set too late for the fetcher to work > > properly, please set the variables earlier in parsing. Erroring instead of > > later obtuse build failures. > > > > > > This error is raised when AUTOREV occurs (__BBAUTOREV_SEEN) but is not > > acted > > > upon (__BBAUTOREV_ACTED_UPON). A recipe with one SCM URI causes the > > > srcrev_internal_helper() function to run, and read the value of SRCREV. > > If > > > SRCREV = "${AUTOREV}", then AUTOREV will be acted upon. > > > > > > If the recipe does not have an SCM URI, SRCREV is not parsed and AUTOREV > > remains > > > unchanged. However, if SRCREV is expanded during the parsing time in > > Python code > > > (either pure or evaluated from BitBake code), then AUTOREV will be > > expanded too > > > and resulting in the setting of __BBAUTOREV_SEEN. However, as there is > > no SCM > > > URI, the code flow that would act upon AUTOREV is never run. > > > > > > The fix slightly changes the way AUTOREV is handled: if no SCM URI is > > provided, > > > AUTOREV will be considered to have never occurred (__BBAUTOREV_SEEN will > > not be > > > defined). > > > > It makes more sense to me to throw an error if AUTOREV is seen but no > > SCM URI is provided, rather than silently ignoring AUTOREV. Perhaps we > > just need to improve the error that is raised. > > > > Is there any valid case for `SRCREV = "${AUTOREV}` when no SCM URI is > > given? Thanks,
> Which patches are you referring to here? I am referring to these patches: - https://lists.openembedded.org/g/openembedded-core/topic/116341789 - https://lists.openembedded.org/g/openembedded-core/message/226505 They introduce and enable a new "insane" test to check whether if a git URI is provided without a revision (SRCREV). > And, does the issue occur on the current master branch without applying any additional patches? No, the issue doesn't occur on the master branch. This happens with my patches because SRCREV is used in the test, and is expanded during the parsing. If SRCREV is equal to "${AUTOREV}" but no SCM URI is present, AUTOREV will be seen but no SCM function will act upon it. This leads to the error. Le mer. 10 déc. 2025 à 11:20, Paul Barker <paul@pbarker.dev> a écrit : > On Tue, 2025-12-09 at 11:43 +0100, Corentin Guillevic via > lists.openembedded.org wrote: > > > Is there any valid case for `SRCREV = "${AUTOREV}` when no SCM URI is > > given? > > > > No, however this case occurs indirectly with the recipetool command (from > > the self-test devtool.DevtoolUpgradeTests). It generates a recipe in > which > > SRCREV is equal to "${AUTOREV}", even though the URI is not a SCM one. > The > > issue arose because I used SRCREV in a new "insane" test, see: > > https://lists.openembedded.org/g/openembedded-core/topic/116341789 > > > > The issue can be reproduced by applying my patches and running the > > following command: > > > > $ recipetool --color=auto create --devtool -o /tmp/devtool4819z5wk ' > > http://downloads.yoctoproject.org/mirror/sources/i2c-tools-3.1.2.tar.bz2 > ' > > -x ${BUILDDIR}/workspace/sources/devtoolsrcdcviuvn9 > > Which patches are you referring to here? And, does the issue occur on the > current master branch without applying any additional patches? > > > > > Le lun. 8 déc. 2025 à 17:59, Paul Barker <paul@pbarker.dev> a écrit : > > > > > On Mon, 2025-12-08 at 17:12 +0100, Corentin Guillevic via > > > lists.openembedded.org wrote: > > > > AUTOREV allows to fetch the latest revision. If no SCM URI is used > (e.g. > > > git://, ...), > > > > AUTOREV has no effect. However there is one case in which a recipe > with > > > AUTOREV > > > > but no SCM URI will lead to a parsing error: > > > > > > > > ERROR: > > > /path/to/yocto/builds/build-qemux86-64/workspace/recipes/i2c-tools/ > > > i2c-tools_3.1.2.bb: AUTOREV/SRCPV set too late for the fetcher to work > > > properly, please set the variables earlier in parsing. Erroring > instead of > > > later obtuse build failures. > > > > > > > > This error is raised when AUTOREV occurs (__BBAUTOREV_SEEN) but is > not > > > acted > > > > upon (__BBAUTOREV_ACTED_UPON). A recipe with one SCM URI causes the > > > > srcrev_internal_helper() function to run, and read the value of > SRCREV. > > > If > > > > SRCREV = "${AUTOREV}", then AUTOREV will be acted upon. > > > > > > > > If the recipe does not have an SCM URI, SRCREV is not parsed and > AUTOREV > > > remains > > > > unchanged. However, if SRCREV is expanded during the parsing time in > > > Python code > > > > (either pure or evaluated from BitBake code), then AUTOREV will be > > > expanded too > > > > and resulting in the setting of __BBAUTOREV_SEEN. However, as there > is > > > no SCM > > > > URI, the code flow that would act upon AUTOREV is never run. > > > > > > > > The fix slightly changes the way AUTOREV is handled: if no SCM URI is > > > provided, > > > > AUTOREV will be considered to have never occurred (__BBAUTOREV_SEEN > will > > > not be > > > > defined). > > > > > > It makes more sense to me to throw an error if AUTOREV is seen but no > > > SCM URI is provided, rather than silently ignoring AUTOREV. Perhaps we > > > just need to improve the error that is raised. > > > > > > Is there any valid case for `SRCREV = "${AUTOREV}` when no SCM URI is > > > given? > > Thanks, > > -- > Paul Barker > > >
Gentle ping Corentin Le mer. 10 déc. 2025 à 19:46, Corentin GUILLEVIC < corentin.guillevic@smile.fr> a écrit : > > Which patches are you referring to here? > I am referring to these patches: > - https://lists.openembedded.org/g/openembedded-core/topic/116341789 > - https://lists.openembedded.org/g/openembedded-core/message/226505 > They introduce and enable a new "insane" test to check whether if a git > URI is provided without a revision (SRCREV). > > > And, does the issue occur on the current master branch without applying > any additional patches? > No, the issue doesn't occur on the master branch. This happens with my > patches because SRCREV is used in the test, and is expanded during the > parsing. If SRCREV is equal to "${AUTOREV}" but no SCM URI is present, > AUTOREV will be seen but no SCM function will act upon it. This leads to > the error. > > Le mer. 10 déc. 2025 à 11:20, Paul Barker <paul@pbarker.dev> a écrit : > >> On Tue, 2025-12-09 at 11:43 +0100, Corentin Guillevic via >> lists.openembedded.org wrote: >> > > Is there any valid case for `SRCREV = "${AUTOREV}` when no SCM URI is >> > given? >> > >> > No, however this case occurs indirectly with the recipetool command >> (from >> > the self-test devtool.DevtoolUpgradeTests). It generates a recipe in >> which >> > SRCREV is equal to "${AUTOREV}", even though the URI is not a SCM one. >> The >> > issue arose because I used SRCREV in a new "insane" test, see: >> > https://lists.openembedded.org/g/openembedded-core/topic/116341789 >> > >> > The issue can be reproduced by applying my patches and running the >> > following command: >> > >> > $ recipetool --color=auto create --devtool -o /tmp/devtool4819z5wk ' >> > >> http://downloads.yoctoproject.org/mirror/sources/i2c-tools-3.1.2.tar.bz2' >> > -x ${BUILDDIR}/workspace/sources/devtoolsrcdcviuvn9 >> >> Which patches are you referring to here? And, does the issue occur on the >> current master branch without applying any additional patches? >> >> > >> > Le lun. 8 déc. 2025 à 17:59, Paul Barker <paul@pbarker.dev> a écrit : >> > >> > > On Mon, 2025-12-08 at 17:12 +0100, Corentin Guillevic via >> > > lists.openembedded.org wrote: >> > > > AUTOREV allows to fetch the latest revision. If no SCM URI is used >> (e.g. >> > > git://, ...), >> > > > AUTOREV has no effect. However there is one case in which a recipe >> with >> > > AUTOREV >> > > > but no SCM URI will lead to a parsing error: >> > > > >> > > > ERROR: >> > > /path/to/yocto/builds/build-qemux86-64/workspace/recipes/i2c-tools/ >> > > i2c-tools_3.1.2.bb: AUTOREV/SRCPV set too late for the fetcher to >> work >> > > properly, please set the variables earlier in parsing. Erroring >> instead of >> > > later obtuse build failures. >> > > > >> > > > This error is raised when AUTOREV occurs (__BBAUTOREV_SEEN) but is >> not >> > > acted >> > > > upon (__BBAUTOREV_ACTED_UPON). A recipe with one SCM URI causes the >> > > > srcrev_internal_helper() function to run, and read the value of >> SRCREV. >> > > If >> > > > SRCREV = "${AUTOREV}", then AUTOREV will be acted upon. >> > > > >> > > > If the recipe does not have an SCM URI, SRCREV is not parsed and >> AUTOREV >> > > remains >> > > > unchanged. However, if SRCREV is expanded during the parsing time in >> > > Python code >> > > > (either pure or evaluated from BitBake code), then AUTOREV will be >> > > expanded too >> > > > and resulting in the setting of __BBAUTOREV_SEEN. However, as there >> is >> > > no SCM >> > > > URI, the code flow that would act upon AUTOREV is never run. >> > > > >> > > > The fix slightly changes the way AUTOREV is handled: if no SCM URI >> is >> > > provided, >> > > > AUTOREV will be considered to have never occurred (__BBAUTOREV_SEEN >> will >> > > not be >> > > > defined). >> > > >> > > It makes more sense to me to throw an error if AUTOREV is seen but no >> > > SCM URI is provided, rather than silently ignoring AUTOREV. Perhaps we >> > > just need to improve the error that is raised. >> > > >> > > Is there any valid case for `SRCREV = "${AUTOREV}` when no SCM URI is >> > > given? >> >> Thanks, >> >> -- >> Paul Barker >> >> >>
On Mon Dec 8, 2025 at 5:12 PM CET, Corentin Guillevic via lists.openembedded.org wrote: > AUTOREV allows to fetch the latest revision. If no SCM URI is used (e.g. git://, ...), > AUTOREV has no effect. However there is one case in which a recipe with AUTOREV > but no SCM URI will lead to a parsing error: > > ERROR: /path/to/yocto/builds/build-qemux86-64/workspace/recipes/i2c-tools/i2c-tools_3.1.2.bb: AUTOREV/SRCPV set too late for the fetcher to work properly, please set the variables earlier in parsing. Erroring instead of later obtuse build failures. > > This error is raised when AUTOREV occurs (__BBAUTOREV_SEEN) but is not acted > upon (__BBAUTOREV_ACTED_UPON). A recipe with one SCM URI causes the > srcrev_internal_helper() function to run, and read the value of SRCREV. If > SRCREV = "${AUTOREV}", then AUTOREV will be acted upon. > > If the recipe does not have an SCM URI, SRCREV is not parsed and AUTOREV remains > unchanged. However, if SRCREV is expanded during the parsing time in Python code > (either pure or evaluated from BitBake code), then AUTOREV will be expanded too > and resulting in the setting of __BBAUTOREV_SEEN. However, as there is no SCM > URI, the code flow that would act upon AUTOREV is never run. > > The fix slightly changes the way AUTOREV is handled: if no SCM URI is provided, > AUTOREV will be considered to have never occurred (__BBAUTOREV_SEEN will not be > defined). > > Signed-off-by: Corentin Guillevic <corentin.guillevic@smile.fr> > --- Hi Corentin, Sorry for the delay, I didn't realise you replied to all questions about this series. I believe this is breaking some QA and selftests, am I missing some other patches? oeqa.utils.subprocesstweak.OETestCalledProcessError: Command '. /srv/pokybuild/yocto-worker/qemuarm64-armhost/build/build/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/testsdkext/environment-setup-cortexa57-poky-linux > /dev/null; devtool add kernel-module-hello-world https://git.yoctoproject.org/git/kernel-module-hello-world;' returned non-zero exit status 1. ... ERROR: ExpansionError during parsing /srv/pokybuild/yocto-worker/qemuarm64-armhost/build/build/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/testsdkext/workspace/recipes/recipetool/tmp-recipetool-g42ny7zl.bb bb.data_smart.ExpansionError: Failure expanding variable AUTOREV, expression was ${@bb.fetch2.get_autorev(d)} which triggered exception RecursionError: maximum recursion depth exceeded The variable dependency chain for the failure is: AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> AUTOREV -> SRCREV -> fetcher_hashes_dummyfunc[vardepvalue] ERROR: Parsing halted due to errors, see error messages above https://autobuilder.yoctoproject.org/valkyrie/#/builders/8/builds/2957 https://autobuilder.yoctoproject.org/valkyrie/#/builders/16/builds/2965 https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/2906 https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/2931 https://autobuilder.yoctoproject.org/valkyrie/#/builders/45/builds/862 https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3007 https://autobuilder.yoctoproject.org/valkyrie/#/builders/80/builds/2769 Failing selftests, all with a similar error as above: pokybleeding.PokyBleeding.test_poky_bleeding_autorev sstatetests.SStateTests.test_autorev_sstate_works devtool.DevtoolAddTests.test_devtool_add_fetch_git devtool.DevtoolAddTests.test_devtool_add_git_style1 recipetool.RecipetoolCreateTests.test_recipetool_create_git_http recipetool.RecipetoolCreateTests.test_recipetool_create_git_srcbranch recipetool.RecipetoolCreateTests.test_recipetool_create_git_srcuri_branch recipetool.RecipetoolCreateTests.test_recipetool_create_git_srcuri_master https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2809 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2916 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3053 Can you have a look at these? Thanks, Mathieu
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index bdd497394..a1e2def95 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -769,8 +769,16 @@ def mark_recipe_nocache(d): d.setVar('BB_DONT_CACHE', '1') def get_autorev(d): - mark_recipe_nocache(d) - d.setVar("__BBAUTOREV_SEEN", True) + fetcher = Fetch(d.getVar('SRC_URI').split(), d) + urldata = fetcher.ud + for u in urldata: + if not urldata[u].method.supports_srcrev(): + continue + + mark_recipe_nocache(d) + d.setVar("__BBAUTOREV_SEEN", True) + break + return "AUTOINC" def _get_srcrev(d, method_name='sortable_revision'):
AUTOREV allows to fetch the latest revision. If no SCM URI is used (e.g. git://, ...), AUTOREV has no effect. However there is one case in which a recipe with AUTOREV but no SCM URI will lead to a parsing error: ERROR: /path/to/yocto/builds/build-qemux86-64/workspace/recipes/i2c-tools/i2c-tools_3.1.2.bb: AUTOREV/SRCPV set too late for the fetcher to work properly, please set the variables earlier in parsing. Erroring instead of later obtuse build failures. This error is raised when AUTOREV occurs (__BBAUTOREV_SEEN) but is not acted upon (__BBAUTOREV_ACTED_UPON). A recipe with one SCM URI causes the srcrev_internal_helper() function to run, and read the value of SRCREV. If SRCREV = "${AUTOREV}", then AUTOREV will be acted upon. If the recipe does not have an SCM URI, SRCREV is not parsed and AUTOREV remains unchanged. However, if SRCREV is expanded during the parsing time in Python code (either pure or evaluated from BitBake code), then AUTOREV will be expanded too and resulting in the setting of __BBAUTOREV_SEEN. However, as there is no SCM URI, the code flow that would act upon AUTOREV is never run. The fix slightly changes the way AUTOREV is handled: if no SCM URI is provided, AUTOREV will be considered to have never occurred (__BBAUTOREV_SEEN will not be defined). Signed-off-by: Corentin Guillevic <corentin.guillevic@smile.fr> --- lib/bb/fetch2/__init__.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)