| Message ID | 20251022113212.3220395-1-alex.kanavin@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [RFC] layer.conf: add a bb_min_version statement to perform an early bitbake version check | expand |
Hi Alex, On 10/22/25 1:32 PM, Alexander Kanavin via lists.openembedded.org wrote: > From: Alexander Kanavin <alex@linutronix.de> > > Please see the corresponding change in bitbake for the rationale. > https://lore.kernel.org/bitbake-devel/20251022113146.3207795-1-alex.kanavin@gmail.com/ I would recommend that we specify which commit from bitbake we're talking about in the commit log here so it's easier to find it. Considering we anyway need the bitbake change to be merged before this one, that should be doable. > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > meta/conf/layer.conf | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf > index 54fa04e213e..6c05c0e1934 100644 > --- a/meta/conf/layer.conf > +++ b/meta/conf/layer.conf > @@ -1,3 +1,5 @@ > +bitbake_min_version 2.15.2 > + Does the sanity check still make sense if it simply cannot be triggered anymore? In other words, should we remove BB_MIN_VERSION variable and its check in meta/classes-global/sanity.bbclass? My worry is that we forget to keep them in sync. If we need to keep it, do you think it'd make sense to move BB_MIN_VERSION to this file here too? And if possible use it as argument to bitbake_min_version so we only have one place to modify for bumping the variable. Another question is whether we should backport this to earlier releases. Another question is whether we can use a newer bitbake with older releases, which this wouldn't help with. E.g. what happens if I use bitbake 2.15.2 with scarthgap layers for example. Should we force the version to be MAJOR.MINOR and ignore any patch if not specified? e.g. bitbake_expect_version 2.15 would match anything from 2.15 to 2.15.999 bitbake_expect_version 2.15.2 would match anything from 2.15.2 to 2.15.999. à-la Rust version resolver, c.f. https://doc.rust-lang.org/cargo/reference/resolver.html#version-requirements Should we rather have an explicit max version (but ok if not defined for e.g. the master branch)? Cheers, Quentin
On Wed, 22 Oct 2025 at 13:56, Quentin Schulz <quentin.schulz@cherry.de> wrote: > Does the sanity check still make sense if it simply cannot be triggered > anymore? > > In other words, should we remove BB_MIN_VERSION variable and its check > in meta/classes-global/sanity.bbclass? My worry is that we forget to > keep them in sync. If we need to keep it, do you think it'd make sense > to move BB_MIN_VERSION to this file here too? And if possible use it as > argument to bitbake_min_version so we only have one place to modify for > bumping the variable. I think the existing sanity check could be dropped, yes. I tried defining BB_MIN_VERSION here, and using it as an argument to bitbake_min_version, but at this point the datastore doesn't seem to be ready, and variable expansion doesn't work. Or maybe I did something incorrectly. Can you try it on your side? > Another question is whether we should backport this to earlier releases. Probably not. It's a new feature, and the issue is most pressing when people move on from integrated poky into separate bitbake/core checkouts. > Another question is whether we can use a newer bitbake with older > releases, which this wouldn't help with. E.g. what happens if I use > bitbake 2.15.2 with scarthgap layers for example. Should we force the > version to be MAJOR.MINOR and ignore any patch if not specified? e.g. > > bitbake_expect_version 2.15 > > would match anything from 2.15 to 2.15.999 > > bitbake_expect_version 2.15.2 > > would match anything from 2.15.2 to 2.15.999. > > à-la Rust version resolver, c.f. > https://doc.rust-lang.org/cargo/reference/resolver.html#version-requirements > > Should we rather have an explicit max version (but ok if not defined for > e.g. the master branch)? I don't know. Enforcing a minimum version is straightforward, but what the maximum version should be is more complicated to define and maintain. It's probably a separate issue; this change simply moves an existing check earlier in bitbake invocations. Alex
On 10/22/25 2:14 PM, Alexander Kanavin wrote: > On Wed, 22 Oct 2025 at 13:56, Quentin Schulz <quentin.schulz@cherry.de> wrote: >> Does the sanity check still make sense if it simply cannot be triggered >> anymore? >> >> In other words, should we remove BB_MIN_VERSION variable and its check >> in meta/classes-global/sanity.bbclass? My worry is that we forget to >> keep them in sync. If we need to keep it, do you think it'd make sense >> to move BB_MIN_VERSION to this file here too? And if possible use it as >> argument to bitbake_min_version so we only have one place to modify for >> bumping the variable. > > I think the existing sanity check could be dropped, yes. I tried > defining BB_MIN_VERSION here, and using it as an argument to > bitbake_min_version, but at this point the datastore doesn't seem to > be ready, and variable expansion doesn't work. Or maybe I did > something incorrectly. Can you try it on your side? > $ bitbake -e mesa-native ERROR: Unable to parse /home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py Traceback (most recent call last): File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 108, in check_bb_version raise ParseError('Bitbake version %s is required and version %s is used\n' % (min_version, bb.__version__), fn, lineno) bb.parse.ParseError: ParseError at /home/qschulz/work/upstream/yocto/openembedded-core/meta/conf/layer.conf:2: Bitbake version $BB_MIN_VERSION is required and version 2.15.2 is used But this was to be expected since in the BB patch it's simply a regexp where we extract whatever's after "bitbake_min_version " with re.match() which means only a string. >> Another question is whether we should backport this to earlier releases. > > Probably not. It's a new feature, and the issue is most pressing when > people move on from integrated poky into separate bitbake/core > checkouts. > Yeah but this happened because the person updated their oe-core repo and not bitbake, but the opposite could happen as well. We would only protect from one scenario and not the other. >> Another question is whether we can use a newer bitbake with older >> releases, which this wouldn't help with. E.g. what happens if I use >> bitbake 2.15.2 with scarthgap layers for example. Should we force the >> version to be MAJOR.MINOR and ignore any patch if not specified? e.g. >> >> bitbake_expect_version 2.15 >> >> would match anything from 2.15 to 2.15.999 >> >> bitbake_expect_version 2.15.2 >> >> would match anything from 2.15.2 to 2.15.999. >> >> à-la Rust version resolver, c.f. >> https://doc.rust-lang.org/cargo/reference/resolver.html#version-requirements >> >> Should we rather have an explicit max version (but ok if not defined for >> e.g. the master branch)? > > I don't know. Enforcing a minimum version is straightforward, but what > the maximum version should be is more complicated to define and > maintain. It's probably a separate issue; this change simply moves an > existing check earlier in bitbake invocations. > We historically (since Denzil) use an even minor version number for releases (and considering we are currently at 2.15 in master, I would say odd numbers for development). We could then match on MAJOR.MINOR only and ignore the rest except if provided (but I assume this should never happen except on the master branch since we would otherwise be breaking backward compatibility of releases (e.g. min_version is now 2.8.98 on scarthgap means we are forced to update bitbake to that version to be able to build newer releases of scarthgap) This assumption would mean tagging bitbake for a release, then using that even minor version number as argument to this new bitbake_expect_version (or whatever name we can find) and then tag oe-core for the new branch. Then the master branch of bitbake gets the new odd version number for the development of the next branch, which is then used also in the development of the next branch in oe-core. I assume something that needs to be added to the branching off process we already have. In any case, considering this patch is about fixing a failsafe that exists but doesn't work, removing the BB_MIN_VERSION check and migrating to this seems fine to me. Thanks! Quentin
On Wed, 22 Oct 2025 at 16:55, Quentin Schulz <quentin.schulz@cherry.de> wrote: > $ bitbake -e mesa-native > ERROR: Unable to parse > /home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py > Traceback (most recent call last): > File > "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py", > line 108, in check_bb_version > raise ParseError('Bitbake version %s is required and version %s is > used\n' % (min_version, bb.__version__), fn, lineno) > bb.parse.ParseError: ParseError at > /home/qschulz/work/upstream/yocto/openembedded-core/meta/conf/layer.conf:2: > Bitbake version $BB_MIN_VERSION is required and version 2.15.2 is used > > But this was to be expected since in the BB patch it's simply a regexp > where we extract whatever's after "bitbake_min_version " with re.match() > which means only a string. You need to declare it thusly: BB_MIN_VERSION = "2.15.2" bb_min_version ${BB_MIN_VERSION} and then run the regexp match (which should be '${BB_MIN_VERSION}') through data.expand() inside eval() in ast.py. It's supposed to work, but for me it did not. Alex
On 10/22/25 5:06 PM, Alexander Kanavin wrote: > On Wed, 22 Oct 2025 at 16:55, Quentin Schulz <quentin.schulz@cherry.de> wrote: >> $ bitbake -e mesa-native >> ERROR: Unable to parse >> /home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py >> Traceback (most recent call last): >> File >> "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py", >> line 108, in check_bb_version >> raise ParseError('Bitbake version %s is required and version %s is >> used\n' % (min_version, bb.__version__), fn, lineno) >> bb.parse.ParseError: ParseError at >> /home/qschulz/work/upstream/yocto/openembedded-core/meta/conf/layer.conf:2: >> Bitbake version $BB_MIN_VERSION is required and version 2.15.2 is used >> >> But this was to be expected since in the BB patch it's simply a regexp >> where we extract whatever's after "bitbake_min_version " with re.match() >> which means only a string. > > You need to declare it thusly: > BB_MIN_VERSION = "2.15.2" > bb_min_version ${BB_MIN_VERSION} > > and then run the regexp match (which should be '${BB_MIN_VERSION}') > through data.expand() inside eval() in ast.py. It's supposed to work, > but for me it did not. > I confirm, same here. (Spent the last few hours reviewing shell code so forgot how to use variables properly in BitBake, thanks for the reminder!) diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf index 54fa04e213..d6dbad040c 100644 --- a/meta/conf/layer.conf +++ b/meta/conf/layer.conf @@ -1,3 +1,6 @@ +BB_MIN_VERSION = "2.15.2" +bitbake_min_version ${BB_MIN_VERSION} + # We have a conf and classes directory, add to BBPATH BBPATH .= ":${LAYERDIR}" # We have recipes-* directories, add to BBFILES Looking into the BB RFC patch, I'm wondering if we don't need to explicitly expand the string passed to bb_min_version? I see that the IncludeNode class does it during its eval() method. I have no clue if this is proper but diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py index 20c6f51a8..9f6874b06 100644 --- a/lib/bb/parse/ast.py +++ b/lib/bb/parse/ast.py @@ -418,6 +418,7 @@ class BbMinVersionNode(AstNode): self.bb_min_version = bb_min_version def eval(self, data): + self.bb_min_version = data.expand(self.bb_min_version) bb.parse.ConfHandler.check_bb_version(self.filename, self.lineno, self.bb_min_version) def handleInclude(statements, filename, lineno, m, force): Made it work (both with ${BB_MIN_VERSION} and the 2.15.2 string). Quentin
On Wed, 22 Oct 2025 at 17:22, Quentin Schulz <quentin.schulz@cherry.de> wrote: > I have no clue if this is proper but > > diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py > index 20c6f51a8..9f6874b06 100644 > --- a/lib/bb/parse/ast.py > +++ b/lib/bb/parse/ast.py > @@ -418,6 +418,7 @@ class BbMinVersionNode(AstNode): > self.bb_min_version = bb_min_version > > def eval(self, data): > + self.bb_min_version = data.expand(self.bb_min_version) > bb.parse.ConfHandler.check_bb_version(self.filename, > self.lineno, self.bb_min_version) > > def handleInclude(statements, filename, lineno, m, force): > > Made it work (both with ${BB_MIN_VERSION} and the 2.15.2 string). Well, this is what I did (data.expand(...)), and yet the expansion didn't happen for me. Can you share a patch where it works? Alex
On 10/22/25 5:26 PM, Alexander Kanavin wrote: > On Wed, 22 Oct 2025 at 17:22, Quentin Schulz <quentin.schulz@cherry.de> wrote: > >> I have no clue if this is proper but >> >> diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py >> index 20c6f51a8..9f6874b06 100644 >> --- a/lib/bb/parse/ast.py >> +++ b/lib/bb/parse/ast.py >> @@ -418,6 +418,7 @@ class BbMinVersionNode(AstNode): >> self.bb_min_version = bb_min_version >> >> def eval(self, data): >> + self.bb_min_version = data.expand(self.bb_min_version) >> bb.parse.ConfHandler.check_bb_version(self.filename, >> self.lineno, self.bb_min_version) >> >> def handleInclude(statements, filename, lineno, m, force): >> >> Made it work (both with ${BB_MIN_VERSION} and the 2.15.2 string). > > Well, this is what I did (data.expand(...)), and yet the expansion > didn't happen for me. Can you share a patch where it works? > BitBake master (35a421cdc) + your patch from https://lore.kernel.org/bitbake-devel/20251022113146.3207795-1-alex.kanavin@gmail.com/T/#u + diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py index 20c6f51a8..9f6874b06 100644 --- a/lib/bb/parse/ast.py +++ b/lib/bb/parse/ast.py @@ -418,6 +418,7 @@ class BbMinVersionNode(AstNode): self.bb_min_version = bb_min_version def eval(self, data): + self.bb_min_version = data.expand(self.bb_min_version) bb.parse.ConfHandler.check_bb_version(self.filename, self.lineno, self.bb_min_version) def handleInclude(statements, filename, lineno, m, force): OE-COre master (7a900ff52d) + diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf index 54fa04e213..5333b4165a 100644 --- a/meta/conf/layer.conf +++ b/meta/conf/layer.conf @@ -1,3 +1,6 @@ +BB_MIN_VERSION = "2.15.7" +bitbake_min_version ${BB_MIN_VERSION} + # We have a conf and classes directory, add to BBPATH BBPATH .= ":${LAYERDIR}" # We have recipes-* directories, add to BBFILES source oe-init-build-env ../build-oe-core-tmp Then run bitbake-getvar BB_MIN_VERSION NOTE: Starting bitbake server... ERROR: Unable to parse /home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py Traceback (most recent call last): File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 108, in check_bb_version raise ParseError('Bitbake version %s is required and version %s is used\n' % (min_version, bb.__version__), fn, lineno) bb.parse.ParseError: ParseError at /home/qschulz/work/upstream/yocto/openembedded-core/meta/conf/layer.conf:2: Bitbake version 2.15.7 is required and version 2.15.2 is used ERROR: Unable to parse /home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py Traceback (most recent call last): File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 108, in check_bb_version raise ParseError('Bitbake version %s is required and version %s is used\n' % (min_version, bb.__version__), fn, lineno) bb.parse.ParseError: ParseError at /home/qschulz/work/upstream/yocto/openembedded-core/meta/conf/layer.conf:2: Bitbake version 2.15.7 is required and version 2.15.2 is used Traceback (most recent call last): File "/home/qschulz/work/upstream/yocto/bitbake/bin/bitbake-getvar", line 54, in <module> tinfoil.prepare(quiet=2, config_only=True) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/tinfoil.py", line 550, in prepare config_params.updateToServer(self.server_connection.connection, os.environ.copy()) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/cookerdata.py", line 74, in updateToServer ret, error = server.runCommand(["updateConfig", options, environment, sys.argv]) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/server/process.py", line 492, in runCommand raise bb.BBHandledException() bb.BBHandledException During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/qschulz/work/upstream/yocto/bitbake/bin/bitbake-getvar", line 46, in <module> with bb.tinfoil.Tinfoil(tracking=True, setup_logging=not quiet) as tinfoil: ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/tinfoil.py", line 477, in __exit__ self.shutdown() ~~~~~~~~~~~~~^^ File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/tinfoil.py", line 996, in shutdown self.run_command('clientComplete') ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/tinfoil.py", line 612, in run_command result = self.server_connection.connection.runCommand(commandline) File "/home/qschulz/work/upstream/yocto/bitbake/lib/bb/server/process.py", line 492, in runCommand raise bb.BBHandledException() bb.BBHandledException Changing BB_MIN_VERSION to 2.15.2 results in NOTE: Starting bitbake server... # # $BB_MIN_VERSION [2 operations] # set /home/qschulz/work/upstream/yocto/openembedded-core/meta/conf/layer.conf:1 # "2.15.2" # set /home/qschulz/work/upstream/yocto/openembedded-core/meta/conf/sanity.conf:6 # "2.15.2" # pre-expansion value: # "2.15.2" BB_MIN_VERSION="2.15.2" as expected. Host distro is fedora 42, no container involved. Quentin
On Wed, 22 Oct 2025 at 17:31, Quentin Schulz <quentin.schulz@cherry.de> wrote: > Changing BB_MIN_VERSION to 2.15.2 results in > > NOTE: Starting bitbake server... > # > # $BB_MIN_VERSION [2 operations] > # set > /home/qschulz/work/upstream/yocto/openembedded-core/meta/conf/layer.conf:1 > # "2.15.2" > # set > /home/qschulz/work/upstream/yocto/openembedded-core/meta/conf/sanity.conf:6 > # "2.15.2" > # pre-expansion value: > # "2.15.2" > BB_MIN_VERSION="2.15.2" > > as expected. Thanks, yes this works for me too. Not sure what I did incorrectly the first time around. I'll incorporate this into the next version, but it would be good to get RP's approval of the overall idea. Maybe this can be done without adding a whole new keyword (which is then used once). Alex
diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf index 54fa04e213e..6c05c0e1934 100644 --- a/meta/conf/layer.conf +++ b/meta/conf/layer.conf @@ -1,3 +1,5 @@ +bitbake_min_version 2.15.2 + # We have a conf and classes directory, add to BBPATH BBPATH .= ":${LAYERDIR}" # We have recipes-* directories, add to BBFILES