| Message ID | 20240502183501.1216856-1-JPEWhacker@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [bitbake-devel] cooker: Handle ImportError for websockets | expand |
On Thu, 2 May 2024 at 20:35, Joshua Watt via lists.openembedded.org <JPEWhacker=gmail.com@lists.openembedded.org> wrote: > - except ConnectionError as e: > + except (ConnectionError, ImportError) as e: > bb.warn("BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash equivalence server at '%s': %s" > % (upstream, repr(e))) > + upstream = None I'm setting up a bitbake-setup configuration that uses BB_HASHSERVE_UPSTREAM and the official yocto sstate cache on the cdn, and I think we have a usability issue here. Specifically, if the user does not have websockets python module installed, then they will only get a warning: WARNING: BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash equivalence server at 'wss://hashserv.yoctoproject.org/ws': ModuleNotFoundError("No module named 'websockets'") without a suggestion about how to resolve it. Worse yet, the build will proceed, but fail to use any sstate, resulting in rebuilds from source, and a bad first experience for yocto newcomers. I would rather turn both of these into two separate hard fails (bb.fatal), and be more specific about why bitbake can't proceed for each of them. Then there's the question of how to handle absent websockets: - we can ask the user to install them on the system, which is not always straightforward - we can somehow detect before running bitbake for the first time that buildtools-tarball is needed (not sure how to do that tbh) - we can extend bitbake-setup configurations with 'install-buildtoos: true' parameter, and then use it in all official configurations that need functional websockets to access pre-built sstate to always install buildtools, even on systems where it's not actually necessary. None of these are great, but I'm leaning towards the last. Looking for better ideas! Alex
On Thu, 20 Feb 2025 at 17:05, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > - we can extend bitbake-setup configurations with 'install-buildtoos: > true' parameter, and then use it in all official configurations that > need functional websockets to access pre-built sstate to always > install buildtools, even on systems where it's not actually necessary. > > None of these are great, but I'm leaning towards the last. Looking for > better ideas! Ok, I've slept on it, and concluded that bitbake-setup should simply always install buildtools by default. It's a front line tool for new users, and it can't afford to fail in mysterious ways. There will be a setting to turn it off, for those who know what they're doing and know they don't need buildtools. Alex
On Fri, 2025-02-21 at 10:11 +0100, Alexander Kanavin wrote: > On Thu, 20 Feb 2025 at 17:05, Alexander Kanavin via > lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> > wrote: > > > - we can extend bitbake-setup configurations with 'install-buildtoos: > > true' parameter, and then use it in all official configurations that > > need functional websockets to access pre-built sstate to always > > install buildtools, even on systems where it's not actually necessary. > > > > None of these are great, but I'm leaning towards the last. Looking for > > better ideas! > > Ok, I've slept on it, and concluded that bitbake-setup should simply > always install buildtools by default. It's a front line tool for new > users, and it can't afford to fail in mysterious ways. There will be a > setting to turn it off, for those who know what they're doing and know > they don't need buildtools. I think that should be an option but I'm reluctant to say to use it by default as we're just not going to find out about host issues until it is too late. Our sanity tests are supposed to detect and give human readable errors for the common issues you'd encounter. Cheers, Richard
On Thu, 2025-02-20 at 17:05 +0100, Alexander Kanavin wrote: > On Thu, 2 May 2024 at 20:35, Joshua Watt via lists.openembedded.org > <JPEWhacker=gmail.com@lists.openembedded.org> wrote: > > - except ConnectionError as e: > > + except (ConnectionError, ImportError) as e: > > bb.warn("BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash equivalence server at '%s': %s" > > % (upstream, repr(e))) > > + upstream = None > > I'm setting up a bitbake-setup configuration that uses > BB_HASHSERVE_UPSTREAM and the official yocto sstate cache on the cdn, > and I think we have a usability issue here. > > Specifically, if the user does not have websockets python module > installed, then they will only get a warning: > > WARNING: BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash > equivalence server at 'wss://hashserv.yoctoproject.org/ws': > ModuleNotFoundError("No module named 'websockets'") > > without a suggestion about how to resolve it. Worse yet, the build > will proceed, but fail to use any sstate, resulting in rebuilds from > source, and a bad first experience for yocto newcomers. I think we should split that exception handler to catch ImportError separately and mention in the message websockets was missing, if wss:// is in the url. Cheers, Richard
On Fri, 21 Feb 2025 at 10:14, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > Ok, I've slept on it, and concluded that bitbake-setup should simply > > always install buildtools by default. It's a front line tool for new > > users, and it can't afford to fail in mysterious ways. There will be a > > setting to turn it off, for those who know what they're doing and know > > they don't need buildtools. > > I think that should be an option but I'm reluctant to say to use it by > default as we're just not going to find out about host issues until it > is too late. Our sanity tests are supposed to detect and give human > readable errors for the common issues you'd encounter. One option is that bitbake-setup always installs buildtools (e.g. runs scripts/install-buildtools which in itself is harmless and doesn't affect builds), but makes their use opt-in. E.g. bitbake-setup already creates build/init-build-env build/build-targets that do not use buildtools. One initializes bitbake environment in the current shell, the other runs pre-defined targets. Additionally bitbake-setup could also create build/init-build-env-with-buildtools build/build-targets-with-buildtools that also source the buildtools environment script. No need for options or settings or guessing in the tools, we just need to highlight the difference, and guide to use one or the other. Alex
On Fri, Feb 21, 2025 at 2:16 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Thu, 2025-02-20 at 17:05 +0100, Alexander Kanavin wrote: > > On Thu, 2 May 2024 at 20:35, Joshua Watt via lists.openembedded.org > > <JPEWhacker=gmail.com@lists.openembedded.org> wrote: > > > - except ConnectionError as e: > > > + except (ConnectionError, ImportError) as e: > > > bb.warn("BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash equivalence server at '%s': %s" > > > % (upstream, repr(e))) > > > + upstream = None > > > > I'm setting up a bitbake-setup configuration that uses > > BB_HASHSERVE_UPSTREAM and the official yocto sstate cache on the cdn, > > and I think we have a usability issue here. > > > > Specifically, if the user does not have websockets python module > > installed, then they will only get a warning: > > > > WARNING: BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash > > equivalence server at 'wss://hashserv.yoctoproject.org/ws': > > ModuleNotFoundError("No module named 'websockets'") > > > > without a suggestion about how to resolve it. Worse yet, the build > > will proceed, but fail to use any sstate, resulting in rebuilds from > > source, and a bad first experience for yocto newcomers. > > I think we should split that exception handler to catch ImportError > separately and mention in the message websockets was missing, if wss:// > is in the url. I agree the error message could be better, and also that buildtools should not be on by default. Perhaps something as simple as: bitbake-setup --buildtools could enable it automatically? I think if there is such a simple option like that it will make it pretty easy for the end users to do the right thing. > > Cheers, > > Richard
On Fri, 21 Feb 2025 at 17:07, Joshua Watt <jpewhacker@gmail.com> wrote: > I agree the error message could be better, and also that buildtools > should not be on by default. Perhaps something as simple as: > > bitbake-setup --buildtools > > could enable it automatically? I think if there is such a simple > option like that it will make it pretty easy for the end users to do > the right thing. How do I put this politely? Such an option is not simple or easy for the end users, and I'm strongly against this sort of 'solution'. The problem is that you, me, RP and everyone else here are suffering from 'yocto experience bias', and such suggestions come from that bias. We know what to do when we see an error that can be resolved by installing buildtools and putting them into PATH. Now please try to put yourself into the shoes of the first time Yocto user. They can't possibly know in advance that they need buildtools. The set up a build with bitbake-setup, they initialize the environment, they run bitbake, and it fails in ways that they will certainly struggle to interpret. How would they discover this magic option that will solve the fails for them, without showing the error to someone experienced? I have to stress again: bitbake-setup is a frontline tool for yocto newcomers. It has to work. I can kinda see why we do not want to sweep host specific issues under the carpet by defaulting to buildtools, but we do need to find a way to clearly point out what the solution to such fails is, in a way that is difficult to miss. I'm still deliberating how to do this. Make a statement during 'bitbake-setup init'? Make another statement in 'build-targets', subject to bitbake failure? Install tools upfront, or provide a script in a standard location in the build directory to do so? If installed upfront, use a separate environment initialization script to enable them, or integrate that into 'special' variants of init-build-env and build-targets (I wrote a separate email suggesting that)? I'm looking for ideas about helping users with this. Alex
On Tue, 2025-02-25 at 09:22 +0100, Alexander Kanavin wrote: > On Fri, 21 Feb 2025 at 17:07, Joshua Watt <jpewhacker@gmail.com> > wrote: > > > I agree the error message could be better, and also that buildtools > > should not be on by default. Perhaps something as simple as: > > > > bitbake-setup --buildtools > > > > could enable it automatically? I think if there is such a simple > > option like that it will make it pretty easy for the end users to > > do > > the right thing. > > How do I put this politely? Such an option is not simple or easy for > the end users, and I'm strongly against this sort of 'solution'. The > problem is that you, me, RP and everyone else here are suffering from > 'yocto experience bias', and such suggestions come from that bias. We > know what to do when we see an error that can be resolved by > installing buildtools and putting them into PATH. There is a tricky balancing act here as you also have to give users a reasonable expectation. We're not a point and click GUI where every possible choice is going to work, in fact the more you deviate from the tested paths, the larger the number of issues you're going to have to handle yourself. I therefore worry that setting an expectation of everything working, just means when something does fail, users are going to more surprised/upset. Hiding host contamination issues behind the scenes means you could end up with a generation of patch submitters who simply don't test/know/care about that set of problems and I'm not sure the project would benefit from that in the long run. So whilst I do see your point, we do have to draw a line somewhere and I'm not convinced always using buildtools is the right one as a default, particularly if we want every day developers using the tool too (which we do). I'd also note that buildtools isn't "free", it does have download time, space use and installation time. Cheers, Richard
On Tue, 25 Feb 2025 at 11:11, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > So whilst I do see your point, we do have to draw a line somewhere and > I'm not convinced always using buildtools is the right one as a > default, particularly if we want every day developers using the tool > too (which we do). I'm no longer proposing always using buildtools by default. The point of my message was something else: should we direct bitbake-setup users to buildtools if they get a fail that can be addressed with them, and if so, how? I'm pretty sure a lot of first timers are going to get the 'missing python websockets module' problem, and they will find a hint about buildtools helpful (and something that is already installed and ready to run even more so). I'd rather think about it now, and not when bitbake-setup, standard configs and associated CDN sstate cache go 'live'. There were several options proposed in my previous message, and they were all skipped in response :( Alex
On Tue, 2025-02-25 at 11:40 +0100, Alexander Kanavin wrote: > On Tue, 25 Feb 2025 at 11:11, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > So whilst I do see your point, we do have to draw a line somewhere and > > I'm not convinced always using buildtools is the right one as a > > default, particularly if we want every day developers using the tool > > too (which we do). > > I'm no longer proposing always using buildtools by default. The point > of my message was something else: should we direct bitbake-setup users > to buildtools if they get a fail that can be addressed with them, and > if so, how? I'm pretty sure a lot of first timers are going to get the > 'missing python websockets module' problem, and they will find a hint > about buildtools helpful (and something that is already installed and > ready to run even more so). I'd rather think about it now, and not > when bitbake-setup, standard configs and associated CDN sstate cache > go 'live'. > > There were several options proposed in my previous message, and they > were all skipped in response :( I'd stand by the idea what failure messages should be good in general and standalone. We have worked on sanity.bbclass in the past to try and ensure that the messages are understandable and give pointers to users. What might be missing is documentation in the quick build docs saying "if you run into host problems, you could use buildtools?" Perhaps if sanity.bbclass is failing the build, there could be a message in there mentioning buildtools as possibly helping/ Could bitbake-setup actually run the sanity tests before the build and wrap the result with a message about buildtools? Cheers, Richard
On Tue, 25 Feb 2025 at 11:52, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > Could bitbake-setup actually run the sanity tests before the build and > wrap the result with a message about buildtools? Perhaps it could run 'bitbake -p', e.g. parse the recipes and quit. This can be done already in 'bitbake-setup init', which will make it slower but will give early feedback if something isn't working, and then bitbake-setup can detect a fail and add a hint about buildtools. I'm going on holiday starting tomorrow, so I'll send the latest revision (without any special handling for buildtools), and we can come back to it later. Alex
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index 25b614f1e44..939a9999746 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -318,9 +318,10 @@ class BBCooker: try: with hashserv.create_client(upstream) as client: client.ping() - except ConnectionError as e: + except (ConnectionError, ImportError) as e: bb.warn("BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash equivalence server at '%s': %s" % (upstream, repr(e))) + upstream = None self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR") self.hashserv = hashserv.create_server(
Handles ImportError when creating a hash equivalence to ping the server. This notifies user earlier with a more precise error if websockets can't be used, and also prevents passing a known bad upstream value to the local server Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- bitbake/lib/bb/cooker.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)