diff mbox series

[V2] runqemu: restore support to run without bitbake

Message ID 20260130030601.388697-1-Qi.Chen@windriver.com
State Under Review
Headers show
Series [V2] runqemu: restore support to run without bitbake | expand

Commit Message

ChenQi Jan. 30, 2026, 3:06 a.m. UTC 
From: Chen Qi <Qi.Chen@windriver.com>

This patch is basically a revert of the following three patches:
- 0c10a78796 scripts/runqemu: raise an error when bitbake was not found
- b931f74442 scripts/runqemu: remove the code block that works around the missing bitbake environment
- 8197be4dd3 runqemu: ensure that bitbake environment is either returned, or an exception is raised

The comment is also changed to reflect the current usage:
"invoked from a running bitbake instance" ->
"invoked from environment with no bitbake (e.g., SDK)"

This code path was deleted by accident based on the reason that nobody
is using it. But in fact, running runqemu from SDK needs this code path.
Such case has been supported for years.

A little more history on this code path:
In 2016/09/09, this code path was introduced[1]. It was introduced
originally to handle running bitbake inside bitbake situation.
In 2026/09/19, running runqemu from SDK is supported using this code
path[2]. This means that this code path was then needed by one more case: SDK.

Supporting running nativesdk-qemu from SDK easily via our script dates back to
more than 18 years ago, as shown by 'git blame' on the nativesdk-qemu-helper recipe[3].

Using runqemu from SDK is useful. Here's a typical case for it:
Team A build different images and SDKs for different machines and copy them to a central place.
Team B test these different images with different methods. For qemu images, they extract
SDK and use runqemu from SDK to test these qemu images.

Below are example steps for the above qemu image copying and running process:
1. mkdir destdir
2. cp -r /path/to/build/tmp/deploy/image/qemux86-64 destdir
3. Install SDK to destdir
4. Source SDK
5. runqemu qemux86-64 nographic slirp

[1] https://git.openembedded.org/openembedded-core/commit/?id=1e8165ea2f19aecdc03ccd102ee44ef0544f0f39
[2] https://git.openembedded.org/openembedded-core/commit/?id=93649edc034f2540ff55dc9b41638797209cfb9c
[3] https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/qemu/nativesdk-qemu-helper_1.0.bb

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 scripts/runqemu | 47 ++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 36 insertions(+), 11 deletions(-)

Comments

Alexander Kanavin Feb. 2, 2026, 3:52 p.m. UTC | #1 
On Fri, 30 Jan 2026 at 04:10, Chen Qi via lists.openembedded.org
<Qi.Chen=windriver.com@lists.openembedded.org> wrote:
> This patch is basically a revert of the following three patches:
> - 0c10a78796 scripts/runqemu: raise an error when bitbake was not found
> - b931f74442 scripts/runqemu: remove the code block that works around the missing bitbake environment
> - 8197be4dd3 runqemu: ensure that bitbake environment is either returned, or an exception is raised

Again, you can't just revert these commits. Please read the commit
messages for the reasons the commits were added. Particularly the
first, and the third one are real improvements, one gives users a
better error message, the other significantly improves the function
call API.

> Using runqemu from SDK is useful. Here's a typical case for it:
> Team A build different images and SDKs for different machines and copy them to a central place.
> Team B test these different images with different methods. For qemu images, they extract
> SDK and use runqemu from SDK to test these qemu images.
>
> Below are example steps for the above qemu image copying and running process:
> 1. mkdir destdir
> 2. cp -r /path/to/build/tmp/deploy/image/qemux86-64 destdir
> 3. Install SDK to destdir
> 4. Source SDK
> 5. runqemu qemux86-64 nographic slirp

So you need to start with this: a test case for these steps. Then when
we can see how the test case fails, we can work out a fix that doesn't
simply undo the changes. I've already said this in the previous
review, so please do not simply resend the patch.

Alex
ChenQi Feb. 3, 2026, 2:21 a.m. UTC | #2 
Hi Alex,

The steps for the test case should be simple and easy to reproduce.
Do you have any suggestion how to fix this issue?
Note that SDK does not have bitbake and OE metadata there.

Or do you want me to first send out a patch to add an OEQA selftest case? Will that be accepted?

Regards,
Qi

-----Original Message-----
From: Alexander Kanavin <alex.kanavin@gmail.com> 
Sent: Monday, February 2, 2026 11:53 PM
To: Chen, Qi <Qi.Chen@windriver.com>
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core][PATCH V2] runqemu: restore support to run without bitbake

On Fri, 30 Jan 2026 at 04:10, Chen Qi via lists.openembedded.org <Qi.Chen=windriver.com@lists.openembedded.org> wrote:
> This patch is basically a revert of the following three patches:
> - 0c10a78796 scripts/runqemu: raise an error when bitbake was not 
> found
> - b931f74442 scripts/runqemu: remove the code block that works around 
> the missing bitbake environment
> - 8197be4dd3 runqemu: ensure that bitbake environment is either 
> returned, or an exception is raised

Again, you can't just revert these commits. Please read the commit messages for the reasons the commits were added. Particularly the first, and the third one are real improvements, one gives users a better error message, the other significantly improves the function call API.

> Using runqemu from SDK is useful. Here's a typical case for it:
> Team A build different images and SDKs for different machines and copy them to a central place.
> Team B test these different images with different methods. For qemu 
> images, they extract SDK and use runqemu from SDK to test these qemu images.
>
> Below are example steps for the above qemu image copying and running process:
> 1. mkdir destdir
> 2. cp -r /path/to/build/tmp/deploy/image/qemux86-64 destdir 3. Install 
> SDK to destdir 4. Source SDK 5. runqemu qemux86-64 nographic slirp

So you need to start with this: a test case for these steps. Then when we can see how the test case fails, we can work out a fix that doesn't simply undo the changes. I've already said this in the previous review, so please do not simply resend the patch.

Alex
Alexander Kanavin Feb. 3, 2026, 10:10 a.m. UTC | #3 
On Tue, 3 Feb 2026 at 03:21, Chen, Qi <Qi.Chen@windriver.com> wrote:
> The steps for the test case should be simple and easy to reproduce.
> Do you have any suggestion how to fix this issue?
> Note that SDK does not have bitbake and OE metadata there.
>
> Or do you want me to first send out a patch to add an OEQA selftest case? Will that be accepted?

I would suggest that you develop that patch first, and send it as a
RFC (because it is expected to fail). I'd then be able to use the
patch to reproduce the failure locally, and come up with suggestions
(or maybe even patches) to address it. That would ensure the use case
is officially supported, and will guard it against future regressions.

Runqemu code is too complex and over-engineered, and to be honest,
needs to be somehow simplified and rewritten, so when we see
opportunities to simplify logic in it, we take it. Your use case fell
victim of that because there were no tests or other indication that
the code is necessary, but it wasn't intentional, and restoring
support for the use case should be fine, but this time it should come
with tests.

Alex
ChenQi Feb. 4, 2026, 2:31 a.m. UTC | #4 
On 2/3/26 18:10, Alexander Kanavin wrote:
> On Tue, 3 Feb 2026 at 03:21, Chen, Qi <Qi.Chen@windriver.com> wrote:
>> The steps for the test case should be simple and easy to reproduce.
>> Do you have any suggestion how to fix this issue?
>> Note that SDK does not have bitbake and OE metadata there.
>>
>> Or do you want me to first send out a patch to add an OEQA selftest case? Will that be accepted?
> I would suggest that you develop that patch first, and send it as a
> RFC (because it is expected to fail). I'd then be able to use the
> patch to reproduce the failure locally, and come up with suggestions
> (or maybe even patches) to address it. That would ensure the use case
> is officially supported, and will guard it against future regressions.
>
> Runqemu code is too complex and over-engineered, and to be honest,
> needs to be somehow simplified and rewritten, so when we see
> opportunities to simplify logic in it, we take it. Your use case fell
> victim of that because there were no tests or other indication that
> the code is necessary, but it wasn't intentional, and restoring
> support for the use case should be fine, but this time it should come
> with tests.
>
> Alex

Got it. I'll add the test case and send out the patch as RFC.

Regards,
Qi
diff mbox series

Patch

diff --git a/scripts/runqemu b/scripts/runqemu
index 32a3d6296a..7e3d11f653 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -1007,12 +1007,34 @@  to your build configuration.
             if not self.bitbake_e:
                 self.load_bitbake_env()
 
-            native_vars = ['STAGING_DIR_NATIVE']
-            for nv in native_vars:
-                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
-                if s and s.group(1) != self.get(nv):
-                    logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
-                    self.set(nv, s.group(1))
+            if self.bitbake_e:
+                native_vars = ['STAGING_DIR_NATIVE']
+                for nv in native_vars:
+                    s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
+                    if s and s.group(1) != self.get(nv):
+                        logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
+                        self.set(nv, s.group(1))
+            else:
+                # when we're invoked from environment with no bitbake (e.g., SDK),
+                # we won't be able to call `bitbake -e`, then try:
+                # - get OE_TMPDIR from environment and guess paths based on it
+                # - get OECORE_NATIVE_SYSROOT from environment (for sdk)
+                tmpdir = self.get('OE_TMPDIR')
+                oecore_native_sysroot = self.get('OECORE_NATIVE_SYSROOT')
+                if tmpdir:
+                    logger.info('Setting STAGING_DIR_NATIVE and STAGING_BINDIR_NATIVE relative to OE_TMPDIR (%s)' % tmpdir)
+                    hostos, _, _, _, machine = os.uname()
+                    buildsys = '%s-%s' % (machine, hostos.lower())
+                    staging_dir_native = '%s/sysroots/%s' % (tmpdir, buildsys)
+                    self.set('STAGING_DIR_NATIVE', staging_dir_native)
+                elif oecore_native_sysroot:
+                    logger.info('Setting STAGING_DIR_NATIVE to OECORE_NATIVE_SYSROOT (%s)' % oecore_native_sysroot)
+                    self.set('STAGING_DIR_NATIVE', oecore_native_sysroot)
+                if self.get('STAGING_DIR_NATIVE'):
+                    # we have to assume that STAGING_BINDIR_NATIVE is at usr/bin
+                    staging_bindir_native = '%s/usr/bin' % self.get('STAGING_DIR_NATIVE')
+                    logger.info('Setting STAGING_BINDIR_NATIVE to %s' % staging_bindir_native)
+                    self.set('STAGING_BINDIR_NATIVE', '%s/usr/bin' % self.get('STAGING_DIR_NATIVE'))
 
     def print_config(self):
         logoutput = ['Continuing with the following parameters:']
@@ -1691,6 +1713,9 @@  to your build configuration.
         self.cleaned = True
 
     def run_bitbake_env(self, mach=None, target=''):
+        bitbake = shutil.which('bitbake')
+        if not bitbake:
+            return
 
         if not mach:
             mach = self.get('MACHINE')
@@ -1707,10 +1732,6 @@  to your build configuration.
         else:
             cmd = 'bitbake -e %s %s' % (multiconfig, target)
 
-        bitbake = shutil.which('bitbake')
-        if not bitbake:
-            raise OEPathError("Bitbake is needed to run '%s', but it is not found in PATH. Please source the bitbake build environment." % cmd.strip())
-
         logger.info('Running %s...' % cmd)
         try:
             return subprocess.check_output(cmd, shell=True).decode('utf-8')
@@ -1722,7 +1743,11 @@  to your build configuration.
                 cmd = 'MACHINE=%s bitbake -e %s %s' % (mach, multiconfig, target)
             else:
                 cmd = 'bitbake -e %s %s' % (multiconfig, target)
-            return subprocess.check_output(cmd, shell=True).decode('utf-8')
+            try:
+                return subprocess.check_output(cmd, shell=True).decode('utf-8')
+            except subprocess.CalledProcessError as err:
+                logger.warning("Couldn't run '%s' to gather environment information, giving up with 'bitbake -e':\n%s" % (cmd, err.output.decode('utf-8')))
+                return ''
 
 
     def load_bitbake_env(self, mach=None, target=None):