diff mbox series

runqemu: restore support to run without bitbake

Message ID 20260129030811.104017-1-Qi.Chen@windriver.com
State New
Headers show
Series runqemu: restore support to run without bitbake | expand

Commit Message

ChenQi Jan. 29, 2026, 3:08 a.m. UTC 
From: Chen Qi <Qi.Chen@windriver.com>

This patch is basically a revert of the following three patches:
- 0c10a78796 scripts/runqemu: raise an error when bitbake was not found
- b931f74442 scripts/runqemu: remove the code block that works around the missing bitbake environment
- 8197be4dd3 runqemu: ensure that bitbake environment is either returned, or an exception is raised

The comment is also changed to reflect the current usage:
"invoked from a running bitbake instance" ->
"invoked from environment with no bitbake (e.g., SDK)"

This code path was deleted by accident based on the reason that nobody
is using it. But in fact, running runqemu from SDK needs this code path.
Such case has been supported for years.

A little more history on this code path:
In 2016/09/09, this code path was introduced[1]. It was introduced
originally to handle running bitbake inside bitbake situation.
In 2026/09/19, running runqemu from SDK is supported using this code
path[2]. This means that this code path was then needed by one more case: SDK.

Supporting running nativesdk-qemu from SDK easily via our script dates back to
more than 18 years ago, as shown by 'git blame' on the nativesdk-qemu-helper recipe[3].

[1] https://git.openembedded.org/openembedded-core/commit/?id=1e8165ea2f19aecdc03ccd102ee44ef0544f0f39
[2] https://git.openembedded.org/openembedded-core/commit/?id=93649edc034f2540ff55dc9b41638797209cfb9c
[3] https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/qemu/nativesdk-qemu-helper_1.0.bb

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 scripts/runqemu | 47 ++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 36 insertions(+), 11 deletions(-)

Comments

Alexander Kanavin Jan. 29, 2026, 9:57 a.m. UTC | #1 
Apologies, but you can't just hastily undo these because something
broke in your setup. We've made these fixes to solve other problems
and improve usability, so if there's a use case that needs to be
supported, you need to show steps to perform that use case, what you
expect to happen, what actually happens, and whether we need to
introduce tests that ensure this use case works. Then we can work
towards a fix that does not re-introduce the same old problems.

Alex

On Thu, 29 Jan 2026 at 04:08, Chen Qi via lists.openembedded.org
<Qi.Chen=windriver.com@lists.openembedded.org> wrote:
>
> From: Chen Qi <Qi.Chen@windriver.com>
>
> This patch is basically a revert of the following three patches:
> - 0c10a78796 scripts/runqemu: raise an error when bitbake was not found
> - b931f74442 scripts/runqemu: remove the code block that works around the missing bitbake environment
> - 8197be4dd3 runqemu: ensure that bitbake environment is either returned, or an exception is raised
>
> The comment is also changed to reflect the current usage:
> "invoked from a running bitbake instance" ->
> "invoked from environment with no bitbake (e.g., SDK)"
>
> This code path was deleted by accident based on the reason that nobody
> is using it. But in fact, running runqemu from SDK needs this code path.
> Such case has been supported for years.
>
> A little more history on this code path:
> In 2016/09/09, this code path was introduced[1]. It was introduced
> originally to handle running bitbake inside bitbake situation.
> In 2026/09/19, running runqemu from SDK is supported using this code
> path[2]. This means that this code path was then needed by one more case: SDK.
>
> Supporting running nativesdk-qemu from SDK easily via our script dates back to
> more than 18 years ago, as shown by 'git blame' on the nativesdk-qemu-helper recipe[3].
>
> [1] https://git.openembedded.org/openembedded-core/commit/?id=1e8165ea2f19aecdc03ccd102ee44ef0544f0f39
> [2] https://git.openembedded.org/openembedded-core/commit/?id=93649edc034f2540ff55dc9b41638797209cfb9c
> [3] https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/qemu/nativesdk-qemu-helper_1.0.bb
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  scripts/runqemu | 47 ++++++++++++++++++++++++++++++++++++-----------
>  1 file changed, 36 insertions(+), 11 deletions(-)
>
> diff --git a/scripts/runqemu b/scripts/runqemu
> index 32a3d6296a..7e3d11f653 100755
> --- a/scripts/runqemu
> +++ b/scripts/runqemu
> @@ -1007,12 +1007,34 @@ to your build configuration.
>              if not self.bitbake_e:
>                  self.load_bitbake_env()
>
> -            native_vars = ['STAGING_DIR_NATIVE']
> -            for nv in native_vars:
> -                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
> -                if s and s.group(1) != self.get(nv):
> -                    logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
> -                    self.set(nv, s.group(1))
> +            if self.bitbake_e:
> +                native_vars = ['STAGING_DIR_NATIVE']
> +                for nv in native_vars:
> +                    s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
> +                    if s and s.group(1) != self.get(nv):
> +                        logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
> +                        self.set(nv, s.group(1))
> +            else:
> +                # when we're invoked from environment with no bitbake (e.g., SDK),
> +                # we won't be able to call `bitbake -e`, then try:
> +                # - get OE_TMPDIR from environment and guess paths based on it
> +                # - get OECORE_NATIVE_SYSROOT from environment (for sdk)
> +                tmpdir = self.get('OE_TMPDIR')
> +                oecore_native_sysroot = self.get('OECORE_NATIVE_SYSROOT')
> +                if tmpdir:
> +                    logger.info('Setting STAGING_DIR_NATIVE and STAGING_BINDIR_NATIVE relative to OE_TMPDIR (%s)' % tmpdir)
> +                    hostos, _, _, _, machine = os.uname()
> +                    buildsys = '%s-%s' % (machine, hostos.lower())
> +                    staging_dir_native = '%s/sysroots/%s' % (tmpdir, buildsys)
> +                    self.set('STAGING_DIR_NATIVE', staging_dir_native)
> +                elif oecore_native_sysroot:
> +                    logger.info('Setting STAGING_DIR_NATIVE to OECORE_NATIVE_SYSROOT (%s)' % oecore_native_sysroot)
> +                    self.set('STAGING_DIR_NATIVE', oecore_native_sysroot)
> +                if self.get('STAGING_DIR_NATIVE'):
> +                    # we have to assume that STAGING_BINDIR_NATIVE is at usr/bin
> +                    staging_bindir_native = '%s/usr/bin' % self.get('STAGING_DIR_NATIVE')
> +                    logger.info('Setting STAGING_BINDIR_NATIVE to %s' % staging_bindir_native)
> +                    self.set('STAGING_BINDIR_NATIVE', '%s/usr/bin' % self.get('STAGING_DIR_NATIVE'))
>
>      def print_config(self):
>          logoutput = ['Continuing with the following parameters:']
> @@ -1691,6 +1713,9 @@ to your build configuration.
>          self.cleaned = True
>
>      def run_bitbake_env(self, mach=None, target=''):
> +        bitbake = shutil.which('bitbake')
> +        if not bitbake:
> +            return
>
>          if not mach:
>              mach = self.get('MACHINE')
> @@ -1707,10 +1732,6 @@ to your build configuration.
>          else:
>              cmd = 'bitbake -e %s %s' % (multiconfig, target)
>
> -        bitbake = shutil.which('bitbake')
> -        if not bitbake:
> -            raise OEPathError("Bitbake is needed to run '%s', but it is not found in PATH. Please source the bitbake build environment." % cmd.strip())
> -
>          logger.info('Running %s...' % cmd)
>          try:
>              return subprocess.check_output(cmd, shell=True).decode('utf-8')
> @@ -1722,7 +1743,11 @@ to your build configuration.
>                  cmd = 'MACHINE=%s bitbake -e %s %s' % (mach, multiconfig, target)
>              else:
>                  cmd = 'bitbake -e %s %s' % (multiconfig, target)
> -            return subprocess.check_output(cmd, shell=True).decode('utf-8')
> +            try:
> +                return subprocess.check_output(cmd, shell=True).decode('utf-8')
> +            except subprocess.CalledProcessError as err:
> +                logger.warning("Couldn't run '%s' to gather environment information, giving up with 'bitbake -e':\n%s" % (cmd, err.output.decode('utf-8')))
> +                return ''
>
>
>      def load_bitbake_env(self, mach=None, target=None):
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#230110): https://lists.openembedded.org/g/openembedded-core/message/230110
> Mute This Topic: https://lists.openembedded.org/mt/117521005/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Jörg Sommer Jan. 29, 2026, 3:48 p.m. UTC | #2 
Chen Qi via lists.openembedded.org schrieb am Do 29. Jan, 11:08 (+0800):
> From: Chen Qi <Qi.Chen@windriver.com>
> 
> This patch is basically a revert of the following three patches:
> - 0c10a78796 scripts/runqemu: raise an error when bitbake was not found
> - b931f74442 scripts/runqemu: remove the code block that works around the missing bitbake environment
> - 8197be4dd3 runqemu: ensure that bitbake environment is either returned, or an exception is raised
> 
> The comment is also changed to reflect the current usage:
> "invoked from a running bitbake instance" ->
> "invoked from environment with no bitbake (e.g., SDK)"
> 
> This code path was deleted by accident based on the reason that nobody
> is using it. But in fact, running runqemu from SDK needs this code path.
> Such case has been supported for years.

By the way: I use runqemu (version from kirkstone) without bitbake, because
bitbake is in a docker container (kas-container):

OECORE_NATIVE_SYSROOT=/ runqemu ....qemuboot.conf


Regards Jörg
ChenQi Jan. 30, 2026, 2:58 a.m. UTC | #3 
Hi Alex,

I read the commit messages carefully but I did not find any use case or 
yocto bug that those patches fix or improve.

The major reason for this code path removal, according to the the commit 
log, is:
"""
this code path is never taken and can be removed.
"""

This reason does not stand, which can be seen from a 'git blame' on the 
codes. I've also found the history and put that in the commit message.

I will send out V2 to add the use case and steps of using runqemu from 
SDK and provide some background.

If there is some use case that these three commits fix, please let me 
know and I'll take that into consideration and send out V3.

Regards,
Qi

On 1/29/26 17:57, Alexander Kanavin wrote:
> Apologies, but you can't just hastily undo these because something
> broke in your setup. We've made these fixes to solve other problems
> and improve usability, so if there's a use case that needs to be
> supported, you need to show steps to perform that use case, what you
> expect to happen, what actually happens, and whether we need to
> introduce tests that ensure this use case works. Then we can work
> towards a fix that does not re-introduce the same old problems.
>
> Alex
>
> On Thu, 29 Jan 2026 at 04:08, Chen Qi via lists.openembedded.org
> <Qi.Chen=windriver.com@lists.openembedded.org> wrote:
>> From: Chen Qi <Qi.Chen@windriver.com>
>>
>> This patch is basically a revert of the following three patches:
>> - 0c10a78796 scripts/runqemu: raise an error when bitbake was not found
>> - b931f74442 scripts/runqemu: remove the code block that works around the missing bitbake environment
>> - 8197be4dd3 runqemu: ensure that bitbake environment is either returned, or an exception is raised
>>
>> The comment is also changed to reflect the current usage:
>> "invoked from a running bitbake instance" ->
>> "invoked from environment with no bitbake (e.g., SDK)"
>>
>> This code path was deleted by accident based on the reason that nobody
>> is using it. But in fact, running runqemu from SDK needs this code path.
>> Such case has been supported for years.
>>
>> A little more history on this code path:
>> In 2016/09/09, this code path was introduced[1]. It was introduced
>> originally to handle running bitbake inside bitbake situation.
>> In 2026/09/19, running runqemu from SDK is supported using this code
>> path[2]. This means that this code path was then needed by one more case: SDK.
>>
>> Supporting running nativesdk-qemu from SDK easily via our script dates back to
>> more than 18 years ago, as shown by 'git blame' on the nativesdk-qemu-helper recipe[3].
>>
>> [1] https://git.openembedded.org/openembedded-core/commit/?id=1e8165ea2f19aecdc03ccd102ee44ef0544f0f39
>> [2] https://git.openembedded.org/openembedded-core/commit/?id=93649edc034f2540ff55dc9b41638797209cfb9c
>> [3] https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/qemu/nativesdk-qemu-helper_1.0.bb
>>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>> ---
>>   scripts/runqemu | 47 ++++++++++++++++++++++++++++++++++++-----------
>>   1 file changed, 36 insertions(+), 11 deletions(-)
>>
>> diff --git a/scripts/runqemu b/scripts/runqemu
>> index 32a3d6296a..7e3d11f653 100755
>> --- a/scripts/runqemu
>> +++ b/scripts/runqemu
>> @@ -1007,12 +1007,34 @@ to your build configuration.
>>               if not self.bitbake_e:
>>                   self.load_bitbake_env()
>>
>> -            native_vars = ['STAGING_DIR_NATIVE']
>> -            for nv in native_vars:
>> -                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
>> -                if s and s.group(1) != self.get(nv):
>> -                    logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
>> -                    self.set(nv, s.group(1))
>> +            if self.bitbake_e:
>> +                native_vars = ['STAGING_DIR_NATIVE']
>> +                for nv in native_vars:
>> +                    s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
>> +                    if s and s.group(1) != self.get(nv):
>> +                        logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
>> +                        self.set(nv, s.group(1))
>> +            else:
>> +                # when we're invoked from environment with no bitbake (e.g., SDK),
>> +                # we won't be able to call `bitbake -e`, then try:
>> +                # - get OE_TMPDIR from environment and guess paths based on it
>> +                # - get OECORE_NATIVE_SYSROOT from environment (for sdk)
>> +                tmpdir = self.get('OE_TMPDIR')
>> +                oecore_native_sysroot = self.get('OECORE_NATIVE_SYSROOT')
>> +                if tmpdir:
>> +                    logger.info('Setting STAGING_DIR_NATIVE and STAGING_BINDIR_NATIVE relative to OE_TMPDIR (%s)' % tmpdir)
>> +                    hostos, _, _, _, machine = os.uname()
>> +                    buildsys = '%s-%s' % (machine, hostos.lower())
>> +                    staging_dir_native = '%s/sysroots/%s' % (tmpdir, buildsys)
>> +                    self.set('STAGING_DIR_NATIVE', staging_dir_native)
>> +                elif oecore_native_sysroot:
>> +                    logger.info('Setting STAGING_DIR_NATIVE to OECORE_NATIVE_SYSROOT (%s)' % oecore_native_sysroot)
>> +                    self.set('STAGING_DIR_NATIVE', oecore_native_sysroot)
>> +                if self.get('STAGING_DIR_NATIVE'):
>> +                    # we have to assume that STAGING_BINDIR_NATIVE is at usr/bin
>> +                    staging_bindir_native = '%s/usr/bin' % self.get('STAGING_DIR_NATIVE')
>> +                    logger.info('Setting STAGING_BINDIR_NATIVE to %s' % staging_bindir_native)
>> +                    self.set('STAGING_BINDIR_NATIVE', '%s/usr/bin' % self.get('STAGING_DIR_NATIVE'))
>>
>>       def print_config(self):
>>           logoutput = ['Continuing with the following parameters:']
>> @@ -1691,6 +1713,9 @@ to your build configuration.
>>           self.cleaned = True
>>
>>       def run_bitbake_env(self, mach=None, target=''):
>> +        bitbake = shutil.which('bitbake')
>> +        if not bitbake:
>> +            return
>>
>>           if not mach:
>>               mach = self.get('MACHINE')
>> @@ -1707,10 +1732,6 @@ to your build configuration.
>>           else:
>>               cmd = 'bitbake -e %s %s' % (multiconfig, target)
>>
>> -        bitbake = shutil.which('bitbake')
>> -        if not bitbake:
>> -            raise OEPathError("Bitbake is needed to run '%s', but it is not found in PATH. Please source the bitbake build environment." % cmd.strip())
>> -
>>           logger.info('Running %s...' % cmd)
>>           try:
>>               return subprocess.check_output(cmd, shell=True).decode('utf-8')
>> @@ -1722,7 +1743,11 @@ to your build configuration.
>>                   cmd = 'MACHINE=%s bitbake -e %s %s' % (mach, multiconfig, target)
>>               else:
>>                   cmd = 'bitbake -e %s %s' % (multiconfig, target)
>> -            return subprocess.check_output(cmd, shell=True).decode('utf-8')
>> +            try:
>> +                return subprocess.check_output(cmd, shell=True).decode('utf-8')
>> +            except subprocess.CalledProcessError as err:
>> +                logger.warning("Couldn't run '%s' to gather environment information, giving up with 'bitbake -e':\n%s" % (cmd, err.output.decode('utf-8')))
>> +                return ''
>>
>>
>>       def load_bitbake_env(self, mach=None, target=None):
>> --
>> 2.34.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#230110): https://lists.openembedded.org/g/openembedded-core/message/230110
>> Mute This Topic: https://lists.openembedded.org/mt/117521005/1686489
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
diff mbox series

Patch

diff --git a/scripts/runqemu b/scripts/runqemu
index 32a3d6296a..7e3d11f653 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -1007,12 +1007,34 @@  to your build configuration.
             if not self.bitbake_e:
                 self.load_bitbake_env()
 
-            native_vars = ['STAGING_DIR_NATIVE']
-            for nv in native_vars:
-                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
-                if s and s.group(1) != self.get(nv):
-                    logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
-                    self.set(nv, s.group(1))
+            if self.bitbake_e:
+                native_vars = ['STAGING_DIR_NATIVE']
+                for nv in native_vars:
+                    s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
+                    if s and s.group(1) != self.get(nv):
+                        logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
+                        self.set(nv, s.group(1))
+            else:
+                # when we're invoked from environment with no bitbake (e.g., SDK),
+                # we won't be able to call `bitbake -e`, then try:
+                # - get OE_TMPDIR from environment and guess paths based on it
+                # - get OECORE_NATIVE_SYSROOT from environment (for sdk)
+                tmpdir = self.get('OE_TMPDIR')
+                oecore_native_sysroot = self.get('OECORE_NATIVE_SYSROOT')
+                if tmpdir:
+                    logger.info('Setting STAGING_DIR_NATIVE and STAGING_BINDIR_NATIVE relative to OE_TMPDIR (%s)' % tmpdir)
+                    hostos, _, _, _, machine = os.uname()
+                    buildsys = '%s-%s' % (machine, hostos.lower())
+                    staging_dir_native = '%s/sysroots/%s' % (tmpdir, buildsys)
+                    self.set('STAGING_DIR_NATIVE', staging_dir_native)
+                elif oecore_native_sysroot:
+                    logger.info('Setting STAGING_DIR_NATIVE to OECORE_NATIVE_SYSROOT (%s)' % oecore_native_sysroot)
+                    self.set('STAGING_DIR_NATIVE', oecore_native_sysroot)
+                if self.get('STAGING_DIR_NATIVE'):
+                    # we have to assume that STAGING_BINDIR_NATIVE is at usr/bin
+                    staging_bindir_native = '%s/usr/bin' % self.get('STAGING_DIR_NATIVE')
+                    logger.info('Setting STAGING_BINDIR_NATIVE to %s' % staging_bindir_native)
+                    self.set('STAGING_BINDIR_NATIVE', '%s/usr/bin' % self.get('STAGING_DIR_NATIVE'))
 
     def print_config(self):
         logoutput = ['Continuing with the following parameters:']
@@ -1691,6 +1713,9 @@  to your build configuration.
         self.cleaned = True
 
     def run_bitbake_env(self, mach=None, target=''):
+        bitbake = shutil.which('bitbake')
+        if not bitbake:
+            return
 
         if not mach:
             mach = self.get('MACHINE')
@@ -1707,10 +1732,6 @@  to your build configuration.
         else:
             cmd = 'bitbake -e %s %s' % (multiconfig, target)
 
-        bitbake = shutil.which('bitbake')
-        if not bitbake:
-            raise OEPathError("Bitbake is needed to run '%s', but it is not found in PATH. Please source the bitbake build environment." % cmd.strip())
-
         logger.info('Running %s...' % cmd)
         try:
             return subprocess.check_output(cmd, shell=True).decode('utf-8')
@@ -1722,7 +1743,11 @@  to your build configuration.
                 cmd = 'MACHINE=%s bitbake -e %s %s' % (mach, multiconfig, target)
             else:
                 cmd = 'bitbake -e %s %s' % (multiconfig, target)
-            return subprocess.check_output(cmd, shell=True).decode('utf-8')
+            try:
+                return subprocess.check_output(cmd, shell=True).decode('utf-8')
+            except subprocess.CalledProcessError as err:
+                logger.warning("Couldn't run '%s' to gather environment information, giving up with 'bitbake -e':\n%s" % (cmd, err.output.decode('utf-8')))
+                return ''
 
 
     def load_bitbake_env(self, mach=None, target=None):