diff mbox series

Poky/Mickledore bitbake menuconfig does not work

Message ID DB6PR01MB3736481355587A417D929794A41EA@DB6PR01MB3736.eurprd01.prod.exchangelabs.com
State New
Headers show
Series Poky/Mickledore bitbake menuconfig does not work | expand

Commit Message

Sebert, Holger.ext Aug. 21, 2023, 1:48 p.m. UTC 
Hi,

I am using a Yocto-Poky-Mickledore (version 4.2.1) based system.

For configuring the kernel, I would like to use the command

    bitbake -c menuconfig virtual/kernel

If I do that, Bitbake opens an xterm window and runs the build steps
for Kconfig. However, after the compiler has finised, the xterm window
immediately closes and Bitbake exists as if nothing happened, i.e. it
does not show any error message or similar.

I investigated the problem and found out that, internally, Bitbake
calls the following command for starting Kconfig:

    make menuconfig CC="ccache x86_64-system-linux-gcc ..." \
        CFLAGS="..." HOSTCC="ccache gcc" HOSTCCFLAGS="..."

The quotes, however, seem to get lost when starting the actual command
in the terminal. This leads to a command line like follows:

    make menuconfig CC=ccache x86_64-system-linux-gcc ... \
        CFLAGS=... HOSTCC="ccache gcc" HOSTCCFLAGS="...

which is clearly wrong.

The following patch fixes the problem in meta/classes-recipe/cml1.bbclass:

setup or does the patch above indeed make sense?

Best regards,
Holger

Comments

Sebert, Holger.ext Sept. 14, 2023, 12:10 p.m. UTC | #1 
Hi Zoran,

I was able to reproduce the behavior described in my original post by enabling CCache. Here are the precise steps to reproduce:

1. Checkout Poky/Mickledore
2. Add the following to `build/conf/local.conf`:
    INHERIT += "ccache"
3. Run `bitbake -c menuconfig virtual/kernel`

The reason for the failure is that due to CCache the compiler is no longer `gcc` but `ccache gcc`, i.e. we have the following variables in the environment:

    CC="ccache x86_64-poky-linux-gcc"
    HOSTCC="ccache gcc"
    ...
    etc.

This is when the wrong quoting in the invocation of `menuconfig` leads to errors.

Could you please verify/check?

Best,
Holger
Yoann Congal Sept. 14, 2023, 9:52 p.m. UTC | #2 
Hi,

Le jeu. 14 sept. 2023 à 14:10, Sebert, Holger.ext <
holger.sebert.ext@karlstorz.com> a écrit :

> Hi Zoran,
>
> I was able to reproduce the behavior described in my original post by
> enabling CCache. Here are the precise steps to reproduce:
>
> 1. Checkout Poky/Mickledore
> 2. Add the following to `build/conf/local.conf`:
>     INHERIT += "ccache"
> 3. Run `bitbake -c menuconfig virtual/kernel`
>
> The reason for the failure is that due to CCache the compiler is no longer
> `gcc` but `ccache gcc`, i.e. we have the following variables in the
> environment:
>
>     CC="ccache x86_64-poky-linux-gcc"
>     HOSTCC="ccache gcc"
>     ...
>     etc.
>
> This is when the wrong quoting in the invocation of `menuconfig` leads to
> errors.
>
> Could you please verify/check?
>


I have reproduced this. It really looks like a bug!


> [...]
> On Mon, Aug 21, 2023 at 3:48 PM Sebert, Holger.ext
> <holger.sebert.ext@karlstorz.com> wrote:
> >
> > Hi,
> >
> > I am using a Yocto-Poky-Mickledore (version 4.2.1) based system.
> >
> > For configuring the kernel, I would like to use the command
> >
> >     bitbake -c menuconfig virtual/kernel
> >
> > If I do that, Bitbake opens an xterm window and runs the build steps
> > for Kconfig. However, after the compiler has finised, the xterm window
> > immediately closes and Bitbake exists as if nothing happened, i.e. it
> > does not show any error message or similar.
> >
> > I investigated the problem and found out that, internally, Bitbake
> > calls the following command for starting Kconfig:
> >
> >     make menuconfig CC="ccache x86_64-system-linux-gcc ..." \
> >         CFLAGS="..." HOSTCC="ccache gcc" HOSTCCFLAGS="..."
> >
> > The quotes, however, seem to get lost when starting the actual command
> > in the terminal. This leads to a command line like follows:
> >
> >     make menuconfig CC=ccache x86_64-system-linux-gcc ... \
> >         CFLAGS=... HOSTCC="ccache gcc" HOSTCCFLAGS="...
> >
> > which is clearly wrong.
> >
> > The following patch fixes the problem in
> meta/classes-recipe/cml1.bbclass:
> >
> > diff --git a/meta/classes-recipe/cml1.bbclass
> b/meta/classes-recipe/cml1.bbclass
> > index a09a042c3f..d6001d6fd0 100644
> > --- a/meta/classes-recipe/cml1.bbclass
> > +++ b/meta/classes-recipe/cml1.bbclass
> > @@ -53,7 +53,8 @@ python do_menuconfig() {
> >      # ensure that environment variables are overwritten with this tasks
> 'd' values
> >      d.appendVar("OE_TERMINAL_EXPORTS", " PKG_CONFIG_DIR PKG_CONFIG_PATH
> PKG_CONFIG_LIBDIR PKG_CONFIG_SYSROOT_DIR")
> >
> > -    oe_terminal("sh -c \"make %s; if [ \\$? -ne 0 ]; then echo 'Command
> failed.'; printf 'Press any key to continue... '; read r; fi\"" %
> d.getVar('KCONFIG_CONFIG_COMMAND'),
> > +    make_cmd = "make %s" %
> d.getVar('KCONFIG_CONFIG_COMMAND').replace('"', '\\"')
> > +    oe_terminal("sh -c \"%s; if [ \\$? -ne 0 ]; then echo 'Command
> failed.'; printf 'Press any key to continue... '; read r; fi\"" % make_cmd,
> >                  d.getVar('PN') + ' Configuration', d)
>

This patch looks interesting!
I would use shlex.join() to safely construct the command :
https://docs.python.org/3/library/shlex.html#shlex.join


> > What do you think? Is my observed behavior a configuration error of my
> > setup or does the patch above indeed make sense?
>

The patch makes sense to me.

I guess your next steps are :
* If you want to contribute your patch : reproduce the bug and rebase your
patch on master and send it to the mailing list
* Or, create a bug in https://bugzilla.yoctoproject.org/

Best Regards,
Sebert, Holger.ext Sept. 19, 2023, 4:57 p.m. UTC | #3 
Hi Yoann, hi Zoran,

The bug is not present on the Poky master branch.

Until the project is rebased onto the next release, I will simply disable CCache whenever I need to do a kernel menuconfig.

Thanks!

-Holger
Yoann Congal Sept. 20, 2023, 5:59 p.m. UTC | #4 
Le mar. 19 sept. 2023 à 18:57, Sebert, Holger.ext <
Holger.Sebert.ext@karlstorz.com> a écrit :

> Hi Yoann, hi Zoran,
>

Hi,


> The bug is not present on the Poky master branch.
>
> Until the project is rebased onto the next release, I will simply disable
> CCache whenever I need to do a kernel menuconfig.
>

You peaked my curiosity and I've tracked down the fix on master : Your
problem is fixed by
https://git.yoctoproject.org/poky/commit/?id=d4664d2b7974354e73d891762ebb2c8a12d62438

I've requested a backport on mickledore :
https://lists.openembedded.org/g/openembedded-core/message/187934

Regards,


> Thanks!
>
> -Holger
>
> ________________________________________
> Von: Yoann CONGAL <yoann.congal@smile.fr>
> Gesendet: Donnerstag, 14. September 2023 23:52
> An: Sebert, Holger.ext
> Cc: Zoran Stojsavljevic; Yocto-mailing-list
> Betreff: [Ext] Re: [yocto] Poky/Mickledore bitbake menuconfig does not work
>
> Hi,
>
> Le jeu. 14 sept. 2023 à 14:10, Sebert, Holger.ext <
> holger.sebert.ext@karlstorz.com<mailto:holger.sebert.ext@karlstorz.com>>
> a écrit :
> Hi Zoran,
>
> I was able to reproduce the behavior described in my original post by
> enabling CCache. Here are the precise steps to reproduce:
>
> 1. Checkout Poky/Mickledore
> 2. Add the following to `build/conf/local.conf`:
>     INHERIT += "ccache"
> 3. Run `bitbake -c menuconfig virtual/kernel`
>
> The reason for the failure is that due to CCache the compiler is no longer
> `gcc` but `ccache gcc`, i.e. we have the following variables in the
> environment:
>
>     CC="ccache x86_64-poky-linux-gcc"
>     HOSTCC="ccache gcc"
>     ...
>     etc.
>
> This is when the wrong quoting in the invocation of `menuconfig` leads to
> errors.
>
> Could you please verify/check?
>
>
> I have reproduced this. It really looks like a bug!
>
> [...]
> On Mon, Aug 21, 2023 at 3:48 PM Sebert, Holger.ext
> <holger.sebert.ext@karlstorz.com<mailto:holger.sebert.ext@karlstorz.com>>
> wrote:
> >
> > Hi,
> >
> > I am using a Yocto-Poky-Mickledore (version 4.2.1) based system.
> >
> > For configuring the kernel, I would like to use the command
> >
> >     bitbake -c menuconfig virtual/kernel
> >
> > If I do that, Bitbake opens an xterm window and runs the build steps
> > for Kconfig. However, after the compiler has finised, the xterm window
> > immediately closes and Bitbake exists as if nothing happened, i.e. it
> > does not show any error message or similar.
> >
> > I investigated the problem and found out that, internally, Bitbake
> > calls the following command for starting Kconfig:
> >
> >     make menuconfig CC="ccache x86_64-system-linux-gcc ..." \
> >         CFLAGS="..." HOSTCC="ccache gcc" HOSTCCFLAGS="..."
> >
> > The quotes, however, seem to get lost when starting the actual command
> > in the terminal. This leads to a command line like follows:
> >
> >     make menuconfig CC=ccache x86_64-system-linux-gcc ... \
> >         CFLAGS=... HOSTCC="ccache gcc" HOSTCCFLAGS="...
> >
> > which is clearly wrong.
> >
> > The following patch fixes the problem in
> meta/classes-recipe/cml1.bbclass:
> >
> > diff --git a/meta/classes-recipe/cml1.bbclass
> b/meta/classes-recipe/cml1.bbclass
> > index a09a042c3f..d6001d6fd0 100644
> > --- a/meta/classes-recipe/cml1.bbclass
> > +++ b/meta/classes-recipe/cml1.bbclass
> > @@ -53,7 +53,8 @@ python do_menuconfig() {
> >      # ensure that environment variables are overwritten with this tasks
> 'd' values
> >      d.appendVar("OE_TERMINAL_EXPORTS", " PKG_CONFIG_DIR PKG_CONFIG_PATH
> PKG_CONFIG_LIBDIR PKG_CONFIG_SYSROOT_DIR")
> >
> > -    oe_terminal("sh -c \"make %s; if [ \\$? -ne 0 ]; then echo 'Command
> failed.'; printf 'Press any key to continue... '; read r; fi\"" %
> d.getVar('KCONFIG_CONFIG_COMMAND'),
> > +    make_cmd = "make %s" %
> d.getVar('KCONFIG_CONFIG_COMMAND').replace('"', '\\"')
> > +    oe_terminal("sh -c \"%s; if [ \\$? -ne 0 ]; then echo 'Command
> failed.'; printf 'Press any key to continue... '; read r; fi\"" % make_cmd,
> >                  d.getVar('PN') + ' Configuration', d)
>
> This patch looks interesting!
> I would use shlex.join() to safely construct the command :
> https://docs.python.org/3/library/shlex.html#shlex.join<
> https://urldefense.com/v3/__https://docs.python.org/3/library/shlex.html*shlex.join__;Iw!!Lw1uGqvHvtJ_psGG2Et-1voSwbo!fQdtwGScMriVPQw2RO0unCFPxKM9NGX19bIH6mNdVfb77dZxiYZEkdNFiD5PZzVoxaOccilrT4L2I56P_pSw6FJuljKxeNTW$
> >
>
> > What do you think? Is my observed behavior a configuration error of my
> > setup or does the patch above indeed make sense?
>
> The patch makes sense to me.
>
> I guess your next steps are :
> * If you want to contribute your patch : reproduce the bug and rebase your
> patch on master and send it to the mailing list
> * Or, create a bug in https://bugzilla.yoctoproject.org/<
> https://urldefense.com/v3/__https://bugzilla.yoctoproject.org/__;!!Lw1uGqvHvtJ_psGG2Et-1voSwbo!fQdtwGScMriVPQw2RO0unCFPxKM9NGX19bIH6mNdVfb77dZxiYZEkdNFiD5PZzVoxaOccilrT4L2I56P_pSw6FJulqw2-PFV$
> >
>
> Best Regards,
>
> --
> Yoann Congal
> Smile ECS - Tech expert
>
Yoann Congal Sept. 30, 2023, 2:12 p.m. UTC | #5 
Le mer. 20 sept. 2023 à 19:59, Yoann CONGAL <yoann.congal@smile.fr> a
écrit :

> Le mar. 19 sept. 2023 à 18:57, Sebert, Holger.ext <
> Holger.Sebert.ext@karlstorz.com> a écrit :
>
>> Hi Yoann, hi Zoran,
>>
>
> Hi,
>
>
>> The bug is not present on the Poky master branch.
>>
>> Until the project is rebased onto the next release, I will simply disable
>> CCache whenever I need to do a kernel menuconfig.
>>
>
> You peaked my curiosity and I've tracked down the fix on master : Your
> problem is fixed by
>
> https://git.yoctoproject.org/poky/commit/?id=d4664d2b7974354e73d891762ebb2c8a12d62438
>
> I've requested a backport on mickledore :
> https://lists.openembedded.org/g/openembedded-core/message/187934
>

Fix has now been merged in mickledore :
https://git.yoctoproject.org/poky/commit/?h=mickledore&id=cd0c6173a375e5c7b617985ca2efcf167c92c827

You would just have to update your poky mickledore branch to fix your
problem :)

Enjoy!

Regards,
>
>
>> Thanks!
>>
>> -Holger
>>
>> ________________________________________
>> Von: Yoann CONGAL <yoann.congal@smile.fr>
>> Gesendet: Donnerstag, 14. September 2023 23:52
>> An: Sebert, Holger.ext
>> Cc: Zoran Stojsavljevic; Yocto-mailing-list
>> Betreff: [Ext] Re: [yocto] Poky/Mickledore bitbake menuconfig does not
>> work
>>
>> Hi,
>>
>> Le jeu. 14 sept. 2023 à 14:10, Sebert, Holger.ext <
>> holger.sebert.ext@karlstorz.com<mailto:holger.sebert.ext@karlstorz.com>>
>> a écrit :
>> Hi Zoran,
>>
>> I was able to reproduce the behavior described in my original post by
>> enabling CCache. Here are the precise steps to reproduce:
>>
>> 1. Checkout Poky/Mickledore
>> 2. Add the following to `build/conf/local.conf`:
>>     INHERIT += "ccache"
>> 3. Run `bitbake -c menuconfig virtual/kernel`
>>
>> The reason for the failure is that due to CCache the compiler is no
>> longer `gcc` but `ccache gcc`, i.e. we have the following variables in the
>> environment:
>>
>>     CC="ccache x86_64-poky-linux-gcc"
>>     HOSTCC="ccache gcc"
>>     ...
>>     etc.
>>
>> This is when the wrong quoting in the invocation of `menuconfig` leads to
>> errors.
>>
>> Could you please verify/check?
>>
>>
>> I have reproduced this. It really looks like a bug!
>>
>> [...]
>> On Mon, Aug 21, 2023 at 3:48 PM Sebert, Holger.ext
>> <holger.sebert.ext@karlstorz.com<mailto:holger.sebert.ext@karlstorz.com>>
>> wrote:
>> >
>> > Hi,
>> >
>> > I am using a Yocto-Poky-Mickledore (version 4.2.1) based system.
>> >
>> > For configuring the kernel, I would like to use the command
>> >
>> >     bitbake -c menuconfig virtual/kernel
>> >
>> > If I do that, Bitbake opens an xterm window and runs the build steps
>> > for Kconfig. However, after the compiler has finised, the xterm window
>> > immediately closes and Bitbake exists as if nothing happened, i.e. it
>> > does not show any error message or similar.
>> >
>> > I investigated the problem and found out that, internally, Bitbake
>> > calls the following command for starting Kconfig:
>> >
>> >     make menuconfig CC="ccache x86_64-system-linux-gcc ..." \
>> >         CFLAGS="..." HOSTCC="ccache gcc" HOSTCCFLAGS="..."
>> >
>> > The quotes, however, seem to get lost when starting the actual command
>> > in the terminal. This leads to a command line like follows:
>> >
>> >     make menuconfig CC=ccache x86_64-system-linux-gcc ... \
>> >         CFLAGS=... HOSTCC="ccache gcc" HOSTCCFLAGS="...
>> >
>> > which is clearly wrong.
>> >
>> > The following patch fixes the problem in
>> meta/classes-recipe/cml1.bbclass:
>> >
>> > diff --git a/meta/classes-recipe/cml1.bbclass
>> b/meta/classes-recipe/cml1.bbclass
>> > index a09a042c3f..d6001d6fd0 100644
>> > --- a/meta/classes-recipe/cml1.bbclass
>> > +++ b/meta/classes-recipe/cml1.bbclass
>> > @@ -53,7 +53,8 @@ python do_menuconfig() {
>> >      # ensure that environment variables are overwritten with this
>> tasks 'd' values
>> >      d.appendVar("OE_TERMINAL_EXPORTS", " PKG_CONFIG_DIR
>> PKG_CONFIG_PATH PKG_CONFIG_LIBDIR PKG_CONFIG_SYSROOT_DIR")
>> >
>> > -    oe_terminal("sh -c \"make %s; if [ \\$? -ne 0 ]; then echo
>> 'Command failed.'; printf 'Press any key to continue... '; read r; fi\"" %
>> d.getVar('KCONFIG_CONFIG_COMMAND'),
>> > +    make_cmd = "make %s" %
>> d.getVar('KCONFIG_CONFIG_COMMAND').replace('"', '\\"')
>> > +    oe_terminal("sh -c \"%s; if [ \\$? -ne 0 ]; then echo 'Command
>> failed.'; printf 'Press any key to continue... '; read r; fi\"" % make_cmd,
>> >                  d.getVar('PN') + ' Configuration', d)
>>
>> This patch looks interesting!
>> I would use shlex.join() to safely construct the command :
>> https://docs.python.org/3/library/shlex.html#shlex.join<
>> https://urldefense.com/v3/__https://docs.python.org/3/library/shlex.html*shlex.join__;Iw!!Lw1uGqvHvtJ_psGG2Et-1voSwbo!fQdtwGScMriVPQw2RO0unCFPxKM9NGX19bIH6mNdVfb77dZxiYZEkdNFiD5PZzVoxaOccilrT4L2I56P_pSw6FJuljKxeNTW$
>> >
>>
>> > What do you think? Is my observed behavior a configuration error of my
>> > setup or does the patch above indeed make sense?
>>
>> The patch makes sense to me.
>>
>> I guess your next steps are :
>> * If you want to contribute your patch : reproduce the bug and rebase
>> your patch on master and send it to the mailing list
>> * Or, create a bug in https://bugzilla.yoctoproject.org/<
>> https://urldefense.com/v3/__https://bugzilla.yoctoproject.org/__;!!Lw1uGqvHvtJ_psGG2Et-1voSwbo!fQdtwGScMriVPQw2RO0unCFPxKM9NGX19bIH6mNdVfb77dZxiYZEkdNFiD5PZzVoxaOccilrT4L2I56P_pSw6FJulqw2-PFV$
>> >
>>
>> Best Regards,
>>
>> --
>> Yoann Congal
>> Smile ECS - Tech expert
>>
>
>
> --
> Yoann Congal
> Smile ECS - Tech expert
>
diff mbox series

Patch

diff --git a/meta/classes-recipe/cml1.bbclass b/meta/classes-recipe/cml1.bbclass
index a09a042c3f..d6001d6fd0 100644
--- a/meta/classes-recipe/cml1.bbclass
+++ b/meta/classes-recipe/cml1.bbclass
@@ -53,7 +53,8 @@  python do_menuconfig() {
     # ensure that environment variables are overwritten with this tasks 'd' values
     d.appendVar("OE_TERMINAL_EXPORTS", " PKG_CONFIG_DIR PKG_CONFIG_PATH PKG_CONFIG_LIBDIR PKG_CONFIG_SYSROOT_DIR")

-    oe_terminal("sh -c \"make %s; if [ \\$? -ne 0 ]; then echo 'Command failed.'; printf 'Press any key to continue... '; read r; fi\"" % d.getVar('KCONFIG_CONFIG_COMMAND'),
+    make_cmd = "make %s" % d.getVar('KCONFIG_CONFIG_COMMAND').replace('"', '\\"')
+    oe_terminal("sh -c \"%s; if [ \\$? -ne 0 ]; then echo 'Command failed.'; printf 'Press any key to continue... '; read r; fi\"" % make_cmd,
                 d.getVar('PN') + ' Configuration', d)

What do you think? Is my observed behavior a configuration error of my