diff mbox series

python3: Remove DEBUG_PREFIX_MAP from sysconfigdata

Message ID GV0P278MB0516B168ACC6F3595C48BFE88BC52@GV0P278MB0516.CHEP278.PROD.OUTLOOK.COM
State New
Headers show
Series python3: Remove DEBUG_PREFIX_MAP from sysconfigdata | expand

Commit Message

Sanjuán García, Jorge Feb. 19, 2025, 8:24 a.m. UTC 
Paths in DEBUG_PREFIX_MAP are not being removed from sysconfigdata
files correctly as they are split into line separated values. Update
the sed command so that it uses any matching patern from
DEBUG_PREFIX_MAP instead an exact match of the full variable.

Signed-off-by: Jorge Sanjuan Garcia <jorge.sanjuangarcia@duagon.com>
---
 meta/recipes-devtools/python/python3_3.13.2.bb | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Comments

Alexander Kanavin Feb. 19, 2025, 1:28 p.m. UTC | #1 
We're not seeing the problem on the autobuilder (which has checks for
host paths leaking into target installations). And I'm not seeing it
in my local build.

Can you show an example of this happening? So that it's clear what
isn't being correctly removed without this fix.

Alex


On Wed, 19 Feb 2025 at 14:03, Sanjuán García, Jorge via
lists.openembedded.org
<Jorge.SanjuanGarcia=duagon.com@lists.openembedded.org> wrote:
>
> Paths in DEBUG_PREFIX_MAP are not being removed from sysconfigdata
> files correctly as they are split into line separated values. Update
> the sed command so that it uses any matching patern from
> DEBUG_PREFIX_MAP instead an exact match of the full variable.
>
> Signed-off-by: Jorge Sanjuan Garcia <jorge.sanjuangarcia@duagon.com>
> ---
>  meta/recipes-devtools/python/python3_3.13.2.bb | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/meta/recipes-devtools/python/python3_3.13.2.bb b/meta/recipes-devtools/python/python3_3.13.2.bb
> index 52fac76c00..656a73f501 100644
> --- a/meta/recipes-devtools/python/python3_3.13.2.bb
> +++ b/meta/recipes-devtools/python/python3_3.13.2.bb
> @@ -291,10 +291,12 @@ SSTATE_HASHEQUIV_FILEMAP = " \
>      "
>  PACKAGE_PREPROCESS_FUNCS += "py_package_preprocess"
>
> +DEBUG_PREFIX_MAP_REGEX = "${@'\\|'.join(d.getVar('DEBUG_PREFIX_MAP').split())}"
> +
>  py_package_preprocess () {
>          # Remove references to buildmachine paths in target Makefile and _sysconfigdata
>          sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g \
> -                -e 's|${DEBUG_PREFIX_MAP}||g' \
> +                -e 's:${DEBUG_PREFIX_MAP_REGEX}::g' \
>                  -e 's:${HOSTTOOLS_DIR}/::g' \
>                  -e 's:${RECIPE_SYSROOT_NATIVE}::g' \
>                  -e 's:${RECIPE_SYSROOT}::g' \
> --
> 2.43.0
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#211707): https://lists.openembedded.org/g/openembedded-core/message/211707
> Mute This Topic: https://lists.openembedded.org/mt/111268642/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Sanjuán García, Jorge Feb. 19, 2025, 3:47 p.m. UTC | #2 
Hello,

Thanks for the quick reply. Let me provide some details of my findings:

So the original DEBUG_PREFIX_MAP variable is a list of space separated values:

```
builder@bc2576acfe0e:/build/tmp/work/core2-64-oe-linux/python3/3.13.2$ bitbake-getvar DEBUG_PREFIX_MAP -r python3 
#
# $DEBUG_PREFIX_MAP
#   set? /build/../work/openembedded-core/meta/conf/bitbake.conf:666
#     "-fcanon-prefix-map  -fmacro-prefix-map=${S}=${TARGET_DBGSRC_DIR}  -fdebug-prefix-map=${S}=${TARGET_DBGSRC_DIR}  -fmacro-prefix-map=${B}=${TARGET_DBGSRC_DIR}  -fdebug-prefix-map=${B}=${TARGET_DBGSRC_DIR}  -fdebug-prefix-map=${STAGING_DIR_HOST}=  -fmacro-prefix-map=${STAGING_DIR_HOST}=  -fdebug-prefix-map=${STAGING_DIR_NATIVE}=  -fmacro-prefix-map=${STAGING_DIR_NATIVE}= "
DEBUG_PREFIX_MAP="-fcanon-prefix-map  -fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  -fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  -fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/build=/usr/src/debug/python3/3.13.2  -fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/build=/usr/src/debug/python3/3.13.2  -fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot=  -fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot=  -fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot-native=  -fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot-native= "
```

Before replacing the buildpaths with `py_package_preprocess` we get this as line separated entries:

```
builder@bc2576acfe0e:/build/tmp/work/core2-64-oe-linux/python3/3.13.2$ grep 'prefix-map' ./image/usr/lib/python-sysconfigdata/_sysconfigdata.py 
              '-fcanon-prefix-map  '
              '-fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  '
              '-fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  '
              '-fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2  '
...
```

Some paths of the original contents of DEBUG_PREFIX_MAP are updated. However, what is really being applied is the sed expression `-e 's:${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}::g'` . The actual values from DEBUG_PREFIX_MAP are still there. That's why the host paths leaking checks are not failing. The original code is trying to fully replace the full DEBUG_PREFIX_MAP variable:

```
builder@bc2576acfe0e:/build/tmp/work/core2-64-oe-linux/python3/3.13.2$ grep 'prefix-map' ./package/usr/lib/python3.13/_sysconfigdata__linux_x86_64-linux-gnu.py
              '-fcanon-prefix-map  '
              '-fmacro-prefix-map=/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  '
              '-fdebug-prefix-map=/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  '
              '-fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2  '
...
```

The reason I found this is beacause I was trying to do something that required building python extensions on a host were the compilation flag `-fcanon-prefix-map` was invalid as that was introduced with gcc13. However, looking a bit deeper I realized that the sed expression is actually not doing what it is meant to when removing DEBUG_PREFIX_MAP.

Regards,
Jorge
Alexander Kanavin Feb. 19, 2025, 4:01 p.m. UTC | #3 
On Wed, 19 Feb 2025 at 16:47, Sanjuán García, Jorge
<Jorge.SanjuanGarcia@duagon.com> wrote:
> The reason I found this is beacause I was trying to do something that required building python extensions on a host were the compilation flag `-fcanon-prefix-map` was invalid as that was introduced with gcc13. However, looking a bit deeper I realized that the sed expression is actually not doing what it is meant to when removing DEBUG_PREFIX_MAP.
>

Apologies, but I don't fully understand what you say. You show how the
entry in sysconfigdata looks before and after without your patch, but
I am not seeing how it looks before and after *with* the patch.
Without it, it's really hard to understand what the problem is, and
how it is being fixed.

Alex
Sanjuán García, Jorge Feb. 20, 2025, 6:59 a.m. UTC | #4 
So with this patch applied we would get this CFLAGS on sysconfigdata on the build directory:

 'CFLAGS': '-fno-strict-overflow '
           '-Wsign-compare '
           '-DNDEBUG '
           '-g '
           '-O3 '
           '-Wall '
           '-O2 '
           '-g '
           '-fcanon-prefix-map  '
           '-fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  '
           '-fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  '
           '-fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2  '
           '-fdebug-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2  '
           '-fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot=  '
           '-fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot=  '
           '-fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot-native=  '
           '-fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot-native=  '
           '-pipe  '
           '-O2 '
           '-g '
           '-fcanon-prefix-map  '
           '-fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  '
           '-fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/Python-3.13.2=/usr/src/debug/python3/3.13.2  '
           '-fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2  '
           '-fdebug-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2  '
           '-fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot=  '
           '-fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot=  '
           '-fdebug-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot-native=  '
           '-fmacro-prefix-map=/build/tmp/work/core2-64-oe-linux/python3/3.13.2/recipe-sysroot-native=  '
           '-pipe',

After replacing buildpaths and DEBUG_PREFIX_MAP on the package directory with `py_package_preprocess()`:

'CFLAGS': '-fno-strict-overflow '
           '-Wsign-compare '
           '-DNDEBUG '
           '-g '
           '-O3 '
           '-Wall '
           '-O2 '
           '-g       '
           '-fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2  '
           '-fdebug-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2          '
           '-pipe  '
           '-O2 '
           '-g       '
           '-fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2  '
           '-fdebug-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.13.2          '
           '-pipe',

There are still some 'prefix-map' stuff that is not properly being replaced because the do_install() task is
doing this sed expression `-e "s,${B},/build/path/unavailable/,g"`. We could probably move that to
`py_package_preprocess()` and remove DEBUG_PREFIX_MAP entries altogether. I can put a v2 doing
that if you agree with that.
Alexander Kanavin Feb. 20, 2025, 9:24 a.m. UTC | #5 
On Thu, 20 Feb 2025 at 08:00, Sanjuán García, Jorge
<Jorge.SanjuanGarcia@duagon.com> wrote:

> There are still some 'prefix-map' stuff that is not properly being replaced because the do_install() task is
> doing this sed expression `-e "s,${B},/build/path/unavailable/,g"`. We could probably move that to
> `py_package_preprocess()` and remove DEBUG_PREFIX_MAP entries altogether. I can put a v2 doing
> that if you agree with that.

I don't have a strong opinion here. The point is to more clearly
document in the commit message what is being fixed, e.g. by showing
how the final output looks before and after the patch.

Alex
diff mbox series

Patch

diff --git a/meta/recipes-devtools/python/python3_3.13.2.bb b/meta/recipes-devtools/python/python3_3.13.2.bb
index 52fac76c00..656a73f501 100644
--- a/meta/recipes-devtools/python/python3_3.13.2.bb
+++ b/meta/recipes-devtools/python/python3_3.13.2.bb
@@ -291,10 +291,12 @@  SSTATE_HASHEQUIV_FILEMAP = " \
     "
 PACKAGE_PREPROCESS_FUNCS += "py_package_preprocess"
 
+DEBUG_PREFIX_MAP_REGEX = "${@'\\|'.join(d.getVar('DEBUG_PREFIX_MAP').split())}"
+
 py_package_preprocess () {
         # Remove references to buildmachine paths in target Makefile and _sysconfigdata
         sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g \
-                -e 's|${DEBUG_PREFIX_MAP}||g' \
+                -e 's:${DEBUG_PREFIX_MAP_REGEX}::g' \
                 -e 's:${HOSTTOOLS_DIR}/::g' \
                 -e 's:${RECIPE_SYSROOT_NATIVE}::g' \
                 -e 's:${RECIPE_SYSROOT}::g' \