| Message ID | 20240603162525.2940419-1-raj.khem@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | python3-rpds-py: Create a symlink for cython module on musl | expand |
On Mon, 2024-06-03 at 09:25 -0700, Khem Raj via lists.openembedded.org wrote: > loader expects it to be called with 'linux-gnu' e.g > rpds.cpython-312-x86_64-linux-musl.so is created with musl > but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so > > Lets create the symlink to please the loader. > > Fixes > ImportError while importing test module '/usr/lib/python3-rpds- > py/ptest/tests/test_hash_trie_map.py'. > Hint: make sure your test modules/packages have valid Python names. > Traceback: > ../../python3.12/importlib/__init__.py:90: in import_module > return _bootstrap._gcd_import(name[level:], package, level) > tests/test_hash_trie_map.py:36: in <module> > from rpds import HashTrieMap > ../../python3.12/site-packages/rpds/__init__.py:1: in <module> > from .rpds import * > E ModuleNotFoundError: No module named 'rpds.rpds' > ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py > > Signed-off-by: Khem Raj <raj.khem@gmail.com> > --- > .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11 > +++++++++++ > 1 file changed, 11 insertions(+) > > diff --git a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > index f46df1115c8..b1f0e100332 100644 > --- a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > +++ b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > @@ -27,4 +27,15 @@ do_install_ptest() { > cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/ > } > > +do_install:append() { > + for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so > + do > + fname=`basename $f` > + lname=`echo $fname | sed 's/musl/gnu/'` > + if [ "$fname" != "$lname" ]; then > + mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname > + fi > + done > +} > + > BBCLASSEXTEND = "native nativesdk" > This feels like the kind of workaround that will live forever. Can we not fix the loader to use the right name? Also, there is not even a comment above saying why we need to do this. Cheers, Richard
On Mon, Jun 3, 2024 at 9:35 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Mon, 2024-06-03 at 09:25 -0700, Khem Raj via lists.openembedded.org > wrote: > > loader expects it to be called with 'linux-gnu' e.g > > rpds.cpython-312-x86_64-linux-musl.so is created with musl > > but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so > > > > Lets create the symlink to please the loader. > > > > Fixes > > ImportError while importing test module '/usr/lib/python3-rpds- > > py/ptest/tests/test_hash_trie_map.py'. > > Hint: make sure your test modules/packages have valid Python names. > > Traceback: > > ../../python3.12/importlib/__init__.py:90: in import_module > > return _bootstrap._gcd_import(name[level:], package, level) > > tests/test_hash_trie_map.py:36: in <module> > > from rpds import HashTrieMap > > ../../python3.12/site-packages/rpds/__init__.py:1: in <module> > > from .rpds import * > > E ModuleNotFoundError: No module named 'rpds.rpds' > > ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py > > > > Signed-off-by: Khem Raj <raj.khem@gmail.com> > > --- > > .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11 > > +++++++++++ > > 1 file changed, 11 insertions(+) > > > > diff --git a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > index f46df1115c8..b1f0e100332 100644 > > --- a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > +++ b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > @@ -27,4 +27,15 @@ do_install_ptest() { > > cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/ > > } > > > > +do_install:append() { > > + for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so > > + do > > + fname=`basename $f` > > + lname=`echo $fname | sed 's/musl/gnu/'` > > + if [ "$fname" != "$lname" ]; then > > + mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname > > + fi > > + done > > +} > > + > > BBCLASSEXTEND = "native nativesdk" > > > > This feels like the kind of workaround that will live forever. Can we > not fix the loader to use the right name? OE's python interpreter is cross-built and we end up using OSABI which is detected during build 'linux-gnu', this seems to work fine for most of python modules since they get the OSABI from the sysconfigdata so it all works even though the OSABI is called linux-gnu, however some modules like this one which are using maturin to build, seems to get the OSABI equivalent differently, I am no expert on maturin but maybe someone knows how it happens. This problem is seen with several python modules built with OE, see https://github.com/meta-homeassistant/meta-homeassistant/issues/89 https://github.com/pypa/auditwheel/issues/349 also see how it is fixed in python3-pydantic-core on meta-python https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38 Hopefully musllinux ABI tag will be sorted in pypa and this will work better however, we also need to see how we are encoding OSABI in python builds e.g. on qemux86-64 musl target I get root@qemux86-64:~# python3 Python 3.12.3 (main, Apr 9 2024, 08:09:14) [Clang 19.0.0 (/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sysconfig >>> sysconfig.get_config_var('SOABI') 'cpython-312-x86_64-linux-gnu' > > Also, there is not even a comment above saying why we need to do this. > I added the log showing how it does not work, I thought that is showing how it's not working at all when using musl. If you have better idea what I should add to commit msg to make it better, let me know I will send a v2. > Cheers, > > Richard
I agree with RP, please get to the source of the mismatch. It’s a really unpleasant workaround. Alex On Mon 3. Jun 2024 at 20.49, Khem Raj via lists.openembedded.org <raj.khem= gmail.com@lists.openembedded.org> wrote: > On Mon, Jun 3, 2024 at 9:35 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Mon, 2024-06-03 at 09:25 -0700, Khem Raj via lists.openembedded.org > > wrote: > > > loader expects it to be called with 'linux-gnu' e.g > > > rpds.cpython-312-x86_64-linux-musl.so is created with musl > > > but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so > > > > > > Lets create the symlink to please the loader. > > > > > > Fixes > > > ImportError while importing test module '/usr/lib/python3-rpds- > > > py/ptest/tests/test_hash_trie_map.py'. > > > Hint: make sure your test modules/packages have valid Python names. > > > Traceback: > > > ../../python3.12/importlib/__init__.py:90: in import_module > > > return _bootstrap._gcd_import(name[level:], package, level) > > > tests/test_hash_trie_map.py:36: in <module> > > > from rpds import HashTrieMap > > > ../../python3.12/site-packages/rpds/__init__.py:1: in <module> > > > from .rpds import * > > > E ModuleNotFoundError: No module named 'rpds.rpds' > > > ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py > > > > > > Signed-off-by: Khem Raj <raj.khem@gmail.com> > > > --- > > > .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11 > > > +++++++++++ > > > 1 file changed, 11 insertions(+) > > > > > > diff --git a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > index f46df1115c8..b1f0e100332 100644 > > > --- a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > +++ b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > @@ -27,4 +27,15 @@ do_install_ptest() { > > > cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/ > > > } > > > > > > +do_install:append() { > > > + for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so > > > + do > > > + fname=`basename $f` > > > + lname=`echo $fname | sed 's/musl/gnu/'` > > > + if [ "$fname" != "$lname" ]; then > > > + mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname > > > + fi > > > + done > > > +} > > > + > > > BBCLASSEXTEND = "native nativesdk" > > > > > > > This feels like the kind of workaround that will live forever. Can we > > not fix the loader to use the right name? > > OE's python interpreter is cross-built and we end up using OSABI which > is detected during build 'linux-gnu', this seems to work fine for most of > python modules since they get the OSABI from the sysconfigdata so it > all works even though the OSABI is called linux-gnu, however some modules > like this one which are using maturin to build, seems to get the OSABI > equivalent differently, I am no expert on maturin but maybe someone knows > how it happens. This problem is seen with several python modules built with > OE, see > > https://github.com/meta-homeassistant/meta-homeassistant/issues/89 > https://github.com/pypa/auditwheel/issues/349 > > also see how it is fixed in python3-pydantic-core on meta-python > > https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38 > > Hopefully musllinux ABI tag will be sorted in pypa and this will work > better > however, we also need to see how we are encoding OSABI in python builds > > e.g. on qemux86-64 musl target I get > > root@qemux86-64:~# python3 > Python 3.12.3 (main, Apr 9 2024, 08:09:14) [Clang 19.0.0 > (/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on > linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import sysconfig > >>> sysconfig.get_config_var('SOABI') > 'cpython-312-x86_64-linux-gnu' > > > > > Also, there is not even a comment above saying why we need to do this. > > > > I added the log showing how it does not work, I thought that is showing how > it's not working at all when using musl. If you have better idea what I > should > add to commit msg to make it better, let me know I will send a v2. > > > Cheers, > > > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#200275): > https://lists.openembedded.org/g/openembedded-core/message/200275 > Mute This Topic: https://lists.openembedded.org/mt/106465307/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Mon, Jun 3, 2024 at 10:49 AM Khem Raj <raj.khem@gmail.com> wrote: > > On Mon, Jun 3, 2024 at 9:35 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Mon, 2024-06-03 at 09:25 -0700, Khem Raj via lists.openembedded.org > > wrote: > > > loader expects it to be called with 'linux-gnu' e.g > > > rpds.cpython-312-x86_64-linux-musl.so is created with musl > > > but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so > > > > > > Lets create the symlink to please the loader. > > > > > > Fixes > > > ImportError while importing test module '/usr/lib/python3-rpds- > > > py/ptest/tests/test_hash_trie_map.py'. > > > Hint: make sure your test modules/packages have valid Python names. > > > Traceback: > > > ../../python3.12/importlib/__init__.py:90: in import_module > > > return _bootstrap._gcd_import(name[level:], package, level) > > > tests/test_hash_trie_map.py:36: in <module> > > > from rpds import HashTrieMap > > > ../../python3.12/site-packages/rpds/__init__.py:1: in <module> > > > from .rpds import * > > > E ModuleNotFoundError: No module named 'rpds.rpds' > > > ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py > > > > > > Signed-off-by: Khem Raj <raj.khem@gmail.com> > > > --- > > > .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11 > > > +++++++++++ > > > 1 file changed, 11 insertions(+) > > > > > > diff --git a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > index f46df1115c8..b1f0e100332 100644 > > > --- a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > +++ b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > @@ -27,4 +27,15 @@ do_install_ptest() { > > > cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/ > > > } > > > > > > +do_install:append() { > > > + for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so > > > + do > > > + fname=`basename $f` > > > + lname=`echo $fname | sed 's/musl/gnu/'` > > > + if [ "$fname" != "$lname" ]; then > > > + mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname > > > + fi > > > + done > > > +} > > > + > > > BBCLASSEXTEND = "native nativesdk" > > > > > > > This feels like the kind of workaround that will live forever. Can we > > not fix the loader to use the right name? > > OE's python interpreter is cross-built and we end up using OSABI which > is detected during build 'linux-gnu', this seems to work fine for most of > python modules since they get the OSABI from the sysconfigdata so it > all works even though the OSABI is called linux-gnu, however some modules > like this one which are using maturin to build, seems to get the OSABI > equivalent differently, I am no expert on maturin but maybe someone knows > how it happens. This problem is seen with several python modules built with > OE, see > > https://github.com/meta-homeassistant/meta-homeassistant/issues/89 > https://github.com/pypa/auditwheel/issues/349 > > also see how it is fixed in python3-pydantic-core on meta-python > https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38 > > Hopefully musllinux ABI tag will be sorted in pypa and this will work better > however, we also need to see how we are encoding OSABI in python builds > > e.g. on qemux86-64 musl target I get > > root@qemux86-64:~# python3 > Python 3.12.3 (main, Apr 9 2024, 08:09:14) [Clang 19.0.0 > (/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on > linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import sysconfig > >>> sysconfig.get_config_var('SOABI') > 'cpython-312-x86_64-linux-gnu' > I looked further in pyo and maturin infrastructure and it is using ustc --target=<tuple> --print cfg to compute EXT_SUFFIX equivalent value which is then used to build the C extension (if any) inside the wheel, thats why we get cpython-312-x86_64-linux-musl for suffix, what pyo/maturin is doing is right, however how OE does it in python itself seems to be wrong to me, its synthesizing the OSABI incorrectly. where 'environment' value is not right. However, this is same everywhere so it works, so long as the modules are built using setuptools. since SOABI is part of "_sysconfigdata" and its an OE generated file, Perhaps it can consider the musl/gnu difference and emit SOABI accordingly. Something like below https://snips.sh/f/L6QspcdxjL might be able to fix the python interpreter's logic. > > > > Also, there is not even a comment above saying why we need to do this. > > > > I added the log showing how it does not work, I thought that is showing how > it's not working at all when using musl. If you have better idea what I should > add to commit msg to make it better, let me know I will send a v2. > > > Cheers, > > > > Richard
On Mon, Jun 3, 2024 at 9:56 PM Khem Raj <raj.khem@gmail.com> wrote: > > On Mon, Jun 3, 2024 at 10:49 AM Khem Raj <raj.khem@gmail.com> wrote: > > > > On Mon, Jun 3, 2024 at 9:35 AM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > On Mon, 2024-06-03 at 09:25 -0700, Khem Raj via lists.openembedded.org > > > wrote: > > > > loader expects it to be called with 'linux-gnu' e.g > > > > rpds.cpython-312-x86_64-linux-musl.so is created with musl > > > > but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so > > > > > > > > Lets create the symlink to please the loader. > > > > > > > > Fixes > > > > ImportError while importing test module '/usr/lib/python3-rpds- > > > > py/ptest/tests/test_hash_trie_map.py'. > > > > Hint: make sure your test modules/packages have valid Python names. > > > > Traceback: > > > > ../../python3.12/importlib/__init__.py:90: in import_module > > > > return _bootstrap._gcd_import(name[level:], package, level) > > > > tests/test_hash_trie_map.py:36: in <module> > > > > from rpds import HashTrieMap > > > > ../../python3.12/site-packages/rpds/__init__.py:1: in <module> > > > > from .rpds import * > > > > E ModuleNotFoundError: No module named 'rpds.rpds' > > > > ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py > > > > > > > > Signed-off-by: Khem Raj <raj.khem@gmail.com> > > > > --- > > > > .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11 > > > > +++++++++++ > > > > 1 file changed, 11 insertions(+) > > > > > > > > diff --git a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > > b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > > index f46df1115c8..b1f0e100332 100644 > > > > --- a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > > +++ b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > > @@ -27,4 +27,15 @@ do_install_ptest() { > > > > cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/ > > > > } > > > > > > > > +do_install:append() { > > > > + for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so > > > > + do > > > > + fname=`basename $f` > > > > + lname=`echo $fname | sed 's/musl/gnu/'` > > > > + if [ "$fname" != "$lname" ]; then > > > > + mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname > > > > + fi > > > > + done > > > > +} > > > > + > > > > BBCLASSEXTEND = "native nativesdk" > > > > > > > > > > This feels like the kind of workaround that will live forever. Can we > > > not fix the loader to use the right name? > > > > OE's python interpreter is cross-built and we end up using OSABI which > > is detected during build 'linux-gnu', this seems to work fine for most of > > python modules since they get the OSABI from the sysconfigdata so it > > all works even though the OSABI is called linux-gnu, however some modules > > like this one which are using maturin to build, seems to get the OSABI > > equivalent differently, I am no expert on maturin but maybe someone knows > > how it happens. This problem is seen with several python modules built with > > OE, see > > > > https://github.com/meta-homeassistant/meta-homeassistant/issues/89 > > https://github.com/pypa/auditwheel/issues/349 > > > > also see how it is fixed in python3-pydantic-core on meta-python > > https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38 > > > > Hopefully musllinux ABI tag will be sorted in pypa and this will work better > > however, we also need to see how we are encoding OSABI in python builds > > > > e.g. on qemux86-64 musl target I get > > > > root@qemux86-64:~# python3 > > Python 3.12.3 (main, Apr 9 2024, 08:09:14) [Clang 19.0.0 > > (/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on > > linux > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import sysconfig > > >>> sysconfig.get_config_var('SOABI') > > 'cpython-312-x86_64-linux-gnu' > > > > I looked further in pyo and maturin infrastructure and it is using > ustc --target=<tuple> --print cfg to compute EXT_SUFFIX equivalent > value which is then used to build the C extension (if any) inside the > wheel, thats why we get cpython-312-x86_64-linux-musl for suffix, > what pyo/maturin is doing is right, however how OE does it in python > itself seems to be wrong to me, its synthesizing the OSABI > incorrectly. > where 'environment' value is not right. However, this is same > everywhere so it works, so long as the modules are built using > setuptools. > since SOABI is part of "_sysconfigdata" and its an OE generated file, > Perhaps it can consider the musl/gnu difference and emit SOABI > accordingly. Something like below > > https://snips.sh/f/L6QspcdxjL > > might be able to fix the python interpreter's logic. To close this thread. I have sent an alternative patch which fixes it in python3 itself. https://lore.kernel.org/openembedded-core/20240605064317.1646597-1-raj.khem@gmail.com/T/#t > > > > > > > Also, there is not even a comment above saying why we need to do this. > > > > > > > I added the log showing how it does not work, I thought that is showing how > > it's not working at all when using musl. If you have better idea what I should > > add to commit msg to make it better, let me know I will send a v2. > > > > > Cheers, > > > > > > Richard
Thanks for getting to the root of the issue. Alex On Wed 5. Jun 2024 at 9.47, Khem Raj via lists.openembedded.org <raj.khem= gmail.com@lists.openembedded.org> wrote: > On Mon, Jun 3, 2024 at 9:56 PM Khem Raj <raj.khem@gmail.com> wrote: > > > > On Mon, Jun 3, 2024 at 10:49 AM Khem Raj <raj.khem@gmail.com> wrote: > > > > > > On Mon, Jun 3, 2024 at 9:35 AM Richard Purdie > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > On Mon, 2024-06-03 at 09:25 -0700, Khem Raj via > lists.openembedded.org > > > > wrote: > > > > > loader expects it to be called with 'linux-gnu' e.g > > > > > rpds.cpython-312-x86_64-linux-musl.so is created with musl > > > > > but loader expects it to called > rpds.cpython-312-x86_64-linux-gnu.so > > > > > > > > > > Lets create the symlink to please the loader. > > > > > > > > > > Fixes > > > > > ImportError while importing test module '/usr/lib/python3-rpds- > > > > > py/ptest/tests/test_hash_trie_map.py'. > > > > > Hint: make sure your test modules/packages have valid Python names. > > > > > Traceback: > > > > > ../../python3.12/importlib/__init__.py:90: in import_module > > > > > return _bootstrap._gcd_import(name[level:], package, level) > > > > > tests/test_hash_trie_map.py:36: in <module> > > > > > from rpds import HashTrieMap > > > > > ../../python3.12/site-packages/rpds/__init__.py:1: in <module> > > > > > from .rpds import * > > > > > E ModuleNotFoundError: No module named 'rpds.rpds' > > > > > ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py > > > > > > > > > > Signed-off-by: Khem Raj <raj.khem@gmail.com> > > > > > --- > > > > > .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11 > > > > > +++++++++++ > > > > > 1 file changed, 11 insertions(+) > > > > > > > > > > diff --git a/meta/recipes-devtools/python/ > python3-rpds-py_0.18.1.bb > > > > > b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > > > index f46df1115c8..b1f0e100332 100644 > > > > > --- a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > > > +++ b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb > > > > > @@ -27,4 +27,15 @@ do_install_ptest() { > > > > > cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/ > > > > > } > > > > > > > > > > +do_install:append() { > > > > > + for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so > > > > > + do > > > > > + fname=`basename $f` > > > > > + lname=`echo $fname | sed 's/musl/gnu/'` > > > > > + if [ "$fname" != "$lname" ]; then > > > > > + mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname > > > > > + fi > > > > > + done > > > > > +} > > > > > + > > > > > BBCLASSEXTEND = "native nativesdk" > > > > > > > > > > > > > This feels like the kind of workaround that will live forever. Can we > > > > not fix the loader to use the right name? > > > > > > OE's python interpreter is cross-built and we end up using OSABI which > > > is detected during build 'linux-gnu', this seems to work fine for most > of > > > python modules since they get the OSABI from the sysconfigdata so it > > > all works even though the OSABI is called linux-gnu, however some > modules > > > like this one which are using maturin to build, seems to get the OSABI > > > equivalent differently, I am no expert on maturin but maybe someone > knows > > > how it happens. This problem is seen with several python modules built > with > > > OE, see > > > > > > https://github.com/meta-homeassistant/meta-homeassistant/issues/89 > > > https://github.com/pypa/auditwheel/issues/349 > > > > > > also see how it is fixed in python3-pydantic-core on meta-python > > > > https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38 > > > > > > Hopefully musllinux ABI tag will be sorted in pypa and this will work > better > > > however, we also need to see how we are encoding OSABI in python builds > > > > > > e.g. on qemux86-64 musl target I get > > > > > > root@qemux86-64:~# python3 > > > Python 3.12.3 (main, Apr 9 2024, 08:09:14) [Clang 19.0.0 > > > (/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on > > > linux > > > Type "help", "copyright", "credits" or "license" for more information. > > > >>> import sysconfig > > > >>> sysconfig.get_config_var('SOABI') > > > 'cpython-312-x86_64-linux-gnu' > > > > > > > I looked further in pyo and maturin infrastructure and it is using > > ustc --target=<tuple> --print cfg to compute EXT_SUFFIX equivalent > > value which is then used to build the C extension (if any) inside the > > wheel, thats why we get cpython-312-x86_64-linux-musl for suffix, > > what pyo/maturin is doing is right, however how OE does it in python > > itself seems to be wrong to me, its synthesizing the OSABI > > incorrectly. > > where 'environment' value is not right. However, this is same > > everywhere so it works, so long as the modules are built using > > setuptools. > > since SOABI is part of "_sysconfigdata" and its an OE generated file, > > Perhaps it can consider the musl/gnu difference and emit SOABI > > accordingly. Something like below > > > > https://snips.sh/f/L6QspcdxjL > > > > might be able to fix the python interpreter's logic. > > To close this thread. I have sent an alternative patch which fixes it > in python3 itself. > > https://lore.kernel.org/openembedded-core/20240605064317.1646597-1-raj.khem@gmail.com/T/#t > > > > > > > > > > > Also, there is not even a comment above saying why we need to do > this. > > > > > > > > > > I added the log showing how it does not work, I thought that is > showing how > > > it's not working at all when using musl. If you have better idea what > I should > > > add to commit msg to make it better, let me know I will send a v2. > > > > > > > Cheers, > > > > > > > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#200355): > https://lists.openembedded.org/g/openembedded-core/message/200355 > Mute This Topic: https://lists.openembedded.org/mt/106465307/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
diff --git a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb index f46df1115c8..b1f0e100332 100644 --- a/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +++ b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb @@ -27,4 +27,15 @@ do_install_ptest() { cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/ } +do_install:append() { + for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so + do + fname=`basename $f` + lname=`echo $fname | sed 's/musl/gnu/'` + if [ "$fname" != "$lname" ]; then + mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname + fi + done +} + BBCLASSEXTEND = "native nativesdk"
loader expects it to be called with 'linux-gnu' e.g rpds.cpython-312-x86_64-linux-musl.so is created with musl but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so Lets create the symlink to please the loader. Fixes ImportError while importing test module '/usr/lib/python3-rpds-py/ptest/tests/test_hash_trie_map.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: ../../python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) tests/test_hash_trie_map.py:36: in <module> from rpds import HashTrieMap ../../python3.12/site-packages/rpds/__init__.py:1: in <module> from .rpds import * E ModuleNotFoundError: No module named 'rpds.rpds' ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py Signed-off-by: Khem Raj <raj.khem@gmail.com> --- .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11 +++++++++++ 1 file changed, 11 insertions(+)