[meta-python] python3-pandas: compile against target version of numpy

python3-numpy headers are required by pandas to compile successfully.
By default, this recipe used python3-numpy-native for compilation,
which usually works. However in case the bitness of the build-host
differs from the target, then problems arise.

For example when compiling for 32-bit ARM on a x86-64 machine, the following
error appears when trying to import the module:

ValueError: Buffer dtype mismatch, expected 'const int64_t' but got 'long long'

When running a diff on all numpy headers across native and target,
only one header differs, but that one in a significant way:

Patch is ok but the commit message should not contain diffs, it
confuses tool like b4 while pulling this patch into my staging area,
so please send a v2 after removing diffs from commit msg.

On Sun, Mar 9, 2025 at 12:43 PM Gyorgy Sarvari via
lists.openembedded.org <skandigraun=gmail.com@lists.openembedded.org>
wrote:
>
> python3-numpy headers are required by pandas to compile successfully.
> By default, this recipe used python3-numpy-native for compilation,
> which usually works. However in case the bitness of the build-host
> differs from the target, then problems arise.
>
> For example when compiling for 32-bit ARM on a x86-64 machine, the following
> error appears when trying to import the module:
>
> ValueError: Buffer dtype mismatch, expected 'const int64_t' but got 'long long'
>
> When running a diff on all numpy headers across native and target,
> only one header differs, but that one in a significant way:
>
> --- ./_numpyconfig.h    2025-03-07 08:08:58.949476926 +0100
> +++ <ARM32_FOLDER>/_numpyconfig.h    2025-03-07 08:09:10.811983538 +0100
> @@ -2,16 +2,16 @@
>
>  #define NPY_SIZEOF_SHORT 2
>  #define NPY_SIZEOF_INT 4
> -#define NPY_SIZEOF_LONG 8
> +#define NPY_SIZEOF_LONG 4
>  #define NPY_SIZEOF_FLOAT 4
>  #define NPY_SIZEOF_COMPLEX_FLOAT 8
>  #define NPY_SIZEOF_DOUBLE 8
>  #define NPY_SIZEOF_COMPLEX_DOUBLE 16
> -#define NPY_SIZEOF_LONGDOUBLE 16
> -#define NPY_SIZEOF_COMPLEX_LONGDOUBLE 32
> -#define NPY_SIZEOF_PY_INTPTR_T 8
> -#define NPY_SIZEOF_INTP 8
> -#define NPY_SIZEOF_UINTP 8
> +#define NPY_SIZEOF_LONGDOUBLE 8
> +#define NPY_SIZEOF_COMPLEX_LONGDOUBLE 16
> +#define NPY_SIZEOF_PY_INTPTR_T 4
> +#define NPY_SIZEOF_INTP 4
> +#define NPY_SIZEOF_UINTP 4
>  #define NPY_SIZEOF_WCHAR_T 4
>  #define NPY_SIZEOF_OFF_T 8
>  #define NPY_SIZEOF_PY_LONG_LONG 8
>
> This change switches from python3-numpy-native dependency to python3-numpy to
> ensure that the correct headers are used. Beside this also patch the meson script,
> so it accepts an option (numpy_inc_dir) to specify the location of these headers,
> since it is not able to query them from the class-target module
>
> The PYTHONPATH variable is extended with the target's RECIPE_SYSROOT, because
> numpy is specified as a dependency in meson, and it needs to find the module
> to continue successfully.
>
> Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
> ---
>  ...fy-numpy-header-path-as-meson-option.patch | 52 +++++++++++++++++++
>  .../python/python3-pandas_2.2.2.bb            |  9 +++-
>  2 files changed, 60 insertions(+), 1 deletion(-)
>  create mode 100644 meta-python/recipes-devtools/python/python3-pandas/0001-specify-numpy-header-path-as-meson-option.patch
>
> diff --git a/meta-python/recipes-devtools/python/python3-pandas/0001-specify-numpy-header-path-as-meson-option.patch b/meta-python/recipes-devtools/python/python3-pandas/0001-specify-numpy-header-path-as-meson-option.patch
> new file mode 100644
> index 0000000000..039f116ff1
> --- /dev/null
> +++ b/meta-python/recipes-devtools/python/python3-pandas/0001-specify-numpy-header-path-as-meson-option.patch
> @@ -0,0 +1,52 @@
> +From 46b3c3ee68f271acbc178260939ca8423f9244a5 Mon Sep 17 00:00:00 2001
> +From: Gyorgy Sarvari <skandigraun@gmail.com>
> +Date: Sun, 9 Mar 2025 19:57:26 +0100
> +Subject: [PATCH] specify numpy header path as meson option
> +
> +In case pandas is compiled against the class-target version of numpy, it is
> +not possible to import the module, due to target-compiled binary files -
> +it just fails with an error. However this also means that meson isn't
> +able to query the include folder location, so the compilation fails.
> +
> +To avoid this, the new "numpy_inc_dir" meson option allows specifying the
> +location of the required headers.
> +
> +Upstream-Status: Inappropriate [oe-specific]
> +---
> + meson.options      | 1 +
> + pandas/meson.build | 6 +++---
> + 2 files changed, 4 insertions(+), 3 deletions(-)
> + create mode 100644 meson.options
> +
> +diff --git a/meson.options b/meson.options
> +new file mode 100644
> +index 0000000000..3d3a18d145
> +--- /dev/null
> ++++ b/meson.options
> +@@ -0,0 +1 @@
> ++option('numpy_inc_dir', type : 'string', description : 'The absolute path to the numpy headers')
> +diff --git a/pandas/meson.build b/pandas/meson.build
> +index 435103a954..32ad04bd8e 100644
> +--- a/pandas/meson.build
> ++++ b/pandas/meson.build
> +@@ -3,17 +3,17 @@ incdir_numpy = run_command(py,
> +     '-c',
> +     '''
> + import os
> +-import numpy as np
> + try:
> +     # Check if include directory is inside the pandas dir
> +     # e.g. a venv created inside the pandas dir
> +     # If so, convert it to a relative path
> +-    incdir = os.path.relpath(np.get_include())
> ++    incdir = os.path.relpath(os.getenv('NUMPY_INC_DIR'))
> + except Exception:
> +-    incdir = np.get_include()
> ++    incdir = os.getenv('NUMPY_INC_DIR')
> + print(incdir)
> +      '''
> +   ],
> ++  env: {'NUMPY_INC_DIR': get_option('numpy_inc_dir')},
> +   check: true
> + ).stdout().strip()
> +
> diff --git a/meta-python/recipes-devtools/python/python3-pandas_2.2.2.bb b/meta-python/recipes-devtools/python/python3-pandas_2.2.2.bb
> index c12a237653..5676d98cd0 100644
> --- a/meta-python/recipes-devtools/python/python3-pandas_2.2.2.bb
> +++ b/meta-python/recipes-devtools/python/python3-pandas_2.2.2.bb
> @@ -11,12 +11,14 @@ SRC_URI += " \
>              file://0001-pyproject.toml-Downgrade-numpy-version-needs-to-1.x.patch \
>              "
>
> +SRC_URI:append:class-target = " file://0001-specify-numpy-header-path-as-meson-option.patch "
> +
>  SRC_URI[sha256sum] = "9e79019aba43cb4fda9e4d983f8e88ca0373adbb697ae9c6c43093218de28b54"
>
>  inherit pkgconfig pypi python_mesonpy cython
>
>  DEPENDS += " \
> -    python3-numpy-native \
> +    python3-numpy \
>      python3-versioneer-native \
>  "
>
> @@ -30,3 +32,8 @@ RDEPENDS:${PN} += " \
>      python3-pytz \
>      python3-profile \
>  "
> +
> +PYTHONPATH:prepend:class-target = "${RECIPE_SYSROOT}${PYTHON_SITEPACKAGES_DIR}:"
> +export PYTHONPATH
> +
> +EXTRA_OEMESON:append:class-target = " -Dnumpy_inc_dir=${RECIPE_SYSROOT}${PYTHON_SITEPACKAGES_DIR}/numpy/_core/include "
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#115816): https://lists.openembedded.org/g/openembedded-devel/message/115816
> Mute This Topic: https://lists.openembedded.org/mt/111606607/1997914
> Group Owner: openembedded-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On 9 Mar 2025, at 19:43, Gyorgy Sarvari via lists.openembedded.org <skandigraun=gmail.com@lists.openembedded.org> wrote:
> +Upstream-Status: Inappropriate [oe-specific]

I’ll quibble with Inappropriate.  This is a “pandas cannot cross-compile” problem and they have an upstream bug, so this should _at least_ be forwarded to them as an example of how we’re fixing the problem. They might decide to take it, they might have a better solution in mind, but either way this isn’t OE-specific.

Ross

On 10.03.25 13:12, Ross Burton wrote:
> On 9 Mar 2025, at 19:43, Gyorgy Sarvari via lists.openembedded.org <skandigraun=gmail.com@lists.openembedded.org> wrote:
>> +Upstream-Status: Inappropriate [oe-specific]
> I’ll quibble with Inappropriate.  This is a “pandas cannot cross-compile” problem and they have an upstream bug, so this should _at least_ be forwarded to them as an example of how we’re fixing the problem. They might decide to take it, they might have a better solution in mind, but either way this isn’t OE-specific.
>
> Ross
ACK, v3 sent.