From patchwork Thu Nov 7 17:23:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 52171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F035D5D674 for ; Thu, 7 Nov 2024 17:24:16 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.78026.1731000250735172167 for ; Thu, 07 Nov 2024 09:24:11 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D0622497 for ; Thu, 7 Nov 2024 09:24:39 -0800 (PST) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8B9AE3F6A8 for ; Thu, 7 Nov 2024 09:24:09 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 1/8] python3-setuptools: add more comments to a patch Date: Thu, 7 Nov 2024 17:23:57 +0000 Message-Id: <20241107172404.4017047-1-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 07 Nov 2024 17:24:16 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206842 Add more explanatory comments, including when we can drop the patch. Signed-off-by: Ross Burton --- ...-conditionally-do-not-fetch-code-by-easy_install.patch | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch b/meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch index e227c2889cf..16b316e9cc2 100644 --- a/meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch +++ b/meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch @@ -3,11 +3,13 @@ From: Hongxu Jia Date: Tue, 17 Jul 2018 10:13:38 +0800 Subject: [PATCH] conditionally do not fetch code by easy_install -If var-NO_FETCH_BUILD is set, do not allow to fetch code from -internet by easy_install. +If NO_FETCH_BUILD is set, do not fetch code in easy_install. This avoids setup.py +trying to fetch missing dependencies at build time. -Upstream-Status: Inappropriate [oe specific] +This is only used by the deprecated codepath as implemented by setuptools3.bbclass, so +when that ever gets removed this patch can also be deleted. +Upstream-Status: Denied [https://github.com/pypa/setuptools/issues/4735] Signed-off-by: Hongxu Jia --- setuptools/command/easy_install.py | 5 +++++ From patchwork Thu Nov 7 17:23:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 52176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B854D5D675 for ; Thu, 7 Nov 2024 17:24:16 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.78027.1731000251269503128 for ; Thu, 07 Nov 2024 09:24:11 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9088B1063 for ; Thu, 7 Nov 2024 09:24:40 -0800 (PST) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4C16A3F6A8 for ; Thu, 7 Nov 2024 09:24:10 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 2/8] classes/setuptools: do parallel builds in a clean directory Date: Thu, 7 Nov 2024 17:23:58 +0000 Message-Id: <20241107172404.4017047-2-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241107172404.4017047-1-ross.burton@arm.com> References: <20241107172404.4017047-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 07 Nov 2024 17:24:16 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206843 It turns out that by setting DIST_EXTRA_CONFIG you can tell setuptools to read a supplementary configuration file. This lets us pass options that we normally cannot reach via the bdist_wheel command. Tell the build task to use a build directory outside of the source tree, so that we can clean it before each rebuild. Tell the build_ext task to do parallel builds, as it defaults to not. In my testing this halves the build time of python3-pandas. Signed-off-by: Ross Burton --- .../python_setuptools_build_meta.bbclass | 14 ++++++++++++++ meta/classes-recipe/setuptools3.bbclass | 12 ++++++++++++ 2 files changed, 26 insertions(+) diff --git a/meta/classes-recipe/python_setuptools_build_meta.bbclass b/meta/classes-recipe/python_setuptools_build_meta.bbclass index 4c84d1e8d0b..3fd42cc9845 100644 --- a/meta/classes-recipe/python_setuptools_build_meta.bbclass +++ b/meta/classes-recipe/python_setuptools_build_meta.bbclass @@ -7,3 +7,17 @@ inherit setuptools3-base python_pep517 DEPENDS += "python3-setuptools-native python3-wheel-native" + +B = "${WORKDIR}/build" +do_configure[cleandirs] += "${B}" + +do_compile:prepend() { + # Write an extra config file to build in parallel + export DIST_EXTRA_CONFIG=${WORKDIR}/setuptools-extra.cfg + cat <$DIST_EXTRA_CONFIG +[build] +build_base = ${B} +[build_ext] +parallel = ${@oe.utils.parallel_make(d)} +EOF +} diff --git a/meta/classes-recipe/setuptools3.bbclass b/meta/classes-recipe/setuptools3.bbclass index 64a78e9a367..be260093b47 100644 --- a/meta/classes-recipe/setuptools3.bbclass +++ b/meta/classes-recipe/setuptools3.bbclass @@ -12,6 +12,8 @@ SETUPTOOLS_BUILD_ARGS ?= "" SETUPTOOLS_SETUP_PATH ?= "${S}" +B = "${WORKDIR}/build" + python do_check_backend() { import re filename = d.expand("${SETUPTOOLS_SETUP_PATH}/pyproject.toml") @@ -29,8 +31,18 @@ addtask check_backend after do_patch before do_configure setuptools3_do_configure() { : } +do_configure[cleandirs] += "${B}" setuptools3_do_compile() { + # Write an extra config file to build in parallel + export DIST_EXTRA_CONFIG=${WORKDIR}/setuptools-extra.cfg + cat <$DIST_EXTRA_CONFIG +[build] +build_base = ${B} +[build_ext] +parallel = ${@oe.utils.parallel_make(d)} +EOF + cd ${SETUPTOOLS_SETUP_PATH} NO_FETCH_BUILD=1 \ STAGING_INCDIR=${STAGING_INCDIR} \ From patchwork Thu Nov 7 17:23:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 52173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C797D5D672 for ; Thu, 7 Nov 2024 17:24:16 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.78028.1731000251900852408 for ; Thu, 07 Nov 2024 09:24:12 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 59219497 for ; Thu, 7 Nov 2024 09:24:41 -0800 (PST) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0C0F63F6A8 for ; Thu, 7 Nov 2024 09:24:10 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 3/8] python3-numpy: don't do out-of-tree builds Date: Thu, 7 Nov 2024 17:23:59 +0000 Message-Id: <20241107172404.4017047-3-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241107172404.4017047-1-ross.burton@arm.com> References: <20241107172404.4017047-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 07 Nov 2024 17:24:16 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206844 The source generation in numpy appears to be fragile and silently fails when an out-of-tree build is done with setuptools. The result is that importing the numpy library results in failing linkage. We should move numpy to build with mesonpy, but until that is done we can simply reset B to be at the default location under the source tree. Signed-off-by: Ross Burton --- meta/recipes-devtools/python/python3-numpy_1.26.4.bb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb b/meta/recipes-devtools/python/python3-numpy_1.26.4.bb index 914245507c7..4ffbe0d3708 100644 --- a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb +++ b/meta/recipes-devtools/python/python3-numpy_1.26.4.bb @@ -23,6 +23,9 @@ DEPENDS += "python3-cython-native" inherit ptest setuptools3 github-releases S = "${WORKDIR}/numpy-${PV}" +# numpy built with setuptools fails to build correctly when out-of-tree, so +# until we switch to mesonpy we need this. +B = "${S}/build" CLEANBROKEN = "1" From patchwork Thu Nov 7 17:24:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 52172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E7DDD5D66F for ; Thu, 7 Nov 2024 17:24:16 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.77813.1731000252806712979 for ; Thu, 07 Nov 2024 09:24:12 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 19171497 for ; Thu, 7 Nov 2024 09:24:42 -0800 (PST) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C8BE33F6A8 for ; Thu, 7 Nov 2024 09:24:11 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 4/8] recipes/python: remove obsolete CLEANBROKEN Date: Thu, 7 Nov 2024 17:24:00 +0000 Message-Id: <20241107172404.4017047-4-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241107172404.4017047-1-ross.burton@arm.com> References: <20241107172404.4017047-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 07 Nov 2024 17:24:16 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206845 These CLEANBROKENs are obsolete: the classes involved set their own do_configure so the clean in base_do_configure doesn't get ran. Signed-off-by: Ross Burton --- meta/recipes-devtools/python/python3-babel_2.16.0.bb | 2 -- meta/recipes-devtools/python/python3-click_8.1.7.bb | 2 -- meta/recipes-devtools/python/python3-jinja2_3.1.4.bb | 2 -- meta/recipes-devtools/python/python3-lxml_5.3.0.bb | 2 -- meta/recipes-devtools/python/python3-numpy_1.26.4.bb | 2 -- 5 files changed, 10 deletions(-) diff --git a/meta/recipes-devtools/python/python3-babel_2.16.0.bb b/meta/recipes-devtools/python/python3-babel_2.16.0.bb index 929d7e99831..34aa7c9a750 100644 --- a/meta/recipes-devtools/python/python3-babel_2.16.0.bb +++ b/meta/recipes-devtools/python/python3-babel_2.16.0.bb @@ -9,8 +9,6 @@ inherit pypi setuptools3 S = "${WORKDIR}/babel-${PV}" -CLEANBROKEN = "1" - RDEPENDS:${PN} += " \ python3-codecs \ python3-difflib \ diff --git a/meta/recipes-devtools/python/python3-click_8.1.7.bb b/meta/recipes-devtools/python/python3-click_8.1.7.bb index 7d91e1af838..3998d0864f5 100644 --- a/meta/recipes-devtools/python/python3-click_8.1.7.bb +++ b/meta/recipes-devtools/python/python3-click_8.1.7.bb @@ -29,8 +29,6 @@ do_install_ptest() { UPSTREAM_CHECK_REGEX = "click/(?P\d+(\.\d+)+)/" -CLEANBROKEN = "1" - RDEPENDS:${PN} += "\ python3-io \ python3-threading \ diff --git a/meta/recipes-devtools/python/python3-jinja2_3.1.4.bb b/meta/recipes-devtools/python/python3-jinja2_3.1.4.bb index a11878496c6..98968ffaec4 100644 --- a/meta/recipes-devtools/python/python3-jinja2_3.1.4.bb +++ b/meta/recipes-devtools/python/python3-jinja2_3.1.4.bb @@ -10,8 +10,6 @@ PYPI_PACKAGE = "jinja2" CVE_PRODUCT = "jinja2 jinja" -CLEANBROKEN = "1" - inherit pypi python_flit_core ptest UPSTREAM_CHECK_PYPI_PACKAGE = "Jinja2" diff --git a/meta/recipes-devtools/python/python3-lxml_5.3.0.bb b/meta/recipes-devtools/python/python3-lxml_5.3.0.bb index 4c9c22e5c40..f130b567225 100644 --- a/meta/recipes-devtools/python/python3-lxml_5.3.0.bb +++ b/meta/recipes-devtools/python/python3-lxml_5.3.0.bb @@ -40,5 +40,3 @@ BBCLASSEXTEND = "native nativesdk" RDEPENDS:${PN} += "libxml2 libxslt python3-compression" CVE_PRODUCT = "lxml" - -CLEANBROKEN = "1" diff --git a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb b/meta/recipes-devtools/python/python3-numpy_1.26.4.bb index 4ffbe0d3708..95614361be0 100644 --- a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb +++ b/meta/recipes-devtools/python/python3-numpy_1.26.4.bb @@ -27,8 +27,6 @@ S = "${WORKDIR}/numpy-${PV}" # until we switch to mesonpy we need this. B = "${S}/build" -CLEANBROKEN = "1" - do_compile:prepend() { export NPY_DISABLE_SVML=1 } From patchwork Thu Nov 7 17:24:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 52175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BF97D5D678 for ; Thu, 7 Nov 2024 17:24:16 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.78030.1731000253271984750 for ; Thu, 07 Nov 2024 09:24:13 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CE531497 for ; Thu, 7 Nov 2024 09:24:42 -0800 (PST) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 897313F6A8 for ; Thu, 7 Nov 2024 09:24:12 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 5/8] python3-cython: add patch to prefix map source paths in generated files Date: Thu, 7 Nov 2024 17:24:01 +0000 Message-Id: <20241107172404.4017047-5-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241107172404.4017047-1-ross.burton@arm.com> References: <20241107172404.4017047-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 07 Nov 2024 17:24:16 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206846 Cython generates C source code that contains mentions to the original .py files, which results in build paths being embedded inside the binaries. Implement prefix remapping to change these build paths to point at the target debug directory, so that we don't leak build paths and have reproducible builds. This patch is currently not submitted upstream, but will be shortly and I expect it to evolve before being merged. Signed-off-by: Ross Burton --- .../python3-cython/0001-WIP-prefix-map.patch | 148 ++++++++++++++++++ .../python/python3-cython_3.0.11.bb | 1 + 2 files changed, 149 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-cython/0001-WIP-prefix-map.patch diff --git a/meta/recipes-devtools/python/python3-cython/0001-WIP-prefix-map.patch b/meta/recipes-devtools/python/python3-cython/0001-WIP-prefix-map.patch new file mode 100644 index 00000000000..adc9463ffa7 --- /dev/null +++ b/meta/recipes-devtools/python/python3-cython/0001-WIP-prefix-map.patch @@ -0,0 +1,148 @@ +From 4d1b7911372561b22e03c7f2b4ec807502b5b9c1 Mon Sep 17 00:00:00 2001 +From: Ross Burton +Date: Mon, 4 Nov 2024 15:36:39 +0000 +Subject: [PATCH] WIP prefix map + +Upstream-Status: Inappropriate +Signed-off-by: Ross Burton +--- + Cython/Compiler/CmdLine.py | 9 ++++++++- + Cython/Compiler/Main.py | 9 +++++---- + Cython/Compiler/Options.py | 1 + + Cython/Compiler/Parsing.py | 1 + + Cython/Compiler/Scanning.py | 9 +++++++-- + 5 files changed, 22 insertions(+), 7 deletions(-) + +diff --git a/Cython/Compiler/CmdLine.py b/Cython/Compiler/CmdLine.py +index 776636c..f5a7c79 100644 +--- a/Cython/Compiler/CmdLine.py ++++ b/Cython/Compiler/CmdLine.py +@@ -74,6 +74,12 @@ class SetAnnotateCoverageAction(Action): + namespace.annotate = True + namespace.annotate_coverage_xml = values + ++class SetPrefixMapAction(Action): ++ def __call__(self, parser, namespace, values, option_string=None): ++ mappings = getattr(namespace, self.dest, {}) ++ k, v = values.split("=", 1) ++ mappings[k] = v ++ setattr(namespace, self.dest, mappings) + + def create_cython_argparser(): + description = "Cython (https://cython.org/) is a compiler for code written in the "\ +@@ -157,9 +163,10 @@ def create_cython_argparser(): + 'deduced from the import path if source file is in ' + 'a package, or equals the filename otherwise.') + parser.add_argument('-M', '--depfile', action='store_true', help='produce depfiles for the sources') ++ # TODO: add help ++ parser.add_argument("--prefix-map", action=SetPrefixMapAction) + parser.add_argument('sources', nargs='*', default=[]) + +- # TODO: add help + parser.add_argument("-z", "--pre-import", dest='pre_import', action='store', type=str, help=SUPPRESS) + parser.add_argument("--convert-range", dest='convert_range', action='store_true', help=SUPPRESS) + parser.add_argument("--no-c-in-traceback", dest='c_line_in_traceback', action='store_false', help=SUPPRESS) +diff --git a/Cython/Compiler/Main.py b/Cython/Compiler/Main.py +index 80946c0..28cfe68 100644 +--- a/Cython/Compiler/Main.py ++++ b/Cython/Compiler/Main.py +@@ -70,7 +70,7 @@ class Context(object): + language_level = None # warn when not set but default to Py2 + + def __init__(self, include_directories, compiler_directives, cpp=False, +- language_level=None, options=None): ++ language_level=None, prefix_map=None, options=None): + # cython_scope is a hack, set to False by subclasses, in order to break + # an infinite loop. + # Better code organization would fix it. +@@ -83,6 +83,7 @@ class Context(object): + self.future_directives = set() + self.compiler_directives = compiler_directives + self.cpp = cpp ++ self.prefix_map = prefix_map or {} + self.options = options + + self.pxds = {} # full name -> node tree +@@ -98,7 +99,7 @@ class Context(object): + @classmethod + def from_options(cls, options): + return cls(options.include_path, options.compiler_directives, +- options.cplus, options.language_level, options=options) ++ options.cplus, options.language_level, prefix_map=options.prefix_map, options=options) + + def set_language_level(self, level): + from .Future import print_function, unicode_literals, absolute_import, division, generator_stop +@@ -259,7 +260,7 @@ class Context(object): + rel_path = module_name.replace('.', os.sep) + os.path.splitext(pxd_pathname)[1] + if not pxd_pathname.endswith(rel_path): + rel_path = pxd_pathname # safety measure to prevent printing incorrect paths +- source_desc = FileSourceDescriptor(pxd_pathname, rel_path) ++ source_desc = FileSourceDescriptor(pxd_pathname, rel_path, prefix_map=self.prefix_map) + err, result = self.process_pxd(source_desc, scope, qualified_name) + if err: + raise err +@@ -509,7 +510,7 @@ def run_pipeline(source, options, full_module_name=None, context=None): + rel_path = source # safety measure to prevent printing incorrect paths + else: + rel_path = abs_path +- source_desc = FileSourceDescriptor(abs_path, rel_path) ++ source_desc = FileSourceDescriptor(abs_path, rel_path, prefix_map=context.prefix_map) + source = CompilationSource(source_desc, full_module_name, cwd) + + # Set up result object +diff --git a/Cython/Compiler/Options.py b/Cython/Compiler/Options.py +index 61950a7..cc52732 100644 +--- a/Cython/Compiler/Options.py ++++ b/Cython/Compiler/Options.py +@@ -796,4 +796,5 @@ default_options = dict( + create_extension=None, + np_pythran=False, + legacy_implicit_noexcept=None, ++ prefix_map=dict(pair.split("=", 1) for pair in os.environ.get("CYTHON_PREFIX_MAP", "").split()), + ) +diff --git a/Cython/Compiler/Parsing.py b/Cython/Compiler/Parsing.py +index 25c0de9..6c0eccf 100644 +--- a/Cython/Compiler/Parsing.py ++++ b/Cython/Compiler/Parsing.py +@@ -2106,6 +2106,7 @@ def p_include_statement(s, ctx): + s.included_files.append(include_file_name) + with Utils.open_source_file(include_file_path) as f: + source_desc = FileSourceDescriptor(include_file_path) ++ print(f"TODO Cannot use prefix map on {include_file_path}") + s2 = PyrexScanner(f, source_desc, s, source_encoding=f.encoding, parse_comments=s.parse_comments) + tree = p_statement_list(s2, ctx) + return tree +diff --git a/Cython/Compiler/Scanning.py b/Cython/Compiler/Scanning.py +index 372392b..0fa3b30 100644 +--- a/Cython/Compiler/Scanning.py ++++ b/Cython/Compiler/Scanning.py +@@ -195,7 +195,7 @@ class FileSourceDescriptor(SourceDescriptor): + optional name argument and will be passed back when asking for + the position()-tuple. + """ +- def __init__(self, filename, path_description=None): ++ def __init__(self, filename, path_description=None, prefix_map={}): + filename = Utils.decode_filename(filename) + self.path_description = path_description or filename + self.filename = filename +@@ -205,6 +205,7 @@ class FileSourceDescriptor(SourceDescriptor): + self.set_file_type_from_name(filename) + self._cmp_name = filename + self._lines = {} ++ self.prefix_map = prefix_map + + def get_lines(self, encoding=None, error_handling=None): + # we cache the lines only the second time this is called, in +@@ -243,7 +244,11 @@ class FileSourceDescriptor(SourceDescriptor): + return path + + def get_filenametable_entry(self): +- return self.file_path ++ entry = self.file_path ++ for k, v in self.prefix_map.items(): ++ # TODO: should just replace the prefix ++ entry = entry.replace(k, v) ++ return entry + + def __eq__(self, other): + return isinstance(other, FileSourceDescriptor) and self.filename == other.filename diff --git a/meta/recipes-devtools/python/python3-cython_3.0.11.bb b/meta/recipes-devtools/python/python3-cython_3.0.11.bb index 980cb046365..efce67d0921 100644 --- a/meta/recipes-devtools/python/python3-cython_3.0.11.bb +++ b/meta/recipes-devtools/python/python3-cython_3.0.11.bb @@ -7,6 +7,7 @@ SECTION = "devel/python" LICENSE = "Apache-2.0" LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=61c3ee8961575861fa86c7e62bc9f69c" +SRC_URI += "file://0001-WIP-prefix-map.patch" SRC_URI[sha256sum] = "7146dd2af8682b4ca61331851e6aebce9fe5158e75300343f80c07ca80b1faff" inherit pypi setuptools3 From patchwork Thu Nov 7 17:24:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 52174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BF0AD5D677 for ; Thu, 7 Nov 2024 17:24:16 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.77814.1731000254039318083 for ; Thu, 07 Nov 2024 09:24:14 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 97998497 for ; Thu, 7 Nov 2024 09:24:43 -0800 (PST) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4A0933F6A8 for ; Thu, 7 Nov 2024 09:24:13 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 6/8] classes-recipe: add cython class Date: Thu, 7 Nov 2024 17:24:02 +0000 Message-Id: <20241107172404.4017047-6-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241107172404.4017047-1-ross.burton@arm.com> References: <20241107172404.4017047-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 07 Nov 2024 17:24:16 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206847 Recipes that use Cython typically also do some bespoke fixup. Add a class to centralise the logic: - Set CYTHON_PREFIX_MAP to stop build paths appearing in generated objects - Strip "Cython Metadata" blocks from generated code that ends up in the -src package Signed-off-by: Ross Burton --- meta/classes-recipe/cython.bbclass | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 meta/classes-recipe/cython.bbclass diff --git a/meta/classes-recipe/cython.bbclass b/meta/classes-recipe/cython.bbclass new file mode 100644 index 00000000000..f37ebeb23fe --- /dev/null +++ b/meta/classes-recipe/cython.bbclass @@ -0,0 +1,11 @@ +DEPENDS:append = " python3-cython-native" + +# Remap the build paths that appear in generated .c code +export CYTHON_PREFIX_MAP = "${S}=${TARGET_DBGSRC_DIR} ${B}=${TARGET_DBGSRC_DIR}" + +do_compile[postfuncs] = "strip_cython_metadata" +strip_cython_metadata() { + # Remove the Cython Metadata headers that we don't need after the build, and + # may contain build paths. + find ${S} -name "*.c" -print0 | xargs -0 sed -i -e "/BEGIN: Cython Metadata/,/END: Cython Metadata/d" +} From patchwork Thu Nov 7 17:24:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 52177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E6D4D5D67A for ; Thu, 7 Nov 2024 17:24:16 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.78033.1731000254849724109 for ; Thu, 07 Nov 2024 09:24:15 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 57887497 for ; Thu, 7 Nov 2024 09:24:44 -0800 (PST) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1323D3F6A8 for ; Thu, 7 Nov 2024 09:24:13 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 7/8] python3-*: inherit cython Date: Thu, 7 Nov 2024 17:24:03 +0000 Message-Id: <20241107172404.4017047-7-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241107172404.4017047-1-ross.burton@arm.com> References: <20241107172404.4017047-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 07 Nov 2024 17:24:16 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206848 Change recipes that depend on python3-cython-native to inherit cython instead. Signed-off-by: Ross Burton --- .../python/python3-cython_3.0.11.bb | 32 +++---------------- .../python/python3-numpy_1.26.4.bb | 4 +-- .../python/python3-pyyaml_6.0.2.bb | 4 +-- 3 files changed, 7 insertions(+), 33 deletions(-) diff --git a/meta/recipes-devtools/python/python3-cython_3.0.11.bb b/meta/recipes-devtools/python/python3-cython_3.0.11.bb index efce67d0921..020a8e7205c 100644 --- a/meta/recipes-devtools/python/python3-cython_3.0.11.bb +++ b/meta/recipes-devtools/python/python3-cython_3.0.11.bb @@ -10,42 +10,20 @@ LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=61c3ee8961575861fa86c7e62bc9f69c" SRC_URI += "file://0001-WIP-prefix-map.patch" SRC_URI[sha256sum] = "7146dd2af8682b4ca61331851e6aebce9fe5158e75300343f80c07ca80b1faff" -inherit pypi setuptools3 -UPSTREAM_CHECK_PYPI_PACKAGE = "Cython" +inherit pypi setuptools3 cython -do_install:append() { - # Make sure we use /usr/bin/env python3 - for PYTHSCRIPT in `grep -rIl '^#!.*python' ${D}`; do - sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' $PYTHSCRIPT - done +# No need to depend on self +DEPENDS:remove = "python3-cython-native" - # remove build paths from generated sources - sed -i -e 's|${WORKDIR}||' ${S}/Cython/*.c ${S}/Cython/Compiler/*.c ${S}/Cython/Plex/*.c +UPSTREAM_CHECK_PYPI_PACKAGE = "Cython" +do_install:append() { # rename scripts that would conflict with the Python 2 build of Cython mv ${D}${bindir}/cython ${D}${bindir}/cython3 mv ${D}${bindir}/cythonize ${D}${bindir}/cythonize3 mv ${D}${bindir}/cygdb ${D}${bindir}/cygdb3 } -PACKAGESPLITFUNCS =+ "cython_fix_sources" - -cython_fix_sources () { - for f in ${PKGD}${TARGET_DBGSRC_DIR}/Cython/Compiler/FlowControl.c \ - ${PKGD}${TARGET_DBGSRC_DIR}/Cython/Compiler/FusedNode.c \ - ${PKGD}${TARGET_DBGSRC_DIR}/Cython/Compiler/Scanning.c \ - ${PKGD}${TARGET_DBGSRC_DIR}/Cython/Compiler/Visitor.c \ - ${PKGD}${TARGET_DBGSRC_DIR}/Cython/Plex/Actions.c \ - ${PKGD}${TARGET_DBGSRC_DIR}/Cython/Plex/Scanners.c \ - ${PKGD}${TARGET_DBGSRC_DIR}/Cython/Runtime/refnanny.c \ - ${PKGD}${TARGET_DBGSRC_DIR}/Cython/Tempita/_tempita.c \ - ${PKGD}${libdir}/${PYTHON_DIR}/site-packages/Cython*/SOURCES.txt; do - if [ -e $f ]; then - sed -i -e 's#${WORKDIR}/cython-${PV}#${TARGET_DBGSRC_DIR}#g' $f - fi - done -} - RDEPENDS:${PN}:class-target += "\ python3-misc \ python3-netserver \ diff --git a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb b/meta/recipes-devtools/python/python3-numpy_1.26.4.bb index 95614361be0..7de92f687c3 100644 --- a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb +++ b/meta/recipes-devtools/python/python3-numpy_1.26.4.bb @@ -18,9 +18,7 @@ SRC_URI[sha256sum] = "2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4 GITHUB_BASE_URI = "https://github.com/numpy/numpy/releases" UPSTREAM_CHECK_REGEX = "releases/tag/v?(?P\d+(\.\d+)+)$" -DEPENDS += "python3-cython-native" - -inherit ptest setuptools3 github-releases +inherit ptest setuptools3 github-releases cython S = "${WORKDIR}/numpy-${PV}" # numpy built with setuptools fails to build correctly when out-of-tree, so diff --git a/meta/recipes-devtools/python/python3-pyyaml_6.0.2.bb b/meta/recipes-devtools/python/python3-pyyaml_6.0.2.bb index 10e61274b07..9f9320a5949 100644 --- a/meta/recipes-devtools/python/python3-pyyaml_6.0.2.bb +++ b/meta/recipes-devtools/python/python3-pyyaml_6.0.2.bb @@ -4,8 +4,6 @@ HOMEPAGE = "https://pyyaml.org/" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://LICENSE;md5=6d8242660a8371add5fe547adf083079" -DEPENDS += "python3-cython-native" - SRC_URI[sha256sum] = "d584d9ec91ad65861cc08d42e834324ef890a082e591037abe114850ff7bbc3e" SRC_URI += "\ @@ -16,7 +14,7 @@ SRC_URI[test.sha256sum] = "b6a8a2825d89fdc8aee226560f66b8196e872012a0ea7118cbef1 UPSTREAM_CHECK_PYPI_PACKAGE = "PyYAML" -inherit pypi python_setuptools_build_meta ptest +inherit pypi python_setuptools_build_meta ptest cython PACKAGECONFIG ?= "libyaml" PACKAGECONFIG[libyaml] = "--with-libyaml,--without-libyaml,libyaml" From patchwork Thu Nov 7 17:24:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 52178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BCFCD5D67E for ; Thu, 7 Nov 2024 17:24:16 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.78034.1731000255545604102 for ; Thu, 07 Nov 2024 09:24:15 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 20325497 for ; Thu, 7 Nov 2024 09:24:45 -0800 (PST) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C71F03F6A8 for ; Thu, 7 Nov 2024 09:24:14 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 8/8] python_mesonpy: remove cython-specific INSANE_SKIPs Date: Thu, 7 Nov 2024 17:24:04 +0000 Message-Id: <20241107172404.4017047-8-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241107172404.4017047-1-ross.burton@arm.com> References: <20241107172404.4017047-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 07 Nov 2024 17:24:16 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/206849 These INSANE_SKIPs are specific to cython and unrelated to mesonpy, so remove them. The new cython.bbclass should resolve the cause. Signed-off-by: Ross Burton --- meta/classes-recipe/python_mesonpy.bbclass | 4 ---- 1 file changed, 4 deletions(-) diff --git a/meta/classes-recipe/python_mesonpy.bbclass b/meta/classes-recipe/python_mesonpy.bbclass index 131fa74bede..81c087c7c75 100644 --- a/meta/classes-recipe/python_mesonpy.bbclass +++ b/meta/classes-recipe/python_mesonpy.bbclass @@ -33,10 +33,6 @@ def mesonpy_get_args(d): PEP517_BUILD_OPTS = "-Cbuilddir='${B}' ${@mesonpy_get_args(d)}" -# Python pyx -> c -> so build leaves absolute build paths in the code -INSANE_SKIP:${PN} += "buildpaths" -INSANE_SKIP:${PN}-src += "buildpaths" - python_mesonpy_do_configure () { python_pep517_do_configure }