diff mbox series

[v2] python3-cython: make generated source file be reproducible

Message ID 20250917135047.2957170-1-hongxu.jia@windriver.com
State Accepted, archived
Commit 61d98d12eca1c7bdf3b7387a820c83d3b8fad965
Headers show
Series [v2] python3-cython: make generated source file be reproducible | expand

Commit Message

Hongxu Jia Sept. 17, 2025, 1:50 p.m. UTC 
While python3 module use cython to build library, the generated source file
is not stable at each build and made the generated library not be reproducible

This commit replaces un-predictable string with hardcode string in generated
source file to assure the generated library should be reproducible

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 ...ictable-build-path-prefix-with-hardc.patch | 72 +++++++++++++++++++
 .../python/python3-cython_3.1.3.bb            |  4 ++
 2 files changed, 76 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-cython/0001-Replace-not-predictable-build-path-prefix-with-hardc.patch

Comments

Ross Burton Sept. 22, 2025, 5:07 p.m. UTC | #1 
On 17 Sep 2025, at 14:50, hongxu via lists.openembedded.org <hongxu.jia=eng.windriver.com@lists.openembedded.org> wrote:
> 
> While python3 module use cython to build library, the generated source file
> is not stable at each build and made the generated library not be reproducible
> 
> This commit replaces un-predictable string with hardcode string in generated
> source file to assure the generated library should be reproducible

I see that python3-frozenlist already inherits the cython class, which should be fixing this problem. However that class searches for just .c files, whereas frozenpath uses .cpp, so that should be fixed.

I did start to submit a change to Cython to support path remapping:

https://github.com/cython/cython/pull/6489

But this was obsoleted by cython writing out _relative_ paths when building:

https://github.com/cython/cython/pull/6341

Of course then python3-build came along and started doing builds in /tmp so the relative paths became absolute again as the files are not under the source anymore…

So I’d say we need to:

1) fix cython.bbclass to also fix .cpp files. This will resolve your reproducibility problem in python3-frozenlist and hopefully others. Was this patch needed for any other recipes?

2) Engage with upstream to resolve this properly. I’ve just commented on the new PR you opened mentioning my prior art, hopefully this has some traction with upstream shortly.

Thanks,
Ross
diff mbox series

Patch

diff --git a/meta/recipes-devtools/python/python3-cython/0001-Replace-not-predictable-build-path-prefix-with-hardc.patch b/meta/recipes-devtools/python/python3-cython/0001-Replace-not-predictable-build-path-prefix-with-hardc.patch
new file mode 100644
index 0000000000..1fbbd8fd48
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-cython/0001-Replace-not-predictable-build-path-prefix-with-hardc.patch
@@ -0,0 +1,72 @@ 
+From 1690c505f1387e1884565021991a162e2f88f2b9 Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Wed, 17 Sep 2025 01:42:08 -0700
+Subject: [PATCH] Replace not predictable build path prefix with hardcode
+ string in the generated output file
+
+The build path may contain tmp dir which is not predictable, it caused
+the generated output file is not stable at each build and made
+the generated library is not reproducible [1] between builds
+
+vim frozenlist/_frozenlist.cpp
+...
+/* BEGIN: Cython Metadata
+{
+    "distutils": {
+        "depends": [],
+        "language": "c++",
+        "name": "frozenlist._frozenlist",
+        "sources": [
+            "/tmp/.tmp-frozenlist-pep517-cfdvygni/src/frozenlist/_frozenlist.pyx"
+        ]
+    },
+    "module_name": "frozenlist._frozenlist"
+}
+END: Cython Metadata */
+...
+
+Replace build path prefix with hardcode `build_path', it is no harm to
+tweak comments in source file, after applied this commit,
+vim frozenlist/_frozenlist.cpp
+...
+/* BEGIN: Cython Metadata
+{
+    "distutils": {
+        "depends": [],
+        "language": "c++",
+        "name": "frozenlist._frozenlist",
+        "sources": [
+            "build_path/frozenlist/_frozenlist.pyx"
+        ]
+    },
+    "module_name": "frozenlist._frozenlist"
+}
+END: Cython Metadata */
+...
+
+[1] https://reproducible-builds.org/
+
+Upstream-Status: Submitted [https://github.com/cython/cython/pull/7162]
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ Cython/Compiler/ModuleNode.py | 4 +++-
+ 1 file changed, 3 insertions(+), 1 deletion(-)
+
+diff --git a/Cython/Compiler/ModuleNode.py b/Cython/Compiler/ModuleNode.py
+index 6672cb986..b1123515e 100644
+--- a/Cython/Compiler/ModuleNode.py
++++ b/Cython/Compiler/ModuleNode.py
+@@ -779,7 +779,9 @@ class ModuleNode(Nodes.Node, Nodes.BlockNode):
+         code.put_generated_by()
+         if metadata:
+             code.putln("/* BEGIN: Cython Metadata")
+-            code.putln(json.dumps(metadata, indent=4, sort_keys=True))
++            _metadata = json.dumps(metadata, indent=4, sort_keys=True)
++            _metadata = _metadata.replace(os.getcwd(), 'build_path')
++            code.putln(_metadata)
+             code.putln("END: Cython Metadata */")
+             code.putln("")
+ 
+-- 
+2.49.0
+
diff --git a/meta/recipes-devtools/python/python3-cython_3.1.3.bb b/meta/recipes-devtools/python/python3-cython_3.1.3.bb
index dcb61a3634..3b2835d9c5 100644
--- a/meta/recipes-devtools/python/python3-cython_3.1.3.bb
+++ b/meta/recipes-devtools/python/python3-cython_3.1.3.bb
@@ -11,6 +11,10 @@  SRC_URI[sha256sum] = "10ee785e42328924b78f75a74f66a813cb956b4a9bc91c44816d089d59
 
 inherit pypi setuptools3 cython
 
+SRC_URI += " \
+    file://0001-Replace-not-predictable-build-path-prefix-with-hardc.patch \
+"
+
 # No need to depend on self
 DEPENDS:remove = "python3-cython-native"