diff mbox series

[meta-python,3/5] python3-fastnumbers: fix denoise of large floats when built with clang

Message ID 20260912235141.2355852-3-khem.raj@oss.qualcomm.com
State New
Headers show
Series [meta-python,1/5] python3-betamax: skip the live-network hook tests in ptest | expand

Commit Message

Khem Raj Sept. 12, 2026, 11:51 p.m. UTC
float_as_int_without_noise() returns early when the floored value fits
in a long, checking it with floor_val == static_cast<long>(floor_val).
For values outside the range of long the conversion is undefined
behavior; clang -O2 folds the round trip into "floor_val is integral",
which is always true, so the denoising is skipped:

  FAIL: tests/test_fastnumbers_examples.py:test_try_real
  FAIL: tests/test_fastnumbers_examples.py:test_try_forceint
  assert 3452999999999999737856 == 3453000000000000000000

Add a patch that only does the conversion when the value is in range.
A reduced reproducer returns the wrong answer with the recipe's clang++
at -O2 and the right one with the fix (and with -O0 or gcc).

AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
---
 ...efined-double-to-long-conversion-in-.patch | 46 +++++++++++++++++++
 .../python/python3-fastnumbers_5.2.0.bb       |  1 +
 2 files changed, 47 insertions(+)
 create mode 100644 meta-python/recipes-devtools/python/python3-fastnumbers/0001-parser-avoid-undefined-double-to-long-conversion-in-.patch
diff mbox series

Patch

diff --git a/meta-python/recipes-devtools/python/python3-fastnumbers/0001-parser-avoid-undefined-double-to-long-conversion-in-.patch b/meta-python/recipes-devtools/python/python3-fastnumbers/0001-parser-avoid-undefined-double-to-long-conversion-in-.patch
new file mode 100644
index 0000000000..52275c1d8e
--- /dev/null
+++ b/meta-python/recipes-devtools/python/python3-fastnumbers/0001-parser-avoid-undefined-double-to-long-conversion-in-.patch
@@ -0,0 +1,46 @@ 
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: Khem Raj <khem.raj@oss.qualcomm.com>
+Date: Sat, 12 Sep 2026 00:00:00 -0700
+Subject: [PATCH] parser: avoid undefined double to long conversion in
+ float_as_int_without_noise
+
+float_as_int_without_noise() returns early when the floored value fits
+in a long:
+
+    if (floor_val == static_cast<long>(floor_val))
+
+For values outside the range of long (e.g. 3.453e21) the conversion is
+undefined behavior. clang (-O2) takes advantage of that and folds the
+round trip into "floor_val is integral", which is always true, so the
+denoising is skipped and try_real(3.453e21, denoise=True) returns
+3452999999999999737856 instead of 3453000000000000000000:
+
+  tests/test_fastnumbers_examples.py::test_try_real FAILED
+  tests/test_fastnumbers_examples.py::test_try_forceint FAILED
+
+Only do the conversion when the value is within the range of long.
+
+Upstream-Status: Pending
+Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
+---
+ src/cpp/parser.cpp | 8 +++++++-
+ 1 file changed, 7 insertions(+), 1 deletion(-)
+
+--- a/src/cpp/parser.cpp
++++ b/src/cpp/parser.cpp
+@@ -92,8 +92,14 @@
+ 
+     // If the given float can fit a C long without loss then no need
+     // to go through the below rounding steps.
++    // Converting a double that is outside the range of long is undefined
++    // behavior, and clang uses that to fold this check into "floor_val is
++    // integral", which is always true and skips the denoising below for large
++    // values. Make sure the value is in range before converting.
+     const double floor_val = std::floor(val);
+-    if (floor_val == static_cast<long>(floor_val)) {
++    constexpr double long_limit = -static_cast<double>(std::numeric_limits<long>::min());
++    if (floor_val >= -long_limit && floor_val < long_limit
++        && floor_val == static_cast<long>(floor_val)) {
+         return val_int;
+     }
+ 
diff --git a/meta-python/recipes-devtools/python/python3-fastnumbers_5.2.0.bb b/meta-python/recipes-devtools/python/python3-fastnumbers_5.2.0.bb
index 65d7fcea3f..d067e44938 100644
--- a/meta-python/recipes-devtools/python/python3-fastnumbers_5.2.0.bb
+++ b/meta-python/recipes-devtools/python/python3-fastnumbers_5.2.0.bb
@@ -4,6 +4,7 @@  LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://LICENSE;md5=d32431d1b650010945da4e078011c8fa"
 
 
+SRC_URI += "file://0001-parser-avoid-undefined-double-to-long-conversion-in-.patch"
 SRC_URI[sha256sum] = "07266a2fca9e08eeb5a6c70be4c8db3637db9b930ea5198facbbb9b0b31d4d03"
 
 inherit pypi python_setuptools_build_meta ptest-python-pytest