mbox series

[RFC,0/4] Enable Python free-threading

Message ID 20260326150225.2083962-1-zboszor@gmail.com
Headers show
Series Enable Python free-threading | expand

Message

Böszörményi Zoltán March 26, 2026, 2:35 p.m. UTC 
Enable Python free-threading and fix some obvious fallouts.
See https://docs.python.org/3/howto/free-threading-python.html

Quote:
==================================================================
Python support for free threading

Starting with the 3.13 release, CPython has support for a build of
Python called free threading where the global interpreter lock
(GIL) is disabled. Free-threaded execution allows for full
utilization of the available processing power by running threads
in parallel on available CPU cores. While not all software will
benefit from this automatically, programs designed with threading
in mind will run faster on multi-core hardware.

Some third-party packages, in particular ones with an extension
module, may not be ready for use in a free-threaded build, and
will re-enable the GIL.
==================================================================

See also for the the post-release announcement on Phoronix and
a benchmark:
https://www.phoronix.com/news/Python-3.14
https://www.phoronix.com/review/python-314-benchmarks

The reasons behind this series are:

* Python free-threading makes threaded Python scripts faster by
  not stalling each other while one of them is parsed.

* Even with the latest 3.13.12, highly threaded scripts may cause
  crashes in the Python interpreter with a backtrace ending similarly
  as below and Google search results pointing to a so called
  "import storm" that makes Python unstable.

  Thread 0x00007f90ebfff6c0 (most recent call first):
  File "<frozen importlib._bootstrap>", line 446 in cb
  File "<frozen importlib._bootstrap>", line 1357 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1415 in _handle_fromlist
  ...

  I was interested to see whether Python 3.14 with free-threading enabled
  can fix it.

This series builds on the previous v2 series with the cover letter
entitled "Use PYTHON_DIR consistently".

Unfortunately, this series by itself is not enough and a lot of recipes
won't build:

* lldb in clang uses embedded Python code in C/C++ and uses the
  internals of the PyObject implementation, which changes when
  free-threading is enabled. As a result, lldb won't build.
  It may be possible to conditionally disable lldb.

* Many Python modules with binary extensions similary do not build
  against the free-threaded Python.

* Many Python modules, although build, do not work correctly.
  Usually they include thio, or something similarly not quite production
  level indicator line in their PKG-INFO or pyproject.toml

  "Programming Language :: Python :: Free Threading :: 1 - Unstable"

In other words: DO NOT TRY THIS AT HOME.