| Message ID | 20251029202250.270947-1-tgamblin@baylibre.com |
|---|---|
| Headers | show |
| Series | python3: upgrade 3.13.9 -> 3.14.0 | expand |
On Wed, 29 Oct 2025 at 21:22, Trevor Gamblin via lists.openembedded.org <tgamblin=baylibre.com@lists.openembedded.org> wrote: > > This is a fairly significant upgrade, so I'd like to request some feedback from > others in the community, especially on the new test skips and _sysconfig_vars > reproducibility solutions to see if anyone has better ideas. I've included some > justification in each patch based on what I found during my investigation. Thanks for working on this! It's good to have several people (and not just me) who can handle major python updates. > I'm also having a couple of issues with testing that maybe someone can identify > the problem with. In particular, the python3 ptests, reproducibility, and > buildall-qemu results seem OK when I test, but when I try to build ptest images > for recipes that depend on the python3-stringold package, such as > core-image-ptest-python3-license-expression, I see errors like: > > |ERROR: python3-unittest-automake-output-0.3-r0 do_package: QA Issue: python3-unittest-automake-output: Files/directories were installed but not shipped in any package: ... > | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/__init__.py I don't think it's to do with stringold. Rather, python3targetconfig.bbclass no longer does what it's supposed to (which is trick native python into reporting target configuration). You can reproduce for example by building target rpm, and then, from its $WORKDIR: with python 3.13 alex@Zen2:/srv/storage/alex/yocto/build-64/tmp/work/x86-64-v3-poky-linux/rpm/4.20.1$ PYTHONPATH=./recipe-sysroot/usr/lib/python-sysconfigdata _PYTHON_SYSCONFIGDATA_NAME="_sysconfigdata" ./recipe-sysroot-native/usr/bin/python3-native/python3 -c "import sysconfig;print(sysconfig.get_path('platlib'))" /usr/lib/python3.13/site-packages with python 3.14 alex@Zen2:/srv/storage/alex/yocto/build-64/tmp/work/x86-64-v3-poky-linux/rpm/4.20.1$ PYTHONPATH=./recipe-sysroot/usr/lib/python-sysconfigdata _PYTHON_SYSCONFIGDATA_NAME="_sysconfigdata" ./recipe-sysroot-native/usr/bin/python3-native/python3 -c "import sysconfig;print(sysconfig.get_path('platlib'))" /srv/storage/alex/yocto/build-64/tmp/work/x86-64-v3-poky-linux/rpm/4.20.1/recipe-sysroot-native/usr/lib/python3.14/site-packages What is supposed to happen is native python should read ./recipe-sysroot/usr/lib/python-sysconfigdata/_sysconfigdata.py with these tweaks and use that to report the directory values. So either the file no longer has correct values, or sysconfig has been somehow refactored to calculate the paths in some other way. I have to stop for now, but yes, cross building python is horrible. > ...1-test_cmd-skip-bang-completion-test.patch | 31 + > ...-test_unix_console.test_cursor_back_.patch | 46 + These two new patches should also include an upstream ticket links that report the issues probably. Alex
This is a fairly significant upgrade, so I'd like to request some feedback from others in the community, especially on the new test skips and _sysconfig_vars reproducibility solutions to see if anyone has better ideas. I've included some justification in each patch based on what I found during my investigation. I'm also having a couple of issues with testing that maybe someone can identify the problem with. In particular, the python3 ptests, reproducibility, and buildall-qemu results seem OK when I test, but when I try to build ptest images for recipes that depend on the python3-stringold package, such as core-image-ptest-python3-license-expression, I see errors like: |ERROR: python3-unittest-automake-output-0.3-r0 do_package: QA Issue: python3-unittest-automake-output: Files/directories were installed but not shipped in any package: | /home | /home/tgamblin | /home/tgamblin/workspace | /home/tgamblin/workspace/yocto | /home/tgamblin/workspace/yocto/openembedded-core | /home/tgamblin/workspace/yocto/openembedded-core/build | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3 | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14 | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao-0.3.dist-info | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/__init__.py | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/pytest.py | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/unittest | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/__pycache__ | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/unittest/__init__.py | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/unittest/__main__.py | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/unittest/__pycache__ | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/unittest/__pycache__/__init__.cpython-314.pyc | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/unittest/__pycache__/__main__.cpython-314.pyc | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/__pycache__/__init__.cpython-314.pyc | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao/__pycache__/pytest.cpython-314.pyc | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao-0.3.dist-info/entry_points.txt | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao-0.3.dist-info/WHEEL | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao-0.3.dist-info/METADATA | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao-0.3.dist-info/licenses | /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/recipe-sysroot-native/usr/lib/python3.14/site-packages/putao-0.3.dist-info/licenses/LICENSE |Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install. |python3-unittest-automake-output: 34 installed and not shipped files. [installed-vs-shipped] The full list of failures in this example is: |Summary: 10 tasks failed: | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-unittest-automake-output_0.3.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-unittest-automake-output/0.3/temp/log.do_package.264314 | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-pretend_1.0.9.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-pretend/1.0.9/temp/log.do_package.264321 | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-booleanpy_5.0.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-booleanpy/5.0/temp/log.do_package.264320 | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-iniconfig_2.1.0.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-iniconfig/2.1.0/temp/log.do_package.264322 | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-pytest_8.4.2.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-pytest/8.4.2/temp/log.do_package.264315 | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-colorama_0.4.6.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-colorama/0.4.6/temp/log.do_package.264324 | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-pluggy_1.6.0.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-pluggy/1.6.0/temp/log.do_package.264323 | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-license-expression_30.4.4.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-license-expression/30.4.4/temp/log.do_package.264345 | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-packaging_25.0.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-packaging/25.0/temp/log.do_package.264326 | /home/tgamblin/workspace/yocto/openembedded-core/meta/recipes-devtools/python/python3-pygments_2.19.2.bb:do_package | log: /home/tgamblin/workspace/yocto/openembedded-core/build/tmp/work/x86-64-v3-poky-linux/python3-pygments/2.19.2/temp/log.do_package.264331 |Summary: There were 6 WARNING messages. |Summary: There were 20 ERROR messages, returning a non-zero exit code. The logs essentially mirror the QA error from above, though. The last thing to highlight for discussion is the renaming of the python3-stringold package split itself - in 3.13.9 (our current version) and older, many string functions are contained in a string.py module (reflected in the upstream source at Lib/string.py). As of 3.14.0, this has been changed to be a directory Lib/string/ with an __init__.py and the new templatelib.py submodule, where the latter contains the new t-string functionality. I think this changeover marks a good time to migrate from using python3-stringold to python3-string everywhere, but maybe there's a better naming convention. If that's OK though, I will make the change and submit a corresponding patch for meta-openembedded once the final patches for this go out. Trevor Gamblin (1): python3: upgrade 3.13.9 -> 3.14.0 meta/classes-recipe/python3-dir.bbclass | 2 +- ...r-OpenSSL-3.4-and-add-it-to-multissl.patch | 1452 ----------------- ...-use-prefix-value-from-build-configu.patch | 35 +- ...sts-due-to-load-variability-on-YP-AB.patch | 37 +- ...ctive_children-skip-problematic-test.patch | 17 +- ...1-test_cmd-skip-bang-completion-test.patch | 31 + ...-test_unix_console.test_cursor_back_.patch | 46 + ...kip-test_sysconfig.test_sysconfigdat.patch | 32 + .../python/python3/python3-manifest.json | 3 +- .../{python3_3.13.9.bb => python3_3.14.0.bb} | 20 +- 10 files changed, 191 insertions(+), 1484 deletions(-) delete mode 100644 meta/recipes-devtools/python/python3/0001-Generate-data-for-OpenSSL-3.4-and-add-it-to-multissl.patch create mode 100644 meta/recipes-devtools/python/python3/0001-test_cmd-skip-bang-completion-test.patch create mode 100644 meta/recipes-devtools/python/python3/0001-test_pyrepl-skip-test_unix_console.test_cursor_back_.patch create mode 100644 meta/recipes-devtools/python/python3/0001-test_sysconfig-skip-test_sysconfig.test_sysconfigdat.patch rename meta/recipes-devtools/python/{python3_3.13.9.bb => python3_3.14.0.bb} (95%)