| Message ID | cover.1645478020.git.tim.orling@konsulko.com |
|---|---|
| State | Not Applicable, archived |
| Headers | show |
this is going into dep loops Try bitbake python3-wheel-native On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> wrote: > > Upstream Python is moving to pyproject.toml rather than setup.py and > wheels rather than eggs. The main place this is documented is in > PEP-517[1]. Installing packages with setup.py install is deprecated. > > We have a choice with the 'kirkstone' LTS release of introducing this > series (and fixing any fall out during the stabilization period) or > living with deprecated Python packaging for the duration of the LTS (at > a minimum 2 years). This seems to be a compelling enough reason to bring > in this series, even if it results in a slight slip of the 'feature > freeze', which is officially tomorrow, Monday 21 Feb, 2022. > > This series bootstraps python3-setuptools-native, > python3-flit-core-native, python3-wheel-native and python3-pip-native, > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows all > other recipes to be installed with pip, as intended by upstream. > > Three new classes are introduced: > (1) flit_core which builds PEP-517 compliant packages which use > flit_core.buildapi as their build-backend. > (2) setuptools_build_meta which builds PEP-517 compliant packages which use > setuptools_build_meta as their build-backend. > (3) pip_install_wheel, which simply installs wheels built by other > classes. > > The original setuptools3.bbclass do_compile is modified to > 'setup.py bdist_wheel' (from python3-wheel-native) rather than the deprecated > 'setup.py build' mechanism. > > The original setuptools3.bbclass do_install is modified to > use the pip_install_wheel.bbclass to install wheels with pip rather than > the deprecated 'setup.py install' mechanism. > > For pip install, we disable dependency checking (--no-deps) because it > proves to be brittle in our environment without providing full wheel > cache that seems like overhead that we do not need. We also disable > fetching wheels from pypi (--no-index). This is no different than our > normal workflow of requiring dependencies to be defined in a recipe. > > Missing from this series is documentation of the new features, classes > and variables (this will be done under separate cover). > > Also desired and previously discussed in YP meetings is a helper > function to warn that a recipe which currently inherits setuptools3 > but has a pyproject.toml and defines a build-system.build-backend should > instead use of the new PEP-517 classes. This function is still in progress. > > During testing, recipes which use python3-setuptools-scm are failing to > determine the version of the python package, For several recipes, the fix > was to use pyproject.toml and PEP-517 build backends. To be sure, other recipes > are still going to fail for the short term, until the root cause can be > determined. This was not seen again in v2 builds. > > Another PEP-517 build-backend, poetry.core.masonry.api has been > identified in recipes in meta-python, but not in oe-core. The > python3-poetry-core recipe is WIP, but brings in a number of > dependencies that are perhaps not worth it to bring into oe-core at this > time. Rather we can support these recipes in meta-python? > > Fixes in v2 include passing --root=${D} to pip install (rather than --target) > and also passing --prefix so that scripts are (usually) installed in the proper > location. pip install also 'helpfully' overwrites python3 intepreter with nativepython3 > in #!, so add a fix to return the values to what is actually correct (similar to what is > done in setuptools3.bbclass). Also several recipes still needed PIP_INSTALL_PACKAGE defined. > > Also added self as maintainer of new recipes. > > [YOCTO #14638] > > [1] https://www.python.org/dev/peps/pep-0517/ > > The following changes since commit 41ae26ecab731f2fb6593df143476c94183be209: > > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) > > are available in the Git repository at: > > git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels > > Tim Orling (32): > python3-wheel: move 0.37.0 from meta-python > python3-flit-core: add recipe for 3.6.0 > flit_core.bbclass: add helper for newer python packaging > python3-wheel: inherit flit_core > pip_install_wheel.bbclass: add helper class > setuptools_build_meta.bbclass: add helper class > python3-pip: inherit setuptools_build_meta > python3-attrs: inherit setuptools_build_meta > python3-git: inherit setuptools_build_meta > python3-pytest: inherit setuptools_build_meta > python3-setuptools-scm: inherit setuptools_build_meta > python3-zipp: inherit setuptools_build_meta > python3-iniconfig: inherit setuptools_build_meta > python3-py: inherit setuptools_build_meta > python3-pluggy: inherit setuptools_build_meta > python3-setuptools: inherit setuptools_base_meta > setuptools3.bbclass: refactor for wheels > python3-more-itertools: set PIP_INSTALL_PACKAGE > meson: inherit setuptools_build_meta > python3-libarchive-d: set PIP_INSTALL_PACKAGE > python3-smartypants: patch hash bang to python3 > python3-scons: merge -native recipe > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE > python3-magic: set PIP_INSTALL_PACKAGE > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER > asciidoc: set PIP_INSTALL_PACKAGE > gi-docgen: set PIP_INSTALL_PACKAGE > numpy: set PIP_INSTALL_PACKAGE > python3-dbusmock: set PIP_INSTALL_PACKAGE > python3-mako: inherit setuptools_build_meta > python3-packaging: inherit setuptools_build_meta > python3-nose: drop recipe > > meta/classes/flit_core.bbclass | 16 +++ > meta/classes/pip_install_wheel.bbclass | 37 +++++++ > meta/classes/setuptools3.bbclass | 20 ++-- > meta/classes/setuptools_build_meta.bbclass | 18 ++++ > meta/conf/distro/include/maintainers.inc | 3 +- > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- > meta/recipes-devtools/python/python-nose.inc | 18 ---- > .../python/python-subunit.inc | 14 --- > .../python/python3-attrs_21.4.0.bb | 2 +- > .../python/python3-dbusmock_0.25.0.bb | 1 + > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ > .../python/python3-git_3.1.26.bb | 2 +- > .../python/python3-iniconfig_1.1.1.bb | 4 +- > .../python/python3-libarchive-c_4.0.bb | 2 + > .../python/python3-magic_0.4.25.bb | 1 + > .../python/python3-mako_1.1.6.bb | 2 +- > .../python/python3-more-itertools_8.12.0.bb | 2 + > .../python/python3-nose_1.3.7.bb | 2 - > .../python/python3-numpy_1.22.2.bb | 1 + > .../python/python3-packaging_21.3.bb | 2 +- > .../python/python3-pip_22.0.3.bb | 19 +++- > .../python/python3-pluggy_1.0.0.bb | 2 +- > .../python/python3-py_1.11.0.bb | 2 +- > .../python/python3-pytest_7.0.1.bb | 4 +- > .../python/python3-scons-native_4.3.0.bb | 7 -- > .../python/python3-scons_4.3.0.bb | 13 ++- > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- > .../python/python3-setuptools_59.5.0.bb | 19 +++- > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ > .../python/python3-smartypants_2.0.0.bb | 1 + > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ > .../python/python3-strict-rfc3339_0.7.bb | 4 +- > .../python/python3-subunit_1.4.0.bb | 17 ++- > ...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++ > .../python/python3-wheel_0.37.1.bb | 24 +++++ > .../python/python3-zipp_3.7.0.bb | 2 +- > .../asciidoc/asciidoc_10.1.1.bb | 1 + > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- > 39 files changed, 412 insertions(+), 78 deletions(-) > create mode 100644 meta/classes/flit_core.bbclass > create mode 100644 meta/classes/pip_install_wheel.bbclass > create mode 100644 meta/classes/setuptools_build_meta.bbclass > delete mode 100644 meta/recipes-devtools/python/python-nose.inc > delete mode 100644 meta/recipes-devtools/python/python-subunit.inc > create mode 100644 meta/recipes-devtools/python/python3-flit-core_3.6.0.bb > delete mode 100644 meta/recipes-devtools/python/python3-nose_1.3.7.bb > delete mode 100644 meta/recipes-devtools/python/python3-scons-native_4.3.0.bb > create mode 100644 meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch > create mode 100644 meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch > create mode 100644 meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch > create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => bmap-tools_git.bb} (86%) > > -- > 2.30.2 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#162076): https://lists.openembedded.org/g/openembedded-core/message/162076 > Mute This Topic: https://lists.openembedded.org/mt/89304007/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com> wrote: > this is going into dep loops Try > > bitbake python3-wheel-native It shouldn’t https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels DEPENDS:remove:class-native = "python3-pip-native" do_install:class-native () { # We need to bootstrap python3-wheel-native install -d ${D}${PYTHON_SITEPACKAGES_DIR} PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ bbfatal_log "Failed to install"} > > > On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> wrote: > > > > Upstream Python is moving to pyproject.toml rather than setup.py and > > wheels rather than eggs. The main place this is documented is in > > PEP-517[1]. Installing packages with setup.py install is deprecated. > > > > We have a choice with the 'kirkstone' LTS release of introducing this > > series (and fixing any fall out during the stabilization period) or > > living with deprecated Python packaging for the duration of the LTS (at > > a minimum 2 years). This seems to be a compelling enough reason to bring > > in this series, even if it results in a slight slip of the 'feature > > freeze', which is officially tomorrow, Monday 21 Feb, 2022. > > > > This series bootstraps python3-setuptools-native, > > python3-flit-core-native, python3-wheel-native and python3-pip-native, > > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows all > > other recipes to be installed with pip, as intended by upstream. > > > > Three new classes are introduced: > > (1) flit_core which builds PEP-517 compliant packages which use > > flit_core.buildapi as their build-backend. > > (2) setuptools_build_meta which builds PEP-517 compliant packages which > use > > setuptools_build_meta as their build-backend. > > (3) pip_install_wheel, which simply installs wheels built by other > > classes. > > > > The original setuptools3.bbclass do_compile is modified to > > 'setup.py bdist_wheel' (from python3-wheel-native) rather than the > deprecated > > 'setup.py build' mechanism. > > > > The original setuptools3.bbclass do_install is modified to > > use the pip_install_wheel.bbclass to install wheels with pip rather than > > the deprecated 'setup.py install' mechanism. > > > > For pip install, we disable dependency checking (--no-deps) because it > > proves to be brittle in our environment without providing full wheel > > cache that seems like overhead that we do not need. We also disable > > fetching wheels from pypi (--no-index). This is no different than our > > normal workflow of requiring dependencies to be defined in a recipe. > > > > Missing from this series is documentation of the new features, classes > > and variables (this will be done under separate cover). > > > > Also desired and previously discussed in YP meetings is a helper > > function to warn that a recipe which currently inherits setuptools3 > > but has a pyproject.toml and defines a build-system.build-backend should > > instead use of the new PEP-517 classes. This function is still in > progress. > > > > During testing, recipes which use python3-setuptools-scm are failing to > > determine the version of the python package, For several recipes, the fix > > was to use pyproject.toml and PEP-517 build backends. To be sure, other > recipes > > are still going to fail for the short term, until the root cause can be > > determined. This was not seen again in v2 builds. > > > > Another PEP-517 build-backend, poetry.core.masonry.api has been > > identified in recipes in meta-python, but not in oe-core. The > > python3-poetry-core recipe is WIP, but brings in a number of > > dependencies that are perhaps not worth it to bring into oe-core at this > > time. Rather we can support these recipes in meta-python? > > > > Fixes in v2 include passing --root=${D} to pip install (rather than > --target) > > and also passing --prefix so that scripts are (usually) installed in the > proper > > location. pip install also 'helpfully' overwrites python3 intepreter > with nativepython3 > > in #!, so add a fix to return the values to what is actually correct > (similar to what is > > done in setuptools3.bbclass). Also several recipes still needed > PIP_INSTALL_PACKAGE defined. > > > > Also added self as maintainer of new recipes. > > > > [YOCTO #14638] > > > > [1] https://www.python.org/dev/peps/pep-0517/ > > > > The following changes since commit > 41ae26ecab731f2fb6593df143476c94183be209: > > > > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) > > > > are available in the Git repository at: > > > > git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels > > > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels > > > > Tim Orling (32): > > python3-wheel: move 0.37.0 from meta-python > > python3-flit-core: add recipe for 3.6.0 > > flit_core.bbclass: add helper for newer python packaging > > python3-wheel: inherit flit_core > > pip_install_wheel.bbclass: add helper class > > setuptools_build_meta.bbclass: add helper class > > python3-pip: inherit setuptools_build_meta > > python3-attrs: inherit setuptools_build_meta > > python3-git: inherit setuptools_build_meta > > python3-pytest: inherit setuptools_build_meta > > python3-setuptools-scm: inherit setuptools_build_meta > > python3-zipp: inherit setuptools_build_meta > > python3-iniconfig: inherit setuptools_build_meta > > python3-py: inherit setuptools_build_meta > > python3-pluggy: inherit setuptools_build_meta > > python3-setuptools: inherit setuptools_base_meta > > setuptools3.bbclass: refactor for wheels > > python3-more-itertools: set PIP_INSTALL_PACKAGE > > meson: inherit setuptools_build_meta > > python3-libarchive-d: set PIP_INSTALL_PACKAGE > > python3-smartypants: patch hash bang to python3 > > python3-scons: merge -native recipe > > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE > > python3-magic: set PIP_INSTALL_PACKAGE > > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER > > asciidoc: set PIP_INSTALL_PACKAGE > > gi-docgen: set PIP_INSTALL_PACKAGE > > numpy: set PIP_INSTALL_PACKAGE > > python3-dbusmock: set PIP_INSTALL_PACKAGE > > python3-mako: inherit setuptools_build_meta > > python3-packaging: inherit setuptools_build_meta > > python3-nose: drop recipe > > > > meta/classes/flit_core.bbclass | 16 +++ > > meta/classes/pip_install_wheel.bbclass | 37 +++++++ > > meta/classes/setuptools3.bbclass | 20 ++-- > > meta/classes/setuptools_build_meta.bbclass | 18 ++++ > > meta/conf/distro/include/maintainers.inc | 3 +- > > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- > > meta/recipes-devtools/python/python-nose.inc | 18 ---- > > .../python/python-subunit.inc | 14 --- > > .../python/python3-attrs_21.4.0.bb | 2 +- > > .../python/python3-dbusmock_0.25.0.bb | 1 + > > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ > > .../python/python3-git_3.1.26.bb | 2 +- > > .../python/python3-iniconfig_1.1.1.bb | 4 +- > > .../python/python3-libarchive-c_4.0.bb | 2 + > > .../python/python3-magic_0.4.25.bb | 1 + > > .../python/python3-mako_1.1.6.bb | 2 +- > > .../python/python3-more-itertools_8.12.0.bb | 2 + > > .../python/python3-nose_1.3.7.bb | 2 - > > .../python/python3-numpy_1.22.2.bb | 1 + > > .../python/python3-packaging_21.3.bb | 2 +- > > .../python/python3-pip_22.0.3.bb | 19 +++- > > .../python/python3-pluggy_1.0.0.bb | 2 +- > > .../python/python3-py_1.11.0.bb | 2 +- > > .../python/python3-pytest_7.0.1.bb | 4 +- > > .../python/python3-scons-native_4.3.0.bb | 7 -- > > .../python/python3-scons_4.3.0.bb | 13 ++- > > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- > > .../python/python3-setuptools_59.5.0.bb | 19 +++- > > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ > > .../python/python3-smartypants_2.0.0.bb | 1 + > > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ > > .../python/python3-strict-rfc3339_0.7.bb | 4 +- > > .../python/python3-subunit_1.4.0.bb | 17 ++- > > ...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++ > > .../python/python3-wheel_0.37.1.bb | 24 +++++ > > .../python/python3-zipp_3.7.0.bb | 2 +- > > .../asciidoc/asciidoc_10.1.1.bb | 1 + > > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + > > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- > > 39 files changed, 412 insertions(+), 78 deletions(-) > > create mode 100644 meta/classes/flit_core.bbclass > > create mode 100644 meta/classes/pip_install_wheel.bbclass > > create mode 100644 meta/classes/setuptools_build_meta.bbclass > > delete mode 100644 meta/recipes-devtools/python/python-nose.inc > > delete mode 100644 meta/recipes-devtools/python/python-subunit.inc > > create mode 100644 meta/recipes-devtools/python/ > python3-flit-core_3.6.0.bb > > delete mode 100644 meta/recipes-devtools/python/python3-nose_1.3.7.bb > > delete mode 100644 meta/recipes-devtools/python/ > python3-scons-native_4.3.0.bb > > create mode 100644 > meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch > > create mode 100644 > meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch > > create mode 100644 > meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch > > create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb > > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => > bmap-tools_git.bb} (86%) > > > > -- > > 2.30.2 > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#162076): > https://lists.openembedded.org/g/openembedded-core/message/162076 > > Mute This Topic: https://lists.openembedded.org/mt/89304007/1997914 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > raj.khem@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > >
On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <ticotimo@gmail.com> wrote: > > > > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com> wrote: >> >> this is going into dep loops Try >> >> bitbake python3-wheel-native > > > It shouldn’t > https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels > I have been trying whats on master-next > DEPENDS:remove:class-native = "python3-pip-native" > > do_install:class-native () { > # We need to bootstrap python3-wheel-native > install -d ${D}${PYTHON_SITEPACKAGES_DIR} > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ > bbfatal_log "Failed to install" > } >> >> >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> wrote: >> > >> > Upstream Python is moving to pyproject.toml rather than setup.py and >> > wheels rather than eggs. The main place this is documented is in >> > PEP-517[1]. Installing packages with setup.py install is deprecated. >> > >> > We have a choice with the 'kirkstone' LTS release of introducing this >> > series (and fixing any fall out during the stabilization period) or >> > living with deprecated Python packaging for the duration of the LTS (at >> > a minimum 2 years). This seems to be a compelling enough reason to bring >> > in this series, even if it results in a slight slip of the 'feature >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. >> > >> > This series bootstraps python3-setuptools-native, >> > python3-flit-core-native, python3-wheel-native and python3-pip-native, >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows all >> > other recipes to be installed with pip, as intended by upstream. >> > >> > Three new classes are introduced: >> > (1) flit_core which builds PEP-517 compliant packages which use >> > flit_core.buildapi as their build-backend. >> > (2) setuptools_build_meta which builds PEP-517 compliant packages which use >> > setuptools_build_meta as their build-backend. >> > (3) pip_install_wheel, which simply installs wheels built by other >> > classes. >> > >> > The original setuptools3.bbclass do_compile is modified to >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than the deprecated >> > 'setup.py build' mechanism. >> > >> > The original setuptools3.bbclass do_install is modified to >> > use the pip_install_wheel.bbclass to install wheels with pip rather than >> > the deprecated 'setup.py install' mechanism. >> > >> > For pip install, we disable dependency checking (--no-deps) because it >> > proves to be brittle in our environment without providing full wheel >> > cache that seems like overhead that we do not need. We also disable >> > fetching wheels from pypi (--no-index). This is no different than our >> > normal workflow of requiring dependencies to be defined in a recipe. >> > >> > Missing from this series is documentation of the new features, classes >> > and variables (this will be done under separate cover). >> > >> > Also desired and previously discussed in YP meetings is a helper >> > function to warn that a recipe which currently inherits setuptools3 >> > but has a pyproject.toml and defines a build-system.build-backend should >> > instead use of the new PEP-517 classes. This function is still in progress. >> > >> > During testing, recipes which use python3-setuptools-scm are failing to >> > determine the version of the python package, For several recipes, the fix >> > was to use pyproject.toml and PEP-517 build backends. To be sure, other recipes >> > are still going to fail for the short term, until the root cause can be >> > determined. This was not seen again in v2 builds. >> > >> > Another PEP-517 build-backend, poetry.core.masonry.api has been >> > identified in recipes in meta-python, but not in oe-core. The >> > python3-poetry-core recipe is WIP, but brings in a number of >> > dependencies that are perhaps not worth it to bring into oe-core at this >> > time. Rather we can support these recipes in meta-python? >> > >> > Fixes in v2 include passing --root=${D} to pip install (rather than --target) >> > and also passing --prefix so that scripts are (usually) installed in the proper >> > location. pip install also 'helpfully' overwrites python3 intepreter with nativepython3 >> > in #!, so add a fix to return the values to what is actually correct (similar to what is >> > done in setuptools3.bbclass). Also several recipes still needed PIP_INSTALL_PACKAGE defined. >> > >> > Also added self as maintainer of new recipes. >> > >> > [YOCTO #14638] >> > >> > [1] https://www.python.org/dev/peps/pep-0517/ >> > >> > The following changes since commit 41ae26ecab731f2fb6593df143476c94183be209: >> > >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) >> > >> > are available in the Git repository at: >> > >> > git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels >> > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels >> > >> > Tim Orling (32): >> > python3-wheel: move 0.37.0 from meta-python >> > python3-flit-core: add recipe for 3.6.0 >> > flit_core.bbclass: add helper for newer python packaging >> > python3-wheel: inherit flit_core >> > pip_install_wheel.bbclass: add helper class >> > setuptools_build_meta.bbclass: add helper class >> > python3-pip: inherit setuptools_build_meta >> > python3-attrs: inherit setuptools_build_meta >> > python3-git: inherit setuptools_build_meta >> > python3-pytest: inherit setuptools_build_meta >> > python3-setuptools-scm: inherit setuptools_build_meta >> > python3-zipp: inherit setuptools_build_meta >> > python3-iniconfig: inherit setuptools_build_meta >> > python3-py: inherit setuptools_build_meta >> > python3-pluggy: inherit setuptools_build_meta >> > python3-setuptools: inherit setuptools_base_meta >> > setuptools3.bbclass: refactor for wheels >> > python3-more-itertools: set PIP_INSTALL_PACKAGE >> > meson: inherit setuptools_build_meta >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE >> > python3-smartypants: patch hash bang to python3 >> > python3-scons: merge -native recipe >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE >> > python3-magic: set PIP_INSTALL_PACKAGE >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER >> > asciidoc: set PIP_INSTALL_PACKAGE >> > gi-docgen: set PIP_INSTALL_PACKAGE >> > numpy: set PIP_INSTALL_PACKAGE >> > python3-dbusmock: set PIP_INSTALL_PACKAGE >> > python3-mako: inherit setuptools_build_meta >> > python3-packaging: inherit setuptools_build_meta >> > python3-nose: drop recipe >> > >> > meta/classes/flit_core.bbclass | 16 +++ >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ >> > meta/classes/setuptools3.bbclass | 20 ++-- >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ >> > meta/conf/distro/include/maintainers.inc | 3 +- >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- >> > .../python/python-subunit.inc | 14 --- >> > .../python/python3-attrs_21.4.0.bb | 2 +- >> > .../python/python3-dbusmock_0.25.0.bb | 1 + >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ >> > .../python/python3-git_3.1.26.bb | 2 +- >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- >> > .../python/python3-libarchive-c_4.0.bb | 2 + >> > .../python/python3-magic_0.4.25.bb | 1 + >> > .../python/python3-mako_1.1.6.bb | 2 +- >> > .../python/python3-more-itertools_8.12.0.bb | 2 + >> > .../python/python3-nose_1.3.7.bb | 2 - >> > .../python/python3-numpy_1.22.2.bb | 1 + >> > .../python/python3-packaging_21.3.bb | 2 +- >> > .../python/python3-pip_22.0.3.bb | 19 +++- >> > .../python/python3-pluggy_1.0.0.bb | 2 +- >> > .../python/python3-py_1.11.0.bb | 2 +- >> > .../python/python3-pytest_7.0.1.bb | 4 +- >> > .../python/python3-scons-native_4.3.0.bb | 7 -- >> > .../python/python3-scons_4.3.0.bb | 13 ++- >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ >> > .../python/python3-smartypants_2.0.0.bb | 1 + >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- >> > .../python/python3-subunit_1.4.0.bb | 17 ++- >> > ...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++ >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ >> > .../python/python3-zipp_3.7.0.bb | 2 +- >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- >> > 39 files changed, 412 insertions(+), 78 deletions(-) >> > create mode 100644 meta/classes/flit_core.bbclass >> > create mode 100644 meta/classes/pip_install_wheel.bbclass >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass >> > delete mode 100644 meta/recipes-devtools/python/python-nose.inc >> > delete mode 100644 meta/recipes-devtools/python/python-subunit.inc >> > create mode 100644 meta/recipes-devtools/python/python3-flit-core_3.6.0.bb >> > delete mode 100644 meta/recipes-devtools/python/python3-nose_1.3.7.bb >> > delete mode 100644 meta/recipes-devtools/python/python3-scons-native_4.3.0.bb >> > create mode 100644 meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch >> > create mode 100644 meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch >> > create mode 100644 meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch >> > create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => bmap-tools_git.bb} (86%) >> > >> > -- >> > 2.30.2 >> > >> > >> > -=-=-=-=-=-=-=-=-=-=-=- >> > Links: You receive all messages sent to this group. >> > View/Reply Online (#162076): https://lists.openembedded.org/g/openembedded-core/message/162076 >> > Mute This Topic: https://lists.openembedded.org/mt/89304007/1997914 >> > Group Owner: openembedded-core+owner@lists.openembedded.org >> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] >> > -=-=-=-=-=-=-=-=-=-=-=- >> >
On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <raj.khem@gmail.com> wrote: > On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <ticotimo@gmail.com> wrote: > > > > > > > > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com> wrote: > >> > >> this is going into dep loops Try > >> > >> bitbake python3-wheel-native > > > > > > It shouldn’t > > > https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels > > > > I have been trying whats on master-next > > I can't reproduce. Can you point me at an error log please? It might have something to do with sstate cache, but that's just a hunch. > > DEPENDS:remove:class-native = "python3-pip-native" > > > > do_install:class-native () { > > # We need to bootstrap python3-wheel-native > > install -d ${D}${PYTHON_SITEPACKAGES_DIR} > > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" > > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ > > bbfatal_log "Failed to install" > > } > >> > >> > >> > >> > >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> wrote: > >> > > >> > Upstream Python is moving to pyproject.toml rather than setup.py and > >> > wheels rather than eggs. The main place this is documented is in > >> > PEP-517[1]. Installing packages with setup.py install is deprecated. > >> > > >> > We have a choice with the 'kirkstone' LTS release of introducing this > >> > series (and fixing any fall out during the stabilization period) or > >> > living with deprecated Python packaging for the duration of the LTS > (at > >> > a minimum 2 years). This seems to be a compelling enough reason to > bring > >> > in this series, even if it results in a slight slip of the 'feature > >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. > >> > > >> > This series bootstraps python3-setuptools-native, > >> > python3-flit-core-native, python3-wheel-native and python3-pip-native, > >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows > all > >> > other recipes to be installed with pip, as intended by upstream. > >> > > >> > Three new classes are introduced: > >> > (1) flit_core which builds PEP-517 compliant packages which use > >> > flit_core.buildapi as their build-backend. > >> > (2) setuptools_build_meta which builds PEP-517 compliant packages > which use > >> > setuptools_build_meta as their build-backend. > >> > (3) pip_install_wheel, which simply installs wheels built by other > >> > classes. > >> > > >> > The original setuptools3.bbclass do_compile is modified to > >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than the > deprecated > >> > 'setup.py build' mechanism. > >> > > >> > The original setuptools3.bbclass do_install is modified to > >> > use the pip_install_wheel.bbclass to install wheels with pip rather > than > >> > the deprecated 'setup.py install' mechanism. > >> > > >> > For pip install, we disable dependency checking (--no-deps) because it > >> > proves to be brittle in our environment without providing full wheel > >> > cache that seems like overhead that we do not need. We also disable > >> > fetching wheels from pypi (--no-index). This is no different than our > >> > normal workflow of requiring dependencies to be defined in a recipe. > >> > > >> > Missing from this series is documentation of the new features, classes > >> > and variables (this will be done under separate cover). > >> > > >> > Also desired and previously discussed in YP meetings is a helper > >> > function to warn that a recipe which currently inherits setuptools3 > >> > but has a pyproject.toml and defines a build-system.build-backend > should > >> > instead use of the new PEP-517 classes. This function is still in > progress. > >> > > >> > During testing, recipes which use python3-setuptools-scm are failing > to > >> > determine the version of the python package, For several recipes, the > fix > >> > was to use pyproject.toml and PEP-517 build backends. To be sure, > other recipes > >> > are still going to fail for the short term, until the root cause can > be > >> > determined. This was not seen again in v2 builds. > >> > > >> > Another PEP-517 build-backend, poetry.core.masonry.api has been > >> > identified in recipes in meta-python, but not in oe-core. The > >> > python3-poetry-core recipe is WIP, but brings in a number of > >> > dependencies that are perhaps not worth it to bring into oe-core at > this > >> > time. Rather we can support these recipes in meta-python? > >> > > >> > Fixes in v2 include passing --root=${D} to pip install (rather than > --target) > >> > and also passing --prefix so that scripts are (usually) installed in > the proper > >> > location. pip install also 'helpfully' overwrites python3 intepreter > with nativepython3 > >> > in #!, so add a fix to return the values to what is actually correct > (similar to what is > >> > done in setuptools3.bbclass). Also several recipes still needed > PIP_INSTALL_PACKAGE defined. > >> > > >> > Also added self as maintainer of new recipes. > >> > > >> > [YOCTO #14638] > >> > > >> > [1] https://www.python.org/dev/peps/pep-0517/ > >> > > >> > The following changes since commit > 41ae26ecab731f2fb6593df143476c94183be209: > >> > > >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) > >> > > >> > are available in the Git repository at: > >> > > >> > git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels > >> > > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels > >> > > >> > Tim Orling (32): > >> > python3-wheel: move 0.37.0 from meta-python > >> > python3-flit-core: add recipe for 3.6.0 > >> > flit_core.bbclass: add helper for newer python packaging > >> > python3-wheel: inherit flit_core > >> > pip_install_wheel.bbclass: add helper class > >> > setuptools_build_meta.bbclass: add helper class > >> > python3-pip: inherit setuptools_build_meta > >> > python3-attrs: inherit setuptools_build_meta > >> > python3-git: inherit setuptools_build_meta > >> > python3-pytest: inherit setuptools_build_meta > >> > python3-setuptools-scm: inherit setuptools_build_meta > >> > python3-zipp: inherit setuptools_build_meta > >> > python3-iniconfig: inherit setuptools_build_meta > >> > python3-py: inherit setuptools_build_meta > >> > python3-pluggy: inherit setuptools_build_meta > >> > python3-setuptools: inherit setuptools_base_meta > >> > setuptools3.bbclass: refactor for wheels > >> > python3-more-itertools: set PIP_INSTALL_PACKAGE > >> > meson: inherit setuptools_build_meta > >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE > >> > python3-smartypants: patch hash bang to python3 > >> > python3-scons: merge -native recipe > >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE > >> > python3-magic: set PIP_INSTALL_PACKAGE > >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER > >> > asciidoc: set PIP_INSTALL_PACKAGE > >> > gi-docgen: set PIP_INSTALL_PACKAGE > >> > numpy: set PIP_INSTALL_PACKAGE > >> > python3-dbusmock: set PIP_INSTALL_PACKAGE > >> > python3-mako: inherit setuptools_build_meta > >> > python3-packaging: inherit setuptools_build_meta > >> > python3-nose: drop recipe > >> > > >> > meta/classes/flit_core.bbclass | 16 +++ > >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ > >> > meta/classes/setuptools3.bbclass | 20 ++-- > >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ > >> > meta/conf/distro/include/maintainers.inc | 3 +- > >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- > >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- > >> > .../python/python-subunit.inc | 14 --- > >> > .../python/python3-attrs_21.4.0.bb | 2 +- > >> > .../python/python3-dbusmock_0.25.0.bb | 1 + > >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ > >> > .../python/python3-git_3.1.26.bb | 2 +- > >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- > >> > .../python/python3-libarchive-c_4.0.bb | 2 + > >> > .../python/python3-magic_0.4.25.bb | 1 + > >> > .../python/python3-mako_1.1.6.bb | 2 +- > >> > .../python/python3-more-itertools_8.12.0.bb | 2 + > >> > .../python/python3-nose_1.3.7.bb | 2 - > >> > .../python/python3-numpy_1.22.2.bb | 1 + > >> > .../python/python3-packaging_21.3.bb | 2 +- > >> > .../python/python3-pip_22.0.3.bb | 19 +++- > >> > .../python/python3-pluggy_1.0.0.bb | 2 +- > >> > .../python/python3-py_1.11.0.bb | 2 +- > >> > .../python/python3-pytest_7.0.1.bb | 4 +- > >> > .../python/python3-scons-native_4.3.0.bb | 7 -- > >> > .../python/python3-scons_4.3.0.bb | 13 ++- > >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- > >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- > >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ > >> > .../python/python3-smartypants_2.0.0.bb | 1 + > >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ > >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- > >> > .../python/python3-subunit_1.4.0.bb | 17 ++- > >> > ...roject.toml-from-flit-backend-branch.patch | 100 > ++++++++++++++++++ > >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ > >> > .../python/python3-zipp_3.7.0.bb | 2 +- > >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + > >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + > >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- > >> > 39 files changed, 412 insertions(+), 78 deletions(-) > >> > create mode 100644 meta/classes/flit_core.bbclass > >> > create mode 100644 meta/classes/pip_install_wheel.bbclass > >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass > >> > delete mode 100644 meta/recipes-devtools/python/python-nose.inc > >> > delete mode 100644 meta/recipes-devtools/python/python-subunit.inc > >> > create mode 100644 meta/recipes-devtools/python/ > python3-flit-core_3.6.0.bb > >> > delete mode 100644 meta/recipes-devtools/python/ > python3-nose_1.3.7.bb > >> > delete mode 100644 meta/recipes-devtools/python/ > python3-scons-native_4.3.0.bb > >> > create mode 100644 > meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch > >> > create mode 100644 > meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch > >> > create mode 100644 > meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch > >> > create mode 100644 meta/recipes-devtools/python/ > python3-wheel_0.37.1.bb > >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => > bmap-tools_git.bb} (86%) > >> > > >> > -- > >> > 2.30.2 > >> > > >> > > >> > -=-=-=-=-=-=-=-=-=-=-=- > >> > Links: You receive all messages sent to this group. > >> > View/Reply Online (#162076): > https://lists.openembedded.org/g/openembedded-core/message/162076 > >> > Mute This Topic: https://lists.openembedded.org/mt/89304007/1997914 > >> > Group Owner: openembedded-core+owner@lists.openembedded.org > >> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub > [raj.khem@gmail.com] > >> > -=-=-=-=-=-=-=-=-=-=-=- > >> > >
On Tue, Feb 22, 2022 at 10:39 AM Tim Orling <ticotimo@gmail.com> wrote: > > > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <raj.khem@gmail.com> wrote: > >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <ticotimo@gmail.com> wrote: >> > >> > >> > >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com> wrote: >> >> >> >> this is going into dep loops Try >> >> >> >> bitbake python3-wheel-native >> > >> > >> > It shouldn’t >> > >> https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels >> > >> >> I have been trying whats on master-next >> >> > I can't reproduce. Can you point me at an error log please? > It might have something to do with sstate cache, but that's just a hunch. > > Ah. I bet you have meta-python in the bblayers and therefore python3-wheel recipe from meta-python is winning. > > DEPENDS:remove:class-native = "python3-pip-native" >> > >> > do_install:class-native () { >> > # We need to bootstrap python3-wheel-native >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} >> > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ >> > bbfatal_log "Failed to install" >> > } >> >> >> >> >> >> >> >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> wrote: >> >> > >> >> > Upstream Python is moving to pyproject.toml rather than setup.py and >> >> > wheels rather than eggs. The main place this is documented is in >> >> > PEP-517[1]. Installing packages with setup.py install is deprecated. >> >> > >> >> > We have a choice with the 'kirkstone' LTS release of introducing this >> >> > series (and fixing any fall out during the stabilization period) or >> >> > living with deprecated Python packaging for the duration of the LTS >> (at >> >> > a minimum 2 years). This seems to be a compelling enough reason to >> bring >> >> > in this series, even if it results in a slight slip of the 'feature >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. >> >> > >> >> > This series bootstraps python3-setuptools-native, >> >> > python3-flit-core-native, python3-wheel-native and >> python3-pip-native, >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows >> all >> >> > other recipes to be installed with pip, as intended by upstream. >> >> > >> >> > Three new classes are introduced: >> >> > (1) flit_core which builds PEP-517 compliant packages which use >> >> > flit_core.buildapi as their build-backend. >> >> > (2) setuptools_build_meta which builds PEP-517 compliant packages >> which use >> >> > setuptools_build_meta as their build-backend. >> >> > (3) pip_install_wheel, which simply installs wheels built by other >> >> > classes. >> >> > >> >> > The original setuptools3.bbclass do_compile is modified to >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than the >> deprecated >> >> > 'setup.py build' mechanism. >> >> > >> >> > The original setuptools3.bbclass do_install is modified to >> >> > use the pip_install_wheel.bbclass to install wheels with pip rather >> than >> >> > the deprecated 'setup.py install' mechanism. >> >> > >> >> > For pip install, we disable dependency checking (--no-deps) because >> it >> >> > proves to be brittle in our environment without providing full wheel >> >> > cache that seems like overhead that we do not need. We also disable >> >> > fetching wheels from pypi (--no-index). This is no different than our >> >> > normal workflow of requiring dependencies to be defined in a recipe. >> >> > >> >> > Missing from this series is documentation of the new features, >> classes >> >> > and variables (this will be done under separate cover). >> >> > >> >> > Also desired and previously discussed in YP meetings is a helper >> >> > function to warn that a recipe which currently inherits setuptools3 >> >> > but has a pyproject.toml and defines a build-system.build-backend >> should >> >> > instead use of the new PEP-517 classes. This function is still in >> progress. >> >> > >> >> > During testing, recipes which use python3-setuptools-scm are failing >> to >> >> > determine the version of the python package, For several recipes, >> the fix >> >> > was to use pyproject.toml and PEP-517 build backends. To be sure, >> other recipes >> >> > are still going to fail for the short term, until the root cause can >> be >> >> > determined. This was not seen again in v2 builds. >> >> > >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has been >> >> > identified in recipes in meta-python, but not in oe-core. The >> >> > python3-poetry-core recipe is WIP, but brings in a number of >> >> > dependencies that are perhaps not worth it to bring into oe-core at >> this >> >> > time. Rather we can support these recipes in meta-python? >> >> > >> >> > Fixes in v2 include passing --root=${D} to pip install (rather than >> --target) >> >> > and also passing --prefix so that scripts are (usually) installed in >> the proper >> >> > location. pip install also 'helpfully' overwrites python3 intepreter >> with nativepython3 >> >> > in #!, so add a fix to return the values to what is actually correct >> (similar to what is >> >> > done in setuptools3.bbclass). Also several recipes still needed >> PIP_INSTALL_PACKAGE defined. >> >> > >> >> > Also added self as maintainer of new recipes. >> >> > >> >> > [YOCTO #14638] >> >> > >> >> > [1] https://www.python.org/dev/peps/pep-0517/ >> >> > >> >> > The following changes since commit >> 41ae26ecab731f2fb6593df143476c94183be209: >> >> > >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) >> >> > >> >> > are available in the Git repository at: >> >> > >> >> > git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels >> >> > >> http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels >> >> > >> >> > Tim Orling (32): >> >> > python3-wheel: move 0.37.0 from meta-python >> >> > python3-flit-core: add recipe for 3.6.0 >> >> > flit_core.bbclass: add helper for newer python packaging >> >> > python3-wheel: inherit flit_core >> >> > pip_install_wheel.bbclass: add helper class >> >> > setuptools_build_meta.bbclass: add helper class >> >> > python3-pip: inherit setuptools_build_meta >> >> > python3-attrs: inherit setuptools_build_meta >> >> > python3-git: inherit setuptools_build_meta >> >> > python3-pytest: inherit setuptools_build_meta >> >> > python3-setuptools-scm: inherit setuptools_build_meta >> >> > python3-zipp: inherit setuptools_build_meta >> >> > python3-iniconfig: inherit setuptools_build_meta >> >> > python3-py: inherit setuptools_build_meta >> >> > python3-pluggy: inherit setuptools_build_meta >> >> > python3-setuptools: inherit setuptools_base_meta >> >> > setuptools3.bbclass: refactor for wheels >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE >> >> > meson: inherit setuptools_build_meta >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE >> >> > python3-smartypants: patch hash bang to python3 >> >> > python3-scons: merge -native recipe >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE >> >> > python3-magic: set PIP_INSTALL_PACKAGE >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER >> >> > asciidoc: set PIP_INSTALL_PACKAGE >> >> > gi-docgen: set PIP_INSTALL_PACKAGE >> >> > numpy: set PIP_INSTALL_PACKAGE >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE >> >> > python3-mako: inherit setuptools_build_meta >> >> > python3-packaging: inherit setuptools_build_meta >> >> > python3-nose: drop recipe >> >> > >> >> > meta/classes/flit_core.bbclass | 16 +++ >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ >> >> > meta/classes/setuptools3.bbclass | 20 ++-- >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ >> >> > meta/conf/distro/include/maintainers.inc | 3 +- >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- >> >> > .../python/python-subunit.inc | 14 --- >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ >> >> > .../python/python3-git_3.1.26.bb | 2 +- >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + >> >> > .../python/python3-magic_0.4.25.bb | 1 + >> >> > .../python/python3-mako_1.1.6.bb | 2 +- >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + >> >> > .../python/python3-nose_1.3.7.bb | 2 - >> >> > .../python/python3-numpy_1.22.2.bb | 1 + >> >> > .../python/python3-packaging_21.3.bb | 2 +- >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- >> >> > .../python/python3-py_1.11.0.bb | 2 +- >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 >> ++++++++++++++++++ >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + >> >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) >> >> > create mode 100644 meta/classes/flit_core.bbclass >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass >> >> > delete mode 100644 meta/recipes-devtools/python/python-nose.inc >> >> > delete mode 100644 meta/recipes-devtools/python/python-subunit.inc >> >> > create mode 100644 meta/recipes-devtools/python/ >> python3-flit-core_3.6.0.bb >> >> > delete mode 100644 meta/recipes-devtools/python/ >> python3-nose_1.3.7.bb >> >> > delete mode 100644 meta/recipes-devtools/python/ >> python3-scons-native_4.3.0.bb >> >> > create mode 100644 >> meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch >> >> > create mode 100644 >> meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch >> >> > create mode 100644 >> meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch >> >> > create mode 100644 meta/recipes-devtools/python/ >> python3-wheel_0.37.1.bb >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => >> bmap-tools_git.bb} (86%) >> >> > >> >> > -- >> >> > 2.30.2 >> >> > >> >> > >> >> > -=-=-=-=-=-=-=-=-=-=-=- >> >> > Links: You receive all messages sent to this group. >> >> > View/Reply Online (#162076): >> https://lists.openembedded.org/g/openembedded-core/message/162076 >> >> > Mute This Topic: https://lists.openembedded.org/mt/89304007/1997914 >> >> > Group Owner: openembedded-core+owner@lists.openembedded.org >> >> > Unsubscribe: >> https://lists.openembedded.org/g/openembedded-core/unsub [ >> raj.khem@gmail.com] >> >> > -=-=-=-=-=-=-=-=-=-=-=- >> >> > >> >
On Tue, Feb 22, 2022 at 10:40 AM Tim Orling <ticotimo@gmail.com> wrote: > > > > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <raj.khem@gmail.com> wrote: >> >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <ticotimo@gmail.com> wrote: >> > >> > >> > >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com> wrote: >> >> >> >> this is going into dep loops Try >> >> >> >> bitbake python3-wheel-native >> > >> > >> > It shouldn’t >> > https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels >> > >> >> I have been trying whats on master-next >> > > I can't reproduce. Can you point me at an error log please? > It might have something to do with sstate cache, but that's just a hunch. > This is because python3-wheel was moved to core and modified but not deleted from meta-python at same time. I have sent a patch for that >> >> > DEPENDS:remove:class-native = "python3-pip-native" >> > >> > do_install:class-native () { >> > # We need to bootstrap python3-wheel-native >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} >> > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ >> > bbfatal_log "Failed to install" >> > } >> >> >> >> >> >> >> >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> wrote: >> >> > >> >> > Upstream Python is moving to pyproject.toml rather than setup.py and >> >> > wheels rather than eggs. The main place this is documented is in >> >> > PEP-517[1]. Installing packages with setup.py install is deprecated. >> >> > >> >> > We have a choice with the 'kirkstone' LTS release of introducing this >> >> > series (and fixing any fall out during the stabilization period) or >> >> > living with deprecated Python packaging for the duration of the LTS (at >> >> > a minimum 2 years). This seems to be a compelling enough reason to bring >> >> > in this series, even if it results in a slight slip of the 'feature >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. >> >> > >> >> > This series bootstraps python3-setuptools-native, >> >> > python3-flit-core-native, python3-wheel-native and python3-pip-native, >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows all >> >> > other recipes to be installed with pip, as intended by upstream. >> >> > >> >> > Three new classes are introduced: >> >> > (1) flit_core which builds PEP-517 compliant packages which use >> >> > flit_core.buildapi as their build-backend. >> >> > (2) setuptools_build_meta which builds PEP-517 compliant packages which use >> >> > setuptools_build_meta as their build-backend. >> >> > (3) pip_install_wheel, which simply installs wheels built by other >> >> > classes. >> >> > >> >> > The original setuptools3.bbclass do_compile is modified to >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than the deprecated >> >> > 'setup.py build' mechanism. >> >> > >> >> > The original setuptools3.bbclass do_install is modified to >> >> > use the pip_install_wheel.bbclass to install wheels with pip rather than >> >> > the deprecated 'setup.py install' mechanism. >> >> > >> >> > For pip install, we disable dependency checking (--no-deps) because it >> >> > proves to be brittle in our environment without providing full wheel >> >> > cache that seems like overhead that we do not need. We also disable >> >> > fetching wheels from pypi (--no-index). This is no different than our >> >> > normal workflow of requiring dependencies to be defined in a recipe. >> >> > >> >> > Missing from this series is documentation of the new features, classes >> >> > and variables (this will be done under separate cover). >> >> > >> >> > Also desired and previously discussed in YP meetings is a helper >> >> > function to warn that a recipe which currently inherits setuptools3 >> >> > but has a pyproject.toml and defines a build-system.build-backend should >> >> > instead use of the new PEP-517 classes. This function is still in progress. >> >> > >> >> > During testing, recipes which use python3-setuptools-scm are failing to >> >> > determine the version of the python package, For several recipes, the fix >> >> > was to use pyproject.toml and PEP-517 build backends. To be sure, other recipes >> >> > are still going to fail for the short term, until the root cause can be >> >> > determined. This was not seen again in v2 builds. >> >> > >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has been >> >> > identified in recipes in meta-python, but not in oe-core. The >> >> > python3-poetry-core recipe is WIP, but brings in a number of >> >> > dependencies that are perhaps not worth it to bring into oe-core at this >> >> > time. Rather we can support these recipes in meta-python? >> >> > >> >> > Fixes in v2 include passing --root=${D} to pip install (rather than --target) >> >> > and also passing --prefix so that scripts are (usually) installed in the proper >> >> > location. pip install also 'helpfully' overwrites python3 intepreter with nativepython3 >> >> > in #!, so add a fix to return the values to what is actually correct (similar to what is >> >> > done in setuptools3.bbclass). Also several recipes still needed PIP_INSTALL_PACKAGE defined. >> >> > >> >> > Also added self as maintainer of new recipes. >> >> > >> >> > [YOCTO #14638] >> >> > >> >> > [1] https://www.python.org/dev/peps/pep-0517/ >> >> > >> >> > The following changes since commit 41ae26ecab731f2fb6593df143476c94183be209: >> >> > >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) >> >> > >> >> > are available in the Git repository at: >> >> > >> >> > git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels >> >> > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels >> >> > >> >> > Tim Orling (32): >> >> > python3-wheel: move 0.37.0 from meta-python >> >> > python3-flit-core: add recipe for 3.6.0 >> >> > flit_core.bbclass: add helper for newer python packaging >> >> > python3-wheel: inherit flit_core >> >> > pip_install_wheel.bbclass: add helper class >> >> > setuptools_build_meta.bbclass: add helper class >> >> > python3-pip: inherit setuptools_build_meta >> >> > python3-attrs: inherit setuptools_build_meta >> >> > python3-git: inherit setuptools_build_meta >> >> > python3-pytest: inherit setuptools_build_meta >> >> > python3-setuptools-scm: inherit setuptools_build_meta >> >> > python3-zipp: inherit setuptools_build_meta >> >> > python3-iniconfig: inherit setuptools_build_meta >> >> > python3-py: inherit setuptools_build_meta >> >> > python3-pluggy: inherit setuptools_build_meta >> >> > python3-setuptools: inherit setuptools_base_meta >> >> > setuptools3.bbclass: refactor for wheels >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE >> >> > meson: inherit setuptools_build_meta >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE >> >> > python3-smartypants: patch hash bang to python3 >> >> > python3-scons: merge -native recipe >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE >> >> > python3-magic: set PIP_INSTALL_PACKAGE >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER >> >> > asciidoc: set PIP_INSTALL_PACKAGE >> >> > gi-docgen: set PIP_INSTALL_PACKAGE >> >> > numpy: set PIP_INSTALL_PACKAGE >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE >> >> > python3-mako: inherit setuptools_build_meta >> >> > python3-packaging: inherit setuptools_build_meta >> >> > python3-nose: drop recipe >> >> > >> >> > meta/classes/flit_core.bbclass | 16 +++ >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ >> >> > meta/classes/setuptools3.bbclass | 20 ++-- >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ >> >> > meta/conf/distro/include/maintainers.inc | 3 +- >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- >> >> > .../python/python-subunit.inc | 14 --- >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ >> >> > .../python/python3-git_3.1.26.bb | 2 +- >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + >> >> > .../python/python3-magic_0.4.25.bb | 1 + >> >> > .../python/python3-mako_1.1.6.bb | 2 +- >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + >> >> > .../python/python3-nose_1.3.7.bb | 2 - >> >> > .../python/python3-numpy_1.22.2.bb | 1 + >> >> > .../python/python3-packaging_21.3.bb | 2 +- >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- >> >> > .../python/python3-py_1.11.0.bb | 2 +- >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++ >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + >> >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) >> >> > create mode 100644 meta/classes/flit_core.bbclass >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass >> >> > delete mode 100644 meta/recipes-devtools/python/python-nose.inc >> >> > delete mode 100644 meta/recipes-devtools/python/python-subunit.inc >> >> > create mode 100644 meta/recipes-devtools/python/python3-flit-core_3.6.0.bb >> >> > delete mode 100644 meta/recipes-devtools/python/python3-nose_1.3.7.bb >> >> > delete mode 100644 meta/recipes-devtools/python/python3-scons-native_4.3.0.bb >> >> > create mode 100644 meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch >> >> > create mode 100644 meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch >> >> > create mode 100644 meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch >> >> > create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => bmap-tools_git.bb} (86%) >> >> > >> >> > -- >> >> > 2.30.2 >> >> > >> >> > >> >> > -=-=-=-=-=-=-=-=-=-=-=- >> >> > Links: You receive all messages sent to this group. >> >> > View/Reply Online (#162076): https://lists.openembedded.org/g/openembedded-core/message/162076 >> >> > Mute This Topic: https://lists.openembedded.org/mt/89304007/1997914 >> >> > Group Owner: openembedded-core+owner@lists.openembedded.org >> >> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] >> >> > -=-=-=-=-=-=-=-=-=-=-=- >> >> >
On Tue, Feb 22, 2022 at 11:18 AM Khem Raj <raj.khem@gmail.com> wrote: > On Tue, Feb 22, 2022 at 10:40 AM Tim Orling <ticotimo@gmail.com> wrote: > > > > > > > > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <raj.khem@gmail.com> wrote: > >> > >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <ticotimo@gmail.com> wrote: > >> > > >> > > >> > > >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com> wrote: > >> >> > >> >> this is going into dep loops Try > >> >> > >> >> bitbake python3-wheel-native > >> > > >> > > >> > It shouldn’t > >> > > https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels > >> > > >> > >> I have been trying whats on master-next > >> > > > > I can't reproduce. Can you point me at an error log please? > > It might have something to do with sstate cache, but that's just a hunch. > > > > This is because python3-wheel was moved to core and modified but not > deleted from meta-python at same time. I have sent a patch for that > > Already sent one: https://patchwork.yoctoproject.org/project/oe/patch/20220222184920.2310391-1-tim.orling@konsulko.com/ > >> > >> > DEPENDS:remove:class-native = "python3-pip-native" > >> > > >> > do_install:class-native () { > >> > # We need to bootstrap python3-wheel-native > >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} > >> > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" > >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ > >> > bbfatal_log "Failed to install" > >> > } > >> >> > >> >> > >> >> > >> >> > >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> > wrote: > >> >> > > >> >> > Upstream Python is moving to pyproject.toml rather than setup.py > and > >> >> > wheels rather than eggs. The main place this is documented is in > >> >> > PEP-517[1]. Installing packages with setup.py install is > deprecated. > >> >> > > >> >> > We have a choice with the 'kirkstone' LTS release of introducing > this > >> >> > series (and fixing any fall out during the stabilization period) or > >> >> > living with deprecated Python packaging for the duration of the > LTS (at > >> >> > a minimum 2 years). This seems to be a compelling enough reason to > bring > >> >> > in this series, even if it results in a slight slip of the 'feature > >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. > >> >> > > >> >> > This series bootstraps python3-setuptools-native, > >> >> > python3-flit-core-native, python3-wheel-native and > python3-pip-native, > >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This > allows all > >> >> > other recipes to be installed with pip, as intended by upstream. > >> >> > > >> >> > Three new classes are introduced: > >> >> > (1) flit_core which builds PEP-517 compliant packages which use > >> >> > flit_core.buildapi as their build-backend. > >> >> > (2) setuptools_build_meta which builds PEP-517 compliant packages > which use > >> >> > setuptools_build_meta as their build-backend. > >> >> > (3) pip_install_wheel, which simply installs wheels built by other > >> >> > classes. > >> >> > > >> >> > The original setuptools3.bbclass do_compile is modified to > >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than the > deprecated > >> >> > 'setup.py build' mechanism. > >> >> > > >> >> > The original setuptools3.bbclass do_install is modified to > >> >> > use the pip_install_wheel.bbclass to install wheels with pip > rather than > >> >> > the deprecated 'setup.py install' mechanism. > >> >> > > >> >> > For pip install, we disable dependency checking (--no-deps) > because it > >> >> > proves to be brittle in our environment without providing full > wheel > >> >> > cache that seems like overhead that we do not need. We also disable > >> >> > fetching wheels from pypi (--no-index). This is no different than > our > >> >> > normal workflow of requiring dependencies to be defined in a > recipe. > >> >> > > >> >> > Missing from this series is documentation of the new features, > classes > >> >> > and variables (this will be done under separate cover). > >> >> > > >> >> > Also desired and previously discussed in YP meetings is a helper > >> >> > function to warn that a recipe which currently inherits setuptools3 > >> >> > but has a pyproject.toml and defines a build-system.build-backend > should > >> >> > instead use of the new PEP-517 classes. This function is still in > progress. > >> >> > > >> >> > During testing, recipes which use python3-setuptools-scm are > failing to > >> >> > determine the version of the python package, For several recipes, > the fix > >> >> > was to use pyproject.toml and PEP-517 build backends. To be sure, > other recipes > >> >> > are still going to fail for the short term, until the root cause > can be > >> >> > determined. This was not seen again in v2 builds. > >> >> > > >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has been > >> >> > identified in recipes in meta-python, but not in oe-core. The > >> >> > python3-poetry-core recipe is WIP, but brings in a number of > >> >> > dependencies that are perhaps not worth it to bring into oe-core > at this > >> >> > time. Rather we can support these recipes in meta-python? > >> >> > > >> >> > Fixes in v2 include passing --root=${D} to pip install (rather > than --target) > >> >> > and also passing --prefix so that scripts are (usually) installed > in the proper > >> >> > location. pip install also 'helpfully' overwrites python3 > intepreter with nativepython3 > >> >> > in #!, so add a fix to return the values to what is actually > correct (similar to what is > >> >> > done in setuptools3.bbclass). Also several recipes still needed > PIP_INSTALL_PACKAGE defined. > >> >> > > >> >> > Also added self as maintainer of new recipes. > >> >> > > >> >> > [YOCTO #14638] > >> >> > > >> >> > [1] https://www.python.org/dev/peps/pep-0517/ > >> >> > > >> >> > The following changes since commit > 41ae26ecab731f2fb6593df143476c94183be209: > >> >> > > >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) > >> >> > > >> >> > are available in the Git repository at: > >> >> > > >> >> > git://git.yoctoproject.org/git/poky-contrib > timo/bootstrap-wheels > >> >> > > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels > >> >> > > >> >> > Tim Orling (32): > >> >> > python3-wheel: move 0.37.0 from meta-python > >> >> > python3-flit-core: add recipe for 3.6.0 > >> >> > flit_core.bbclass: add helper for newer python packaging > >> >> > python3-wheel: inherit flit_core > >> >> > pip_install_wheel.bbclass: add helper class > >> >> > setuptools_build_meta.bbclass: add helper class > >> >> > python3-pip: inherit setuptools_build_meta > >> >> > python3-attrs: inherit setuptools_build_meta > >> >> > python3-git: inherit setuptools_build_meta > >> >> > python3-pytest: inherit setuptools_build_meta > >> >> > python3-setuptools-scm: inherit setuptools_build_meta > >> >> > python3-zipp: inherit setuptools_build_meta > >> >> > python3-iniconfig: inherit setuptools_build_meta > >> >> > python3-py: inherit setuptools_build_meta > >> >> > python3-pluggy: inherit setuptools_build_meta > >> >> > python3-setuptools: inherit setuptools_base_meta > >> >> > setuptools3.bbclass: refactor for wheels > >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE > >> >> > meson: inherit setuptools_build_meta > >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE > >> >> > python3-smartypants: patch hash bang to python3 > >> >> > python3-scons: merge -native recipe > >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE > >> >> > python3-magic: set PIP_INSTALL_PACKAGE > >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER > >> >> > asciidoc: set PIP_INSTALL_PACKAGE > >> >> > gi-docgen: set PIP_INSTALL_PACKAGE > >> >> > numpy: set PIP_INSTALL_PACKAGE > >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE > >> >> > python3-mako: inherit setuptools_build_meta > >> >> > python3-packaging: inherit setuptools_build_meta > >> >> > python3-nose: drop recipe > >> >> > > >> >> > meta/classes/flit_core.bbclass | 16 +++ > >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ > >> >> > meta/classes/setuptools3.bbclass | 20 ++-- > >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ > >> >> > meta/conf/distro/include/maintainers.inc | 3 +- > >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- > >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- > >> >> > .../python/python-subunit.inc | 14 --- > >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- > >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + > >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ > >> >> > .../python/python3-git_3.1.26.bb | 2 +- > >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- > >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + > >> >> > .../python/python3-magic_0.4.25.bb | 1 + > >> >> > .../python/python3-mako_1.1.6.bb | 2 +- > >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + > >> >> > .../python/python3-nose_1.3.7.bb | 2 - > >> >> > .../python/python3-numpy_1.22.2.bb | 1 + > >> >> > .../python/python3-packaging_21.3.bb | 2 +- > >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- > >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- > >> >> > .../python/python3-py_1.11.0.bb | 2 +- > >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- > >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- > >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- > >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- > >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- > >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ > >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + > >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ > >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- > >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- > >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 > ++++++++++++++++++ > >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ > >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- > >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + > >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + > >> >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- > >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) > >> >> > create mode 100644 meta/classes/flit_core.bbclass > >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass > >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass > >> >> > delete mode 100644 meta/recipes-devtools/python/python-nose.inc > >> >> > delete mode 100644 meta/recipes-devtools/python/python-subunit.inc > >> >> > create mode 100644 meta/recipes-devtools/python/ > python3-flit-core_3.6.0.bb > >> >> > delete mode 100644 meta/recipes-devtools/python/ > python3-nose_1.3.7.bb > >> >> > delete mode 100644 meta/recipes-devtools/python/ > python3-scons-native_4.3.0.bb > >> >> > create mode 100644 > meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch > >> >> > create mode 100644 > meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch > >> >> > create mode 100644 > meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch > >> >> > create mode 100644 meta/recipes-devtools/python/ > python3-wheel_0.37.1.bb > >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => > bmap-tools_git.bb} (86%) > >> >> > > >> >> > -- > >> >> > 2.30.2 > >> >> > > >> >> > > >> >> > -=-=-=-=-=-=-=-=-=-=-=- > >> >> > Links: You receive all messages sent to this group. > >> >> > View/Reply Online (#162076): > https://lists.openembedded.org/g/openembedded-core/message/162076 > >> >> > Mute This Topic: > https://lists.openembedded.org/mt/89304007/1997914 > >> >> > Group Owner: openembedded-core+owner@lists.openembedded.org > >> >> > Unsubscribe: > https://lists.openembedded.org/g/openembedded-core/unsub [ > raj.khem@gmail.com] > >> >> > -=-=-=-=-=-=-=-=-=-=-=- > >> >> > >
Right now approx 150 out of 470 recipes [1] in meta-python are directly failing, perhaps there will be more once these are fixed then there are other layers which might be carrying some impacted python modules. Also YP compatibility is broken due to python3-nose removal. If we are planning to include this in the upcoming release,we would need some commitment and strategy to fix these and other ripple effects throughout the ecosystem especially since this is an LTS release. I wish this changeset was proposed in a release prior to LTS release, so it could have given enough time to adjust for the ecosystem. This is will take some effort in stabilizing I also wonder, what do we lose if we do not put this into the LTS release? [1] https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/1460 On Tue, Feb 22, 2022 at 11:23 AM Tim Orling <ticotimo@gmail.com> wrote: > > > > On Tue, Feb 22, 2022 at 11:18 AM Khem Raj <raj.khem@gmail.com> wrote: >> >> On Tue, Feb 22, 2022 at 10:40 AM Tim Orling <ticotimo@gmail.com> wrote: >> > >> > >> > >> > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <raj.khem@gmail.com> wrote: >> >> >> >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <ticotimo@gmail.com> wrote: >> >> > >> >> > >> >> > >> >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com> wrote: >> >> >> >> >> >> this is going into dep loops Try >> >> >> >> >> >> bitbake python3-wheel-native >> >> > >> >> > >> >> > It shouldn’t >> >> > https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels >> >> > >> >> >> >> I have been trying whats on master-next >> >> >> > >> > I can't reproduce. Can you point me at an error log please? >> > It might have something to do with sstate cache, but that's just a hunch. >> > >> >> This is because python3-wheel was moved to core and modified but not >> deleted from meta-python at same time. I have sent a patch for that >> > Already sent one: > https://patchwork.yoctoproject.org/project/oe/patch/20220222184920.2310391-1-tim.orling@konsulko.com/ > >> >> >> >> >> > DEPENDS:remove:class-native = "python3-pip-native" >> >> > >> >> > do_install:class-native () { >> >> > # We need to bootstrap python3-wheel-native >> >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} >> >> > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" >> >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ >> >> > bbfatal_log "Failed to install" >> >> > } >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> wrote: >> >> >> > >> >> >> > Upstream Python is moving to pyproject.toml rather than setup.py and >> >> >> > wheels rather than eggs. The main place this is documented is in >> >> >> > PEP-517[1]. Installing packages with setup.py install is deprecated. >> >> >> > >> >> >> > We have a choice with the 'kirkstone' LTS release of introducing this >> >> >> > series (and fixing any fall out during the stabilization period) or >> >> >> > living with deprecated Python packaging for the duration of the LTS (at >> >> >> > a minimum 2 years). This seems to be a compelling enough reason to bring >> >> >> > in this series, even if it results in a slight slip of the 'feature >> >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. >> >> >> > >> >> >> > This series bootstraps python3-setuptools-native, >> >> >> > python3-flit-core-native, python3-wheel-native and python3-pip-native, >> >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows all >> >> >> > other recipes to be installed with pip, as intended by upstream. >> >> >> > >> >> >> > Three new classes are introduced: >> >> >> > (1) flit_core which builds PEP-517 compliant packages which use >> >> >> > flit_core.buildapi as their build-backend. >> >> >> > (2) setuptools_build_meta which builds PEP-517 compliant packages which use >> >> >> > setuptools_build_meta as their build-backend. >> >> >> > (3) pip_install_wheel, which simply installs wheels built by other >> >> >> > classes. >> >> >> > >> >> >> > The original setuptools3.bbclass do_compile is modified to >> >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than the deprecated >> >> >> > 'setup.py build' mechanism. >> >> >> > >> >> >> > The original setuptools3.bbclass do_install is modified to >> >> >> > use the pip_install_wheel.bbclass to install wheels with pip rather than >> >> >> > the deprecated 'setup.py install' mechanism. >> >> >> > >> >> >> > For pip install, we disable dependency checking (--no-deps) because it >> >> >> > proves to be brittle in our environment without providing full wheel >> >> >> > cache that seems like overhead that we do not need. We also disable >> >> >> > fetching wheels from pypi (--no-index). This is no different than our >> >> >> > normal workflow of requiring dependencies to be defined in a recipe. >> >> >> > >> >> >> > Missing from this series is documentation of the new features, classes >> >> >> > and variables (this will be done under separate cover). >> >> >> > >> >> >> > Also desired and previously discussed in YP meetings is a helper >> >> >> > function to warn that a recipe which currently inherits setuptools3 >> >> >> > but has a pyproject.toml and defines a build-system.build-backend should >> >> >> > instead use of the new PEP-517 classes. This function is still in progress. >> >> >> > >> >> >> > During testing, recipes which use python3-setuptools-scm are failing to >> >> >> > determine the version of the python package, For several recipes, the fix >> >> >> > was to use pyproject.toml and PEP-517 build backends. To be sure, other recipes >> >> >> > are still going to fail for the short term, until the root cause can be >> >> >> > determined. This was not seen again in v2 builds. >> >> >> > >> >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has been >> >> >> > identified in recipes in meta-python, but not in oe-core. The >> >> >> > python3-poetry-core recipe is WIP, but brings in a number of >> >> >> > dependencies that are perhaps not worth it to bring into oe-core at this >> >> >> > time. Rather we can support these recipes in meta-python? >> >> >> > >> >> >> > Fixes in v2 include passing --root=${D} to pip install (rather than --target) >> >> >> > and also passing --prefix so that scripts are (usually) installed in the proper >> >> >> > location. pip install also 'helpfully' overwrites python3 intepreter with nativepython3 >> >> >> > in #!, so add a fix to return the values to what is actually correct (similar to what is >> >> >> > done in setuptools3.bbclass). Also several recipes still needed PIP_INSTALL_PACKAGE defined. >> >> >> > >> >> >> > Also added self as maintainer of new recipes. >> >> >> > >> >> >> > [YOCTO #14638] >> >> >> > >> >> >> > [1] https://www.python.org/dev/peps/pep-0517/ >> >> >> > >> >> >> > The following changes since commit 41ae26ecab731f2fb6593df143476c94183be209: >> >> >> > >> >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) >> >> >> > >> >> >> > are available in the Git repository at: >> >> >> > >> >> >> > git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels >> >> >> > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels >> >> >> > >> >> >> > Tim Orling (32): >> >> >> > python3-wheel: move 0.37.0 from meta-python >> >> >> > python3-flit-core: add recipe for 3.6.0 >> >> >> > flit_core.bbclass: add helper for newer python packaging >> >> >> > python3-wheel: inherit flit_core >> >> >> > pip_install_wheel.bbclass: add helper class >> >> >> > setuptools_build_meta.bbclass: add helper class >> >> >> > python3-pip: inherit setuptools_build_meta >> >> >> > python3-attrs: inherit setuptools_build_meta >> >> >> > python3-git: inherit setuptools_build_meta >> >> >> > python3-pytest: inherit setuptools_build_meta >> >> >> > python3-setuptools-scm: inherit setuptools_build_meta >> >> >> > python3-zipp: inherit setuptools_build_meta >> >> >> > python3-iniconfig: inherit setuptools_build_meta >> >> >> > python3-py: inherit setuptools_build_meta >> >> >> > python3-pluggy: inherit setuptools_build_meta >> >> >> > python3-setuptools: inherit setuptools_base_meta >> >> >> > setuptools3.bbclass: refactor for wheels >> >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE >> >> >> > meson: inherit setuptools_build_meta >> >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE >> >> >> > python3-smartypants: patch hash bang to python3 >> >> >> > python3-scons: merge -native recipe >> >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE >> >> >> > python3-magic: set PIP_INSTALL_PACKAGE >> >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER >> >> >> > asciidoc: set PIP_INSTALL_PACKAGE >> >> >> > gi-docgen: set PIP_INSTALL_PACKAGE >> >> >> > numpy: set PIP_INSTALL_PACKAGE >> >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE >> >> >> > python3-mako: inherit setuptools_build_meta >> >> >> > python3-packaging: inherit setuptools_build_meta >> >> >> > python3-nose: drop recipe >> >> >> > >> >> >> > meta/classes/flit_core.bbclass | 16 +++ >> >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ >> >> >> > meta/classes/setuptools3.bbclass | 20 ++-- >> >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ >> >> >> > meta/conf/distro/include/maintainers.inc | 3 +- >> >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- >> >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- >> >> >> > .../python/python-subunit.inc | 14 --- >> >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- >> >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + >> >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ >> >> >> > .../python/python3-git_3.1.26.bb | 2 +- >> >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- >> >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + >> >> >> > .../python/python3-magic_0.4.25.bb | 1 + >> >> >> > .../python/python3-mako_1.1.6.bb | 2 +- >> >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + >> >> >> > .../python/python3-nose_1.3.7.bb | 2 - >> >> >> > .../python/python3-numpy_1.22.2.bb | 1 + >> >> >> > .../python/python3-packaging_21.3.bb | 2 +- >> >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- >> >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- >> >> >> > .../python/python3-py_1.11.0.bb | 2 +- >> >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- >> >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- >> >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- >> >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- >> >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- >> >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ >> >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + >> >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ >> >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- >> >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- >> >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++ >> >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ >> >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- >> >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + >> >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + >> >> >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- >> >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) >> >> >> > create mode 100644 meta/classes/flit_core.bbclass >> >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass >> >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass >> >> >> > delete mode 100644 meta/recipes-devtools/python/python-nose.inc >> >> >> > delete mode 100644 meta/recipes-devtools/python/python-subunit.inc >> >> >> > create mode 100644 meta/recipes-devtools/python/python3-flit-core_3.6.0.bb >> >> >> > delete mode 100644 meta/recipes-devtools/python/python3-nose_1.3.7.bb >> >> >> > delete mode 100644 meta/recipes-devtools/python/python3-scons-native_4.3.0.bb >> >> >> > create mode 100644 meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch >> >> >> > create mode 100644 meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch >> >> >> > create mode 100644 meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch >> >> >> > create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb >> >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => bmap-tools_git.bb} (86%) >> >> >> > >> >> >> > -- >> >> >> > 2.30.2 >> >> >> > >> >> >> > >> >> >> > -=-=-=-=-=-=-=-=-=-=-=- >> >> >> > Links: You receive all messages sent to this group. >> >> >> > View/Reply Online (#162076): https://lists.openembedded.org/g/openembedded-core/message/162076 >> >> >> > Mute This Topic: https://lists.openembedded.org/mt/89304007/1997914 >> >> >> > Group Owner: openembedded-core+owner@lists.openembedded.org >> >> >> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] >> >> >> > -=-=-=-=-=-=-=-=-=-=-=- >> >> >> >
On Wed, Feb 23, 2022 at 8:07 PM Khem Raj <raj.khem@gmail.com> wrote: > Right now approx 150 out of 470 recipes [1] in meta-python are directly > failing, > perhaps there will be more once these are fixed then there are other layers > which might be carrying some impacted python modules. The issue has mostly been detection of the wheel filename and this should be improved by: https://lore.kernel.org/openembedded-core/20220224041200.2645054-1-tim.orling@konsulko.com/T/#t Also YP compatibility > is broken due to python3-nose removal. We can try to run 2to3 on the python3-nose plugins or remove them surgically to reintroduce cleanly Python 3 only python3-nose. I propose that we move it to meta-python even so because nothing in oe-core depends on nose and the code has been completely bit-rotting since 2016. It could not be built anymore in oe-core and something had to give. > If we are planning > to include this in the upcoming release,we would need some commitment > and strategy to fix these and other ripple effects throughout the ecosystem > especially since this is an LTS release. I wish this changeset was proposed > in a release prior to LTS release, so it could have given enough time to > adjust for the ecosystem. This is will take some effort in stabilizing > This has been months of work on my part to attempt to get us to where upstream Python Packaging Authority has been pushing since PEP-517 was announced in 2015. We are very far behind the curve of upstream Python packaging. While I too would love to have had this ready sooner, that was not possible for many reasons. But no matter what, this would have been painful until recipes could be migrated, even if had been available at M1. > I also wonder, what do we lose if we do not put this into the LTS > release? We already have recipes in oe-core such as python3-jsonschema which can no longer be upgraded without moving to pyproject.toml based package builds. I'm sure there are others in meta-python and other layers in a similar story. We once again are in a situation that is a bad image for the project because we are using the now deprecated "setup.py install" and still building eggs. To step into an LTS release with the python packaging already entirely deprecated is just not a good impression. I understand your frustration with the build failures, but it has been a very solid week of getting oe-core stabilized (see https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3280). I am running a build now for meta-python with the improved wheel filename patch as well as a patch to check for whether the current recipe already has pyproject.toml back-end declared: https://lore.kernel.org/openembedded-core/20220224041200.2645054-2-tim.orling@konsulko.com/T/#u > [1] https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/1460 > > On Tue, Feb 22, 2022 at 11:23 AM Tim Orling <ticotimo@gmail.com> wrote: > > > > > > > > On Tue, Feb 22, 2022 at 11:18 AM Khem Raj <raj.khem@gmail.com> wrote: > >> > >> On Tue, Feb 22, 2022 at 10:40 AM Tim Orling <ticotimo@gmail.com> wrote: > >> > > >> > > >> > > >> > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <raj.khem@gmail.com> wrote: > >> >> > >> >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <ticotimo@gmail.com> > wrote: > >> >> > > >> >> > > >> >> > > >> >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com> > wrote: > >> >> >> > >> >> >> this is going into dep loops Try > >> >> >> > >> >> >> bitbake python3-wheel-native > >> >> > > >> >> > > >> >> > It shouldn’t > >> >> > > https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels > >> >> > > >> >> > >> >> I have been trying whats on master-next > >> >> > >> > > >> > I can't reproduce. Can you point me at an error log please? > >> > It might have something to do with sstate cache, but that's just a > hunch. > >> > > >> > >> This is because python3-wheel was moved to core and modified but not > >> deleted from meta-python at same time. I have sent a patch for that > >> > > Already sent one: > > > https://patchwork.yoctoproject.org/project/oe/patch/20220222184920.2310391-1-tim.orling@konsulko.com/ > > > >> > >> >> > >> >> > DEPENDS:remove:class-native = "python3-pip-native" > >> >> > > >> >> > do_install:class-native () { > >> >> > # We need to bootstrap python3-wheel-native > >> >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} > >> >> > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" > >> >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ > >> >> > bbfatal_log "Failed to install" > >> >> > } > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> > wrote: > >> >> >> > > >> >> >> > Upstream Python is moving to pyproject.toml rather than > setup.py and > >> >> >> > wheels rather than eggs. The main place this is documented is in > >> >> >> > PEP-517[1]. Installing packages with setup.py install is > deprecated. > >> >> >> > > >> >> >> > We have a choice with the 'kirkstone' LTS release of > introducing this > >> >> >> > series (and fixing any fall out during the stabilization > period) or > >> >> >> > living with deprecated Python packaging for the duration of the > LTS (at > >> >> >> > a minimum 2 years). This seems to be a compelling enough reason > to bring > >> >> >> > in this series, even if it results in a slight slip of the > 'feature > >> >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. > >> >> >> > > >> >> >> > This series bootstraps python3-setuptools-native, > >> >> >> > python3-flit-core-native, python3-wheel-native and > python3-pip-native, > >> >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This > allows all > >> >> >> > other recipes to be installed with pip, as intended by upstream. > >> >> >> > > >> >> >> > Three new classes are introduced: > >> >> >> > (1) flit_core which builds PEP-517 compliant packages which use > >> >> >> > flit_core.buildapi as their build-backend. > >> >> >> > (2) setuptools_build_meta which builds PEP-517 compliant > packages which use > >> >> >> > setuptools_build_meta as their build-backend. > >> >> >> > (3) pip_install_wheel, which simply installs wheels built by > other > >> >> >> > classes. > >> >> >> > > >> >> >> > The original setuptools3.bbclass do_compile is modified to > >> >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than > the deprecated > >> >> >> > 'setup.py build' mechanism. > >> >> >> > > >> >> >> > The original setuptools3.bbclass do_install is modified to > >> >> >> > use the pip_install_wheel.bbclass to install wheels with pip > rather than > >> >> >> > the deprecated 'setup.py install' mechanism. > >> >> >> > > >> >> >> > For pip install, we disable dependency checking (--no-deps) > because it > >> >> >> > proves to be brittle in our environment without providing full > wheel > >> >> >> > cache that seems like overhead that we do not need. We also > disable > >> >> >> > fetching wheels from pypi (--no-index). This is no different > than our > >> >> >> > normal workflow of requiring dependencies to be defined in a > recipe. > >> >> >> > > >> >> >> > Missing from this series is documentation of the new features, > classes > >> >> >> > and variables (this will be done under separate cover). > >> >> >> > > >> >> >> > Also desired and previously discussed in YP meetings is a helper > >> >> >> > function to warn that a recipe which currently inherits > setuptools3 > >> >> >> > but has a pyproject.toml and defines a > build-system.build-backend should > >> >> >> > instead use of the new PEP-517 classes. This function is still > in progress. > >> >> >> > > >> >> >> > During testing, recipes which use python3-setuptools-scm are > failing to > >> >> >> > determine the version of the python package, For several > recipes, the fix > >> >> >> > was to use pyproject.toml and PEP-517 build backends. To be > sure, other recipes > >> >> >> > are still going to fail for the short term, until the root > cause can be > >> >> >> > determined. This was not seen again in v2 builds. > >> >> >> > > >> >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has been > >> >> >> > identified in recipes in meta-python, but not in oe-core. The > >> >> >> > python3-poetry-core recipe is WIP, but brings in a number of > >> >> >> > dependencies that are perhaps not worth it to bring into > oe-core at this > >> >> >> > time. Rather we can support these recipes in meta-python? > >> >> >> > > >> >> >> > Fixes in v2 include passing --root=${D} to pip install (rather > than --target) > >> >> >> > and also passing --prefix so that scripts are (usually) > installed in the proper > >> >> >> > location. pip install also 'helpfully' overwrites python3 > intepreter with nativepython3 > >> >> >> > in #!, so add a fix to return the values to what is actually > correct (similar to what is > >> >> >> > done in setuptools3.bbclass). Also several recipes still needed > PIP_INSTALL_PACKAGE defined. > >> >> >> > > >> >> >> > Also added self as maintainer of new recipes. > >> >> >> > > >> >> >> > [YOCTO #14638] > >> >> >> > > >> >> >> > [1] https://www.python.org/dev/peps/pep-0517/ > >> >> >> > > >> >> >> > The following changes since commit > 41ae26ecab731f2fb6593df143476c94183be209: > >> >> >> > > >> >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) > >> >> >> > > >> >> >> > are available in the Git repository at: > >> >> >> > > >> >> >> > git://git.yoctoproject.org/git/poky-contrib > timo/bootstrap-wheels > >> >> >> > > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels > >> >> >> > > >> >> >> > Tim Orling (32): > >> >> >> > python3-wheel: move 0.37.0 from meta-python > >> >> >> > python3-flit-core: add recipe for 3.6.0 > >> >> >> > flit_core.bbclass: add helper for newer python packaging > >> >> >> > python3-wheel: inherit flit_core > >> >> >> > pip_install_wheel.bbclass: add helper class > >> >> >> > setuptools_build_meta.bbclass: add helper class > >> >> >> > python3-pip: inherit setuptools_build_meta > >> >> >> > python3-attrs: inherit setuptools_build_meta > >> >> >> > python3-git: inherit setuptools_build_meta > >> >> >> > python3-pytest: inherit setuptools_build_meta > >> >> >> > python3-setuptools-scm: inherit setuptools_build_meta > >> >> >> > python3-zipp: inherit setuptools_build_meta > >> >> >> > python3-iniconfig: inherit setuptools_build_meta > >> >> >> > python3-py: inherit setuptools_build_meta > >> >> >> > python3-pluggy: inherit setuptools_build_meta > >> >> >> > python3-setuptools: inherit setuptools_base_meta > >> >> >> > setuptools3.bbclass: refactor for wheels > >> >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE > >> >> >> > meson: inherit setuptools_build_meta > >> >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE > >> >> >> > python3-smartypants: patch hash bang to python3 > >> >> >> > python3-scons: merge -native recipe > >> >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE > >> >> >> > python3-magic: set PIP_INSTALL_PACKAGE > >> >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER > >> >> >> > asciidoc: set PIP_INSTALL_PACKAGE > >> >> >> > gi-docgen: set PIP_INSTALL_PACKAGE > >> >> >> > numpy: set PIP_INSTALL_PACKAGE > >> >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE > >> >> >> > python3-mako: inherit setuptools_build_meta > >> >> >> > python3-packaging: inherit setuptools_build_meta > >> >> >> > python3-nose: drop recipe > >> >> >> > > >> >> >> > meta/classes/flit_core.bbclass | 16 +++ > >> >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ > >> >> >> > meta/classes/setuptools3.bbclass | 20 ++-- > >> >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ > >> >> >> > meta/conf/distro/include/maintainers.inc | 3 +- > >> >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- > >> >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- > >> >> >> > .../python/python-subunit.inc | 14 --- > >> >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- > >> >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + > >> >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ > >> >> >> > .../python/python3-git_3.1.26.bb | 2 +- > >> >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- > >> >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + > >> >> >> > .../python/python3-magic_0.4.25.bb | 1 + > >> >> >> > .../python/python3-mako_1.1.6.bb | 2 +- > >> >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + > >> >> >> > .../python/python3-nose_1.3.7.bb | 2 - > >> >> >> > .../python/python3-numpy_1.22.2.bb | 1 + > >> >> >> > .../python/python3-packaging_21.3.bb | 2 +- > >> >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- > >> >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- > >> >> >> > .../python/python3-py_1.11.0.bb | 2 +- > >> >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- > >> >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- > >> >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- > >> >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- > >> >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- > >> >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ > >> >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + > >> >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ > >> >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- > >> >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- > >> >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 > ++++++++++++++++++ > >> >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ > >> >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- > >> >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + > >> >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + > >> >> >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- > >> >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) > >> >> >> > create mode 100644 meta/classes/flit_core.bbclass > >> >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass > >> >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass > >> >> >> > delete mode 100644 meta/recipes-devtools/python/python-nose.inc > >> >> >> > delete mode 100644 > meta/recipes-devtools/python/python-subunit.inc > >> >> >> > create mode 100644 meta/recipes-devtools/python/ > python3-flit-core_3.6.0.bb > >> >> >> > delete mode 100644 meta/recipes-devtools/python/ > python3-nose_1.3.7.bb > >> >> >> > delete mode 100644 meta/recipes-devtools/python/ > python3-scons-native_4.3.0.bb > >> >> >> > create mode 100644 > meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch > >> >> >> > create mode 100644 > meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch > >> >> >> > create mode 100644 > meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch > >> >> >> > create mode 100644 meta/recipes-devtools/python/ > python3-wheel_0.37.1.bb > >> >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => > bmap-tools_git.bb} (86%) > >> >> >> > > >> >> >> > -- > >> >> >> > 2.30.2 > >> >> >> > > >> >> >> > > >> >> >> > -=-=-=-=-=-=-=-=-=-=-=- > >> >> >> > Links: You receive all messages sent to this group. > >> >> >> > View/Reply Online (#162076): > https://lists.openembedded.org/g/openembedded-core/message/162076 > >> >> >> > Mute This Topic: > https://lists.openembedded.org/mt/89304007/1997914 > >> >> >> > Group Owner: openembedded-core+owner@lists.openembedded.org > >> >> >> > Unsubscribe: > https://lists.openembedded.org/g/openembedded-core/unsub [ > raj.khem@gmail.com] > >> >> >> > -=-=-=-=-=-=-=-=-=-=-=- > >> >> >> > >
On Wed, Feb 23, 2022 at 8:50 PM Tim Orling via lists.openembedded.org <ticotimo=gmail.com@lists.openembedded.org> wrote: > > > On Wed, Feb 23, 2022 at 8:07 PM Khem Raj <raj.khem@gmail.com> wrote: > >> Right now approx 150 out of 470 recipes [1] in meta-python are directly >> failing, >> perhaps there will be more once these are fixed then there are other >> layers >> which might be carrying some impacted python modules. > > > The issue has mostly been detection of the wheel filename and this should > be improved by: > > https://lore.kernel.org/openembedded-core/20220224041200.2645054-1-tim.orling@konsulko.com/T/#t > > Also YP compatibility >> is broken due to python3-nose removal. > > > We can try to run 2to3 on the python3-nose plugins or remove them > surgically to reintroduce cleanly Python 3 only > python3-nose. I propose that we move it to meta-python even so because > nothing in oe-core depends on nose and the > code has been completely bit-rotting since 2016. It could not be built > anymore in oe-core and something had to give. > Perhaps we can switch to this repo: https://github.com/atsb/nose-py3 > > >> If we are planning >> to include this in the upcoming release,we would need some commitment >> and strategy to fix these and other ripple effects throughout the >> ecosystem >> especially since this is an LTS release. I wish this changeset was >> proposed >> in a release prior to LTS release, so it could have given enough time to >> adjust for the ecosystem. This is will take some effort in stabilizing >> > > This has been months of work on my part to attempt to get us to where > upstream > Python Packaging Authority has been pushing since PEP-517 was announced in > 2015. > We are very far behind the curve of upstream Python packaging. > > While I too would love to have had this ready sooner, that was not > possible for many reasons. > But no matter what, this would have been painful until recipes could be > migrated, even if had > been available at M1. > > >> I also wonder, what do we lose if we do not put this into the LTS >> release? > > > We already have recipes in oe-core such as python3-jsonschema which can no > longer be upgraded > without moving to pyproject.toml based package builds. I'm sure there are > others in meta-python and > other layers in a similar story. We once again are in a situation that is > a bad image for the project because > we are using the now deprecated "setup.py install" and still building > eggs. To step into an LTS release > with the python packaging already entirely deprecated is just not a good > impression. > > I understand your frustration with the build failures, but it has been a > very solid week of getting oe-core > stabilized (see > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3280). > > I am running a build now for meta-python with the improved wheel filename > patch as well as a patch to > check for whether the current recipe already has pyproject.toml back-end > declared: > > https://lore.kernel.org/openembedded-core/20220224041200.2645054-2-tim.orling@konsulko.com/T/#u > > >> > [1] >> https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/1460 > > >> >> On Tue, Feb 22, 2022 at 11:23 AM Tim Orling <ticotimo@gmail.com> wrote: >> > >> > >> > >> > On Tue, Feb 22, 2022 at 11:18 AM Khem Raj <raj.khem@gmail.com> wrote: >> >> >> >> On Tue, Feb 22, 2022 at 10:40 AM Tim Orling <ticotimo@gmail.com> >> wrote: >> >> > >> >> > >> >> > >> >> > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <raj.khem@gmail.com> >> wrote: >> >> >> >> >> >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <ticotimo@gmail.com> >> wrote: >> >> >> > >> >> >> > >> >> >> > >> >> >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com> >> wrote: >> >> >> >> >> >> >> >> this is going into dep loops Try >> >> >> >> >> >> >> >> bitbake python3-wheel-native >> >> >> > >> >> >> > >> >> >> > It shouldn’t >> >> >> > >> https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels >> >> >> > >> >> >> >> >> >> I have been trying whats on master-next >> >> >> >> >> > >> >> > I can't reproduce. Can you point me at an error log please? >> >> > It might have something to do with sstate cache, but that's just a >> hunch. >> >> > >> >> >> >> This is because python3-wheel was moved to core and modified but not >> >> deleted from meta-python at same time. I have sent a patch for that >> >> >> > Already sent one: >> > >> https://patchwork.yoctoproject.org/project/oe/patch/20220222184920.2310391-1-tim.orling@konsulko.com/ >> > >> >> >> >> >> >> >> >> > DEPENDS:remove:class-native = "python3-pip-native" >> >> >> > >> >> >> > do_install:class-native () { >> >> >> > # We need to bootstrap python3-wheel-native >> >> >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} >> >> >> > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" >> >> >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ >> >> >> > bbfatal_log "Failed to install" >> >> >> > } >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <ticotimo@gmail.com> >> wrote: >> >> >> >> > >> >> >> >> > Upstream Python is moving to pyproject.toml rather than >> setup.py and >> >> >> >> > wheels rather than eggs. The main place this is documented is >> in >> >> >> >> > PEP-517[1]. Installing packages with setup.py install is >> deprecated. >> >> >> >> > >> >> >> >> > We have a choice with the 'kirkstone' LTS release of >> introducing this >> >> >> >> > series (and fixing any fall out during the stabilization >> period) or >> >> >> >> > living with deprecated Python packaging for the duration of >> the LTS (at >> >> >> >> > a minimum 2 years). This seems to be a compelling enough >> reason to bring >> >> >> >> > in this series, even if it results in a slight slip of the >> 'feature >> >> >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. >> >> >> >> > >> >> >> >> > This series bootstraps python3-setuptools-native, >> >> >> >> > python3-flit-core-native, python3-wheel-native and >> python3-pip-native, >> >> >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This >> allows all >> >> >> >> > other recipes to be installed with pip, as intended by >> upstream. >> >> >> >> > >> >> >> >> > Three new classes are introduced: >> >> >> >> > (1) flit_core which builds PEP-517 compliant packages which use >> >> >> >> > flit_core.buildapi as their build-backend. >> >> >> >> > (2) setuptools_build_meta which builds PEP-517 compliant >> packages which use >> >> >> >> > setuptools_build_meta as their build-backend. >> >> >> >> > (3) pip_install_wheel, which simply installs wheels built by >> other >> >> >> >> > classes. >> >> >> >> > >> >> >> >> > The original setuptools3.bbclass do_compile is modified to >> >> >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than >> the deprecated >> >> >> >> > 'setup.py build' mechanism. >> >> >> >> > >> >> >> >> > The original setuptools3.bbclass do_install is modified to >> >> >> >> > use the pip_install_wheel.bbclass to install wheels with pip >> rather than >> >> >> >> > the deprecated 'setup.py install' mechanism. >> >> >> >> > >> >> >> >> > For pip install, we disable dependency checking (--no-deps) >> because it >> >> >> >> > proves to be brittle in our environment without providing full >> wheel >> >> >> >> > cache that seems like overhead that we do not need. We also >> disable >> >> >> >> > fetching wheels from pypi (--no-index). This is no different >> than our >> >> >> >> > normal workflow of requiring dependencies to be defined in a >> recipe. >> >> >> >> > >> >> >> >> > Missing from this series is documentation of the new features, >> classes >> >> >> >> > and variables (this will be done under separate cover). >> >> >> >> > >> >> >> >> > Also desired and previously discussed in YP meetings is a >> helper >> >> >> >> > function to warn that a recipe which currently inherits >> setuptools3 >> >> >> >> > but has a pyproject.toml and defines a >> build-system.build-backend should >> >> >> >> > instead use of the new PEP-517 classes. This function is still >> in progress. >> >> >> >> > >> >> >> >> > During testing, recipes which use python3-setuptools-scm are >> failing to >> >> >> >> > determine the version of the python package, For several >> recipes, the fix >> >> >> >> > was to use pyproject.toml and PEP-517 build backends. To be >> sure, other recipes >> >> >> >> > are still going to fail for the short term, until the root >> cause can be >> >> >> >> > determined. This was not seen again in v2 builds. >> >> >> >> > >> >> >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has been >> >> >> >> > identified in recipes in meta-python, but not in oe-core. The >> >> >> >> > python3-poetry-core recipe is WIP, but brings in a number of >> >> >> >> > dependencies that are perhaps not worth it to bring into >> oe-core at this >> >> >> >> > time. Rather we can support these recipes in meta-python? >> >> >> >> > >> >> >> >> > Fixes in v2 include passing --root=${D} to pip install (rather >> than --target) >> >> >> >> > and also passing --prefix so that scripts are (usually) >> installed in the proper >> >> >> >> > location. pip install also 'helpfully' overwrites python3 >> intepreter with nativepython3 >> >> >> >> > in #!, so add a fix to return the values to what is actually >> correct (similar to what is >> >> >> >> > done in setuptools3.bbclass). Also several recipes still >> needed PIP_INSTALL_PACKAGE defined. >> >> >> >> > >> >> >> >> > Also added self as maintainer of new recipes. >> >> >> >> > >> >> >> >> > [YOCTO #14638] >> >> >> >> > >> >> >> >> > [1] https://www.python.org/dev/peps/pep-0517/ >> >> >> >> > >> >> >> >> > The following changes since commit >> 41ae26ecab731f2fb6593df143476c94183be209: >> >> >> >> > >> >> >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) >> >> >> >> > >> >> >> >> > are available in the Git repository at: >> >> >> >> > >> >> >> >> > git://git.yoctoproject.org/git/poky-contrib >> timo/bootstrap-wheels >> >> >> >> > >> http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels >> >> >> >> > >> >> >> >> > Tim Orling (32): >> >> >> >> > python3-wheel: move 0.37.0 from meta-python >> >> >> >> > python3-flit-core: add recipe for 3.6.0 >> >> >> >> > flit_core.bbclass: add helper for newer python packaging >> >> >> >> > python3-wheel: inherit flit_core >> >> >> >> > pip_install_wheel.bbclass: add helper class >> >> >> >> > setuptools_build_meta.bbclass: add helper class >> >> >> >> > python3-pip: inherit setuptools_build_meta >> >> >> >> > python3-attrs: inherit setuptools_build_meta >> >> >> >> > python3-git: inherit setuptools_build_meta >> >> >> >> > python3-pytest: inherit setuptools_build_meta >> >> >> >> > python3-setuptools-scm: inherit setuptools_build_meta >> >> >> >> > python3-zipp: inherit setuptools_build_meta >> >> >> >> > python3-iniconfig: inherit setuptools_build_meta >> >> >> >> > python3-py: inherit setuptools_build_meta >> >> >> >> > python3-pluggy: inherit setuptools_build_meta >> >> >> >> > python3-setuptools: inherit setuptools_base_meta >> >> >> >> > setuptools3.bbclass: refactor for wheels >> >> >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE >> >> >> >> > meson: inherit setuptools_build_meta >> >> >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE >> >> >> >> > python3-smartypants: patch hash bang to python3 >> >> >> >> > python3-scons: merge -native recipe >> >> >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE >> >> >> >> > python3-magic: set PIP_INSTALL_PACKAGE >> >> >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER >> >> >> >> > asciidoc: set PIP_INSTALL_PACKAGE >> >> >> >> > gi-docgen: set PIP_INSTALL_PACKAGE >> >> >> >> > numpy: set PIP_INSTALL_PACKAGE >> >> >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE >> >> >> >> > python3-mako: inherit setuptools_build_meta >> >> >> >> > python3-packaging: inherit setuptools_build_meta >> >> >> >> > python3-nose: drop recipe >> >> >> >> > >> >> >> >> > meta/classes/flit_core.bbclass | 16 +++ >> >> >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ >> >> >> >> > meta/classes/setuptools3.bbclass | 20 ++-- >> >> >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ >> >> >> >> > meta/conf/distro/include/maintainers.inc | 3 +- >> >> >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- >> >> >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- >> >> >> >> > .../python/python-subunit.inc | 14 --- >> >> >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- >> >> >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + >> >> >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ >> >> >> >> > .../python/python3-git_3.1.26.bb | 2 +- >> >> >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- >> >> >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + >> >> >> >> > .../python/python3-magic_0.4.25.bb | 1 + >> >> >> >> > .../python/python3-mako_1.1.6.bb | 2 +- >> >> >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + >> >> >> >> > .../python/python3-nose_1.3.7.bb | 2 - >> >> >> >> > .../python/python3-numpy_1.22.2.bb | 1 + >> >> >> >> > .../python/python3-packaging_21.3.bb | 2 +- >> >> >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- >> >> >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- >> >> >> >> > .../python/python3-py_1.11.0.bb | 2 +- >> >> >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- >> >> >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- >> >> >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- >> >> >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- >> >> >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- >> >> >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ >> >> >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + >> >> >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ >> >> >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- >> >> >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- >> >> >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 >> ++++++++++++++++++ >> >> >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ >> >> >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- >> >> >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + >> >> >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + >> >> >> >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- >> >> >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) >> >> >> >> > create mode 100644 meta/classes/flit_core.bbclass >> >> >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass >> >> >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass >> >> >> >> > delete mode 100644 >> meta/recipes-devtools/python/python-nose.inc >> >> >> >> > delete mode 100644 >> meta/recipes-devtools/python/python-subunit.inc >> >> >> >> > create mode 100644 meta/recipes-devtools/python/ >> python3-flit-core_3.6.0.bb >> >> >> >> > delete mode 100644 meta/recipes-devtools/python/ >> python3-nose_1.3.7.bb >> >> >> >> > delete mode 100644 meta/recipes-devtools/python/ >> python3-scons-native_4.3.0.bb >> >> >> >> > create mode 100644 >> meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch >> >> >> >> > create mode 100644 >> meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch >> >> >> >> > create mode 100644 >> meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch >> >> >> >> > create mode 100644 meta/recipes-devtools/python/ >> python3-wheel_0.37.1.bb >> >> >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => >> bmap-tools_git.bb} (86%) >> >> >> >> > >> >> >> >> > -- >> >> >> >> > 2.30.2 >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#162292): > https://lists.openembedded.org/g/openembedded-core/message/162292 > Mute This Topic: https://lists.openembedded.org/mt/89304007/924729 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > ticotimo@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Upstream Python is moving to pyproject.toml rather than setup.py and wheels rather than eggs. The main place this is documented is in PEP-517[1]. Installing packages with setup.py install is deprecated. We have a choice with the 'kirkstone' LTS release of introducing this series (and fixing any fall out during the stabilization period) or living with deprecated Python packaging for the duration of the LTS (at a minimum 2 years). This seems to be a compelling enough reason to bring in this series, even if it results in a slight slip of the 'feature freeze', which is officially tomorrow, Monday 21 Feb, 2022. This series bootstraps python3-setuptools-native, python3-flit-core-native, python3-wheel-native and python3-pip-native, simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows all other recipes to be installed with pip, as intended by upstream. Three new classes are introduced: (1) flit_core which builds PEP-517 compliant packages which use flit_core.buildapi as their build-backend. (2) setuptools_build_meta which builds PEP-517 compliant packages which use setuptools_build_meta as their build-backend. (3) pip_install_wheel, which simply installs wheels built by other classes. The original setuptools3.bbclass do_compile is modified to 'setup.py bdist_wheel' (from python3-wheel-native) rather than the deprecated 'setup.py build' mechanism. The original setuptools3.bbclass do_install is modified to use the pip_install_wheel.bbclass to install wheels with pip rather than the deprecated 'setup.py install' mechanism. For pip install, we disable dependency checking (--no-deps) because it proves to be brittle in our environment without providing full wheel cache that seems like overhead that we do not need. We also disable fetching wheels from pypi (--no-index). This is no different than our normal workflow of requiring dependencies to be defined in a recipe. Missing from this series is documentation of the new features, classes and variables (this will be done under separate cover). Also desired and previously discussed in YP meetings is a helper function to warn that a recipe which currently inherits setuptools3 but has a pyproject.toml and defines a build-system.build-backend should instead use of the new PEP-517 classes. This function is still in progress. During testing, recipes which use python3-setuptools-scm are failing to determine the version of the python package, For several recipes, the fix was to use pyproject.toml and PEP-517 build backends. To be sure, other recipes are still going to fail for the short term, until the root cause can be determined. This was not seen again in v2 builds. Another PEP-517 build-backend, poetry.core.masonry.api has been identified in recipes in meta-python, but not in oe-core. The python3-poetry-core recipe is WIP, but brings in a number of dependencies that are perhaps not worth it to bring into oe-core at this time. Rather we can support these recipes in meta-python? Fixes in v2 include passing --root=${D} to pip install (rather than --target) and also passing --prefix so that scripts are (usually) installed in the proper location. pip install also 'helpfully' overwrites python3 intepreter with nativepython3 in #!, so add a fix to return the values to what is actually correct (similar to what is done in setuptools3.bbclass). Also several recipes still needed PIP_INSTALL_PACKAGE defined. Also added self as maintainer of new recipes. [YOCTO #14638] [1] https://www.python.org/dev/peps/pep-0517/ The following changes since commit 41ae26ecab731f2fb6593df143476c94183be209: releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) are available in the Git repository at: git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels Tim Orling (32): python3-wheel: move 0.37.0 from meta-python python3-flit-core: add recipe for 3.6.0 flit_core.bbclass: add helper for newer python packaging python3-wheel: inherit flit_core pip_install_wheel.bbclass: add helper class setuptools_build_meta.bbclass: add helper class python3-pip: inherit setuptools_build_meta python3-attrs: inherit setuptools_build_meta python3-git: inherit setuptools_build_meta python3-pytest: inherit setuptools_build_meta python3-setuptools-scm: inherit setuptools_build_meta python3-zipp: inherit setuptools_build_meta python3-iniconfig: inherit setuptools_build_meta python3-py: inherit setuptools_build_meta python3-pluggy: inherit setuptools_build_meta python3-setuptools: inherit setuptools_base_meta setuptools3.bbclass: refactor for wheels python3-more-itertools: set PIP_INSTALL_PACKAGE meson: inherit setuptools_build_meta python3-libarchive-d: set PIP_INSTALL_PACKAGE python3-smartypants: patch hash bang to python3 python3-scons: merge -native recipe python3-subunit: merge inc; set PIP_INSTALL_PACKAGE python3-magic: set PIP_INSTALL_PACKAGE bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER asciidoc: set PIP_INSTALL_PACKAGE gi-docgen: set PIP_INSTALL_PACKAGE numpy: set PIP_INSTALL_PACKAGE python3-dbusmock: set PIP_INSTALL_PACKAGE python3-mako: inherit setuptools_build_meta python3-packaging: inherit setuptools_build_meta python3-nose: drop recipe meta/classes/flit_core.bbclass | 16 +++ meta/classes/pip_install_wheel.bbclass | 37 +++++++ meta/classes/setuptools3.bbclass | 20 ++-- meta/classes/setuptools_build_meta.bbclass | 18 ++++ meta/conf/distro/include/maintainers.inc | 3 +- meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- meta/recipes-devtools/python/python-nose.inc | 18 ---- .../python/python-subunit.inc | 14 --- .../python/python3-attrs_21.4.0.bb | 2 +- .../python/python3-dbusmock_0.25.0.bb | 1 + .../python/python3-flit-core_3.6.0.bb | 41 +++++++ .../python/python3-git_3.1.26.bb | 2 +- .../python/python3-iniconfig_1.1.1.bb | 4 +- .../python/python3-libarchive-c_4.0.bb | 2 + .../python/python3-magic_0.4.25.bb | 1 + .../python/python3-mako_1.1.6.bb | 2 +- .../python/python3-more-itertools_8.12.0.bb | 2 + .../python/python3-nose_1.3.7.bb | 2 - .../python/python3-numpy_1.22.2.bb | 1 + .../python/python3-packaging_21.3.bb | 2 +- .../python/python3-pip_22.0.3.bb | 19 +++- .../python/python3-pluggy_1.0.0.bb | 2 +- .../python/python3-py_1.11.0.bb | 2 +- .../python/python3-pytest_7.0.1.bb | 4 +- .../python/python3-scons-native_4.3.0.bb | 7 -- .../python/python3-scons_4.3.0.bb | 13 ++- .../python/python3-setuptools-scm_6.4.2.bb | 2 +- .../python/python3-setuptools_59.5.0.bb | 19 +++- .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ .../python/python3-smartypants_2.0.0.bb | 1 + ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ .../python/python3-strict-rfc3339_0.7.bb | 4 +- .../python/python3-subunit_1.4.0.bb | 17 ++- ...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++ .../python/python3-wheel_0.37.1.bb | 24 +++++ .../python/python3-zipp_3.7.0.bb | 2 +- .../asciidoc/asciidoc_10.1.1.bb | 1 + meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- 39 files changed, 412 insertions(+), 78 deletions(-) create mode 100644 meta/classes/flit_core.bbclass create mode 100644 meta/classes/pip_install_wheel.bbclass create mode 100644 meta/classes/setuptools_build_meta.bbclass delete mode 100644 meta/recipes-devtools/python/python-nose.inc delete mode 100644 meta/recipes-devtools/python/python-subunit.inc create mode 100644 meta/recipes-devtools/python/python3-flit-core_3.6.0.bb delete mode 100644 meta/recipes-devtools/python/python3-nose_1.3.7.bb delete mode 100644 meta/recipes-devtools/python/python3-scons-native_4.3.0.bb create mode 100644 meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch create mode 100644 meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch create mode 100644 meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => bmap-tools_git.bb} (86%)