From patchwork Thu Mar 10 17:16:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 5077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94EEAC433EF for ; Thu, 10 Mar 2022 17:17:00 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.12211.1646932619680684346 for ; Thu, 10 Mar 2022 09:17:00 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8AC051692 for ; Thu, 10 Mar 2022 09:16:58 -0800 (PST) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 38BF23F99C for ; Thu, 10 Mar 2022 09:16:58 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 1/2] python3-installer: add installer module Date: Thu, 10 Mar 2022 17:16:55 +0000 Message-Id: <20220310171656.3106028-1-ross.burton@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Mar 2022 17:17:00 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/163032 Add a recipe for Installer, a minimal library/tool to install Python Wheels. Unlike PIP, it explicitly only installs wheels and does nothing else. Signed-off-by: Ross Burton --- .../python/python3-installer_0.5.0.bb | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-installer_0.5.0.bb diff --git a/meta/recipes-devtools/python/python3-installer_0.5.0.bb b/meta/recipes-devtools/python/python3-installer_0.5.0.bb new file mode 100644 index 0000000000..bde9397569 --- /dev/null +++ b/meta/recipes-devtools/python/python3-installer_0.5.0.bb @@ -0,0 +1,20 @@ +SUMMARY = "A library for installing Python wheels" +DESCRIPTION = "a low-level library for installing a Python package from a wheel distribution." +HOMEPAGE = "https://installer.readthedocs.io/" +BUGTRACKER = "https://github.com/pradyunsg/installer/issues" + +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://LICENSE;md5=5038641aec7a77451e31da828ebfae00" + +SRC_URI[sha256sum] = "0cd6bdab3b358cf7e8749370b99aef9e12202751271c5ddb22126599b34dc665" + +inherit pypi flit_core + +DEPENDS:remove:class-native = "python3-installer-native" +DEPENDS:append:class-native = " unzip-native" + +do_install:class-native () { + pip_install_wheel_do_bootstrap_install +} + +BBCLASSEXTEND = "native nativesdk" From patchwork Thu Mar 10 17:16:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 5078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 969D3C433FE for ; Thu, 10 Mar 2022 17:17:00 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web09.12056.1646932619684278523 for ; Thu, 10 Mar 2022 09:17:00 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1B10816A3 for ; Thu, 10 Mar 2022 09:16:59 -0800 (PST) Received: from oss-tx204.lab.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BF8713F99C for ; Thu, 10 Mar 2022 09:16:58 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 2/2] pip_install_wheel: use installer instead of pip Date: Thu, 10 Mar 2022 17:16:56 +0000 Message-Id: <20220310171656.3106028-2-ross.burton@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220310171656.3106028-1-ross.burton@arm.com> References: <20220310171656.3106028-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Mar 2022 17:17:00 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/163033 Instead of battling pip to install a wheel, use installer. Installer does one thing, so it's faster and easier to work with. This means setuptools, pip, and wheel are no longer part of the bootstrap phase, so they can be built normally. To avoid sysroot file conflicts these three recipes can't install .pyc files to the native sysroot. Signed-off-by: Ross Burton --- meta/classes/pip_install_wheel.bbclass | 16 +++------ .../python/python3-flit-core_3.7.1.bb | 2 +- .../python/python3-pip_22.0.3.bb | 35 +++---------------- .../python/python3-setuptools_59.5.0.bb | 12 +++---- .../python/python3-wheel_0.37.1.bb | 26 +++----------- 5 files changed, 18 insertions(+), 73 deletions(-) diff --git a/meta/classes/pip_install_wheel.bbclass b/meta/classes/pip_install_wheel.bbclass index 29cd544aa3..497f69260e 100644 --- a/meta/classes/pip_install_wheel.bbclass +++ b/meta/classes/pip_install_wheel.bbclass @@ -1,23 +1,15 @@ -DEPENDS:append = " python3-pip-native" +DEPENDS:append = " python3-installer-native" # The directory where wheels should be written too. Build classes # will ideally [cleandirs] this but we don't do that here in case # a recipe wants to install prebuilt wheels. PIP_INSTALL_DIST_PATH ?= "${WORKDIR}/dist" -PIP_INSTALL_ARGS = "\ - -vvvv \ - --ignore-installed \ - --no-cache \ - --no-deps \ - --no-index \ - --root=${D} \ - --prefix=${prefix} \ -" - PIP_INSTALL_PYTHON = "python3" PIP_INSTALL_PYTHON:class-native = "nativepython3" +INSTALL_WHEEL_COMPILE_BYTECODE ?= "--compile-bytecode=0" + pip_install_wheel_do_install () { COUNT=$(find ${PIP_INSTALL_DIST_PATH} -name '*.whl' | wc -l) if test $COUNT -eq 0; then @@ -26,7 +18,7 @@ pip_install_wheel_do_install () { bbfatal More than one wheel found in ${PIP_INSTALL_DIST_PATH}, this should not happen fi - nativepython3 -m pip install ${PIP_INSTALL_ARGS} ${PIP_INSTALL_DIST_PATH}/*.whl + nativepython3 -m installer ${INSTALL_WHEEL_COMPILE_BYTECODE} --destdir=${D} ${PIP_INSTALL_DIST_PATH}/*.whl cd ${D} for i in ${D}${bindir}/* ${D}${sbindir}/*; do diff --git a/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb b/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb index 1f14345d50..d4993fee69 100644 --- a/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb +++ b/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb @@ -13,7 +13,7 @@ SRC_URI[sha256sum] = "3c9bd9c140515bfe62dd938c6610d10d6efb9e35cc647fc614fe5fb3a5 inherit pypi flit_core # Need to install by hand as there's a dependency loop -DEPENDS:remove:class-native = " python3-pip-native" +DEPENDS:remove:class-native = " python3-installer-native" DEPENDS:append:class-native = " unzip-native" # We need the full flit tarball diff --git a/meta/recipes-devtools/python/python3-pip_22.0.3.bb b/meta/recipes-devtools/python/python3-pip_22.0.3.bb index bce3b68861..9ca8fbc1e5 100644 --- a/meta/recipes-devtools/python/python3-pip_22.0.3.bb +++ b/meta/recipes-devtools/python/python3-pip_22.0.3.bb @@ -6,43 +6,14 @@ LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=63ec52baf95163b597008bb46db68030" inherit pypi setuptools_build_meta -DEPENDS += "python3" - -# To avoid a dependency loop; we bootstrap -native -DEPENDS:remove:class-native = "python3-pip-native" -DEPENDS:append:class-native = " unzip-native" - SRC_URI += "file://0001-change-shebang-to-python3.patch" SRC_URI += "file://no_shebang_mangling.patch" SRC_URI += "file://reproducible.patch" SRC_URI[sha256sum] = "f29d589df8c8ab99c060e68ad294c4a9ed896624f6368c5349d70aa581b333d0" -do_install:class-native() { - pip_install_wheel_do_bootstrap_install - - # pip install would normally generate [console_scripts] in ${bindir} - install -d ${D}/${bindir} - # We will skip the ${bindir}/pip variant as we would just remove it in the do_install:append - cat << EOF >> ${D}/${bindir}/pip3 | tee ${D}/${bindir}/pip${PYTHON_BASEVERSION} -#!/bin/sh -'''exec' ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} "\$0" "\$@" -' ''' -# -*- coding: utf-8 -*- -import re -import sys -from pip._internal.cli.main import main -if __name__ == '__main__': - sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) - sys.exit(main()) -EOF - chmod 0755 ${D}${bindir}/pip3 ${D}${bindir}/pip${PYTHON_BASEVERSION} -} - do_install:append() { - if [ -e ${D}/${bindir}/pip ]; then - rm ${D}/${bindir}/pip - fi + rm -f ${D}/${bindir}/pip } RDEPENDS:${PN} = "\ @@ -59,3 +30,7 @@ RDEPENDS:${PN} = "\ " BBCLASSEXTEND = "native nativesdk" + +# This used to use the bootstrap install which didn't compile. Until we bump the +# tmpdir version we can't compile the native otherwise the sysroot unpack fails +INSTALL_WHEEL_COMPILE_BYTECODE:class-native = "--no-compile-bytecode" diff --git a/meta/recipes-devtools/python/python3-setuptools_59.5.0.bb b/meta/recipes-devtools/python/python3-setuptools_59.5.0.bb index 7cd67b85f9..35bec19e33 100644 --- a/meta/recipes-devtools/python/python3-setuptools_59.5.0.bb +++ b/meta/recipes-devtools/python/python3-setuptools_59.5.0.bb @@ -17,14 +17,6 @@ SRC_URI[sha256sum] = "d144f85102f999444d06f9c0e8c737fd0194f10f2f7e5fdb77573f6e2f DEPENDS += "${PYTHON_PN}" -# Avoid dependency loop; we bootstrap -native -DEPENDS:remove:class-native = "python3-pip-native python3-setuptools-native" -DEPENDS:append:class-native = " unzip-native" - -do_install:class-native() { - pip_install_wheel_do_bootstrap_install -} - RDEPENDS:${PN} = "\ ${PYTHON_PN}-2to3 \ ${PYTHON_PN}-compile \ @@ -59,3 +51,7 @@ RDEPENDS:${PYTHON_PN}-pkg-resources = "\ ${PYTHON_PN}-plistlib \ ${PYTHON_PN}-pprint \ " + +# This used to use the bootstrap install which didn't compile. Until we bump the +# tmpdir version we can't compile the native otherwise the sysroot unpack fails +INSTALL_WHEEL_COMPILE_BYTECODE:class-native = "--no-compile-bytecode" diff --git a/meta/recipes-devtools/python/python3-wheel_0.37.1.bb b/meta/recipes-devtools/python/python3-wheel_0.37.1.bb index cf0d8191b4..efd6c2f968 100644 --- a/meta/recipes-devtools/python/python3-wheel_0.37.1.bb +++ b/meta/recipes-devtools/python/python3-wheel_0.37.1.bb @@ -10,26 +10,8 @@ inherit flit_core pypi SRC_URI += " file://0001-Backport-pyproject.toml-from-flit-backend-branch.patch" -DEPENDS:remove:class-native = "python3-pip-native" - -do_install:class-native () { - pip_install_wheel_do_bootstrap_install - - # pip install would normally generate [project.scripts] in ${bindir} - install -d ${D}/${bindir} - cat << EOF >> ${D}/${bindir}/wheel -#!/bin/sh -'''exec' ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} "\$0" "\$@" -' ''' -# -*- coding: utf-8 -*- -import re -import sys -from wheel.cli import main -if __name__ == '__main__': - sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) - sys.exit(main()) -EOF - chmod 0755 ${D}${bindir}/wheel -} - BBCLASSEXTEND = "native nativesdk" + +# This used to use the bootstrap install which didn't compile. Until we bump the +# tmpdir version we can't compile the native otherwise the sysroot unpack fails +INSTALL_WHEEL_COMPILE_BYTECODE:class-native = "--no-compile-bytecode"