| Message ID | 20251212091927.3741063-1-m.schwan@phytec.de |
|---|---|
| State | New |
| Headers | show |
| Series | [v2,1/5] cargo-update-recipe-crates: Make do_update_crates() a Python function | expand |
On Fri, 12 Dec 2025 at 10:19, Martin Schwan via lists.openembedded.org <m.schwan=phytec.de@lists.openembedded.org> wrote: > +python do_update_crates() { > + import tomllib Sadly this is a problem right here. tomllib is available since python 3.11, and the minimum python requirement for oe-core is 3.9, as seen in sanity.bbclass. Yocto CI builds are executed on older distibutions to check that dependencies on newer python features do not creep in. I suspect this is why this was and still should be piped into python3-native, since that is provided from a recipe in oe-core. Alex
On Fri, 2025-12-12 at 15:58 +0100, Alexander Kanavin wrote: > On Fri, 12 Dec 2025 at 10:19, Martin Schwan via > lists.openembedded.org > <m.schwan=phytec.de@lists.openembedded.org> wrote: > > +python do_update_crates() { > > + import tomllib > > Sadly this is a problem right here. > > tomllib is available since python 3.11, and the minimum python > requirement for oe-core is 3.9, as seen in sanity.bbclass. Okay, understood. Are there any plans to move to Python 3.11? Python 3.9 is now at its EOL for some months. I don't want to open a can of worms, but just like to understand the future/intentions in this regard. > Yocto CI > builds are executed on older distibutions to check that dependencies > on newer python features do not creep in. I suspect this is why this > was and still should be piped into python3-native, since that is > provided from a recipe in oe-core. Alright, I'll revert that thing back then and pipe it to python3-native again. Martin > > Alex
On Wed, 17 Dec 2025 at 09:57, Martin Schwan <M.Schwan@phytec.de> wrote: > Are there any plans to move to Python 3.11? Python 3.9 is now at its > EOL for some months. I don't want to open a can of worms, but just like > to understand the future/intentions in this regard. There is no specific process or criteria for bumping the minimum python version, it happens every sometimes - you can probably dig through git history and read the commit messages for any such version increases. Doing that should not clash too much with what is in the set of supported host distributions, e.g. maybe one or two soon-obsolete distros may not have it, and so installing buildtools would be required for them going forward, but not more. Note that sanity.bbclass is not the only place where python version is checked. There's also a check in meta/scripts/oe-buildenv-internal, and bitbake/lib/bb/__init__.py Alex
On 17 Dec 2025, at 08:57, Martin Schwan via lists.openembedded.org <m.schwan=phytec.de@lists.openembedded.org> wrote: > Are there any plans to move to Python 3.11? Python 3.9 is now at its > EOL for some months. I don't want to open a can of worms, but just like > to understand the future/intentions in this regard. I’ll happily endorse a patch to upgrade the minimal python version for master, but the following needs to happen: For every supported distribution [1] where buildtools is not already required (which will provide a new python), what is the Python version? There’s a script in yocto-autobuilder-helper to assist here, and it’s easier for me to do: [1] 12:23:16 [SUCCESS] alma9-vk-2.yocto.io Python 3.9.23 [2] 12:23:16 [SUCCESS] fedora41-vk-1.yocto.io Python 3.13.3 [3] 12:23:16 [SUCCESS] fedora39-vk-1.yocto.io Python 3.12.7 [4] 12:23:17 [SUCCESS] rocky8-vk-1.yocto.io Python 3.6.8 [5] 12:23:18 [SUCCESS] fedora42-vk-1.yocto.io Python 3.13.6 [6] 12:23:18 [SUCCESS] fedora40-vk-1.yocto.io Python 3.12.10 [7] 12:23:18 [SUCCESS] opensuse155-vk-1.yocto.io Python 3.6.15 [8] 12:23:18 [SUCCESS] ubuntu2504-vk-1.yocto.io Python 3.13.3 [9] 12:23:19 [SUCCESS] ubuntu2404-vk-1.yocto.io Python 3.12.3 [10] 12:23:19 [SUCCESS] rocky9-vk-1.yocto.io Python 3.9.23 [11] 12:23:19 [SUCCESS] ubuntu2204-vk-1.yocto.io Python 3.10.12 [12] 12:23:19 [SUCCESS] stream9-vk-1.yocto.io Python 3.9.25 [13] 12:23:20 [SUCCESS] debian11-vk-1.yocto.io Python 3.9.2 [14] 12:23:20 [SUCCESS] alma8-vk-1.yocto.io Python 3.6.8 [15] 12:23:20 [SUCCESS] opensuse156-vk-1.yocto.io Python 3.6.15 [16] 12:23:21 [SUCCESS] debian13-vk-1.yocto.io Python 3.13.5 [17] 12:23:22 [SUCCESS] debian12-vk-1.yocto.io Python 3.11.2 [18] 12:23:23 [SUCCESS] fedora43-vk-2.yocto.io Python 3.14.0 [19] 12:23:23 [SUCCESS] ubuntu2510-vk-1.yocto.io Python 3.13.7 The distros which have <3.9 need buildtools already, but that leaves Alma 9, Debian 11, Rocky 9 and Stream 9 which are all long-term support and won’t be EOLing shortly. Maybe something will happen that means we need to use buildtools on these, but tomllib is probably not going to be the impetus here. Ross [1] https://docs.yoctoproject.org/ref-manual/system-requirements.html#supported-linux-distributions
Thanks for the answer! Martin On Wed, 2025-12-17 at 12:35 +0000, Ross Burton wrote: > On 17 Dec 2025, at 08:57, Martin Schwan via lists.openembedded.org > <m.schwan=phytec.de@lists.openembedded.org> wrote: > > Are there any plans to move to Python 3.11? Python 3.9 is now at > > its > > EOL for some months. I don't want to open a can of worms, but just > > like > > to understand the future/intentions in this regard. > > I’ll happily endorse a patch to upgrade the minimal python version > for master, but the following needs to happen: > > For every supported distribution [1] where buildtools is not already > required (which will provide a new python), what is the Python > version? > > There’s a script in yocto-autobuilder-helper to assist here, and it’s > easier for me to do: > > [1] 12:23:16 [SUCCESS] alma9-vk-2.yocto.io > Python 3.9.23 > [2] 12:23:16 [SUCCESS] fedora41-vk-1.yocto.io > Python 3.13.3 > [3] 12:23:16 [SUCCESS] fedora39-vk-1.yocto.io > Python 3.12.7 > [4] 12:23:17 [SUCCESS] rocky8-vk-1.yocto.io > Python 3.6.8 > [5] 12:23:18 [SUCCESS] fedora42-vk-1.yocto.io > Python 3.13.6 > [6] 12:23:18 [SUCCESS] fedora40-vk-1.yocto.io > Python 3.12.10 > [7] 12:23:18 [SUCCESS] opensuse155-vk-1.yocto.io > Python 3.6.15 > [8] 12:23:18 [SUCCESS] ubuntu2504-vk-1.yocto.io > Python 3.13.3 > [9] 12:23:19 [SUCCESS] ubuntu2404-vk-1.yocto.io > Python 3.12.3 > [10] 12:23:19 [SUCCESS] rocky9-vk-1.yocto.io > Python 3.9.23 > [11] 12:23:19 [SUCCESS] ubuntu2204-vk-1.yocto.io > Python 3.10.12 > [12] 12:23:19 [SUCCESS] stream9-vk-1.yocto.io > Python 3.9.25 > [13] 12:23:20 [SUCCESS] debian11-vk-1.yocto.io > Python 3.9.2 > [14] 12:23:20 [SUCCESS] alma8-vk-1.yocto.io > Python 3.6.8 > [15] 12:23:20 [SUCCESS] opensuse156-vk-1.yocto.io > Python 3.6.15 > [16] 12:23:21 [SUCCESS] debian13-vk-1.yocto.io > Python 3.13.5 > [17] 12:23:22 [SUCCESS] debian12-vk-1.yocto.io > Python 3.11.2 > [18] 12:23:23 [SUCCESS] fedora43-vk-2.yocto.io > Python 3.14.0 > [19] 12:23:23 [SUCCESS] ubuntu2510-vk-1.yocto.io > Python 3.13.7 > > The distros which have <3.9 need buildtools already, but that leaves > Alma 9, Debian 11, Rocky 9 and Stream 9 which are all long-term > support and won’t be EOLing shortly. Maybe something will happen > that means we need to use buildtools on these, but tomllib is > probably not going to be the impetus here. > > Ross > > [1] > https://docs.yoctoproject.org/ref-manual/system-requirements.html#supported-linux-distributions
diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass index 3251d5ef2e..2ccaf80e40 100644 --- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass +++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass @@ -23,59 +23,58 @@ RECIPE_UPGRADE_EXTRA_TASKS += "do_update_crates" # The directory where to search for Cargo.lock files CARGO_LOCK_SRC_DIR ??= "${S}" -do_update_crates() { - TARGET_FILE="${THISDIR}/${BPN}-crates.inc" +python do_update_crates() { + import tomllib - nativepython3 - <<EOF + target_file = d.expand('${THISDIR}/${BPN}-crates.inc') + cargo_lock_src_dir = d.getVar('CARGO_LOCK_SRC_DIR') -def get_crates(f): - import tomllib - c_list = '# from %s' % os.path.relpath(f, '${CARGO_LOCK_SRC_DIR}') - c_list += '\nSRC_URI += " \\\' - crates = tomllib.load(open(f, 'rb')) + def get_crates(f): + c_list = '# from %s' % os.path.relpath(f, cargo_lock_src_dir) + c_list += '\nSRC_URI += " \\' + crates = tomllib.load(open(f, 'rb')) - # Build a list with crates info that have crates.io in the source - crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package'])) + # Build a list with crates info that have crates.io in the source + crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package'])) - if not crates_candidates: - raise ValueError("Unable to find any candidate crates that use crates.io") + if not crates_candidates: + raise ValueError("Unable to find any candidate crates that use crates.io") - # Update crates uri and their checksum, to avoid name clashing on the checksum - # we need to rename crates with name and version to have a unique key - cksum_list = '' - for c in crates_candidates: - rename = "%s-%s" % (c['name'], c['version']) - c_list += '\n crate://crates.io/%s/%s \\\' % (c['name'], c['version']) - if 'checksum' in c: - cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum']) + # Update crates uri and their checksum, to avoid name clashing on the checksum + # we need to rename crates with name and version to have a unique key + cksum_list = '' + for c in crates_candidates: + rename = "%s-%s" % (c['name'], c['version']) + c_list += '\n crate://crates.io/%s/%s \\' % (c['name'], c['version']) + if 'checksum' in c: + cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum']) - c_list += '\n"\n' - c_list += cksum_list - c_list += '\n' - return c_list + c_list += '\n"\n' + c_list += cksum_list + c_list += '\n' + return c_list -import os -crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n" -found = False -for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'): - # ignore git and patches directories - if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.pc')): - continue - if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.git')): - continue - for file in files: - if file == 'Cargo.lock': - try: - cargo_lock_path = os.path.join(root, file) - crates += get_crates(os.path.join(root, file)) - except Exception as e: - raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e - else: - found = True -if not found: - raise ValueError("Unable to find any Cargo.lock in ${CARGO_LOCK_SRC_DIR}") -open("${TARGET_FILE}", 'w').write(crates) -EOF + import os + crates = d.expand("# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n") + found = False + for root, dirs, files in os.walk(cargo_lock_src_dir): + # ignore git and patches directories + if root.startswith(os.path.join(cargo_lock_src_dir, '.pc')): + continue + if root.startswith(os.path.join(cargo_lock_src_dir, '.git')): + continue + for file in files: + if file == 'Cargo.lock': + try: + cargo_lock_path = os.path.join(root, file) + crates += get_crates(os.path.join(root, file)) + except Exception as e: + raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e + else: + found = True + if not found: + raise ValueError(f"Unable to find any Cargo.lock in {cargo_lock_src_dir}") + open(target_file, 'w').write(crates) - bbnote "Successfully update crates inside '${TARGET_FILE}'" + bb.note(f"Successfully update crates inside '{target_file}'") }
Port the do_update_crates() from Shell to a real Python function. This improves readability by directly executing Python code, instead of redirecting it through a string. In the process, two new variables "target_file" and "cargo_lock_src_dir" are introduced, which just contain the previously, similarly named Bitbake variables in the Python function. No functional changes are made. Signed-off-by: Martin Schwan <m.schwan@phytec.de> --- Changes in v2: - Split changes into multiple patches, for improved readability --- .../cargo-update-recipe-crates.bbclass | 93 +++++++++---------- 1 file changed, 46 insertions(+), 47 deletions(-)