[RFC,1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

For better or worse, more and more rust components are appearing that do
not include their dependencies in tarballs (or git trees), and rely on cargo
to fetch them. On the other hand, bitbake does not use cargo (and quite possible
won't ever be able to), and relies on having each item explicitly listed in SRC_URI
with a crate:// prefix. This however creates a problem of both making such lists in
the first place and updating them when a recipe is updated to a newer version.

So this class can be used to perform such updates by implementing a task that does it;
the next commit shows the outcome for python3-bcrypt (which has been tested to work
and produce a successful build).

Note: the python script relies on tomllib library, which appears in Python 3.11 and
does not exist in earlier versions - I've tested this by first updating python to 3.11-rc2
in oe-core.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 .../cargo-update-recipe-crates.bbclass        | 41 +++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 meta/classes-recipe/cargo-update-recipe-crates.bbclass

Hi Alex,

On 9/30/22 19:55, Alexander Kanavin wrote:
> For better or worse, more and more rust components are appearing that do
> not include their dependencies in tarballs (or git trees), and rely on cargo
> to fetch them. On the other hand, bitbake does not use cargo (and quite possible
> won't ever be able to), and relies on having each item explicitly listed in SRC_URI
> with a crate:// prefix. This however creates a problem of both making such lists in
> the first place and updating them when a recipe is updated to a newer version.
> 
> So this class can be used to perform such updates by implementing a task that does it;
> the next commit shows the outcome for python3-bcrypt (which has been tested to work
> and produce a successful build).
> 
> Note: the python script relies on tomllib library, which appears in Python 3.11 and
> does not exist in earlier versions - I've tested this by first updating python to 3.11-rc2
> in oe-core.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   .../cargo-update-recipe-crates.bbclass        | 41 +++++++++++++++++++
>   1 file changed, 41 insertions(+)
>   create mode 100644 meta/classes-recipe/cargo-update-recipe-crates.bbclass
> 
> diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
> new file mode 100644
> index 0000000000..f90938c734
> --- /dev/null
> +++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
> @@ -0,0 +1,41 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +##
> +## Purpose:
> +## This class is used to update the list of crates in SRC_URI
> +## by reading Cargo.lock in the source tree.
> +##
> +## See meta/recipes-devtools/python/python3-bcrypt_*.bb for an example
> +##
> +## To perform the update: bitbake -c update_crates recipe-name
> +
> +addtask do_update_crates after do_patch
> +do_update_crates[depends] = "python3-native:do_populate_sysroot"
> +
> +do_update_crates() {
> +    nativepython3 - <<EOF
> +
> +def get_crates(f):
> +    import tomllib
> +    c_list = 'SRC_URI += " \\ \n'
> +    crates = tomllib.load(open(f, 'rb'))

Aren't we supposed to use context for this or close the file descriptor 
after opening it manually?

e.g.
with open(f, 'rb') as file:
     crates = tommlib.load(file)

?

> +    for c in crates['package']:
> +        if 'source' in c and 'crates.io' in c['source']:

if 'crates.io' in c.get('source', '')

> +            c_list += "        crate://crates.io/{}/{} \\ \n".format(c['name'], c['version'])
> +    c_list += '"\n'
> +    return c_list
> +
> +import os
> +crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
> +for root, dirs, files in os.walk('${S}'):
> +    for file in files:
> +        if file == 'Cargo.lock':
> +            crates += get_crates(os.path.join(root, file))

from pathlib import Path

for file in Path('${S}').glob('**/Cargo.lock'):
     crates += get_crates(str(file))


> +open(os.path.join('${THISDIR}', '${PN}'+"-crates.inc"), 'w').write(crates)
> +

with open... here again?

Cheers,
Quentin

On Mon, 3 Oct 2022 at 10:09, Quentin Schulz
<quentin.schulz@theobroma-systems.com> wrote:
> > +def get_crates(f):
> > +    import tomllib
> > +    c_list = 'SRC_URI += " \\ \n'
> > +    crates = tomllib.load(open(f, 'rb'))
>
> Aren't we supposed to use context for this or close the file descriptor
> after opening it manually?
>
> e.g.
> with open(f, 'rb') as file:
>      crates = tommlib.load(file)
>
> ?

python3.11 no longer prints warnings about this, so I assume it is now ok.

For the other two suggested changes please explain why.


Alex

On Fri, Sep 30, 2022 at 10:55 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

%> Snip %<


> On the other hand, bitbake does not use cargo (and quite possible
> won't ever be able to),
>

Can you elaborate on this?

I only ask because we are using this[1] and this[2] to build this[3] recipe
for hardknott.

1. https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py
2. https://crates.io/crates/cargo-bitbake
3.
https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3-cryptography-vectors_36.0.2.bb?h=kirkstone

..Ch:W..

On Wed, 5 Oct 2022 at 07:32, Chuck Wolber <chuckwolber@gmail.com> wrote:
> Can you elaborate on this?
>
> I only ask because we are using this[1] and this[2] to build this[3] recipe for hardknott.
>
> 1. https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py
> 2. https://crates.io/crates/cargo-bitbake
> 3. https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3-cryptography-vectors_36.0.2.bb?h=kirkstone

As you can see in [1], crate:// fetcher is not using cargo, it's using
wget - so the question of resolving and listing dependencies from
Cargo.lock falls on the tooling that writes and updates the recipe
itself. cargo-bitbake failed at the installation step for me (both
with --locked and without it), and in general I do not think we should
rely on external tools with unknown maintenance status. The lists of
open issues and pull requests are not at all reassuring:
https://github.com/meta-rust/cargo-bitbake

Alex

On Wed, Oct 5, 2022 at 1:56 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> On Wed, 5 Oct 2022 at 07:32, Chuck Wolber <chuckwolber@gmail.com> wrote:
> > Can you elaborate on this?
> >
> > I only ask because we are using this[1] and this[2] to build this[3]
> recipe for hardknott.
> >
> > 1. https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py
> > 2. https://crates.io/crates/cargo-bitbake
> > 3.
> https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3-cryptography-vectors_36.0.2.bb?h=kirkstone
>
> As you can see in [1], crate:// fetcher is not using cargo, it's using
> wget - so the question of resolving and listing dependencies from
> Cargo.lock falls on the tooling that writes and updates the recipe
> itself. cargo-bitbake failed at the installation step for me (both
> with --locked and without it), and in general I do not think we should
> rely on external tools with unknown maintenance status. The lists of
> open issues and pull requests are not at all reassuring:
> https://github.com/meta-rust/cargo-bitbake
>
>
I have cargo bitbake running locally, but in recent python3-cryptography
and python3-bcrypt upgrades
it produced crate://* SRC_URI that actually failed to build. I had to
manually loop through builds until
I sussed out all the versions that would work. This proves Alex's point.

Having looked at the implementation, I am very much in favor of this
proposal. In fact, I wonder if
a similar approach could help with golang and maybe even nodejs/npm. But
that is for another day.

Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#171453):
> https://lists.openembedded.org/g/openembedded-core/message/171453
> Mute This Topic: https://lists.openembedded.org/mt/94022674/924729
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>