cargo_common.bbclass: do not use buit-in git to fetch crates

From: Frederic Martinsons <frederic.martinsons@gmail.com>

When a project has a dependencies from a git repository and ssh
access, this kind of thing in Cargo.toml for example:

[dependencies]
my-crate = { git = "ssh://git@smyserver:7999/myrepo.git, branch="main" }

do_compile step failed with authentication error:

|     Updating git repository `ssh://git@myserver:7999/myrepo.git`
| error: failed to get `my-crate` as a dependency of package `bscli v0.0.1 (/home/fmartinsons/bscli)`
|
| Caused by:
|   failed to load source for dependency `my-crate`
|
| Caused by:
|   Unable to update ssh://git@myserver:7999/myrepo.git?branch=main
|
| Caused by:
|   failed to fetch into: /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/bscli/v1.0.0+gitAUTOINC+5b5ba2aa3e-r0/cargo_home/git/db/cra-d8e163876777fda6
|
| Caused by:
|   failed to authenticate when downloading repository
|
|   * attempted ssh-agent authentication, but no usernames succeeded: `git`
|
|   if the git CLI succeeds then `net.git-fetch-with-cli` may help here
|   https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
|
| Caused by:
|   error authenticating: no auth sock variable; class=Ssh (23)

No problem if I run "cargo build" manually.

As the compiler suggested, it is corrected by adding git-fetch-with-cli in
cargo config file.

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
---
 meta/classes-recipe/cargo_common.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

Not sure if I understand this fully, but network access in do_compile
is not allowed. You need to place all crates into SRC_URI.

Alex

On Sun, 12 Mar 2023 at 07:55, Frederic Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> From: Frederic Martinsons <frederic.martinsons@gmail.com>
>
> When a project has a dependencies from a git repository and ssh
> access, this kind of thing in Cargo.toml for example:
>
> [dependencies]
> my-crate = { git = "ssh://git@smyserver:7999/myrepo.git, branch="main" }
>
> do_compile step failed with authentication error:
>
> |     Updating git repository `ssh://git@myserver:7999/myrepo.git`
> | error: failed to get `my-crate` as a dependency of package `bscli v0.0.1 (/home/fmartinsons/bscli)`
> |
> | Caused by:
> |   failed to load source for dependency `my-crate`
> |
> | Caused by:
> |   Unable to update ssh://git@myserver:7999/myrepo.git?branch=main
> |
> | Caused by:
> |   failed to fetch into: /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/bscli/v1.0.0+gitAUTOINC+5b5ba2aa3e-r0/cargo_home/git/db/cra-d8e163876777fda6
> |
> | Caused by:
> |   failed to authenticate when downloading repository
> |
> |   * attempted ssh-agent authentication, but no usernames succeeded: `git`
> |
> |   if the git CLI succeeds then `net.git-fetch-with-cli` may help here
> |   https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
> |
> | Caused by:
> |   error authenticating: no auth sock variable; class=Ssh (23)
>
> No problem if I run "cargo build" manually.
>
> As the compiler suggested, it is corrected by adding git-fetch-with-cli in
> cargo config file.
>
> Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
> ---
>  meta/classes-recipe/cargo_common.bbclass | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass
> index f503a001dd..c16aa4934b 100644
> --- a/meta/classes-recipe/cargo_common.bbclass
> +++ b/meta/classes-recipe/cargo_common.bbclass
> @@ -114,6 +114,12 @@ cargo_common_do_configure () {
>         progress.when = 'always'
>         progress.width = 80
>         EOF
> +
> +       cat <<- EOF >> ${CARGO_HOME}/config
> +
> +       [net]
> +       git-fetch-with-cli = true
> +       EOF
>  }
>
>  oe_cargo_fix_env () {
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#178375): https://lists.openembedded.org/g/openembedded-core/message/178375
> Mute This Topic: https://lists.openembedded.org/mt/97555635/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Yes I did do that in a first place (because this is what cargo bitbake
give to me in a first place), but for git dependencies this doesn't
work.
So , to take the example again I had that in SRC_URI (after correcting
malformed url as tracked by
https://github.com/meta-rust/cargo-bitbake/issues/41):

"git://git@myserver:7999/myrepo.git;protocol=ssh;nobranch=1;name=my-crate;destsuffix=my-crate"

And also the SRCREV:
SRCREV_FORMAT .= "_my-crate"
SRCREV_my-crate = "main"
EXTRA_OECARGO_PATHS += "${WORKDIR}/my-crate"

But the compile step tried to git clone anyway and so it failed like I
showed in my first email.

On Sun, 12 Mar 2023 at 08:36, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> Not sure if I understand this fully, but network access in do_compile
> is not allowed. You need to place all crates into SRC_URI.
>
> Alex
>
> On Sun, 12 Mar 2023 at 07:55, Frederic Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > From: Frederic Martinsons <frederic.martinsons@gmail.com>
> >
> > When a project has a dependencies from a git repository and ssh
> > access, this kind of thing in Cargo.toml for example:
> >
> > [dependencies]
> > my-crate = { git = "ssh://git@smyserver:7999/myrepo.git, branch="main" }
> >
> > do_compile step failed with authentication error:
> >
> > |     Updating git repository `ssh://git@myserver:7999/myrepo.git`
> > | error: failed to get `my-crate` as a dependency of package `bscli v0.0.1 (/home/fmartinsons/bscli)`
> > |
> > | Caused by:
> > |   failed to load source for dependency `my-crate`
> > |
> > | Caused by:
> > |   Unable to update ssh://git@myserver:7999/myrepo.git?branch=main
> > |
> > | Caused by:
> > |   failed to fetch into: /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/bscli/v1.0.0+gitAUTOINC+5b5ba2aa3e-r0/cargo_home/git/db/cra-d8e163876777fda6
> > |
> > | Caused by:
> > |   failed to authenticate when downloading repository
> > |
> > |   * attempted ssh-agent authentication, but no usernames succeeded: `git`
> > |
> > |   if the git CLI succeeds then `net.git-fetch-with-cli` may help here
> > |   https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
> > |
> > | Caused by:
> > |   error authenticating: no auth sock variable; class=Ssh (23)
> >
> > No problem if I run "cargo build" manually.
> >
> > As the compiler suggested, it is corrected by adding git-fetch-with-cli in
> > cargo config file.
> >
> > Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
> > ---
> >  meta/classes-recipe/cargo_common.bbclass | 6 ++++++
> >  1 file changed, 6 insertions(+)
> >
> > diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass
> > index f503a001dd..c16aa4934b 100644
> > --- a/meta/classes-recipe/cargo_common.bbclass
> > +++ b/meta/classes-recipe/cargo_common.bbclass
> > @@ -114,6 +114,12 @@ cargo_common_do_configure () {
> >         progress.when = 'always'
> >         progress.width = 80
> >         EOF
> > +
> > +       cat <<- EOF >> ${CARGO_HOME}/config
> > +
> > +       [net]
> > +       git-fetch-with-cli = true
> > +       EOF
> >  }
> >
> >  oe_cargo_fix_env () {
> > --
> > 2.34.1
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#178375): https://lists.openembedded.org/g/openembedded-core/message/178375
> > Mute This Topic: https://lists.openembedded.org/mt/97555635/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >

On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:
> But the compile step tried to git clone anyway and so it failed like I
> showed in my first email.

You need to figure out why this happens, because it should not. We
have plenty of recipes that list crates in SRC_URI in oe-core, so you
could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for
reproducibility and license compliance reasons (both break if that
happens).

'cargo bitbake' is a 3rd party project with questionable maintenance
status that we do not support. The official way is to use
cargo-update-recipe-crates class.

Alex

I understand there should not be a fetch during build step , I'll try
to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo
build" with  --offline option ?

On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> > But the compile step tried to git clone anyway and so it failed like I
> > showed in my first email.
>
> You need to figure out why this happens, because it should not. We
> have plenty of recipes that list crates in SRC_URI in oe-core, so you
> could start by studying how they work.
>
> Accessing network anywhere except do_fetch is not allowed for
> reproducibility and license compliance reasons (both break if that
> happens).
>
> 'cargo bitbake' is a 3rd party project with questionable maintenance
> status that we do not support. The official way is to use
> cargo-update-recipe-crates class.
>
> Alex

Try with this patch:

https://patchwork.yoctoproject.org/project/oe-core/patch/20221030173815.10212-2-alex.kiernan@gmail.com/

On Mon, Mar 13, 2023 at 9:14 AM Frederic Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> I understand there should not be a fetch during build step , I'll try
> to find why .
> In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> build" with  --offline option ?
>
> On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> >
> > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > <frederic.martinsons@gmail.com> wrote:
> > > But the compile step tried to git clone anyway and so it failed like I
> > > showed in my first email.
> >
> > You need to figure out why this happens, because it should not. We
> > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > could start by studying how they work.
> >
> > Accessing network anywhere except do_fetch is not allowed for
> > reproducibility and license compliance reasons (both break if that
> > happens).
> >
> > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > status that we do not support. The official way is to use
> > cargo-update-recipe-crates class.
> >
> > Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#178417): https://lists.openembedded.org/g/openembedded-core/message/178417
> Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Note that we're all rust neophytes, and rust support in yocto is still
evolving towards maturity. If you can see better ways to do things,
patches would be most welcome - but 'no network access except in
bitbake fetchers' is a core rule.

Alex

On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> I understand there should not be a fetch during build step , I'll try
> to find why .
> In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> build" with  --offline option ?
>
> On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> >
> > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > <frederic.martinsons@gmail.com> wrote:
> > > But the compile step tried to git clone anyway and so it failed like I
> > > showed in my first email.
> >
> > You need to figure out why this happens, because it should not. We
> > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > could start by studying how they work.
> >
> > Accessing network anywhere except do_fetch is not allowed for
> > reproducibility and license compliance reasons (both break if that
> > happens).
> >
> > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > status that we do not support. The official way is to use
> > cargo-update-recipe-crates class.
> >
> > Alex

ISTR I tried this and ran into a different set of different problems.

On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> Note that we're all rust neophytes, and rust support in yocto is still
> evolving towards maturity. If you can see better ways to do things,
> patches would be most welcome - but 'no network access except in
> bitbake fetchers' is a core rule.
>
> Alex
>
> On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > I understand there should not be a fetch during build step , I'll try
> > to find why .
> > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > build" with  --offline option ?
> >
> > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > >
> > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > <frederic.martinsons@gmail.com> wrote:
> > > > But the compile step tried to git clone anyway and so it failed like I
> > > > showed in my first email.
> > >
> > > You need to figure out why this happens, because it should not. We
> > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > could start by studying how they work.
> > >
> > > Accessing network anywhere except do_fetch is not allowed for
> > > reproducibility and license compliance reasons (both break if that
> > > happens).
> > >
> > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > status that we do not support. The official way is to use
> > > cargo-update-recipe-crates class.
> > >
> > > Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
> Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Thanks for the patch alex, I did try it and the same happened (git
clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of
supporting git dependencies in Cargo.toml

On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
>
> ISTR I tried this and ran into a different set of different problems.
>
> On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> <alex.kanavin@gmail.com> wrote:
> >
> > Note that we're all rust neophytes, and rust support in yocto is still
> > evolving towards maturity. If you can see better ways to do things,
> > patches would be most welcome - but 'no network access except in
> > bitbake fetchers' is a core rule.
> >
> > Alex
> >
> > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > <frederic.martinsons@gmail.com> wrote:
> > >
> > > I understand there should not be a fetch during build step , I'll try
> > > to find why .
> > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > build" with  --offline option ?
> > >
> > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > >
> > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > <frederic.martinsons@gmail.com> wrote:
> > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > showed in my first email.
> > > >
> > > > You need to figure out why this happens, because it should not. We
> > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > could start by studying how they work.
> > > >
> > > > Accessing network anywhere except do_fetch is not allowed for
> > > > reproducibility and license compliance reasons (both break if that
> > > > happens).
> > > >
> > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > status that we do not support. The official way is to use
> > > > cargo-update-recipe-crates class.
> > > >
> > > > Alex
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
> > Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alex Kiernan

Are you listing your ssh repos in your SRC_URI? We run with network
disabled everywhere other than fetch and with that patch we don't see
fetches from local ssh repos in compile.

On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> Thanks for the patch alex, I did try it and the same happened (git
> clone failure during do_compile), I'll
> dig more on that and possibly submit patches if I found a way of
> supporting git dependencies in Cargo.toml
>
> On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> >
> > ISTR I tried this and ran into a different set of different problems.
> >
> > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > <alex.kanavin@gmail.com> wrote:
> > >
> > > Note that we're all rust neophytes, and rust support in yocto is still
> > > evolving towards maturity. If you can see better ways to do things,
> > > patches would be most welcome - but 'no network access except in
> > > bitbake fetchers' is a core rule.
> > >
> > > Alex
> > >
> > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > <frederic.martinsons@gmail.com> wrote:
> > > >
> > > > I understand there should not be a fetch during build step , I'll try
> > > > to find why .
> > > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > > build" with  --offline option ?
> > > >
> > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > > >
> > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > > showed in my first email.
> > > > >
> > > > > You need to figure out why this happens, because it should not. We
> > > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > > could start by studying how they work.
> > > > >
> > > > > Accessing network anywhere except do_fetch is not allowed for
> > > > > reproducibility and license compliance reasons (both break if that
> > > > > happens).
> > > > >
> > > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > > status that we do not support. The official way is to use
> > > > > cargo-update-recipe-crates class.
> > > > >
> > > > > Alex
> > >
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > Links: You receive all messages sent to this group.
> > > View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
> > > Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > >
> >
> >
> > --
> > Alex Kiernan

For reference this was my test case:

https://github.com/akiernan/uuid-test

I was wondering if this was an ssh vs. https problem, but just
checking we've a mix of ssh and https access to local git repos.

On Mon, Mar 13, 2023 at 9:46 AM Alex Kiernan via
lists.openembedded.org <alex.kiernan=gmail.com@lists.openembedded.org>
wrote:
>
> Are you listing your ssh repos in your SRC_URI? We run with network
> disabled everywhere other than fetch and with that patch we don't see
> fetches from local ssh repos in compile.
>
> On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > Thanks for the patch alex, I did try it and the same happened (git
> > clone failure during do_compile), I'll
> > dig more on that and possibly submit patches if I found a way of
> > supporting git dependencies in Cargo.toml
> >
> > On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > >
> > > ISTR I tried this and ran into a different set of different problems.
> > >
> > > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > > <alex.kanavin@gmail.com> wrote:
> > > >
> > > > Note that we're all rust neophytes, and rust support in yocto is still
> > > > evolving towards maturity. If you can see better ways to do things,
> > > > patches would be most welcome - but 'no network access except in
> > > > bitbake fetchers' is a core rule.
> > > >
> > > > Alex
> > > >
> > > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > > <frederic.martinsons@gmail.com> wrote:
> > > > >
> > > > > I understand there should not be a fetch during build step , I'll try
> > > > > to find why .
> > > > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > > > build" with  --offline option ?
> > > > >
> > > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > > > >
> > > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > > > showed in my first email.
> > > > > >
> > > > > > You need to figure out why this happens, because it should not. We
> > > > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > > > could start by studying how they work.
> > > > > >
> > > > > > Accessing network anywhere except do_fetch is not allowed for
> > > > > > reproducibility and license compliance reasons (both break if that
> > > > > > happens).
> > > > > >
> > > > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > > > status that we do not support. The official way is to use
> > > > > > cargo-update-recipe-crates class.
> > > > > >
> > > > > > Alex
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Alex Kiernan
>
>
>
> --
> Alex Kiernan
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#178424): https://lists.openembedded.org/g/openembedded-core/message/178424
> Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

hmm, how is the network disabled ?

Because with the patch I provided , the git clone in do_compile is successful.

One major thing I forget to mention is that I use dunfell branch where
I backport all the cargo/rust relatives from master.
The backport was functional and straightforward, I just have to keep
the custom crate fetcher in openembedded-core (meta/lib/crate.py)
because I'm stuck with bitbake 1.46.

   -

On Mon, 13 Mar 2023 at 10:46, Alex Kiernan <alex.kiernan@gmail.com> wrote:
>
> Are you listing your ssh repos in your SRC_URI? We run with network
> disabled everywhere other than fetch and with that patch we don't see
> fetches from local ssh repos in compile.
>
> On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > Thanks for the patch alex, I did try it and the same happened (git
> > clone failure during do_compile), I'll
> > dig more on that and possibly submit patches if I found a way of
> > supporting git dependencies in Cargo.toml
> >
> > On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > >
> > > ISTR I tried this and ran into a different set of different problems.
> > >
> > > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > > <alex.kanavin@gmail.com> wrote:
> > > >
> > > > Note that we're all rust neophytes, and rust support in yocto is still
> > > > evolving towards maturity. If you can see better ways to do things,
> > > > patches would be most welcome - but 'no network access except in
> > > > bitbake fetchers' is a core rule.
> > > >
> > > > Alex
> > > >
> > > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > > <frederic.martinsons@gmail.com> wrote:
> > > > >
> > > > > I understand there should not be a fetch during build step , I'll try
> > > > > to find why .
> > > > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > > > build" with  --offline option ?
> > > > >
> > > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > > > >
> > > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > > > showed in my first email.
> > > > > >
> > > > > > You need to figure out why this happens, because it should not. We
> > > > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > > > could start by studying how they work.
> > > > > >
> > > > > > Accessing network anywhere except do_fetch is not allowed for
> > > > > > reproducibility and license compliance reasons (both break if that
> > > > > > happens).
> > > > > >
> > > > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > > > status that we do not support. The official way is to use
> > > > > > cargo-update-recipe-crates class.
> > > > > >
> > > > > > Alex
> > > >
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > Links: You receive all messages sent to this group.
> > > > View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
> > > > Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > >
> > >
> > >
> > > --
> > > Alex Kiernan
>
>
>
> --
> Alex Kiernan

Thank you very much, I tested your project and it worked.

So I'll try https for my git instead of ssh.

On Mon, 13 Mar 2023 at 10:49, Alex Kiernan <alex.kiernan@gmail.com> wrote:
>
> For reference this was my test case:
>
> https://github.com/akiernan/uuid-test
>
> I was wondering if this was an ssh vs. https problem, but just
> checking we've a mix of ssh and https access to local git repos.
>
> On Mon, Mar 13, 2023 at 9:46 AM Alex Kiernan via
> lists.openembedded.org <alex.kiernan=gmail.com@lists.openembedded.org>
> wrote:
> >
> > Are you listing your ssh repos in your SRC_URI? We run with network
> > disabled everywhere other than fetch and with that patch we don't see
> > fetches from local ssh repos in compile.
> >
> > On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
> > <frederic.martinsons@gmail.com> wrote:
> > >
> > > Thanks for the patch alex, I did try it and the same happened (git
> > > clone failure during do_compile), I'll
> > > dig more on that and possibly submit patches if I found a way of
> > > supporting git dependencies in Cargo.toml
> > >
> > > On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > > >
> > > > ISTR I tried this and ran into a different set of different problems.
> > > >
> > > > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > > > <alex.kanavin@gmail.com> wrote:
> > > > >
> > > > > Note that we're all rust neophytes, and rust support in yocto is still
> > > > > evolving towards maturity. If you can see better ways to do things,
> > > > > patches would be most welcome - but 'no network access except in
> > > > > bitbake fetchers' is a core rule.
> > > > >
> > > > > Alex
> > > > >
> > > > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > >
> > > > > > I understand there should not be a fetch during build step , I'll try
> > > > > > to find why .
> > > > > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > > > > build" with  --offline option ?
> > > > > >
> > > > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > > > > >
> > > > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > > > > showed in my first email.
> > > > > > >
> > > > > > > You need to figure out why this happens, because it should not. We
> > > > > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > > > > could start by studying how they work.
> > > > > > >
> > > > > > > Accessing network anywhere except do_fetch is not allowed for
> > > > > > > reproducibility and license compliance reasons (both break if that
> > > > > > > happens).
> > > > > > >
> > > > > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > > > > status that we do not support. The official way is to use
> > > > > > > cargo-update-recipe-crates class.
> > > > > > >
> > > > > > > Alex
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Alex Kiernan
> >
> >
> >
> > --
> > Alex Kiernan
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#178424): https://lists.openembedded.org/g/openembedded-core/message/178424
> > Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alex Kiernan

You need to use kirkstone or newer to see the bitbake-driven network
disabling in action. Can you try on current poky master? You don't
need to port the whole build to that, just one sample recipe.

It is also expected that if you submit patches for master, they were
tested on master.

Alex

On Mon, 13 Mar 2023 at 11:06, Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> hmm, how is the network disabled ?
>
> Because with the patch I provided , the git clone in do_compile is successful.
>
> One major thing I forget to mention is that I use dunfell branch where
> I backport all the cargo/rust relatives from master.
> The backport was functional and straightforward, I just have to keep
> the custom crate fetcher in openembedded-core (meta/lib/crate.py)
> because I'm stuck with bitbake 1.46.
>
>    -
>
> On Mon, 13 Mar 2023 at 10:46, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> >
> > Are you listing your ssh repos in your SRC_URI? We run with network
> > disabled everywhere other than fetch and with that patch we don't see
> > fetches from local ssh repos in compile.
> >
> > On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
> > <frederic.martinsons@gmail.com> wrote:
> > >
> > > Thanks for the patch alex, I did try it and the same happened (git
> > > clone failure during do_compile), I'll
> > > dig more on that and possibly submit patches if I found a way of
> > > supporting git dependencies in Cargo.toml
> > >
> > > On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > > >
> > > > ISTR I tried this and ran into a different set of different problems.
> > > >
> > > > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > > > <alex.kanavin@gmail.com> wrote:
> > > > >
> > > > > Note that we're all rust neophytes, and rust support in yocto is still
> > > > > evolving towards maturity. If you can see better ways to do things,
> > > > > patches would be most welcome - but 'no network access except in
> > > > > bitbake fetchers' is a core rule.
> > > > >
> > > > > Alex
> > > > >
> > > > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > >
> > > > > > I understand there should not be a fetch during build step , I'll try
> > > > > > to find why .
> > > > > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > > > > build" with  --offline option ?
> > > > > >
> > > > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > > > > >
> > > > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > > > > showed in my first email.
> > > > > > >
> > > > > > > You need to figure out why this happens, because it should not. We
> > > > > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > > > > could start by studying how they work.
> > > > > > >
> > > > > > > Accessing network anywhere except do_fetch is not allowed for
> > > > > > > reproducibility and license compliance reasons (both break if that
> > > > > > > happens).
> > > > > > >
> > > > > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > > > > status that we do not support. The official way is to use
> > > > > > > cargo-update-recipe-crates class.
> > > > > > >
> > > > > > > Alex
> > > > >
> > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > Links: You receive all messages sent to this group.
> > > > > View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
> > > > > Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> > > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > >
> > > >
> > > >
> > > > --
> > > > Alex Kiernan
> >
> >
> >
> > --
> > Alex Kiernan

I understand, I'll pursue all my tests with the current poky master. Thanks

On Mon, 13 Mar 2023 at 11:42, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> You need to use kirkstone or newer to see the bitbake-driven network
> disabling in action. Can you try on current poky master? You don't
> need to port the whole build to that, just one sample recipe.
>
> It is also expected that if you submit patches for master, they were
> tested on master.
>
> Alex
>
> On Mon, 13 Mar 2023 at 11:06, Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > hmm, how is the network disabled ?
> >
> > Because with the patch I provided , the git clone in do_compile is successful.
> >
> > One major thing I forget to mention is that I use dunfell branch where
> > I backport all the cargo/rust relatives from master.
> > The backport was functional and straightforward, I just have to keep
> > the custom crate fetcher in openembedded-core (meta/lib/crate.py)
> > because I'm stuck with bitbake 1.46.
> >
> >    -
> >
> > On Mon, 13 Mar 2023 at 10:46, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > >
> > > Are you listing your ssh repos in your SRC_URI? We run with network
> > > disabled everywhere other than fetch and with that patch we don't see
> > > fetches from local ssh repos in compile.
> > >
> > > On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
> > > <frederic.martinsons@gmail.com> wrote:
> > > >
> > > > Thanks for the patch alex, I did try it and the same happened (git
> > > > clone failure during do_compile), I'll
> > > > dig more on that and possibly submit patches if I found a way of
> > > > supporting git dependencies in Cargo.toml
> > > >
> > > > On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > > > >
> > > > > ISTR I tried this and ran into a different set of different problems.
> > > > >
> > > > > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > > > > <alex.kanavin@gmail.com> wrote:
> > > > > >
> > > > > > Note that we're all rust neophytes, and rust support in yocto is still
> > > > > > evolving towards maturity. If you can see better ways to do things,
> > > > > > patches would be most welcome - but 'no network access except in
> > > > > > bitbake fetchers' is a core rule.
> > > > > >
> > > > > > Alex
> > > > > >
> > > > > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > >
> > > > > > > I understand there should not be a fetch during build step , I'll try
> > > > > > > to find why .
> > > > > > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > > > > > build" with  --offline option ?
> > > > > > >
> > > > > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > > > > > >
> > > > > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > > > > > showed in my first email.
> > > > > > > >
> > > > > > > > You need to figure out why this happens, because it should not. We
> > > > > > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > > > > > could start by studying how they work.
> > > > > > > >
> > > > > > > > Accessing network anywhere except do_fetch is not allowed for
> > > > > > > > reproducibility and license compliance reasons (both break if that
> > > > > > > > happens).
> > > > > > > >
> > > > > > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > > > > > status that we do not support. The official way is to use
> > > > > > > > cargo-update-recipe-crates class.
> > > > > > > >
> > > > > > > > Alex
> > > > > >
> > > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > > Links: You receive all messages sent to this group.
> > > > > > View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
> > > > > > Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> > > > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> > > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alex Kiernan
> > >
> > >
> > >
> > > --
> > > Alex Kiernan

Hello, I finally found why my setup didn't work. That was not related
to https versus ssh.
For the patch provided in
https://patchwork.yoctoproject.org/project/oe-core/patch/20221030173815.10212-2-alex.kiernan@gmail.com/
to work, the repository should have a Cargo.lock file.
WIthout the Cargo.lock file, the cargo build will try to generate it
(and so needs network access).

Note that I made a little change to the patch to support user in url
(the patch process in cargo only work if url is exact match):

@@ -135,7 +135,10 @@ python cargo_common_do_patch_paths() {
             name = ud.parm.get('name')
             destsuffix = ud.parm.get('destsuffix')
             if name is not None and destsuffix is not None:
-                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
+                if ud.user:
+                    repo = '%s://%s@%s%s' % (ud.proto, ud.user,
ud.host, ud.path)
+                else:
+                    repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
                 path = '%s = { path = "%s" }' % (name,
os.path.join(workdir, destsuffix))
                 patches.setdefault(repo, []).append(path)

Anyway, I would like to know how to you feel with this need for
Cargo.lock problem ?
Because for it to be generated, the project needs to be built once and
if this is expected for binary type to have Cargo.lock under version
control:
 https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
It is not for "library" project.

Maybe we can add a "cargo generate-lockfile"
(https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
at the end of do_fetch ?
Moreover, I think we should add the `--offline` option to cargo build
because the error generated will be more clear:

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from
/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
| NOTE: cargo =
/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo
| NOTE: cargo build -v --offline --target x86_64-poky-linux-gnu
--release --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
| error: failed to get `zbus-git-dep-test` as a dependency of package
`zbus-test v0.0.1
(/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`
|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  can't checkout from
'ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git': you are in
the offline mode (--offline)
| WARNING: exit code 101 from a shell command.

Instead of

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from
/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
| NOTE: cargo =
/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo
| NOTE: cargo build -v --target x86_64-poky-linux-gnu --release
--manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
|   Updating git repository
`ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git`
| warning: spurious network error (2 tries remaining): failed to
resolve address for gitlab.com: Temporary failure in name resolution;
class=Net (12)
| warning: spurious network error (1 tries remaining): failed to
resolve address for gitlab.com: Temporary failure in name resolution;
class=Net (12)
| error: failed to get `zbus-git-dep-test` as a dependency of package
`zbus-test v0.0.1
(/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`
|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  failed to clone into:
/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/cargo_home/git/db/zbus-git-dep-test-7f2b6322e3bb0cd5
|
| Caused by:
|  network failure seems to have happened
|  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
|  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
|
| Caused by:
|  failed to resolve address for gitlab.com: Temporary failure in name
resolution; class=Net (12)|
|  WARNING: exit code 101 from a shell command.


In any case, I'll be glad to work on this patch if you tell me what
strategy to adopt.

On Mon, 13 Mar 2023 at 17:08, Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> I understand, I'll pursue all my tests with the current poky master. Thanks
>
> On Mon, 13 Mar 2023 at 11:42, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> >
> > You need to use kirkstone or newer to see the bitbake-driven network
> > disabling in action. Can you try on current poky master? You don't
> > need to port the whole build to that, just one sample recipe.
> >
> > It is also expected that if you submit patches for master, they were
> > tested on master.
> >
> > Alex
> >
> > On Mon, 13 Mar 2023 at 11:06, Frédéric Martinsons
> > <frederic.martinsons@gmail.com> wrote:
> > >
> > > hmm, how is the network disabled ?
> > >
> > > Because with the patch I provided , the git clone in do_compile is successful.
> > >
> > > One major thing I forget to mention is that I use dunfell branch where
> > > I backport all the cargo/rust relatives from master.
> > > The backport was functional and straightforward, I just have to keep
> > > the custom crate fetcher in openembedded-core (meta/lib/crate.py)
> > > because I'm stuck with bitbake 1.46.
> > >
> > >    -
> > >
> > > On Mon, 13 Mar 2023 at 10:46, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > > >
> > > > Are you listing your ssh repos in your SRC_URI? We run with network
> > > > disabled everywhere other than fetch and with that patch we don't see
> > > > fetches from local ssh repos in compile.
> > > >
> > > > On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
> > > > <frederic.martinsons@gmail.com> wrote:
> > > > >
> > > > > Thanks for the patch alex, I did try it and the same happened (git
> > > > > clone failure during do_compile), I'll
> > > > > dig more on that and possibly submit patches if I found a way of
> > > > > supporting git dependencies in Cargo.toml
> > > > >
> > > > > On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > > > > >
> > > > > > ISTR I tried this and ran into a different set of different problems.
> > > > > >
> > > > > > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > > > > > <alex.kanavin@gmail.com> wrote:
> > > > > > >
> > > > > > > Note that we're all rust neophytes, and rust support in yocto is still
> > > > > > > evolving towards maturity. If you can see better ways to do things,
> > > > > > > patches would be most welcome - but 'no network access except in
> > > > > > > bitbake fetchers' is a core rule.
> > > > > > >
> > > > > > > Alex
> > > > > > >
> > > > > > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > >
> > > > > > > > I understand there should not be a fetch during build step , I'll try
> > > > > > > > to find why .
> > > > > > > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > > > > > > build" with  --offline option ?
> > > > > > > >
> > > > > > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > > > > > > showed in my first email.
> > > > > > > > >
> > > > > > > > > You need to figure out why this happens, because it should not. We
> > > > > > > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > > > > > > could start by studying how they work.
> > > > > > > > >
> > > > > > > > > Accessing network anywhere except do_fetch is not allowed for
> > > > > > > > > reproducibility and license compliance reasons (both break if that
> > > > > > > > > happens).
> > > > > > > > >
> > > > > > > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > > > > > > status that we do not support. The official way is to use
> > > > > > > > > cargo-update-recipe-crates class.
> > > > > > > > >
> > > > > > > > > Alex
> > > > > > >
> > > > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > > > Links: You receive all messages sent to this group.
> > > > > > > View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
> > > > > > > Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> > > > > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> > > > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Alex Kiernan
> > > >
> > > >
> > > >
> > > > --
> > > > Alex Kiernan

--offline seems like the right thing to add, if it produces better errors.

Generating Cargo.lock on the other hand is not right. We rely on rust
checking the source tree against the checksums in Cargo.lock (to
prevent supply chain attacks), and this would completely subvert that.
There could be better diagnostics around missing Cargo.lock, and steps
to address the issue, but such steps must be manually taken (even
though they could be codified in a special task, similar to how
generating list of crates in SRC_URI is a task).

Alex

On Tue, 14 Mar 2023 at 11:25, Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> Hello, I finally found why my setup didn't work. That was not related
> to https versus ssh.
> For the patch provided in
> https://patchwork.yoctoproject.org/project/oe-core/patch/20221030173815.10212-2-alex.kiernan@gmail.com/
> to work, the repository should have a Cargo.lock file.
> WIthout the Cargo.lock file, the cargo build will try to generate it
> (and so needs network access).
>
> Note that I made a little change to the patch to support user in url
> (the patch process in cargo only work if url is exact match):
>
> @@ -135,7 +135,10 @@ python cargo_common_do_patch_paths() {
>              name = ud.parm.get('name')
>              destsuffix = ud.parm.get('destsuffix')
>              if name is not None and destsuffix is not None:
> -                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
> +                if ud.user:
> +                    repo = '%s://%s@%s%s' % (ud.proto, ud.user,
> ud.host, ud.path)
> +                else:
> +                    repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
>                  path = '%s = { path = "%s" }' % (name,
> os.path.join(workdir, destsuffix))
>                  patches.setdefault(repo, []).append(path)
>
> Anyway, I would like to know how to you feel with this need for
> Cargo.lock problem ?
> Because for it to be generated, the project needs to be built once and
> if this is expected for binary type to have Cargo.lock under version
> control:
>  https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
> It is not for "library" project.
>
> Maybe we can add a "cargo generate-lockfile"
> (https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
> at the end of do_fetch ?
> Moreover, I think we should add the `--offline` option to cargo build
> because the error generated will be more clear:
>
> | DEBUG: Executing shell function do_compile
> | NOTE: Using rust targets from
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
> | NOTE: cargo =
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo
> | NOTE: cargo build -v --offline --target x86_64-poky-linux-gnu
> --release --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
> | error: failed to get `zbus-git-dep-test` as a dependency of package
> `zbus-test v0.0.1
> (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`
> |
> | Caused by:
> |  failed to load source for dependency `zbus-git-dep-test`
> |
> | Caused by:
> |  Unable to update
> ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git?branch=main
> |
> | Caused by:
> |  can't checkout from
> 'ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git': you are in
> the offline mode (--offline)
> | WARNING: exit code 101 from a shell command.
>
> Instead of
>
> | DEBUG: Executing shell function do_compile
> | NOTE: Using rust targets from
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
> | NOTE: cargo =
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo
> | NOTE: cargo build -v --target x86_64-poky-linux-gnu --release
> --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
> |   Updating git repository
> `ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git`
> | warning: spurious network error (2 tries remaining): failed to
> resolve address for gitlab.com: Temporary failure in name resolution;
> class=Net (12)
> | warning: spurious network error (1 tries remaining): failed to
> resolve address for gitlab.com: Temporary failure in name resolution;
> class=Net (12)
> | error: failed to get `zbus-git-dep-test` as a dependency of package
> `zbus-test v0.0.1
> (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`
> |
> | Caused by:
> |  failed to load source for dependency `zbus-git-dep-test`
> |
> | Caused by:
> |  Unable to update
> ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git?branch=main
> |
> | Caused by:
> |  failed to clone into:
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/cargo_home/git/db/zbus-git-dep-test-7f2b6322e3bb0cd5
> |
> | Caused by:
> |  network failure seems to have happened
> |  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
> |  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
> |
> | Caused by:
> |  failed to resolve address for gitlab.com: Temporary failure in name
> resolution; class=Net (12)|
> |  WARNING: exit code 101 from a shell command.
>
>
> In any case, I'll be glad to work on this patch if you tell me what
> strategy to adopt.
>
> On Mon, 13 Mar 2023 at 17:08, Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > I understand, I'll pursue all my tests with the current poky master. Thanks
> >
> > On Mon, 13 Mar 2023 at 11:42, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > >
> > > You need to use kirkstone or newer to see the bitbake-driven network
> > > disabling in action. Can you try on current poky master? You don't
> > > need to port the whole build to that, just one sample recipe.
> > >
> > > It is also expected that if you submit patches for master, they were
> > > tested on master.
> > >
> > > Alex
> > >
> > > On Mon, 13 Mar 2023 at 11:06, Frédéric Martinsons
> > > <frederic.martinsons@gmail.com> wrote:
> > > >
> > > > hmm, how is the network disabled ?
> > > >
> > > > Because with the patch I provided , the git clone in do_compile is successful.
> > > >
> > > > One major thing I forget to mention is that I use dunfell branch where
> > > > I backport all the cargo/rust relatives from master.
> > > > The backport was functional and straightforward, I just have to keep
> > > > the custom crate fetcher in openembedded-core (meta/lib/crate.py)
> > > > because I'm stuck with bitbake 1.46.
> > > >
> > > >    -
> > > >
> > > > On Mon, 13 Mar 2023 at 10:46, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > > > >
> > > > > Are you listing your ssh repos in your SRC_URI? We run with network
> > > > > disabled everywhere other than fetch and with that patch we don't see
> > > > > fetches from local ssh repos in compile.
> > > > >
> > > > > On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
> > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > >
> > > > > > Thanks for the patch alex, I did try it and the same happened (git
> > > > > > clone failure during do_compile), I'll
> > > > > > dig more on that and possibly submit patches if I found a way of
> > > > > > supporting git dependencies in Cargo.toml
> > > > > >
> > > > > > On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > > > > > >
> > > > > > > ISTR I tried this and ran into a different set of different problems.
> > > > > > >
> > > > > > > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > > > > > > <alex.kanavin@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Note that we're all rust neophytes, and rust support in yocto is still
> > > > > > > > evolving towards maturity. If you can see better ways to do things,
> > > > > > > > patches would be most welcome - but 'no network access except in
> > > > > > > > bitbake fetchers' is a core rule.
> > > > > > > >
> > > > > > > > Alex
> > > > > > > >
> > > > > > > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > I understand there should not be a fetch during build step , I'll try
> > > > > > > > > to find why .
> > > > > > > > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > > > > > > > build" with  --offline option ?
> > > > > > > > >
> > > > > > > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > > > > > > > showed in my first email.
> > > > > > > > > >
> > > > > > > > > > You need to figure out why this happens, because it should not. We
> > > > > > > > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > > > > > > > could start by studying how they work.
> > > > > > > > > >
> > > > > > > > > > Accessing network anywhere except do_fetch is not allowed for
> > > > > > > > > > reproducibility and license compliance reasons (both break if that
> > > > > > > > > > happens).
> > > > > > > > > >
> > > > > > > > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > > > > > > > status that we do not support. The official way is to use
> > > > > > > > > > cargo-update-recipe-crates class.
> > > > > > > > > >
> > > > > > > > > > Alex
> > > > > > > >
> > > > > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > > > > Links: You receive all messages sent to this group.
> > > > > > > > View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
> > > > > > > > Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> > > > > > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > > > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> > > > > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Alex Kiernan
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alex Kiernan

The other option is to add checksumming support to crate fetcher in
bitbake. If all items in src_uri are verified by fetchers directly for
integrity from checksums in the recipe, the cargo can generate cargo.lock
anytime it wants to.

Alex

On Tue 14. Mar 2023 at 11.35, Alexander Kanavin via lists.openembedded.org
<alex.kanavin=gmail.com@lists.openembedded.org> wrote:

> --offline seems like the right thing to add, if it produces better errors.
>
> Generating Cargo.lock on the other hand is not right. We rely on rust
> checking the source tree against the checksums in Cargo.lock (to
> prevent supply chain attacks), and this would completely subvert that.
> There could be better diagnostics around missing Cargo.lock, and steps
> to address the issue, but such steps must be manually taken (even
> though they could be codified in a special task, similar to how
> generating list of crates in SRC_URI is a task).
>
> Alex
>
> On Tue, 14 Mar 2023 at 11:25, Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > Hello, I finally found why my setup didn't work. That was not related
> > to https versus ssh.
> > For the patch provided in
> >
> https://patchwork.yoctoproject.org/project/oe-core/patch/20221030173815.10212-2-alex.kiernan@gmail.com/
> > to work, the repository should have a Cargo.lock file.
> > WIthout the Cargo.lock file, the cargo build will try to generate it
> > (and so needs network access).
> >
> > Note that I made a little change to the patch to support user in url
> > (the patch process in cargo only work if url is exact match):
> >
> > @@ -135,7 +135,10 @@ python cargo_common_do_patch_paths() {
> >              name = ud.parm.get('name')
> >              destsuffix = ud.parm.get('destsuffix')
> >              if name is not None and destsuffix is not None:
> > -                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
> > +                if ud.user:
> > +                    repo = '%s://%s@%s%s' % (ud.proto, ud.user,
> > ud.host, ud.path)
> > +                else:
> > +                    repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
> >                  path = '%s = { path = "%s" }' % (name,
> > os.path.join(workdir, destsuffix))
> >                  patches.setdefault(repo, []).append(path)
> >
> > Anyway, I would like to know how to you feel with this need for
> > Cargo.lock problem ?
> > Because for it to be generated, the project needs to be built once and
> > if this is expected for binary type to have Cargo.lock under version
> > control:
> >
> https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
> > It is not for "library" project.
> >
> > Maybe we can add a "cargo generate-lockfile"
> > (https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
> > at the end of do_fetch ?
> > Moreover, I think we should add the `--offline` option to cargo build
> > because the error generated will be more clear:
> >
> > | DEBUG: Executing shell function do_compile
> > | NOTE: Using rust targets from
> >
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
> > | NOTE: cargo =
> >
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo
> > | NOTE: cargo build -v --offline --target x86_64-poky-linux-gnu
> > --release
> --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
> > | error: failed to get `zbus-git-dep-test` as a dependency of package
> > `zbus-test v0.0.1
> >
> (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`
> > |
> > | Caused by:
> > |  failed to load source for dependency `zbus-git-dep-test`
> > |
> > | Caused by:
> > |  Unable to update
> > ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git?branch=main
> > |
> > | Caused by:
> > |  can't checkout from
> > 'ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git': you are in
> > the offline mode (--offline)
> > | WARNING: exit code 101 from a shell command.
> >
> > Instead of
> >
> > | DEBUG: Executing shell function do_compile
> > | NOTE: Using rust targets from
> >
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
> > | NOTE: cargo =
> >
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo
> > | NOTE: cargo build -v --target x86_64-poky-linux-gnu --release
> >
> --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
> > |   Updating git repository
> > `ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git`
> <http://git@gitlab.com/fmartinsons/zbus-git-dep-test.git>
> > | warning: spurious network error (2 tries remaining): failed to
> > resolve address for gitlab.com: Temporary failure in name resolution;
> > class=Net (12)
> > | warning: spurious network error (1 tries remaining): failed to
> > resolve address for gitlab.com: Temporary failure in name resolution;
> > class=Net (12)
> > | error: failed to get `zbus-git-dep-test` as a dependency of package
> > `zbus-test v0.0.1
> >
> (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`
> > |
> > | Caused by:
> > |  failed to load source for dependency `zbus-git-dep-test`
> > |
> > | Caused by:
> > |  Unable to update
> > ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git?branch=main
> > |
> > | Caused by:
> > |  failed to clone into:
> >
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/cargo_home/git/db/zbus-git-dep-test-7f2b6322e3bb0cd5
> > |
> > | Caused by:
> > |  network failure seems to have happened
> > |  if a proxy or similar is necessary `net.git-fetch-with-cli` may help
> here
> > |
> https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
> > |
> > | Caused by:
> > |  failed to resolve address for gitlab.com: Temporary failure in name
> > resolution; class=Net (12)|
> > |  WARNING: exit code 101 from a shell command.
> >
> >
> > In any case, I'll be glad to work on this patch if you tell me what
> > strategy to adopt.
> >
> > On Mon, 13 Mar 2023 at 17:08, Frédéric Martinsons
> > <frederic.martinsons@gmail.com> wrote:
> > >
> > > I understand, I'll pursue all my tests with the current poky master.
> Thanks
> > >
> > > On Mon, 13 Mar 2023 at 11:42, Alexander Kanavin <
> alex.kanavin@gmail.com> wrote:
> > > >
> > > > You need to use kirkstone or newer to see the bitbake-driven network
> > > > disabling in action. Can you try on current poky master? You don't
> > > > need to port the whole build to that, just one sample recipe.
> > > >
> > > > It is also expected that if you submit patches for master, they were
> > > > tested on master.
> > > >
> > > > Alex
> > > >
> > > > On Mon, 13 Mar 2023 at 11:06, Frédéric Martinsons
> > > > <frederic.martinsons@gmail.com> wrote:
> > > > >
> > > > > hmm, how is the network disabled ?
> > > > >
> > > > > Because with the patch I provided , the git clone in do_compile is
> successful.
> > > > >
> > > > > One major thing I forget to mention is that I use dunfell branch
> where
> > > > > I backport all the cargo/rust relatives from master.
> > > > > The backport was functional and straightforward, I just have to
> keep
> > > > > the custom crate fetcher in openembedded-core (meta/lib/crate.py)
> > > > > because I'm stuck with bitbake 1.46.
> > > > >
> > > > >    -
> > > > >
> > > > > On Mon, 13 Mar 2023 at 10:46, Alex Kiernan <alex.kiernan@gmail.com>
> wrote:
> > > > > >
> > > > > > Are you listing your ssh repos in your SRC_URI? We run with
> network
> > > > > > disabled everywhere other than fetch and with that patch we
> don't see
> > > > > > fetches from local ssh repos in compile.
> > > > > >
> > > > > > On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
> > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > >
> > > > > > > Thanks for the patch alex, I did try it and the same happened
> (git
> > > > > > > clone failure during do_compile), I'll
> > > > > > > dig more on that and possibly submit patches if I found a way
> of
> > > > > > > supporting git dependencies in Cargo.toml
> > > > > > >
> > > > > > > On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <
> alex.kiernan@gmail.com> wrote:
> > > > > > > >
> > > > > > > > ISTR I tried this and ran into a different set of different
> problems.
> > > > > > > >
> > > > > > > > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > > > > > > > <alex.kanavin@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Note that we're all rust neophytes, and rust support in
> yocto is still
> > > > > > > > > evolving towards maturity. If you can see better ways to
> do things,
> > > > > > > > > patches would be most welcome - but 'no network access
> except in
> > > > > > > > > bitbake fetchers' is a core rule.
> > > > > > > > >
> > > > > > > > > Alex
> > > > > > > > >
> > > > > > > > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > I understand there should not be a fetch during build
> step , I'll try
> > > > > > > > > > to find why .
> > > > > > > > > > In the mean time, if this is a strong rule, shouldn't we
> invoke "cargo
> > > > > > > > > > build" with  --offline option ?
> > > > > > > > > >
> > > > > > > > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <
> alex.kanavin@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > > > > > But the compile step tried to git clone anyway and
> so it failed like I
> > > > > > > > > > > > showed in my first email.
> > > > > > > > > > >
> > > > > > > > > > > You need to figure out why this happens, because it
> should not. We
> > > > > > > > > > > have plenty of recipes that list crates in SRC_URI in
> oe-core, so you
> > > > > > > > > > > could start by studying how they work.
> > > > > > > > > > >
> > > > > > > > > > > Accessing network anywhere except do_fetch is not
> allowed for
> > > > > > > > > > > reproducibility and license compliance reasons (both
> break if that
> > > > > > > > > > > happens).
> > > > > > > > > > >
> > > > > > > > > > > 'cargo bitbake' is a 3rd party project with
> questionable maintenance
> > > > > > > > > > > status that we do not support. The official way is to
> use
> > > > > > > > > > > cargo-update-recipe-crates class.
> > > > > > > > > > >
> > > > > > > > > > > Alex
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Alex Kiernan
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Alex Kiernan
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#178480):
> https://lists.openembedded.org/g/openembedded-core/message/178480
> Mute This Topic: https://lists.openembedded.org/mt/97555635/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

On Tue, Mar 14, 2023 at 10:25 AM Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> Hello, I finally found why my setup didn't work. That was not related
> to https versus ssh.
> For the patch provided in
> https://patchwork.yoctoproject.org/project/oe-core/patch/20221030173815.10212-2-alex.kiernan@gmail.com/
> to work, the repository should have a Cargo.lock file.
> WIthout the Cargo.lock file, the cargo build will try to generate it
> (and so needs network access).
>
> Note that I made a little change to the patch to support user in url
> (the patch process in cargo only work if url is exact match):
>
> @@ -135,7 +135,10 @@ python cargo_common_do_patch_paths() {
>              name = ud.parm.get('name')
>              destsuffix = ud.parm.get('destsuffix')
>              if name is not None and destsuffix is not None:
> -                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
> +                if ud.user:
> +                    repo = '%s://%s@%s%s' % (ud.proto, ud.user,
> ud.host, ud.path)
> +                else:
> +                    repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
>                  path = '%s = { path = "%s" }' % (name,
> os.path.join(workdir, destsuffix))
>                  patches.setdefault(repo, []).append(path)
>

Cool, will pick that up... the code needs some tests which is where it
stalled last time.

> Anyway, I would like to know how to you feel with this need for
> Cargo.lock problem ?
> Because for it to be generated, the project needs to be built once and
> if this is expected for binary type to have Cargo.lock under version
> control:
>  https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
> It is not for "library" project.
>

So I'm clear, you're building a library project here?

Could you consume your library into a wrapper project which brings
Cargo.lock along with it? Other than running tests on it, I'm not sure
what the value in building a library project is.

> Maybe we can add a "cargo generate-lockfile"
> (https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
> at the end of do_fetch ?
> Moreover, I think we should add the `--offline` option to cargo build
> because the error generated will be more clear:
>

Let me chuck that across our code base, I was fairly sure I'd seen
different problems with it, but it may have been whilst I was
debugging this first time around and am mis-remembering.

> | DEBUG: Executing shell function do_compile
> | NOTE: Using rust targets from
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
> | NOTE: cargo =
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo
> | NOTE: cargo build -v --offline --target x86_64-poky-linux-gnu
> --release --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
> | error: failed to get `zbus-git-dep-test` as a dependency of package
> `zbus-test v0.0.1
> (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`
> |
> | Caused by:
> |  failed to load source for dependency `zbus-git-dep-test`
> |
> | Caused by:
> |  Unable to update
> ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git?branch=main
> |
> | Caused by:
> |  can't checkout from
> 'ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git': you are in
> the offline mode (--offline)
> | WARNING: exit code 101 from a shell command.
>
> Instead of
>
> | DEBUG: Executing shell function do_compile
> | NOTE: Using rust targets from
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
> | NOTE: cargo =
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo
> | NOTE: cargo build -v --target x86_64-poky-linux-gnu --release
> --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
> |   Updating git repository
> `ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git`
> | warning: spurious network error (2 tries remaining): failed to
> resolve address for gitlab.com: Temporary failure in name resolution;
> class=Net (12)
> | warning: spurious network error (1 tries remaining): failed to
> resolve address for gitlab.com: Temporary failure in name resolution;
> class=Net (12)
> | error: failed to get `zbus-git-dep-test` as a dependency of package
> `zbus-test v0.0.1
> (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`
> |
> | Caused by:
> |  failed to load source for dependency `zbus-git-dep-test`
> |
> | Caused by:
> |  Unable to update
> ssh://git@gitlab.com/fmartinsons/zbus-git-dep-test.git?branch=main
> |
> | Caused by:
> |  failed to clone into:
> /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/cargo_home/git/db/zbus-git-dep-test-7f2b6322e3bb0cd5
> |
> | Caused by:
> |  network failure seems to have happened
> |  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
> |  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
> |
> | Caused by:
> |  failed to resolve address for gitlab.com: Temporary failure in name
> resolution; class=Net (12)|
> |  WARNING: exit code 101 from a shell command.
>
>
> In any case, I'll be glad to work on this patch if you tell me what
> strategy to adopt.
>
> On Mon, 13 Mar 2023 at 17:08, Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > I understand, I'll pursue all my tests with the current poky master. Thanks
> >
> > On Mon, 13 Mar 2023 at 11:42, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > >
> > > You need to use kirkstone or newer to see the bitbake-driven network
> > > disabling in action. Can you try on current poky master? You don't
> > > need to port the whole build to that, just one sample recipe.
> > >
> > > It is also expected that if you submit patches for master, they were
> > > tested on master.
> > >
> > > Alex
> > >
> > > On Mon, 13 Mar 2023 at 11:06, Frédéric Martinsons
> > > <frederic.martinsons@gmail.com> wrote:
> > > >
> > > > hmm, how is the network disabled ?
> > > >
> > > > Because with the patch I provided , the git clone in do_compile is successful.
> > > >
> > > > One major thing I forget to mention is that I use dunfell branch where
> > > > I backport all the cargo/rust relatives from master.
> > > > The backport was functional and straightforward, I just have to keep
> > > > the custom crate fetcher in openembedded-core (meta/lib/crate.py)
> > > > because I'm stuck with bitbake 1.46.
> > > >
> > > >    -
> > > >
> > > > On Mon, 13 Mar 2023 at 10:46, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > > > >
> > > > > Are you listing your ssh repos in your SRC_URI? We run with network
> > > > > disabled everywhere other than fetch and with that patch we don't see
> > > > > fetches from local ssh repos in compile.
> > > > >
> > > > > On Mon, Mar 13, 2023 at 9:32 AM Frédéric Martinsons
> > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > >
> > > > > > Thanks for the patch alex, I did try it and the same happened (git
> > > > > > clone failure during do_compile), I'll
> > > > > > dig more on that and possibly submit patches if I found a way of
> > > > > > supporting git dependencies in Cargo.toml
> > > > > >
> > > > > > On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <alex.kiernan@gmail.com> wrote:
> > > > > > >
> > > > > > > ISTR I tried this and ran into a different set of different problems.
> > > > > > >
> > > > > > > On Mon, Mar 13, 2023 at 9:20 AM Alexander Kanavin
> > > > > > > <alex.kanavin@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Note that we're all rust neophytes, and rust support in yocto is still
> > > > > > > > evolving towards maturity. If you can see better ways to do things,
> > > > > > > > patches would be most welcome - but 'no network access except in
> > > > > > > > bitbake fetchers' is a core rule.
> > > > > > > >
> > > > > > > > Alex
> > > > > > > >
> > > > > > > > On Mon, 13 Mar 2023 at 10:14, Frédéric Martinsons
> > > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > I understand there should not be a fetch during build step , I'll try
> > > > > > > > > to find why .
> > > > > > > > > In the mean time, if this is a strong rule, shouldn't we invoke "cargo
> > > > > > > > > build" with  --offline option ?
> > > > > > > > >
> > > > > > > > > On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Mon, 13 Mar 2023 at 09:29, Frédéric Martinsons
> > > > > > > > > > <frederic.martinsons@gmail.com> wrote:
> > > > > > > > > > > But the compile step tried to git clone anyway and so it failed like I
> > > > > > > > > > > showed in my first email.
> > > > > > > > > >
> > > > > > > > > > You need to figure out why this happens, because it should not. We
> > > > > > > > > > have plenty of recipes that list crates in SRC_URI in oe-core, so you
> > > > > > > > > > could start by studying how they work.
> > > > > > > > > >
> > > > > > > > > > Accessing network anywhere except do_fetch is not allowed for
> > > > > > > > > > reproducibility and license compliance reasons (both break if that
> > > > > > > > > > happens).
> > > > > > > > > >
> > > > > > > > > > 'cargo bitbake' is a 3rd party project with questionable maintenance
> > > > > > > > > > status that we do not support. The official way is to use
> > > > > > > > > > cargo-update-recipe-crates class.
> > > > > > > > > >
> > > > > > > > > > Alex
> > > > > > > >
> > > > > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > > > > Links: You receive all messages sent to this group.
> > > > > > > > View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
> > > > > > > > Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
> > > > > > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > > > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> > > > > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Alex Kiernan
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alex Kiernan

> Cool, will pick that up... the code needs some tests which is where it
> stalled last time.

Any help I can provide to you for having this patch merge ? Are you
talking about unit testing ?

> >  https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
> > It is not for "library" project.
> >
>
> So I'm clear, you're building a library project here?
>
> Could you consume your library into a wrapper project which brings
> Cargo.lock along with it? Other than running tests on it, I'm not sure
> what the value in building a library project is.
>

No, I build a binary but was not aware of the recommendation to add Cargo.lock
in version control (and was not aware that was the missing of this
file which make
cargo execute network operation during build),
I just started using rust  several weeks ago so every day I learn new
things ;)

> > Maybe we can add a "cargo generate-lockfile"
> > (https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
> > at the end of do_fetch ?
> > Moreover, I think we should add the `--offline` option to cargo build
> > because the error generated will be more clear:
> >
>
> Let me chuck that across our code base, I was fairly sure I'd seen
> different problems with it, but it may have been whilst I was
> debugging this first time around and am mis-remembering.
>

Alright.

On Tue, 14 Mar 2023 at 11:35, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> --offline seems like the right thing to add, if it produces better errors.
>
> Generating Cargo.lock on the other hand is not right. We rely on rust
> checking the source tree against the checksums in Cargo.lock (to
> prevent supply chain attacks), and this would completely subvert that.
> There could be better diagnostics around missing Cargo.lock, and steps
> to address the issue, but such steps must be manually taken (even
> though they could be codified in a special task, similar to how
> generating list of crates in SRC_URI is a task).
>
> Alex
>

Ok I see what you mean, generating the Cargo.lock in a dedicated task
inside  meta/classes-recipe/cargo-update-recipe-crates.bbclass
or maybe a dedicated meta/classes-recipe/cargo-generate-lockfile.bbclass ?

On Tue, 14 Mar 2023 at 12:25, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> The other option is to add checksumming support to crate fetcher in bitbake. If all items in src_uri are verified by fetchers directly for integrity from checksums in the recipe, the cargo can generate cargo.lock anytime it wants to.
>
> Alex
>

Ok, so If I understand correctly, that would mean a modification
inside bitbake/lib/bb/fetch2/crate.py to add such support ?
And if we have that, where the Cargo.lock generation should be made ?
it can only be during do_fetch step since it will
require network if a "git" dependencies is to be found in the project manifest.

On Tue, Mar 14, 2023 at 1:00 PM Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> On Tue, 14 Mar 2023 at 12:25, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> >
> > The other option is to add checksumming support to crate fetcher in bitbake. If all items in src_uri are verified by fetchers directly for integrity from checksums in the recipe, the cargo can generate cargo.lock anytime it wants to.
> >
> > Alex
> >
>
> Ok, so If I understand correctly, that would mean a modification
> inside bitbake/lib/bb/fetch2/crate.py to add such support ?
> And if we have that, where the Cargo.lock generation should be made ?
> it can only be during do_fetch step since it will
> require network if a "git" dependencies is to be found in the project manifest.

You'd have to list all the crate checksums in your recipe, so the
crate fetcher could check them.

--
Alex Kiernan

On Tue, Mar 14, 2023 at 12:58 PM Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:
>
> On Tue, 14 Mar 2023 at 11:35, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> >
> > --offline seems like the right thing to add, if it produces better errors.
> >
> > Generating Cargo.lock on the other hand is not right. We rely on rust
> > checking the source tree against the checksums in Cargo.lock (to
> > prevent supply chain attacks), and this would completely subvert that.
> > There could be better diagnostics around missing Cargo.lock, and steps
> > to address the issue, but such steps must be manually taken (even
> > though they could be codified in a special task, similar to how
> > generating list of crates in SRC_URI is a task).
> >
> > Alex
> >
>
> Ok I see what you mean, generating the Cargo.lock in a dedicated task
> inside  meta/classes-recipe/cargo-update-recipe-crates.bbclass
> or maybe a dedicated meta/classes-recipe/cargo-generate-lockfile.bbclass ?

I'm really not sure we want Cargo.lock generation as part of the OE
workflow - Cargo.lock should be part of your checked in sources for
non-library projects.


--
Alex Kiernan

Understood, I searched how to add checksum support in crate fetcher but I
don't know enough of bitbake to be effective.

I'll check in the Cargo.lock in my project, backport your patch and
continue my exploration of the wonderful rust world on embedded device ^^

Thank you very much for the help on the subject guys and like I said
earlier, if you need me to do some more tests to help the patch being
merged in the master, I'm available.



Le mar. 14 mars 2023, 15:59, Alex Kiernan <alex.kiernan@gmail.com> a écrit :

> On Tue, Mar 14, 2023 at 12:58 PM Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > On Tue, 14 Mar 2023 at 11:35, Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
> > >
> > > --offline seems like the right thing to add, if it produces better
> errors.
> > >
> > > Generating Cargo.lock on the other hand is not right. We rely on rust
> > > checking the source tree against the checksums in Cargo.lock (to
> > > prevent supply chain attacks), and this would completely subvert that.
> > > There could be better diagnostics around missing Cargo.lock, and steps
> > > to address the issue, but such steps must be manually taken (even
> > > though they could be codified in a special task, similar to how
> > > generating list of crates in SRC_URI is a task).
> > >
> > > Alex
> > >
> >
> > Ok I see what you mean, generating the Cargo.lock in a dedicated task
> > inside  meta/classes-recipe/cargo-update-recipe-crates.bbclass
> > or maybe a dedicated meta/classes-recipe/cargo-generate-lockfile.bbclass
> ?
>
> I'm really not sure we want Cargo.lock generation as part of the OE
> workflow - Cargo.lock should be part of your checked in sources for
> non-library projects.
>
>
> --
> Alex Kiernan
>

On Tue, 14 Mar 2023 at 16:10, Frédéric Martinsons
<frederic.martinsons@gmail.com> wrote:

> Understood, I searched how to add checksum support in crate fetcher but I don't know enough of bitbake to be effective.

You can check how for example http:// fetcher does it by utilizing
verify_checksum() from lib/bb/fetch2/__init__.py in
lib/bb/fetch2/wget.py.

Something like that should happen in crate.py as well.

Alex

On Tue, Mar 14, 2023 at 12:22 PM Alex Kiernan <alex.kiernan@gmail.com> wrote:
>
> On Tue, Mar 14, 2023 at 10:25 AM Frédéric Martinsons
> <frederic.martinsons@gmail.com> wrote:
> >
> > Moreover, I think we should add the `--offline` option to cargo build
> > because the error generated will be more clear:
> >
>
> Let me chuck that across our code base, I was fairly sure I'd seen
> different problems with it, but it may have been whilst I was
> debugging this first time around and am mis-remembering.
>

I'm wrong, seems to be fine.

I think I'll dig into the checksum capability on crate fetcher in the
coming days.

For the offline option and the modication of the patch mentioned above, how
do you want me to proceed?

Submit a dedicated patch for each of these (one for offline option and one
for checksum if I can come up with a suitable patch)?

For the modification of the patch for using a local directory and not
relying on git clone during build time, do you want me to provide something
in particular?

Note that I saw the downloaded crates not being cleaned up from DL_DIR when
I issue a bitbake -c cleanall but I need to confirm this on master branch.

Le mar. 14 mars 2023, 17:49, Alex Kiernan <alex.kiernan@gmail.com> a écrit :

> On Tue, Mar 14, 2023 at 12:22 PM Alex Kiernan <alex.kiernan@gmail.com>
> wrote:
> >
> > On Tue, Mar 14, 2023 at 10:25 AM Frédéric Martinsons
> > <frederic.martinsons@gmail.com> wrote:
> > >
> > > Moreover, I think we should add the `--offline` option to cargo build
> > > because the error generated will be more clear:
> > >
> >
> > Let me chuck that across our code base, I was fairly sure I'd seen
> > different problems with it, but it may have been whilst I was
> > debugging this first time around and am mis-remembering.
> >
>
> I'm wrong, seems to be fine.
>
> --
> Alex Kiernan
>