| Message ID | 20250220172706.3850722-1-stefan-koch@siemens.com |
|---|---|
| State | Accepted, archived |
| Commit | 457288b2fda86fd00cdcaefac616129b0029e1f9 |
| Headers | show |
| Series | [v3,1/4] fetch2/git: Add support for fast initial shallow fetch | expand |
On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > When `ud.shallow == 1`: > - Prefer an initial shallow clone over an initial full bare clone, > while still utilizing any already existing full bare clones. > > This improves: > - Resolve timeout issues during initial clones on slow internet connections > by reducing the amount of data transferred. > - Eliminate the need to use an HTTPS tarball `SRC_URI` > to reduce data transfer. > - Allow SSH-based authentication (e.g. cert and agent-based) when > using non-public repos, so additional HTTPS tokens may not be required. > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > --- > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++----------- > 1 file changed, 74 insertions(+), 26 deletions(-) In testing it looks like this series has some issue on Ubuntu 20.04 systems (both x86_64 and arm): https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 This is probably due to the version of git there... Cheers, Richard
On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > When `ud.shallow == 1`: > > - Prefer an initial shallow clone over an initial full bare clone, > > while still utilizing any already existing full bare clones. > > > > This improves: > > - Resolve timeout issues during initial clones on slow internet > > connections > > by reducing the amount of data transferred. > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > to reduce data transfer. > > - Allow SSH-based authentication (e.g. cert and agent-based) when > > using non-public repos, so additional HTTPS tokens may not be > > required. > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > --- > > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++------- > > ---- > > 1 file changed, 74 insertions(+), 26 deletions(-) > > In testing it looks like this series has some issue on Ubuntu 20.04 > systems (both x86_64 and arm): > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > This is probably due to the version of git there... This issue could occur in two cases: a) old Git client version this could be relevant: https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ b) Git server version, that has disabled announcements see: https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r So, I have reintroduced the switch from v2 patch with inverse logic. 1. enabled by default 2. old behaviour can be forced by variable 3. old behaviour is triggered automatically in mentioned failure case (see above) I'll send updated v4 patch series. > > Cheers, > > Richard Thanks, Stefan Koch Siemens AG http://www.siemens.com/
On Fri, 2025-02-21 at 16:40 +0000, Koch, Stefan wrote: > On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > > When `ud.shallow == 1`: > > > - Prefer an initial shallow clone over an initial full bare > > > clone, > > > while still utilizing any already existing full bare clones. > > > > > > This improves: > > > - Resolve timeout issues during initial clones on slow internet > > > connections > > > by reducing the amount of data transferred. > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > to reduce data transfer. > > > - Allow SSH-based authentication (e.g. cert and agent-based) when > > > using non-public repos, so additional HTTPS tokens may not be > > > required. > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > --- > > > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++----- > > > -- > > > ---- > > > 1 file changed, 74 insertions(+), 26 deletions(-) > > > > In testing it looks like this series has some issue on Ubuntu 20.04 > > systems (both x86_64 and arm): > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > > > This is probably due to the version of git there... > > This issue could occur in two cases: > a) old Git client version > this could be relevant: > https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ I suspect the git version is the cause of the issue. > b) Git server version, that has disabled announcements > see: > https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r We're testing against the same server and we see some passes and only failures on Ubuntu 20.04 so it is unlikely to be this. > So, I have reintroduced the switch from v2 patch with inverse logic. > 1. enabled by default > 2. old behaviour can be forced by variable So in CI on the autobuilder, we need to force the variable to allow the selftests to pass on 20.04 hosts? We also need to error if anyone enables that form of fetching on 20.04 hosts too? > 3. old behaviour is triggered automatically in mentioned failure case > (see above) > > I'll send updated v4 patch series. > I'm afraid this solution won't work for us. We either need to find a work around, fall back to a different way of fetching or I guess we have to drop support for Ubuntu 20.04 without using builtools tarball and bump our minimum git version requirements. Cheers, Richard
On Fri, 2025-02-21 at 17:11 +0000, Richard Purdie wrote: > On Fri, 2025-02-21 at 16:40 +0000, Koch, Stefan wrote: > > On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > > > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > > > When `ud.shallow == 1`: > > > > - Prefer an initial shallow clone over an initial full bare > > > > clone, > > > > while still utilizing any already existing full bare clones. > > > > > > > > This improves: > > > > - Resolve timeout issues during initial clones on slow internet > > > > connections > > > > by reducing the amount of data transferred. > > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > > to reduce data transfer. > > > > - Allow SSH-based authentication (e.g. cert and agent-based) > > > > when > > > > using non-public repos, so additional HTTPS tokens may not be > > > > required. > > > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > > --- > > > > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++--- > > > > -- > > > > -- > > > > ---- > > > > 1 file changed, 74 insertions(+), 26 deletions(-) > > > > > > In testing it looks like this series has some issue on Ubuntu > > > 20.04 > > > systems (both x86_64 and arm): > > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > > > > > This is probably due to the version of git there... > > > > This issue could occur in two cases: > > a) old Git client version > > this could be relevant: > > https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ > > I suspect the git version is the cause of the issue. > > > b) Git server version, that has disabled announcements > > see: > > https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r > > We're testing against the same server and we see some passes and only > failures on Ubuntu 20.04 so it is unlikely to be this. > > > So, I have reintroduced the switch from v2 patch with inverse > > logic. > > 1. enabled by default > > 2. old behaviour can be forced by variable > > So in CI on the autobuilder, we need to force the variable to allow > the > selftests to pass on 20.04 hosts? We also need to error if anyone > enables that form of fetching on 20.04 hosts too? My current v4 implementation does it automatically in failure case. Because the disabled server side issue is possible at end users but not BB CI/CD (there only 20.04) > > > 3. old behaviour is triggered automatically in mentioned failure > > case > > (see above) > > > > I'll send updated v4 patch series. This patch should work with 20.04, too. > > > > I'm afraid this solution won't work for us. We either need to find a > work around, fall back to a different way of fetching or I guess we > have to drop support for Ubuntu 20.04 without using builtools tarball > and bump our minimum git version requirements. Drop 20.04 might be possible to. The EOL of 20.04 comes in a few months... I would prefer the automatic error handling, to support old git server versions at end users, too. > > Cheers, > > Richard > > -- Stefan Koch Siemens AG http://www.siemens.com/
On Fri, 2025-02-21 at 17:18 +0000, Koch, Stefan wrote: > On Fri, 2025-02-21 at 17:11 +0000, Richard Purdie wrote: > > On Fri, 2025-02-21 at 16:40 +0000, Koch, Stefan wrote: > > > On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > > > > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > > > > When `ud.shallow == 1`: > > > > > - Prefer an initial shallow clone over an initial full bare > > > > > clone, > > > > > while still utilizing any already existing full bare clones. > > > > > > > > > > This improves: > > > > > - Resolve timeout issues during initial clones on slow internet > > > > > connections > > > > > by reducing the amount of data transferred. > > > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > > > to reduce data transfer. > > > > > - Allow SSH-based authentication (e.g. cert and agent-based) > > > > > when > > > > > using non-public repos, so additional HTTPS tokens may not be > > > > > required. > > > > > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > > > --- > > > > > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++--- > > > > > -- > > > > > -- > > > > > ---- > > > > > 1 file changed, 74 insertions(+), 26 deletions(-) > > > > > > > > In testing it looks like this series has some issue on Ubuntu > > > > 20.04 > > > > systems (both x86_64 and arm): > > > > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > > > > > > > This is probably due to the version of git there... > > > > > > This issue could occur in two cases: > > > a) old Git client version > > > this could be relevant: > > > https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ > > > > I suspect the git version is the cause of the issue. > > > > > b) Git server version, that has disabled announcements > > > see: > > > https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r > > > > We're testing against the same server and we see some passes and only > > failures on Ubuntu 20.04 so it is unlikely to be this. > > > > > So, I have reintroduced the switch from v2 patch with inverse > > > logic. > > > 1. enabled by default > > > 2. old behaviour can be forced by variable > > > > So in CI on the autobuilder, we need to force the variable to allow > > the > > selftests to pass on 20.04 hosts? We also need to error if anyone > > enables that form of fetching on 20.04 hosts too? > > My current v4 implementation does it automatically in failure case. > Because the disabled server side issue is possible at end users but not > BB CI/CD (there only 20.04) Ok, automatic fallback sounds ok as long as it doesn't complicate the code too much. I'm not sure we want to have a variable for it though, I don't like users needing to care about setting such things from a usability standpoint. > > > 3. old behaviour is triggered automatically in mentioned failure > > > case > > > (see above) > > > > > > I'll send updated v4 patch series. > This patch should work with 20.04, too. > > > > > > > I'm afraid this solution won't work for us. We either need to find a > > work around, fall back to a different way of fetching or I guess we > > have to drop support for Ubuntu 20.04 without using builtools tarball > > and bump our minimum git version requirements. > Drop 20.04 might be possible to. The EOL of 20.04 comes in a few > months... > > I would prefer the automatic error handling, to support old git server > versions at end users, too. There are other reasons we've wondered about using buildtools on 20.04 but I would prefer not to so this seems reasonable assuming the patch isn't too bad... Cheers, Richard
On Fri, 2025-02-21 at 17:22 +0000, Richard Purdie wrote: > On Fri, 2025-02-21 at 17:18 +0000, Koch, Stefan wrote: > > On Fri, 2025-02-21 at 17:11 +0000, Richard Purdie wrote: > > > On Fri, 2025-02-21 at 16:40 +0000, Koch, Stefan wrote: > > > > On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > > > > > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > > > > > When `ud.shallow == 1`: > > > > > > - Prefer an initial shallow clone over an initial full bare > > > > > > clone, > > > > > > while still utilizing any already existing full bare > > > > > > clones. > > > > > > > > > > > > This improves: > > > > > > - Resolve timeout issues during initial clones on slow > > > > > > internet > > > > > > connections > > > > > > by reducing the amount of data transferred. > > > > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > > > > to reduce data transfer. > > > > > > - Allow SSH-based authentication (e.g. cert and agent- > > > > > > based) > > > > > > when > > > > > > using non-public repos, so additional HTTPS tokens may > > > > > > not be > > > > > > required. > > > > > > > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > > > > --- > > > > > > lib/bb/fetch2/git.py | 100 > > > > > > ++++++++++++++++++++++++++++++++--- > > > > > > -- > > > > > > -- > > > > > > ---- > > > > > > 1 file changed, 74 insertions(+), 26 deletions(-) > > > > > > > > > > In testing it looks like this series has some issue on Ubuntu > > > > > 20.04 > > > > > systems (both x86_64 and arm): > > > > > > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > > > > > > > > > This is probably due to the version of git there... > > > > > > > > This issue could occur in two cases: > > > > a) old Git client version > > > > this could be relevant: > > > > https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ > > > > > > I suspect the git version is the cause of the issue. > > > > > > > b) Git server version, that has disabled announcements > > > > see: > > > > https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r > > > > > > We're testing against the same server and we see some passes and > > > only > > > failures on Ubuntu 20.04 so it is unlikely to be this. > > > > > > > So, I have reintroduced the switch from v2 patch with inverse > > > > logic. > > > > 1. enabled by default > > > > 2. old behaviour can be forced by variable > > > > > > So in CI on the autobuilder, we need to force the variable to > > > allow > > > the > > > selftests to pass on 20.04 hosts? We also need to error if anyone > > > enables that form of fetching on 20.04 hosts too? > > > > My current v4 implementation does it automatically in failure case. > > Because the disabled server side issue is possible at end users but > > not > > BB CI/CD (there only 20.04) > > Ok, automatic fallback sounds ok as long as it doesn't complicate the > code too much. I'm not sure we want to have a variable for it though, > I > don't like users needing to care about setting such things from a > usability standpoint. Sent already v4, but it would possible to a) make BB_GIT_SHALLOW_SKIP_FAST an undocumented feature b) only using ud.shallow_skip_fast for that internally (but is it possible to specify shallow_skip_fast=1 via SRC_URI?) c) using not `ud` at all > > > > > 3. old behaviour is triggered automatically in mentioned > > > > failure > > > > case > > > > (see above) > > > > > > > > I'll send updated v4 patch series. > > This patch should work with 20.04, too. > > > > > > > > > > I'm afraid this solution won't work for us. We either need to > > > find a > > > work around, fall back to a different way of fetching or I guess > > > we > > > have to drop support for Ubuntu 20.04 without using builtools > > > tarball > > > and bump our minimum git version requirements. > > Drop 20.04 might be possible to. The EOL of 20.04 comes in a few > > months... > > > > I would prefer the automatic error handling, to support old git > > server > > versions at end users, too. > > There are other reasons we've wondered about using buildtools on > 20.04 > but I would prefer not to so this seems reasonable assuming the patch > isn't too bad... And there might be some (legacy) GIT servers on the internet that doesn't provide support for fetching a commit hash directly. > > Cheers, > > Richard -- Stefan Koch Siemens AG http://www.siemens.com/
On Fri, 2025-02-21 at 17:31 +0000, Koch, Stefan wrote: > On Fri, 2025-02-21 at 17:22 +0000, Richard Purdie wrote: > > On Fri, 2025-02-21 at 17:18 +0000, Koch, Stefan wrote: > > > On Fri, 2025-02-21 at 17:11 +0000, Richard Purdie wrote: > > > > > > > > > My current v4 implementation does it automatically in failure > > > case. > > > Because the disabled server side issue is possible at end users > > > but > > > not > > > BB CI/CD (there only 20.04) > > > > Ok, automatic fallback sounds ok as long as it doesn't complicate > > the > > code too much. I'm not sure we want to have a variable for it > > though, > > I > > don't like users needing to care about setting such things from a > > usability standpoint. > > Sent already v4, but it would possible to > a) make BB_GIT_SHALLOW_SKIP_FAST an undocumented feature This is surprisingly hard to do since someone generally finds it and assumes it is a bug in the documentation. There were a lot of these filed just in the last week. As soon as you list them somewhere, someone asks they be documented. > b) only using ud.shallow_skip_fast for that internally (but is it > possible to specify shallow_skip_fast=1 via SRC_URI?) > c) using not `ud` at all A parameter in the SRC_URI url could be a possibility but it also has the same issue as a), people then will ask it be documented. People also expect options to be tested and so on. I'm really torn on exposing it at all for that reason... I appreciate it sounds simple but I also see all these requests for docs, testing and so on :/ Cheers, Richard
On Thu Feb 20, 2025 at 6:27 PM CET, Stefan via lists.openembedded.org Koch wrote: > When `ud.shallow == 1`: > - Prefer an initial shallow clone over an initial full bare clone, > while still utilizing any already existing full bare clones. > > This improves: > - Resolve timeout issues during initial clones on slow internet connections > by reducing the amount of data transferred. > - Eliminate the need to use an HTTPS tarball `SRC_URI` > to reduce data transfer. > - Allow SSH-based authentication (e.g. cert and agent-based) when > using non-public repos, so additional HTTPS tokens may not be required. > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > --- Hi Stefan, Thanks for your patch. We got a new failure on the autobuilder, specifically with the oe-selftest-ubuntu build. I'm not entirely sure this comes from this series, so I will try a build without it to confirm. Here is the error: NOTE: Running task 2 of 4 (/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-3502943/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb:do_fetch) NOTE: recipe git-submodule-test-1.0-r0: task do_fetch: Started WARNING: git-submodule-test-1.0-r0 do_fetch: Creating fast initial shallow clone failed, try initial regular clone now. WARNING: git-submodule-test-1.0-r0 do_fetch: Failed to fetch URL gitsm://git.openembedded.org/bitbake;protocol=git;name=bitbake;subpath=bitbake;nobranch=1;lfs=False;bareclone=1;nobranch=1, attempting MIRRORS if available ERROR: git-submodule-test-1.0-r0 do_fetch: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export PATH="/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/sysroots-uninative/x86_64-linux/usr/bin:/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/scripts:/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot/usr/bin/crossscripts:/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/usr/sbin:/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/usr/bin:/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/sbin:/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/bin:/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/bitbake/bin:/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/hosttools"; export HOME="/srv/pokybuild"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin fetch origin 52a144a7daa94b2bd239d582cb71d1f03119918f --depth 1 failed with exit code 1, output: error: Server does not allow request for unadvertised object 52a144a7daa94b2bd239d582cb71d1f03119918f https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1031/steps/14/logs/stdio Can you have a look at this issue please?
On Mon, 2025-02-24 at 12:58 +0100, Mathieu Dubois-Briand wrote: > On Thu Feb 20, 2025 at 6:27 PM CET, Stefan via lists.openembedded.org > Koch wrote: > > When `ud.shallow == 1`: > > - Prefer an initial shallow clone over an initial full bare clone, > > while still utilizing any already existing full bare clones. > > > > This improves: > > - Resolve timeout issues during initial clones on slow internet > > connections > > by reducing the amount of data transferred. > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > to reduce data transfer. > > - Allow SSH-based authentication (e.g. cert and agent-based) when > > using non-public repos, so additional HTTPS tokens may not be > > required. > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > --- > > Hi Stefan, > > Thanks for your patch. > > We got a new failure on the autobuilder, specifically with the > oe-selftest-ubuntu build. I'm not entirely sure this comes from this > series, so I will try a build without it to confirm. > > Here is the error: > > NOTE: Running task 2 of 4 (/srv/pokybuild/yocto-worker/oe-selftest- > ubuntu/build/build-st-3502943/meta-selftest/recipes-test/git- > submodule-test/git-submodule-test.bb:do_fetch) > NOTE: recipe git-submodule-test-1.0-r0: task do_fetch: Started > WARNING: git-submodule-test-1.0-r0 do_fetch: Creating fast initial > shallow clone failed, try initial regular clone now. > WARNING: git-submodule-test-1.0-r0 do_fetch: Failed to fetch URL > gitsm://git.openembedded.org/bitbake;protocol=git;name=bitbake;subpat > h=bitbake;nobranch=1;lfs=False;bareclone=1;nobranch=1, attempting > MIRRORS if available > ERROR: git-submodule-test-1.0-r0 do_fetch: Fetcher failure: Fetch > command export PSEUDO_DISABLED=1; export PATH="/srv/pokybuild/yocto- > worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/sysroots- > uninative/x86_64-linux/usr/bin:/srv/pokybuild/yocto-worker/oe- > selftest-ubuntu/build/scripts:/srv/pokybuild/yocto-worker/oe- > selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky- > linux/git-submodule-test/1.0/recipe-sysroot-native/usr/bin/x86_64- > poky-linux:/srv/pokybuild/yocto-worker/oe-selftest- > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > submodule-test/1.0/recipe- > sysroot/usr/bin/crossscripts:/srv/pokybuild/yocto-worker/oe-selftest- > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > submodule-test/1.0/recipe-sysroot- > native/usr/sbin:/srv/pokybuild/yocto-worker/oe-selftest- > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > submodule-test/1.0/recipe-sysroot- > native/usr/bin:/srv/pokybuild/yocto-worker/oe-selftest- > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > submodule-test/1.0/recipe-sysroot-native/sbin:/srv/pokybuild/yocto- > worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64- > poky-linux/git-submodule-test/1.0/recipe-sysroot- > native/bin:/srv/pokybuild/yocto-worker/oe-selftest- > ubuntu/build/bitbake/bin:/srv/pokybuild/yocto-worker/oe-selftest- > ubuntu/build/build-st-3502943/tmp/hosttools"; export > HOME="/srv/pokybuild"; git -c gc.autoDetach=false -c core.pager=cat - > c safe.bareRepository=all -c clone.defaultRemoteName=origin fetch > origin 52a144a7daa94b2bd239d582cb71d1f03119918f --depth 1 failed with > exit code 1, output: > error: Server does not allow request for unadvertised object > 52a144a7daa94b2bd239d582cb71d1f03119918f > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1031/steps/14/logs/stdio > > Can you have a look at this issue please? Do you using 1716ad67bbe9f4cecf1e3bcfc2007b150ba77741 from bitbakes master-next branch? Because that is PATCH v2 with that issue specially on ubuntu 20.04... > -- Stefan Koch Siemens AG http://www.siemens.com/
On Mon, 2025-02-24 at 13:55 +0100, Stefan Koch wrote: > On Mon, 2025-02-24 at 12:58 +0100, Mathieu Dubois-Briand wrote: > > On Thu Feb 20, 2025 at 6:27 PM CET, Stefan via > > lists.openembedded.org > > Koch wrote: > > > When `ud.shallow == 1`: > > > - Prefer an initial shallow clone over an initial full bare > > > clone, > > > while still utilizing any already existing full bare clones. > > > > > > This improves: > > > - Resolve timeout issues during initial clones on slow internet > > > connections > > > by reducing the amount of data transferred. > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > to reduce data transfer. > > > - Allow SSH-based authentication (e.g. cert and agent-based) when > > > using non-public repos, so additional HTTPS tokens may not be > > > required. > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > --- > > > > Hi Stefan, > > > > Thanks for your patch. > > > > We got a new failure on the autobuilder, specifically with the > > oe-selftest-ubuntu build. I'm not entirely sure this comes from > > this > > series, so I will try a build without it to confirm. > > > > Here is the error: > > > > NOTE: Running task 2 of 4 (/srv/pokybuild/yocto-worker/oe-selftest- > > ubuntu/build/build-st-3502943/meta-selftest/recipes-test/git- > > submodule-test/git-submodule-test.bb:do_fetch) > > NOTE: recipe git-submodule-test-1.0-r0: task do_fetch: Started > > WARNING: git-submodule-test-1.0-r0 do_fetch: Creating fast initial > > shallow clone failed, try initial regular clone now. > > WARNING: git-submodule-test-1.0-r0 do_fetch: Failed to fetch URL > > gitsm://git.openembedded.org/bitbake;protocol=git;name=bitbake;subp > > at > > h=bitbake;nobranch=1;lfs=False;bareclone=1;nobranch=1, attempting > > MIRRORS if available > > ERROR: git-submodule-test-1.0-r0 do_fetch: Fetcher failure: Fetch > > command export PSEUDO_DISABLED=1; export > > PATH="/srv/pokybuild/yocto- > > worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/sysroots- > > uninative/x86_64-linux/usr/bin:/srv/pokybuild/yocto-worker/oe- > > selftest-ubuntu/build/scripts:/srv/pokybuild/yocto-worker/oe- > > selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky- > > linux/git-submodule-test/1.0/recipe-sysroot-native/usr/bin/x86_64- > > poky-linux:/srv/pokybuild/yocto-worker/oe-selftest- > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > submodule-test/1.0/recipe- > > sysroot/usr/bin/crossscripts:/srv/pokybuild/yocto-worker/oe- > > selftest- > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > submodule-test/1.0/recipe-sysroot- > > native/usr/sbin:/srv/pokybuild/yocto-worker/oe-selftest- > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > submodule-test/1.0/recipe-sysroot- > > native/usr/bin:/srv/pokybuild/yocto-worker/oe-selftest- > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > submodule-test/1.0/recipe-sysroot-native/sbin:/srv/pokybuild/yocto- > > worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64- > > poky-linux/git-submodule-test/1.0/recipe-sysroot- > > native/bin:/srv/pokybuild/yocto-worker/oe-selftest- > > ubuntu/build/bitbake/bin:/srv/pokybuild/yocto-worker/oe-selftest- > > ubuntu/build/build-st-3502943/tmp/hosttools"; export > > HOME="/srv/pokybuild"; git -c gc.autoDetach=false -c core.pager=cat > > - > > c safe.bareRepository=all -c clone.defaultRemoteName=origin fetch > > origin 52a144a7daa94b2bd239d582cb71d1f03119918f --depth 1 failed > > with > > exit code 1, output: > > error: Server does not allow request for unadvertised object > > 52a144a7daa94b2bd239d582cb71d1f03119918f > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1031/steps/14/logs/stdio > > > > Can you have a look at this issue please? > Do you using 1716ad67bbe9f4cecf1e3bcfc2007b150ba77741 from bitbakes > master-next branch? Because that is PATCH v2 with that issue > specially > on ubuntu 20.04... Sorry, v3 is meant. v4 PATCH from Friday should fix this. > > > -- Stefan Koch Siemens AG http://www.siemens.com/
On Mon Feb 24, 2025 at 2:11 PM CET, Stefan Koch wrote: > On Mon, 2025-02-24 at 13:55 +0100, Stefan Koch wrote: > > On Mon, 2025-02-24 at 12:58 +0100, Mathieu Dubois-Briand wrote: > > > On Thu Feb 20, 2025 at 6:27 PM CET, Stefan via > > > lists.openembedded.org > > > Koch wrote: > > > > When `ud.shallow == 1`: > > > > - Prefer an initial shallow clone over an initial full bare > > > > clone, > > > > while still utilizing any already existing full bare clones. > > > > > > > > This improves: > > > > - Resolve timeout issues during initial clones on slow internet > > > > connections > > > > by reducing the amount of data transferred. > > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > > to reduce data transfer. > > > > - Allow SSH-based authentication (e.g. cert and agent-based) when > > > > using non-public repos, so additional HTTPS tokens may not be > > > > required. > > > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > > --- > > > > > > Hi Stefan, > > > > > > Thanks for your patch. > > > > > > We got a new failure on the autobuilder, specifically with the > > > oe-selftest-ubuntu build. I'm not entirely sure this comes from > > > this > > > series, so I will try a build without it to confirm. > > > > > > Here is the error: > > > > > > NOTE: Running task 2 of 4 (/srv/pokybuild/yocto-worker/oe-selftest- > > > ubuntu/build/build-st-3502943/meta-selftest/recipes-test/git- > > > submodule-test/git-submodule-test.bb:do_fetch) > > > NOTE: recipe git-submodule-test-1.0-r0: task do_fetch: Started > > > WARNING: git-submodule-test-1.0-r0 do_fetch: Creating fast initial > > > shallow clone failed, try initial regular clone now. > > > WARNING: git-submodule-test-1.0-r0 do_fetch: Failed to fetch URL > > > gitsm://git.openembedded.org/bitbake;protocol=git;name=bitbake;subp > > > at > > > h=bitbake;nobranch=1;lfs=False;bareclone=1;nobranch=1, attempting > > > MIRRORS if available > > > ERROR: git-submodule-test-1.0-r0 do_fetch: Fetcher failure: Fetch > > > command export PSEUDO_DISABLED=1; export > > > PATH="/srv/pokybuild/yocto- > > > worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/sysroots- > > > uninative/x86_64-linux/usr/bin:/srv/pokybuild/yocto-worker/oe- > > > selftest-ubuntu/build/scripts:/srv/pokybuild/yocto-worker/oe- > > > selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky- > > > linux/git-submodule-test/1.0/recipe-sysroot-native/usr/bin/x86_64- > > > poky-linux:/srv/pokybuild/yocto-worker/oe-selftest- > > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > > submodule-test/1.0/recipe- > > > sysroot/usr/bin/crossscripts:/srv/pokybuild/yocto-worker/oe- > > > selftest- > > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > > submodule-test/1.0/recipe-sysroot- > > > native/usr/sbin:/srv/pokybuild/yocto-worker/oe-selftest- > > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > > submodule-test/1.0/recipe-sysroot- > > > native/usr/bin:/srv/pokybuild/yocto-worker/oe-selftest- > > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > > submodule-test/1.0/recipe-sysroot-native/sbin:/srv/pokybuild/yocto- > > > worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64- > > > poky-linux/git-submodule-test/1.0/recipe-sysroot- > > > native/bin:/srv/pokybuild/yocto-worker/oe-selftest- > > > ubuntu/build/bitbake/bin:/srv/pokybuild/yocto-worker/oe-selftest- > > > ubuntu/build/build-st-3502943/tmp/hosttools"; export > > > HOME="/srv/pokybuild"; git -c gc.autoDetach=false -c core.pager=cat > > > - > > > c safe.bareRepository=all -c clone.defaultRemoteName=origin fetch > > > origin 52a144a7daa94b2bd239d582cb71d1f03119918f --depth 1 failed > > > with > > > exit code 1, output: > > > error: Server does not allow request for unadvertised object > > > 52a144a7daa94b2bd239d582cb71d1f03119918f > > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1031/steps/14/logs/stdio > > > > > > Can you have a look at this issue please? > > Do you using 1716ad67bbe9f4cecf1e3bcfc2007b150ba77741 from bitbakes > > master-next branch? Because that is PATCH v2 with that issue > > specially > > on ubuntu 20.04... > Sorry, v3 is meant. > v4 PATCH from Friday should fix this. > > > > > Sorry, I sent this in the wrong mail thread, but I confirm I was indeed using the v4.
On Mon, 2025-02-24 at 18:10 +0100, Mathieu Dubois-Briand wrote: > On Mon Feb 24, 2025 at 2:11 PM CET, Stefan Koch wrote: > > On Mon, 2025-02-24 at 13:55 +0100, Stefan Koch wrote: > > > On Mon, 2025-02-24 at 12:58 +0100, Mathieu Dubois-Briand wrote: > > > > On Thu Feb 20, 2025 at 6:27 PM CET, Stefan via > > > > lists.openembedded.org > > > > Koch wrote: > > > > > When `ud.shallow == 1`: > > > > > - Prefer an initial shallow clone over an initial full bare > > > > > clone, > > > > > while still utilizing any already existing full bare > > > > > clones. > > > > > > > > > > This improves: > > > > > - Resolve timeout issues during initial clones on slow > > > > > internet > > > > > connections > > > > > by reducing the amount of data transferred. > > > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > > > to reduce data transfer. > > > > > - Allow SSH-based authentication (e.g. cert and agent-based) > > > > > when > > > > > using non-public repos, so additional HTTPS tokens may not > > > > > be > > > > > required. > > > > > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > > > --- > > > > > > > > Hi Stefan, > > > > > > > > Thanks for your patch. > > > > > > > > We got a new failure on the autobuilder, specifically with the > > > > oe-selftest-ubuntu build. I'm not entirely sure this comes from > > > > this > > > > series, so I will try a build without it to confirm. > > > > > > > > Here is the error: > > > > > > > > NOTE: Running task 2 of 4 (/srv/pokybuild/yocto-worker/oe- > > > > selftest- > > > > ubuntu/build/build-st-3502943/meta-selftest/recipes-test/git- > > > > submodule-test/git-submodule-test.bb:do_fetch) > > > > NOTE: recipe git-submodule-test-1.0-r0: task do_fetch: Started > > > > WARNING: git-submodule-test-1.0-r0 do_fetch: Creating fast > > > > initial > > > > shallow clone failed, try initial regular clone now. > > > > WARNING: git-submodule-test-1.0-r0 do_fetch: Failed to fetch > > > > URL > > > > gitsm://git.openembedded.org/bitbake;protocol=git;name=bitbake; > > > > subp > > > > at > > > > h=bitbake;nobranch=1;lfs=False;bareclone=1;nobranch=1, > > > > attempting > > > > MIRRORS if available > > > > ERROR: git-submodule-test-1.0-r0 do_fetch: Fetcher failure: > > > > Fetch > > > > command export PSEUDO_DISABLED=1; export > > > > PATH="/srv/pokybuild/yocto- > > > > worker/oe-selftest-ubuntu/build/build-st-3502943/tmp/sysroots- > > > > uninative/x86_64-linux/usr/bin:/srv/pokybuild/yocto-worker/oe- > > > > selftest-ubuntu/build/scripts:/srv/pokybuild/yocto-worker/oe- > > > > selftest-ubuntu/build/build-st-3502943/tmp/work/core2-64-poky- > > > > linux/git-submodule-test/1.0/recipe-sysroot- > > > > native/usr/bin/x86_64- > > > > poky-linux:/srv/pokybuild/yocto-worker/oe-selftest- > > > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > > > submodule-test/1.0/recipe- > > > > sysroot/usr/bin/crossscripts:/srv/pokybuild/yocto-worker/oe- > > > > selftest- > > > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > > > submodule-test/1.0/recipe-sysroot- > > > > native/usr/sbin:/srv/pokybuild/yocto-worker/oe-selftest- > > > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > > > submodule-test/1.0/recipe-sysroot- > > > > native/usr/bin:/srv/pokybuild/yocto-worker/oe-selftest- > > > > ubuntu/build/build-st-3502943/tmp/work/core2-64-poky-linux/git- > > > > submodule-test/1.0/recipe-sysroot- > > > > native/sbin:/srv/pokybuild/yocto- > > > > worker/oe-selftest-ubuntu/build/build-st- > > > > 3502943/tmp/work/core2-64- > > > > poky-linux/git-submodule-test/1.0/recipe-sysroot- > > > > native/bin:/srv/pokybuild/yocto-worker/oe-selftest- > > > > ubuntu/build/bitbake/bin:/srv/pokybuild/yocto-worker/oe- > > > > selftest- > > > > ubuntu/build/build-st-3502943/tmp/hosttools"; export > > > > HOME="/srv/pokybuild"; git -c gc.autoDetach=false -c > > > > core.pager=cat > > > > - > > > > c safe.bareRepository=all -c clone.defaultRemoteName=origin > > > > fetch > > > > origin 52a144a7daa94b2bd239d582cb71d1f03119918f --depth 1 > > > > failed > > > > with > > > > exit code 1, output: > > > > error: Server does not allow request for unadvertised object > > > > 52a144a7daa94b2bd239d582cb71d1f03119918f > > > > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1031/steps/14/logs/stdio > > > > > > > > Can you have a look at this issue please? > > > Do you using 1716ad67bbe9f4cecf1e3bcfc2007b150ba77741 from > > > bitbakes > > > master-next branch? Because that is PATCH v2 with that issue > > > specially > > > on ubuntu 20.04... > > Sorry, v3 is meant. > > v4 PATCH from Friday should fix this. > > > > > > > > > Sorry, I sent this in the wrong mail thread, but I confirm I was > indeed > using the v4. So far, I can't reproduce the issue using a ubuntu 20.04 container. git clone git://git.yoctoproject.org/poky Patching subdir bitbake/ oe-selftest -r devtool.DevtoolUpdateTests.test_devtool_git_submodules bitbake git-submodule-test The current master-next branch from bitbake contains the v4 patches, now. Do you still have the issue? > -- Stefan Koch Siemens AG http://www.siemens.com/
On Tue Feb 25, 2025 at 6:04 PM CET, Stefan Koch wrote: > So far, I can't reproduce the issue using a ubuntu 20.04 container. > git clone git://git.yoctoproject.org/poky > Patching subdir bitbake/ > oe-selftest -r devtool.DevtoolUpdateTests.test_devtool_git_submodules > bitbake git-submodule-test > > The current master-next branch from bitbake contains the v4 patches, > now. Do you still have the issue? > > > > -- > Stefan Koch > Siemens AG > http://www.siemens.com/ Hi Stefan, Yes, it does look like Richard had the same issue on his branch: https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1045/steps/14/logs/stdio I will try to provide you with an easy way to reproduce this failure.
On Wed, 2025-02-26 at 09:39 +0100, Mathieu Dubois-Briand wrote: > On Tue Feb 25, 2025 at 6:04 PM CET, Stefan Koch wrote: > > So far, I can't reproduce the issue using a ubuntu 20.04 container. > > git clone git://git.yoctoproject.org/poky > > Patching subdir bitbake/ > > oe-selftest -r > > devtool.DevtoolUpdateTests.test_devtool_git_submodules > > bitbake git-submodule-test > > > > The current master-next branch from bitbake contains the v4 > > patches, > > now. Do you still have the issue? > > > > > > > -- > > Stefan Koch > > Siemens AG > > http://www.siemens.com/ > > > Hi Stefan, > > Yes, it does look like Richard had the same issue on his branch: > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1045/steps/14/logs/stdio > > I will try to provide you with an easy way to reproduce this failure. The problem has now moved into an issue with the gitsm fetcher which wraps the git fetcher. oe-seltest -r archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow would be the command to run that specific test. I suspect the retry on failure logic isn't working in gitsm. I'm also still thinking we shouldn't expose that skip variable to users as we're creating problematic API we'll then just have to maintain. I agree we probably need the fallback logic just in case there are older servers out there. Cheers, Richard
On Wed Feb 26, 2025 at 9:39 AM CET, Mathieu Dubois-Briand wrote: > On Tue Feb 25, 2025 at 6:04 PM CET, Stefan Koch wrote: > > So far, I can't reproduce the issue using a ubuntu 20.04 container. > > git clone git://git.yoctoproject.org/poky > > Patching subdir bitbake/ > > oe-selftest -r devtool.DevtoolUpdateTests.test_devtool_git_submodules > > bitbake git-submodule-test > > > > The current master-next branch from bitbake contains the v4 patches, > > now. Do you still have the issue? > > > > > > > -- > > Stefan Koch > > Siemens AG > > http://www.siemens.com/ > > > Hi Stefan, > > Yes, it does look like Richard had the same issue on his branch: > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1045/steps/14/logs/stdio > > I will try to provide you with an easy way to reproduce this failure. I successfully reproduced this on uubuntu 20.04 with the following commands: git clone git://git.yoctoproject.org/poky-ci-archive -b autobuilder.yoctoproject.org/valkyrie/a-full-1069 . poky-ci-archive/oe-init-build-env echo "SANITY_TESTED_DISTROS = ''" >> conf/local.conf oe-selftest -r archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow WARNING: git-submodule-test-1.0-r0 do_fetch: Creating fast initial shallow clone failed, try initial regular clone now. WARNING: git-submodule-test-1.0-r0 do_fetch: Failed to fetch URL gitsm://git.openembedded.org/bitbake;protocol=git;name=bitbake;subpath=bitbake;nobranch=1;lfs=False;bareclone=1;nobranch=1, attempting MIRRORS if available ERROR: git-submodule-test-1.0-r0 do_fetch: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export PATH="/home/mdubois-briand/swat/fetch2-issue/build-st/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/mdubois-briand/swat/fetch2-issue/poky-ci-archive/scripts:/home/mdubois-briand/swat/fetch2-issue/build-st/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/mdubois-briand/swat/fetch2-issue/build-st/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot/usr/bin/crossscripts:/home/mdubois-briand/swat/fetch2-issue/build-st/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/usr/sbin:/home/mdubois-briand/swat/fetch2-issue/build-st/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/usr/bin:/home/mdubois-briand/swat/fetch2-issue/build-st/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/sbin:/home/mdubois-briand/swat/fetch2-issue/build-st/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/recipe-sysroot-native/bin:/home/mdubois-briand/swat/fetch2-issue/poky-ci-archive/bitbake/bin:/home/mdubois-briand/swat/fetch2-issue/build-st/tmp/hosttools"; export HOME="/home/build"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin fetch origin 52a144a7daa94b2bd239d582cb71d1f03119918f --depth 1 failed with exit code 1, output: error: Server does not allow request for unadvertised object 52a144a7daa94b2bd239d582cb71d1f03119918f ERROR: git-submodule-test-1.0-r0 do_fetch: gitsm: submodule download failed: FetchError Fetcher failure for URL: 'gitsm://git.openembedded.org/bitbake;protocol=git;name=bitbake;subpath=bitbake;nobranch=1;lfs=False;bareclone=1;nobranch=1'. Unable to fetch URL from any source. WARNING: git-submodule-test-1.0-r0 do_fetch: Failed to fetch URL gitsm://git.yoctoproject.org/bitbake-gitsm-test1;protocol=git;name=bitbake-gitsm-test1;subpath=bitbake-gitsm-test1;nobranch=1;lfs=True;bareclone=1;nobranch=1, attempting MIRRORS if available NOTE: recipe git-submodule-test-1.0-r0: task do_fetch: Succeeded NOTE: Running task 3 of 4 (/home/mdubois-briand/swat/fetch2-issue/build-st/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb:do_ar_mirror) NOTE: recipe git-submodule-test-1.0-r0: task do_ar_mirror: Started WARNING: git-submodule-test-1.0-r0 do_ar_mirror: Mirror tarballs are listed for a source but none are present. Falling back to original download. SRC_URI = gitsm://git.openembedded.org/bitbake;protocol=git;name=bitbake;subpath=bitbake;nobranch=1;lfs=False;bareclone=1;nobranch=1 WARNING: git-submodule-test-1.0-r0 do_ar_mirror: Mirror tarballs are listed for a source but none are present. Falling back to original download. SRC_URI = gitsm://git.yoctoproject.org/bitbake-gitsm-test2;protocol=git;name=bitbake-gitsm-test2;subpath=bitbake-gitsm-test2;nobranch=1;lfs=False;bareclone=1;nobranch=1 ERROR: git-submodule-test-1.0-r0 do_ar_mirror: Original download is missing for a source. SRC_URI = gitsm://git.yoctoproject.org/bitbake-gitsm-test2;protocol=git;name=bitbake-gitsm-test2;subpath=bitbake-gitsm-test2;nobranch=1;lfs=False;bareclone=1;nobranch=1 ERROR: Logfile of failure stored in: /home/mdubois-briand/swat/fetch2-issue/build-st/tmp/work/core2-64-poky-linux/git-submodule-test/1.0/temp/log.do_ar_mirror.982 NOTE: recipe git-submodule-test-1.0-r0: task do_ar_mirror: Failed ERROR: Task (/home/mdubois-briand/swat/fetch2-issue/build-st/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb:do_ar_mirror) failed with exit code '1' NOTE: Tasks Summary: Attempted 3 tasks of which 0 didn't need to be rerun and 1 failed.
On Wed, 2025-02-26 at 08:58 +0000, Richard Purdie wrote: > On Wed, 2025-02-26 at 09:39 +0100, Mathieu Dubois-Briand wrote: > > On Tue Feb 25, 2025 at 6:04 PM CET, Stefan Koch wrote: > > > So far, I can't reproduce the issue using a ubuntu 20.04 > > > container. > > > git clone git://git.yoctoproject.org/poky > > > Patching subdir bitbake/ > > > oe-selftest -r > > > devtool.DevtoolUpdateTests.test_devtool_git_submodules > > > bitbake git-submodule-test > > > > > > The current master-next branch from bitbake contains the v4 > > > patches, > > > now. Do you still have the issue? > > > > > > > > > > -- > > > Stefan Koch > > > Siemens AG > > > http://www.siemens.com/ > > > > > > Hi Stefan, > > > > Yes, it does look like Richard had the same issue on his branch: > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1045/steps/14/logs/stdio > > > > I will try to provide you with an easy way to reproduce this > > failure. > > The problem has now moved into an issue with the gitsm fetcher which > wraps the git fetcher. > > oe-seltest -r > archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow Thanks, can reproduce it with that cmd. > > would be the command to run that specific test. I suspect the retry > on > failure logic isn't working in gitsm. > > I'm also still thinking we shouldn't expose that skip variable to > users > as we're creating problematic API we'll then just have to maintain. I > agree we probably need the fallback logic just in case there are > older > servers out there. > > Cheers, > > Richard -- Stefan Koch Siemens AG http://www.siemens.com/
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 6badda597..d5c7f4332 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -366,6 +366,33 @@ class Git(FetchMethod): def tarball_need_update(self, ud): return ud.write_tarballs and not os.path.exists(ud.fullmirror) + def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None): + """Helper method for fetching Git LFS data""" + try: + if self._need_lfs(ud) and self._contains_lfs(ud, d, clonedir) and self._find_git_lfs(d) and len(revision): + # Using worktree with the revision because .lfsconfig may exists + worktree_add_cmd = "%s worktree add wt %s" % (ud.basecmd, revision) + runfetchcmd(worktree_add_cmd, d, log=progresshandler, workdir=clonedir) + lfs_fetch_cmd = "%s lfs fetch %s" % (ud.basecmd, "--all" if fetchall else "") + runfetchcmd(lfs_fetch_cmd, d, log=progresshandler, workdir=(clonedir + "/wt")) + worktree_rem_cmd = "%s worktree remove -f wt" % ud.basecmd + runfetchcmd(worktree_rem_cmd, d, log=progresshandler, workdir=clonedir) + except: + logger.warning("Fetching LFS did not succeed.") + + @contextmanager + def create_atomic(self, filename): + """Create as a temp file and move atomically into position to avoid races""" + fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) + try: + yield tfile + umask = os.umask(0o666) + os.umask(umask) + os.chmod(tfile, (0o666 & ~umask)) + os.rename(tfile, filename) + finally: + os.close(fd) + def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout with only premirrors # is not possible @@ -446,7 +473,40 @@ class Git(FetchMethod): if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url) progresshandler = GitProgressHandler(d) - runfetchcmd(clone_cmd, d, log=progresshandler) + + # When ud.shallow is enabled: + # Try creating an initial shallow clone + shallow_fast_state = False + if ud.shallow: + tempdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR')) + shallowclone = os.path.join(tempdir, 'git') + try: + self.clone_shallow_local(ud, shallowclone, d) + shallow_fast_state = True + except: + logger.warning("Creating initial shallow clone failed, try regular clone now.") + + # When the shallow clone has succeeded: + # Create shallow tarball + if shallow_fast_state: + logger.info("Creating tarball of git repository") + with self.create_atomic(ud.fullshallow) as tfile: + runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) + runfetchcmd("touch %s.done" % ud.fullshallow, d) + + # Always cleanup tempdir + bb.utils.remove(tempdir, recurse=True) + + # When the shallow clone has succeeded: + # Use shallow tarball + if shallow_fast_state: + ud.localpath = ud.fullshallow + return + + # When ud.shallow is disabled or the shallow clone failed: + # Create an initial regular clone + if not shallow_fast_state: + runfetchcmd(clone_cmd, d, log=progresshandler) # Update the checkout if needed if self.clonedir_need_update(ud, d): @@ -509,20 +569,6 @@ class Git(FetchMethod): runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clonedir, d, workdir="%s/.git" % ud.destdir) def build_mirror_data(self, ud, d): - - # Create as a temp file and move atomically into position to avoid races - @contextmanager - def create_atomic(filename): - fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) - try: - yield tfile - umask = os.umask(0o666) - os.umask(umask) - os.chmod(tfile, (0o666 & ~umask)) - os.rename(tfile, filename) - finally: - os.close(fd) - if ud.shallow and ud.write_shallow_tarballs: if not os.path.exists(ud.fullshallow): if os.path.islink(ud.fullshallow): @@ -533,7 +579,7 @@ class Git(FetchMethod): self.clone_shallow_local(ud, shallowclone, d) logger.info("Creating tarball of git repository") - with create_atomic(ud.fullshallow) as tfile: + with self.create_atomic(ud.fullshallow) as tfile: runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) runfetchcmd("touch %s.done" % ud.fullshallow, d) finally: @@ -543,7 +589,7 @@ class Git(FetchMethod): os.unlink(ud.fullmirror) logger.info("Creating tarball of git repository") - with create_atomic(ud.fullmirror) as tfile: + with self.create_atomic(ud.fullmirror) as tfile: mtime = runfetchcmd("{} log --all -1 --format=%cD".format(ud.basecmd), d, quiet=True, workdir=ud.clonedir) runfetchcmd("tar -czf %s --owner oe:0 --group oe:0 --mtime \"%s\" ." @@ -557,12 +603,20 @@ class Git(FetchMethod): - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev """ + progresshandler = GitProgressHandler(d) + repourl = self._get_repo_url(ud) bb.utils.mkdirhier(dest) init_cmd = "%s init -q" % ud.basecmd if ud.bareclone: init_cmd += " --bare" runfetchcmd(init_cmd, d, workdir=dest) - runfetchcmd("%s remote add origin %s" % (ud.basecmd, ud.clonedir), d, workdir=dest) + # Use repourl when creating a fast initial shallow clone + # Prefer already existing full bare clones if available + if ud.shallow and not os.path.exists(ud.clonedir): + remote = shlex.quote(repourl) + else: + remote = ud.clonedir + runfetchcmd("%s remote add origin %s" % (ud.basecmd, remote), d, workdir=dest) # Check the histories which should be excluded shallow_exclude = '' @@ -595,15 +649,10 @@ class Git(FetchMethod): if shallow_exclude: fetch_cmd += shallow_exclude - # Advertise the revision for lower version git such as 2.25.1: - # error: Server does not allow request for unadvertised object. - # The ud.clonedir is a local temporary dir, will be removed when - # fetch is done, so we can do anything on it. - adv_cmd = 'git branch -f advertise-%s %s' % (revision, revision) - runfetchcmd(adv_cmd, d, workdir=ud.clonedir) - runfetchcmd(fetch_cmd, d, workdir=dest) runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) + # Fetch Git LFS data + self.lfs_fetch(ud, d, dest, ud.revisions[ud.names[0]]) # Apply extra ref wildcards all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \ @@ -629,7 +678,6 @@ class Git(FetchMethod): runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) # The url is local ud.clonedir, set it to upstream one - repourl = self._get_repo_url(ud) runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=dest) def unpack(self, ud, destdir, d):
When `ud.shallow == 1`: - Prefer an initial shallow clone over an initial full bare clone, while still utilizing any already existing full bare clones. This improves: - Resolve timeout issues during initial clones on slow internet connections by reducing the amount of data transferred. - Eliminate the need to use an HTTPS tarball `SRC_URI` to reduce data transfer. - Allow SSH-based authentication (e.g. cert and agent-based) when using non-public repos, so additional HTTPS tokens may not be required. Signed-off-by: Stefan Koch <stefan-koch@siemens.com> --- lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++----------- 1 file changed, 74 insertions(+), 26 deletions(-)