| Message ID | 20250220172706.3850722-1-stefan-koch@siemens.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v3,1/4] fetch2/git: Add support for fast initial shallow fetch | expand |
On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > When `ud.shallow == 1`: > - Prefer an initial shallow clone over an initial full bare clone, > while still utilizing any already existing full bare clones. > > This improves: > - Resolve timeout issues during initial clones on slow internet connections > by reducing the amount of data transferred. > - Eliminate the need to use an HTTPS tarball `SRC_URI` > to reduce data transfer. > - Allow SSH-based authentication (e.g. cert and agent-based) when > using non-public repos, so additional HTTPS tokens may not be required. > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > --- > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++----------- > 1 file changed, 74 insertions(+), 26 deletions(-) In testing it looks like this series has some issue on Ubuntu 20.04 systems (both x86_64 and arm): https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 This is probably due to the version of git there... Cheers, Richard
On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > When `ud.shallow == 1`: > > - Prefer an initial shallow clone over an initial full bare clone, > > while still utilizing any already existing full bare clones. > > > > This improves: > > - Resolve timeout issues during initial clones on slow internet > > connections > > by reducing the amount of data transferred. > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > to reduce data transfer. > > - Allow SSH-based authentication (e.g. cert and agent-based) when > > using non-public repos, so additional HTTPS tokens may not be > > required. > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > --- > > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++------- > > ---- > > 1 file changed, 74 insertions(+), 26 deletions(-) > > In testing it looks like this series has some issue on Ubuntu 20.04 > systems (both x86_64 and arm): > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > This is probably due to the version of git there... This issue could occur in two cases: a) old Git client version this could be relevant: https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ b) Git server version, that has disabled announcements see: https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r So, I have reintroduced the switch from v2 patch with inverse logic. 1. enabled by default 2. old behaviour can be forced by variable 3. old behaviour is triggered automatically in mentioned failure case (see above) I'll send updated v4 patch series. > > Cheers, > > Richard Thanks, Stefan Koch Siemens AG http://www.siemens.com/
On Fri, 2025-02-21 at 16:40 +0000, Koch, Stefan wrote: > On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > > When `ud.shallow == 1`: > > > - Prefer an initial shallow clone over an initial full bare > > > clone, > > > while still utilizing any already existing full bare clones. > > > > > > This improves: > > > - Resolve timeout issues during initial clones on slow internet > > > connections > > > by reducing the amount of data transferred. > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > to reduce data transfer. > > > - Allow SSH-based authentication (e.g. cert and agent-based) when > > > using non-public repos, so additional HTTPS tokens may not be > > > required. > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > --- > > > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++----- > > > -- > > > ---- > > > 1 file changed, 74 insertions(+), 26 deletions(-) > > > > In testing it looks like this series has some issue on Ubuntu 20.04 > > systems (both x86_64 and arm): > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > > > This is probably due to the version of git there... > > This issue could occur in two cases: > a) old Git client version > this could be relevant: > https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ I suspect the git version is the cause of the issue. > b) Git server version, that has disabled announcements > see: > https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r We're testing against the same server and we see some passes and only failures on Ubuntu 20.04 so it is unlikely to be this. > So, I have reintroduced the switch from v2 patch with inverse logic. > 1. enabled by default > 2. old behaviour can be forced by variable So in CI on the autobuilder, we need to force the variable to allow the selftests to pass on 20.04 hosts? We also need to error if anyone enables that form of fetching on 20.04 hosts too? > 3. old behaviour is triggered automatically in mentioned failure case > (see above) > > I'll send updated v4 patch series. > I'm afraid this solution won't work for us. We either need to find a work around, fall back to a different way of fetching or I guess we have to drop support for Ubuntu 20.04 without using builtools tarball and bump our minimum git version requirements. Cheers, Richard
On Fri, 2025-02-21 at 17:11 +0000, Richard Purdie wrote: > On Fri, 2025-02-21 at 16:40 +0000, Koch, Stefan wrote: > > On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > > > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > > > When `ud.shallow == 1`: > > > > - Prefer an initial shallow clone over an initial full bare > > > > clone, > > > > while still utilizing any already existing full bare clones. > > > > > > > > This improves: > > > > - Resolve timeout issues during initial clones on slow internet > > > > connections > > > > by reducing the amount of data transferred. > > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > > to reduce data transfer. > > > > - Allow SSH-based authentication (e.g. cert and agent-based) > > > > when > > > > using non-public repos, so additional HTTPS tokens may not be > > > > required. > > > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > > --- > > > > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++--- > > > > -- > > > > -- > > > > ---- > > > > 1 file changed, 74 insertions(+), 26 deletions(-) > > > > > > In testing it looks like this series has some issue on Ubuntu > > > 20.04 > > > systems (both x86_64 and arm): > > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > > > > > This is probably due to the version of git there... > > > > This issue could occur in two cases: > > a) old Git client version > > this could be relevant: > > https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ > > I suspect the git version is the cause of the issue. > > > b) Git server version, that has disabled announcements > > see: > > https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r > > We're testing against the same server and we see some passes and only > failures on Ubuntu 20.04 so it is unlikely to be this. > > > So, I have reintroduced the switch from v2 patch with inverse > > logic. > > 1. enabled by default > > 2. old behaviour can be forced by variable > > So in CI on the autobuilder, we need to force the variable to allow > the > selftests to pass on 20.04 hosts? We also need to error if anyone > enables that form of fetching on 20.04 hosts too? My current v4 implementation does it automatically in failure case. Because the disabled server side issue is possible at end users but not BB CI/CD (there only 20.04) > > > 3. old behaviour is triggered automatically in mentioned failure > > case > > (see above) > > > > I'll send updated v4 patch series. This patch should work with 20.04, too. > > > > I'm afraid this solution won't work for us. We either need to find a > work around, fall back to a different way of fetching or I guess we > have to drop support for Ubuntu 20.04 without using builtools tarball > and bump our minimum git version requirements. Drop 20.04 might be possible to. The EOL of 20.04 comes in a few months... I would prefer the automatic error handling, to support old git server versions at end users, too. > > Cheers, > > Richard > > -- Stefan Koch Siemens AG http://www.siemens.com/
On Fri, 2025-02-21 at 17:18 +0000, Koch, Stefan wrote: > On Fri, 2025-02-21 at 17:11 +0000, Richard Purdie wrote: > > On Fri, 2025-02-21 at 16:40 +0000, Koch, Stefan wrote: > > > On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > > > > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > > > > When `ud.shallow == 1`: > > > > > - Prefer an initial shallow clone over an initial full bare > > > > > clone, > > > > > while still utilizing any already existing full bare clones. > > > > > > > > > > This improves: > > > > > - Resolve timeout issues during initial clones on slow internet > > > > > connections > > > > > by reducing the amount of data transferred. > > > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > > > to reduce data transfer. > > > > > - Allow SSH-based authentication (e.g. cert and agent-based) > > > > > when > > > > > using non-public repos, so additional HTTPS tokens may not be > > > > > required. > > > > > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > > > --- > > > > > lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++--- > > > > > -- > > > > > -- > > > > > ---- > > > > > 1 file changed, 74 insertions(+), 26 deletions(-) > > > > > > > > In testing it looks like this series has some issue on Ubuntu > > > > 20.04 > > > > systems (both x86_64 and arm): > > > > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > > > > > > > This is probably due to the version of git there... > > > > > > This issue could occur in two cases: > > > a) old Git client version > > > this could be relevant: > > > https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ > > > > I suspect the git version is the cause of the issue. > > > > > b) Git server version, that has disabled announcements > > > see: > > > https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r > > > > We're testing against the same server and we see some passes and only > > failures on Ubuntu 20.04 so it is unlikely to be this. > > > > > So, I have reintroduced the switch from v2 patch with inverse > > > logic. > > > 1. enabled by default > > > 2. old behaviour can be forced by variable > > > > So in CI on the autobuilder, we need to force the variable to allow > > the > > selftests to pass on 20.04 hosts? We also need to error if anyone > > enables that form of fetching on 20.04 hosts too? > > My current v4 implementation does it automatically in failure case. > Because the disabled server side issue is possible at end users but not > BB CI/CD (there only 20.04) Ok, automatic fallback sounds ok as long as it doesn't complicate the code too much. I'm not sure we want to have a variable for it though, I don't like users needing to care about setting such things from a usability standpoint. > > > 3. old behaviour is triggered automatically in mentioned failure > > > case > > > (see above) > > > > > > I'll send updated v4 patch series. > This patch should work with 20.04, too. > > > > > > > I'm afraid this solution won't work for us. We either need to find a > > work around, fall back to a different way of fetching or I guess we > > have to drop support for Ubuntu 20.04 without using builtools tarball > > and bump our minimum git version requirements. > Drop 20.04 might be possible to. The EOL of 20.04 comes in a few > months... > > I would prefer the automatic error handling, to support old git server > versions at end users, too. There are other reasons we've wondered about using buildtools on 20.04 but I would prefer not to so this seems reasonable assuming the patch isn't too bad... Cheers, Richard
On Fri, 2025-02-21 at 17:22 +0000, Richard Purdie wrote: > On Fri, 2025-02-21 at 17:18 +0000, Koch, Stefan wrote: > > On Fri, 2025-02-21 at 17:11 +0000, Richard Purdie wrote: > > > On Fri, 2025-02-21 at 16:40 +0000, Koch, Stefan wrote: > > > > On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote: > > > > > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote: > > > > > > When `ud.shallow == 1`: > > > > > > - Prefer an initial shallow clone over an initial full bare > > > > > > clone, > > > > > > while still utilizing any already existing full bare > > > > > > clones. > > > > > > > > > > > > This improves: > > > > > > - Resolve timeout issues during initial clones on slow > > > > > > internet > > > > > > connections > > > > > > by reducing the amount of data transferred. > > > > > > - Eliminate the need to use an HTTPS tarball `SRC_URI` > > > > > > to reduce data transfer. > > > > > > - Allow SSH-based authentication (e.g. cert and agent- > > > > > > based) > > > > > > when > > > > > > using non-public repos, so additional HTTPS tokens may > > > > > > not be > > > > > > required. > > > > > > > > > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com> > > > > > > --- > > > > > > lib/bb/fetch2/git.py | 100 > > > > > > ++++++++++++++++++++++++++++++++--- > > > > > > -- > > > > > > -- > > > > > > ---- > > > > > > 1 file changed, 74 insertions(+), 26 deletions(-) > > > > > > > > > > In testing it looks like this series has some issue on Ubuntu > > > > > 20.04 > > > > > systems (both x86_64 and arm): > > > > > > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097 > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016 > > > > > > > > > > This is probably due to the version of git there... > > > > > > > > This issue could occur in two cases: > > > > a) old Git client version > > > > this could be relevant: > > > > https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/ > > > > > > I suspect the git version is the cause of the issue. > > > > > > > b) Git server version, that has disabled announcements > > > > see: > > > > https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r > > > > > > We're testing against the same server and we see some passes and > > > only > > > failures on Ubuntu 20.04 so it is unlikely to be this. > > > > > > > So, I have reintroduced the switch from v2 patch with inverse > > > > logic. > > > > 1. enabled by default > > > > 2. old behaviour can be forced by variable > > > > > > So in CI on the autobuilder, we need to force the variable to > > > allow > > > the > > > selftests to pass on 20.04 hosts? We also need to error if anyone > > > enables that form of fetching on 20.04 hosts too? > > > > My current v4 implementation does it automatically in failure case. > > Because the disabled server side issue is possible at end users but > > not > > BB CI/CD (there only 20.04) > > Ok, automatic fallback sounds ok as long as it doesn't complicate the > code too much. I'm not sure we want to have a variable for it though, > I > don't like users needing to care about setting such things from a > usability standpoint. Sent already v4, but it would possible to a) make BB_GIT_SHALLOW_SKIP_FAST an undocumented feature b) only using ud.shallow_skip_fast for that internally (but is it possible to specify shallow_skip_fast=1 via SRC_URI?) c) using not `ud` at all > > > > > 3. old behaviour is triggered automatically in mentioned > > > > failure > > > > case > > > > (see above) > > > > > > > > I'll send updated v4 patch series. > > This patch should work with 20.04, too. > > > > > > > > > > I'm afraid this solution won't work for us. We either need to > > > find a > > > work around, fall back to a different way of fetching or I guess > > > we > > > have to drop support for Ubuntu 20.04 without using builtools > > > tarball > > > and bump our minimum git version requirements. > > Drop 20.04 might be possible to. The EOL of 20.04 comes in a few > > months... > > > > I would prefer the automatic error handling, to support old git > > server > > versions at end users, too. > > There are other reasons we've wondered about using buildtools on > 20.04 > but I would prefer not to so this seems reasonable assuming the patch > isn't too bad... And there might be some (legacy) GIT servers on the internet that doesn't provide support for fetching a commit hash directly. > > Cheers, > > Richard -- Stefan Koch Siemens AG http://www.siemens.com/
On Fri, 2025-02-21 at 17:31 +0000, Koch, Stefan wrote: > On Fri, 2025-02-21 at 17:22 +0000, Richard Purdie wrote: > > On Fri, 2025-02-21 at 17:18 +0000, Koch, Stefan wrote: > > > On Fri, 2025-02-21 at 17:11 +0000, Richard Purdie wrote: > > > > > > > > > My current v4 implementation does it automatically in failure > > > case. > > > Because the disabled server side issue is possible at end users > > > but > > > not > > > BB CI/CD (there only 20.04) > > > > Ok, automatic fallback sounds ok as long as it doesn't complicate > > the > > code too much. I'm not sure we want to have a variable for it > > though, > > I > > don't like users needing to care about setting such things from a > > usability standpoint. > > Sent already v4, but it would possible to > a) make BB_GIT_SHALLOW_SKIP_FAST an undocumented feature This is surprisingly hard to do since someone generally finds it and assumes it is a bug in the documentation. There were a lot of these filed just in the last week. As soon as you list them somewhere, someone asks they be documented. > b) only using ud.shallow_skip_fast for that internally (but is it > possible to specify shallow_skip_fast=1 via SRC_URI?) > c) using not `ud` at all A parameter in the SRC_URI url could be a possibility but it also has the same issue as a), people then will ask it be documented. People also expect options to be tested and so on. I'm really torn on exposing it at all for that reason... I appreciate it sounds simple but I also see all these requests for docs, testing and so on :/ Cheers, Richard
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 6badda597..d5c7f4332 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -366,6 +366,33 @@ class Git(FetchMethod): def tarball_need_update(self, ud): return ud.write_tarballs and not os.path.exists(ud.fullmirror) + def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None): + """Helper method for fetching Git LFS data""" + try: + if self._need_lfs(ud) and self._contains_lfs(ud, d, clonedir) and self._find_git_lfs(d) and len(revision): + # Using worktree with the revision because .lfsconfig may exists + worktree_add_cmd = "%s worktree add wt %s" % (ud.basecmd, revision) + runfetchcmd(worktree_add_cmd, d, log=progresshandler, workdir=clonedir) + lfs_fetch_cmd = "%s lfs fetch %s" % (ud.basecmd, "--all" if fetchall else "") + runfetchcmd(lfs_fetch_cmd, d, log=progresshandler, workdir=(clonedir + "/wt")) + worktree_rem_cmd = "%s worktree remove -f wt" % ud.basecmd + runfetchcmd(worktree_rem_cmd, d, log=progresshandler, workdir=clonedir) + except: + logger.warning("Fetching LFS did not succeed.") + + @contextmanager + def create_atomic(self, filename): + """Create as a temp file and move atomically into position to avoid races""" + fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) + try: + yield tfile + umask = os.umask(0o666) + os.umask(umask) + os.chmod(tfile, (0o666 & ~umask)) + os.rename(tfile, filename) + finally: + os.close(fd) + def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout with only premirrors # is not possible @@ -446,7 +473,40 @@ class Git(FetchMethod): if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url) progresshandler = GitProgressHandler(d) - runfetchcmd(clone_cmd, d, log=progresshandler) + + # When ud.shallow is enabled: + # Try creating an initial shallow clone + shallow_fast_state = False + if ud.shallow: + tempdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR')) + shallowclone = os.path.join(tempdir, 'git') + try: + self.clone_shallow_local(ud, shallowclone, d) + shallow_fast_state = True + except: + logger.warning("Creating initial shallow clone failed, try regular clone now.") + + # When the shallow clone has succeeded: + # Create shallow tarball + if shallow_fast_state: + logger.info("Creating tarball of git repository") + with self.create_atomic(ud.fullshallow) as tfile: + runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) + runfetchcmd("touch %s.done" % ud.fullshallow, d) + + # Always cleanup tempdir + bb.utils.remove(tempdir, recurse=True) + + # When the shallow clone has succeeded: + # Use shallow tarball + if shallow_fast_state: + ud.localpath = ud.fullshallow + return + + # When ud.shallow is disabled or the shallow clone failed: + # Create an initial regular clone + if not shallow_fast_state: + runfetchcmd(clone_cmd, d, log=progresshandler) # Update the checkout if needed if self.clonedir_need_update(ud, d): @@ -509,20 +569,6 @@ class Git(FetchMethod): runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clonedir, d, workdir="%s/.git" % ud.destdir) def build_mirror_data(self, ud, d): - - # Create as a temp file and move atomically into position to avoid races - @contextmanager - def create_atomic(filename): - fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) - try: - yield tfile - umask = os.umask(0o666) - os.umask(umask) - os.chmod(tfile, (0o666 & ~umask)) - os.rename(tfile, filename) - finally: - os.close(fd) - if ud.shallow and ud.write_shallow_tarballs: if not os.path.exists(ud.fullshallow): if os.path.islink(ud.fullshallow): @@ -533,7 +579,7 @@ class Git(FetchMethod): self.clone_shallow_local(ud, shallowclone, d) logger.info("Creating tarball of git repository") - with create_atomic(ud.fullshallow) as tfile: + with self.create_atomic(ud.fullshallow) as tfile: runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) runfetchcmd("touch %s.done" % ud.fullshallow, d) finally: @@ -543,7 +589,7 @@ class Git(FetchMethod): os.unlink(ud.fullmirror) logger.info("Creating tarball of git repository") - with create_atomic(ud.fullmirror) as tfile: + with self.create_atomic(ud.fullmirror) as tfile: mtime = runfetchcmd("{} log --all -1 --format=%cD".format(ud.basecmd), d, quiet=True, workdir=ud.clonedir) runfetchcmd("tar -czf %s --owner oe:0 --group oe:0 --mtime \"%s\" ." @@ -557,12 +603,20 @@ class Git(FetchMethod): - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev """ + progresshandler = GitProgressHandler(d) + repourl = self._get_repo_url(ud) bb.utils.mkdirhier(dest) init_cmd = "%s init -q" % ud.basecmd if ud.bareclone: init_cmd += " --bare" runfetchcmd(init_cmd, d, workdir=dest) - runfetchcmd("%s remote add origin %s" % (ud.basecmd, ud.clonedir), d, workdir=dest) + # Use repourl when creating a fast initial shallow clone + # Prefer already existing full bare clones if available + if ud.shallow and not os.path.exists(ud.clonedir): + remote = shlex.quote(repourl) + else: + remote = ud.clonedir + runfetchcmd("%s remote add origin %s" % (ud.basecmd, remote), d, workdir=dest) # Check the histories which should be excluded shallow_exclude = '' @@ -595,15 +649,10 @@ class Git(FetchMethod): if shallow_exclude: fetch_cmd += shallow_exclude - # Advertise the revision for lower version git such as 2.25.1: - # error: Server does not allow request for unadvertised object. - # The ud.clonedir is a local temporary dir, will be removed when - # fetch is done, so we can do anything on it. - adv_cmd = 'git branch -f advertise-%s %s' % (revision, revision) - runfetchcmd(adv_cmd, d, workdir=ud.clonedir) - runfetchcmd(fetch_cmd, d, workdir=dest) runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) + # Fetch Git LFS data + self.lfs_fetch(ud, d, dest, ud.revisions[ud.names[0]]) # Apply extra ref wildcards all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \ @@ -629,7 +678,6 @@ class Git(FetchMethod): runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) # The url is local ud.clonedir, set it to upstream one - repourl = self._get_repo_url(ud) runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=dest) def unpack(self, ud, destdir, d):
When `ud.shallow == 1`: - Prefer an initial shallow clone over an initial full bare clone, while still utilizing any already existing full bare clones. This improves: - Resolve timeout issues during initial clones on slow internet connections by reducing the amount of data transferred. - Eliminate the need to use an HTTPS tarball `SRC_URI` to reduce data transfer. - Allow SSH-based authentication (e.g. cert and agent-based) when using non-public repos, so additional HTTPS tokens may not be required. Signed-off-by: Stefan Koch <stefan-koch@siemens.com> --- lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++----------- 1 file changed, 74 insertions(+), 26 deletions(-)