mbox series

[0/3] fetch2/git: Improve shallow, lfs, and tag support

Message ID 20250129112406.1660522-1-stefan-koch@siemens.com
Headers show
Series fetch2/git: Improve shallow, lfs, and tag support | expand

Message

Stefan Koch Jan. 29, 2025, 11:24 a.m. UTC 
Hi

This patch series improves BitBake's git fetch mechanisms.

The first patch is a _working_ *proof-of-concept* and should serve as a discussion base on the mailing list.
Currently, it uses the `ud.shallow == 1` switch. Maybe it makes sense to add a separate switch.
This reduces the initial clone size from CIP linux kernel git repo from over 3GiB to under 0.5GiB

The second and third patches add support for tags and lfs when used with offline builds.
Offline building is enabled when `BB_NO_NETWORK = "1"` is set.

General example using raw GIT command with the Linux-CIP repo `git://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git`:
- `depth=1` (shallow) data transfer for Linux-CIP via git/ssh is about 148 MiB (default branch)
- bare mirror clone data transfer is about 3.91 GiB
=> So the bare clone needs 27 times more data transfer than one shallow clone.
=> The shallow clone size is similar to the size of usual kernel tarballs.

As long as `ud.shallow == 1`,
* with shallow (first) patch:
- BitBake always makes a new shallow clone for any update - as long as no bare clone already exists.
- Therefore, a shallow clone comes to a similar transfer size like the ususal size of tarballs.

- For non-public repos: It's possible to use SSH agent-based cert auth thanks to GIT/SSH protocol.
- So there is no need for an additional HTTPS token, as small cloning is possible directly via the GIT/SSH protocol.

* without shallow (first) patch:
- BitBake makes an initial Linux-CIP bare clone. When `ud.shallow == 1`, it works afterwards on that base for shallow functionality.
- This is an advantage for updates because the data transfer is probably many times smaller than for a new shallow clone.
- On the other hand, there may be very many shallow clones needed to get the same amount of the initial bare transfer data rate.

Patches:
1) fetch2/git: Add support for fast initial shallow fetch
When `ud.shallow == 1`:
Prefer an initial shallow clone over an initial bare clone,
while still utilizing any already existing bare clones.

2) fetch2/git: Add offline build support for tags
This adds support for offline builds when tags are used within `SRCREV`.

3) fetch2/git: Add offline build support for LFS
This adds support for offline builds for repos that use git-lfs.

Best regards

Stefan

Stefan Koch (3):
  fetch2/git: Add support for fast initial shallow fetch
  fetch2/git: Add offline build support for tags
  fetch2/git: Add offline build support for LFS

 lib/bb/fetch2/git.py | 113 ++++++++++++++++++++++++++++++++-----------
 1 file changed, 84 insertions(+), 29 deletions(-)