| Message ID | 20260430123847.25046-2-marcio.henriques@ctw.bmwgroup.com |
|---|---|
| State | New |
| Headers | show |
| Series | Subject: [RFC PATCH] bitbake: fetch2/git: add switch to disable fast shallow path | expand |
Hi, On Thu Apr 30, 2026 at 2:38 PM CEST, Marcio Henriques via lists.openembedded.org wrote: [...] > diff --git a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > index 8d8e8b8b..1def16e7 100644 > --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > @@ -390,6 +390,25 @@ overview of their function and contents. > # This defaults to enabled if both BB_GIT_SHALLOW and > # BB_GENERATE_MIRROR_TARBALLS are enabled > BB_GENERATE_SHALLOW_TARBALLS ?= "1" > + There are some whitespaces here you can remove. > + :term:`BB_GIT_SHALLOW_SKIP_FAST` > + When :term:`BB_GIT_SHALLOW` is enabled, BitBake will by default attempt a > + fast initial shallow clone directly from the upstream repository, bypassing > + the creation of a local full clone in :term:`DL_DIR`. Setting this variable > + to ``"1"`` disables that fast path so the fetcher always creates and > + maintains a local clone in :term:`DL_DIR`, allowing subsequent builds to > + fetch only the delta of changes rather than re-downloading the full shallow > + history. > + > + This is useful in CI environments where shallow tarballs are treated as > + fallback artifacts and incremental network efficiency is preferred. > + > + Example usage:: > + > + BB_GIT_SHALLOW ?= "1" > + BB_GIT_SHALLOW_SKIP_FAST ?= "1" > + > + See also :term:`BB_GIT_SHALLOW` and :term:`BB_GIT_SHALLOW_DEPTH`. This paragraph should be indented with 3 spaces below :term:`BB_GIT_SHALLOW_SKIP_FAST`. Antonin
diff --git a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst index 8d8e8b8b..1def16e7 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -390,6 +390,25 @@ overview of their function and contents. # This defaults to enabled if both BB_GIT_SHALLOW and # BB_GENERATE_MIRROR_TARBALLS are enabled BB_GENERATE_SHALLOW_TARBALLS ?= "1" + + :term:`BB_GIT_SHALLOW_SKIP_FAST` + When :term:`BB_GIT_SHALLOW` is enabled, BitBake will by default attempt a + fast initial shallow clone directly from the upstream repository, bypassing + the creation of a local full clone in :term:`DL_DIR`. Setting this variable + to ``"1"`` disables that fast path so the fetcher always creates and + maintains a local clone in :term:`DL_DIR`, allowing subsequent builds to + fetch only the delta of changes rather than re-downloading the full shallow + history. + + This is useful in CI environments where shallow tarballs are treated as + fallback artifacts and incremental network efficiency is preferred. + + Example usage:: + + BB_GIT_SHALLOW ?= "1" + BB_GIT_SHALLOW_SKIP_FAST ?= "1" + + See also :term:`BB_GIT_SHALLOW` and :term:`BB_GIT_SHALLOW_DEPTH`. :term:`BB_GIT_SHALLOW_DEPTH` When used with :term:`BB_GENERATE_SHALLOW_TARBALLS`, this variable sets diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 21f5f28a..f72173b9 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -196,7 +196,9 @@ class Git(FetchMethod): if ud.bareclone: ud.cloneflags += " --mirror" - ud.shallow_skip_fast = False + # Allow disabling the fast shallow path so DL_DIR keeps a local clone + # that can be incrementally updated in CI. + ud.shallow_skip_fast = d.getVar("BB_GIT_SHALLOW_SKIP_FAST") == "1" ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1" ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or "").split() if 'tag' in ud.parm:
In some CI environments, shallow tarballs are preferred as fallback artifacts, not as the primary source path. Today, enabling shallow mode also enables an initial fast shallow path that may avoid creating/using the regular local clone flow in DL_DIR. This can reduce the effectiveness of incremental fetches and lead to larger re-downloads. Add BB_GIT_SHALLOW_SKIP_FAST as an opt-in switch. When set to 1, the fetcher skips the fast shallow path from initialization and proceeds with the regular clone/update flow, while still allowing shallow behavior and tarball fallback. Default behavior is preserved when the variable is unset. Signed-off-by: Marcio Henriques <marcio.henriques@ctw.bmwgroup.com> --- .../bitbake-user-manual-ref-variables.rst | 19 +++++++++++++++++++ lib/bb/fetch2/git.py | 4 +++- 2 files changed, 22 insertions(+), 1 deletion(-)