| Message ID | 20260520051800.1951624-1-jamin_lin@aspeedtech.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v1] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace | expand |
There have been patches fixing similar issues, for which oe-core has no tests or ways to reproduce. Rather than ask reviewers to set up zephyr builds so they can observe the issue, can you add a testcase for this to devtool's oe-selftest? Alex On Wed, 20 May 2026 at 07:18, Jamin Lin via lists.openembedded.org <jamin_lin=aspeedtech.com@lists.openembedded.org> wrote: > > When a recipe uses multiple git SRC_URI entries with different destsuffix > values (e.g. Zephyr-based recipes with separate repositories for the > kernel, modules and application), do_unpack clones each source tree with > 'git clone -n -s'. > > The -s flag uses git's shared-object mechanism: > instead of copying objects locally it writes a .git/objects/info/alternates file > pointing back to the bare repository under the downloads directory (DL_DIR/git2/). > > git_convert_standalone_clone() is called by devtool_post_unpack to make > the workspace standalone: it runs 'git repack -a' to copy all objects > into the local object store and then removes the alternates file. > > However it only processes the top-level source directory. Each nested > git repo created by a separate SRC_URI entry retains its own alternates > file still pointing into downloads/. > > Steps to reproduce: > 1. devtool modify <recipe-with-multiple-git-SRC_URI> > 2. bitbake -c cleanall <recipe> > 3. bitbake <recipe> > > At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes > the bare repositories from downloads/git2/. The top-level workspace > repo is standalone (alternates already removed by the original code), > but the nested repos still hold alternates pointing to the now-deleted > paths. > > At step 3, srctree_hash_files() runs 'git add -A .' with a custom > GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on > each nested repo to check for changes; this fails with exit 128 because > the nested alternates are broken: > error: unable to normalize alternate object path: > .../downloads/git2/github.com.zephyrproject-rtos.acpica//objects > fatal: bad object HEAD > fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica > > This halts the BitBake parse phase with a CalledProcessError and leaves > the workspace in an unrecoverable state without manual intervention. > > Fix by extending git_convert_standalone_clone() to walk the source tree > and apply the same 'git repack -a' + remove-alternates treatment to > every nested git repository found, making the entire workspace fully > standalone at devtool modify time. The walk uses dirs[:] = [] to stop > at each git boundary so it never descends into already-converted repos. > > Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com> > --- > scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++---- > 1 file changed, 24 insertions(+), 4 deletions(-) > > diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py > index 32e749dbb1..0a83470373 100644 > --- a/scripts/lib/scriptutils.py > +++ b/scripts/lib/scriptutils.py > @@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath): > > > def git_convert_standalone_clone(repodir): > - """If specified directory is a git repository, ensure it's a standalone clone""" > + """ > + If specified directory is a git repository, ensure it's a standalone clone. > + Also converts any nested git repositories (created by multiple SRC_URI git > + entries with different destsuffix values) so that none of their contents > + depend on the shared downloads directory via alternates. > + """ > + > import bb.process > - if os.path.exists(os.path.join(repodir, '.git')): > - alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates') > + > + def _convert(gitdir, workdir): > + alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates') > if os.path.exists(alternatesfile): > # This will have been cloned with -s, so we need to convert it so none > # of the contents is shared > - bb.process.run('git repack -a', cwd=repodir) > + bb.process.run('git repack -a', cwd=workdir) > os.remove(alternatesfile) > > + if os.path.exists(os.path.join(repodir, '.git')): > + _convert(os.path.join(repodir, '.git'), repodir) > + > + # Also handle nested git repos created by multiple SRC_URI git entries > + # with different destsuffix values. Each nested repo is cloned with -s > + # and has its own alternates pointing to the downloads directory. > + for root, dirs, files in os.walk(repodir): > + if root == repodir: > + continue > + if '.git' in dirs: > + _convert(os.path.join(root, '.git'), root) > + dirs[:] = [] # don't recurse into nested repos > + > def _get_temp_recipe_dir(d): > # This is a little bit hacky but we need to find a place where we can put > # the recipe so that bitbake can find it. We're going to delete it at the > -- > 2.43.0 > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#237382): https://lists.openembedded.org/g/openembedded-core/message/237382 > Mute This Topic: https://lists.openembedded.org/mt/119403462/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py index 32e749dbb1..0a83470373 100644 --- a/scripts/lib/scriptutils.py +++ b/scripts/lib/scriptutils.py @@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath): def git_convert_standalone_clone(repodir): - """If specified directory is a git repository, ensure it's a standalone clone""" + """ + If specified directory is a git repository, ensure it's a standalone clone. + Also converts any nested git repositories (created by multiple SRC_URI git + entries with different destsuffix values) so that none of their contents + depend on the shared downloads directory via alternates. + """ + import bb.process - if os.path.exists(os.path.join(repodir, '.git')): - alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates') + + def _convert(gitdir, workdir): + alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates') if os.path.exists(alternatesfile): # This will have been cloned with -s, so we need to convert it so none # of the contents is shared - bb.process.run('git repack -a', cwd=repodir) + bb.process.run('git repack -a', cwd=workdir) os.remove(alternatesfile) + if os.path.exists(os.path.join(repodir, '.git')): + _convert(os.path.join(repodir, '.git'), repodir) + + # Also handle nested git repos created by multiple SRC_URI git entries + # with different destsuffix values. Each nested repo is cloned with -s + # and has its own alternates pointing to the downloads directory. + for root, dirs, files in os.walk(repodir): + if root == repodir: + continue + if '.git' in dirs: + _convert(os.path.join(root, '.git'), root) + dirs[:] = [] # don't recurse into nested repos + def _get_temp_recipe_dir(d): # This is a little bit hacky but we need to find a place where we can put # the recipe so that bitbake can find it. We're going to delete it at the
When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with 'git clone -n -s'. The -s flag uses git's shared-object mechanism: instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/). git_convert_standalone_clone() is called by devtool_post_unpack to make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file. However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/. Steps to reproduce: 1. devtool modify <recipe-with-multiple-git-SRC_URI> 2. bitbake -c cleanall <recipe> 3. bitbake <recipe> At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths. At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken: error: unable to normalize alternate object path: .../downloads/git2/github.com.zephyrproject-rtos.acpica//objects fatal: bad object HEAD fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention. Fix by extending git_convert_standalone_clone() to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted repos. Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com> --- scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)