| Message ID | e763d75fb33b69bbfbf3350b2592b4eea7d3878d.1679641659.git.liezhi.yang@windriver.com |
|---|---|
| State | Accepted, archived |
| Commit | a0d8108eba8d542707740d00c66c1c5f5b963f18 |
| Headers | show |
| Series | [1/1] fetch/git: Fix local clone url to make it work with repo | expand |
On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@windriver.com> wrote: > > The "git clone /path/to/git/objects_symlink" couldn't work after the following > change: > > https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 > > But repo command manages the git repo as symlinks, so check whether the objects > is an symlink to fix the problem: > > * Nothing is changed if git/objects is not a symlink > * Use "git clone file://" when git/objects is a symlink > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Tested-by: Theodore A. Roth <troth@openavr.org>
Thanks for responding to this. Does this method allow symlinking for local clones? I have a large git repository checked out using the repo --mirror option, which is then cloned for each target workspace. I'd like to avoid doing large local clones for each MACHINE workspace (there's a good dozen) but instead just symlink git objects as has been working previously and according to the patch I submitted. Sam On Fri, 24 Mar 2023, 19:51 Theodore A. Roth, <troth@openavr.org> wrote: > On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@windriver.com> > wrote: > > > > The "git clone /path/to/git/objects_symlink" couldn't work after the > following > > change: > > > > > https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 > > > > But repo command manages the git repo as symlinks, so check whether the > objects > > is an symlink to fix the problem: > > > > * Nothing is changed if git/objects is not a symlink > > * Use "git clone file://" when git/objects is a symlink > > > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > > Tested-by: Theodore A. Roth <troth@openavr.org> >
Hi Sam, On 3/26/23 10:18 PM, Sam Liddicott wrote: > Thanks for responding to this. > > Does this method allow symlinking for local clones? Did you mean "git clone -s/--shared"? No, it doesn't support that as the code shows. Use "git clone --shared" would save time and disk spaces, but it would cause problems, for example, it doesn't work with BB_GENERATE_MIRROR_TARBALL since the .git/objects/info/alternates will be invalid on other hosts. I'm not sure whether we need fix other parts to make "--shared" work, let's see whether RP has any suggestions. > > I have a large git repository checked out using the repo --mirror option, which > is then cloned for each target workspace. Maybe consider using a shared DL_DIR among the builds, e.g.: DL_DIR = "/path/to/shared/downloads" // Robert > > I'd like to avoid doing large local clones for each MACHINE workspace (there's a > good dozen) but instead just symlink git objects as has been working previously > and according to the patch I submitted. > > Sam > > On Fri, 24 Mar 2023, 19:51 Theodore A. Roth, <troth@openavr.org > <mailto:troth@openavr.org>> wrote: > > On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>> wrote: > > > > The "git clone /path/to/git/objects_symlink" couldn't work after the > following > > change: > > > > > https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 > <https://urldefense.com/v3/__https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56__;!!AjveYdw8EvQ!bAhzaY_csImWvjvDB4Fa1yvAmmEJ32SC4duzLyDarpR7o36SD7vwI2lra1HmH7jt_x22MPCDWbmT2knQRA$> > > > > But repo command manages the git repo as symlinks, so check whether the > objects > > is an symlink to fix the problem: > > > > * Nothing is changed if git/objects is not a symlink > > * Use "git clone file://" when git/objects is a symlink > > > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>> > > Tested-by: Theodore A. Roth <troth@openavr.org <mailto:troth@openavr.org>> >
On Sun, 2023-03-26 at 15:18 +0100, Sam Liddicott wrote: > Thanks for responding to this. > > Does this method allow symlinking for local clones? > > I have a large git repository checked out using the repo --mirror > option, which is then cloned for each target workspace. > > I'd like to avoid doing large local clones for each MACHINE workspace > (there's a good dozen) but instead just symlink git objects as has > been working previously and according to the patch I submitted. I don't think we can make everything work for everyone. As Robert mentioned, using the alternates mechanism will break the mirror code. I think some users will also have issues if DL_DIR changes break the checkout from under them. We are going to have to do something to address the changes made in upstream git and Robert's patch looks like a reasonable step to me to fix that. I'm open to other proposals but we can't regress other workflows. I'm also reluctant to complicate the code more than we need to, there are already far too many different codepaths we have to worry about :/. Cheers, Richard
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index d0d68538e2..2a3c06fe4e 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -367,9 +367,13 @@ class Git(FetchMethod): # If the repo still doesn't exist, fallback to cloning it if not os.path.exists(ud.clonedir): - # We do this since git will use a "-l" option automatically for local urls where possible + # We do this since git will use a "-l" option automatically for local urls where possible, + # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): - repourl = repourl[7:] + repourl_path = repourl[7:] + objects = os.path.join(repourl_path, 'objects') + if os.path.isdir(objects) and not os.path.islink(objects): + repourl = repourl_path clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir) if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url)
The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- bitbake/lib/bb/fetch2/git.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)