diff mbox series

[1/1] fetch/git: Fix local clone url to make it work with repo

Message ID e763d75fb33b69bbfbf3350b2592b4eea7d3878d.1679641659.git.liezhi.yang@windriver.com
State Accepted, archived
Commit a0d8108eba8d542707740d00c66c1c5f5b963f18
Headers show
Series [1/1] fetch/git: Fix local clone url to make it work with repo | expand

Commit Message

Robert Yang March 24, 2023, 7:09 a.m. UTC
The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:

https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56

But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:

* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/fetch2/git.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

Comments

Theodore A. Roth March 24, 2023, 7:50 p.m. UTC | #1
On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@windriver.com> wrote:
>
> The "git clone /path/to/git/objects_symlink" couldn't work after the following
> change:
>
> https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56
>
> But repo command manages the git repo as symlinks, so check whether the objects
> is an symlink to fix the problem:
>
> * Nothing is changed if git/objects is not a symlink
> * Use "git clone file://" when git/objects is a symlink
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>

Tested-by: Theodore A. Roth <troth@openavr.org>
Sam Liddicott March 26, 2023, 2:18 p.m. UTC | #2
Thanks for responding to this.

Does this method allow symlinking for local clones?

I have a large git repository checked out using the repo --mirror option,
which is then cloned for each target workspace.

I'd like to avoid doing large local clones for each MACHINE workspace
(there's a good dozen) but instead just symlink git objects as has been
working previously and according to the patch I submitted.

Sam

On Fri, 24 Mar 2023, 19:51 Theodore A. Roth, <troth@openavr.org> wrote:

> On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@windriver.com>
> wrote:
> >
> > The "git clone /path/to/git/objects_symlink" couldn't work after the
> following
> > change:
> >
> >
> https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56
> >
> > But repo command manages the git repo as symlinks, so check whether the
> objects
> > is an symlink to fix the problem:
> >
> > * Nothing is changed if git/objects is not a symlink
> > * Use "git clone file://" when git/objects is a symlink
> >
> > Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>
> Tested-by: Theodore A. Roth <troth@openavr.org>
>
Robert Yang March 27, 2023, 5:06 a.m. UTC | #3
Hi Sam,

On 3/26/23 10:18 PM, Sam Liddicott wrote:
> Thanks for responding to this.
> 
> Does this method allow symlinking for local clones?

Did you mean "git clone -s/--shared"? No, it doesn't support that as the code 
shows. Use "git clone --shared" would save time and disk spaces, but
it would cause problems, for example, it doesn't work with 
BB_GENERATE_MIRROR_TARBALL since the .git/objects/info/alternates will be
invalid on other hosts.

I'm not sure whether we need fix other parts to make "--shared" work, let's see
whether RP has any suggestions.

> 
> I have a large git repository checked out using the repo --mirror option, which 
> is then cloned for each target workspace.

Maybe consider using a shared DL_DIR among the builds, e.g.:

DL_DIR = "/path/to/shared/downloads"

// Robert

> 
> I'd like to avoid doing large local clones for each MACHINE workspace (there's a 
> good dozen) but instead just symlink git objects as has been working previously 
> and according to the patch I submitted.
> 
> Sam
> 
> On Fri, 24 Mar 2023, 19:51 Theodore A. Roth, <troth@openavr.org 
> <mailto:troth@openavr.org>> wrote:
> 
>     On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@windriver.com
>     <mailto:liezhi.yang@windriver.com>> wrote:
>      >
>      > The "git clone /path/to/git/objects_symlink" couldn't work after the
>     following
>      > change:
>      >
>      >
>     https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56
>     <https://urldefense.com/v3/__https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56__;!!AjveYdw8EvQ!bAhzaY_csImWvjvDB4Fa1yvAmmEJ32SC4duzLyDarpR7o36SD7vwI2lra1HmH7jt_x22MPCDWbmT2knQRA$>
>      >
>      > But repo command manages the git repo as symlinks, so check whether the
>     objects
>      > is an symlink to fix the problem:
>      >
>      > * Nothing is changed if git/objects is not a symlink
>      > * Use "git clone file://" when git/objects is a symlink
>      >
>      > Signed-off-by: Robert Yang <liezhi.yang@windriver.com
>     <mailto:liezhi.yang@windriver.com>>
> 
>     Tested-by: Theodore A. Roth <troth@openavr.org <mailto:troth@openavr.org>>
>
Richard Purdie March 27, 2023, 9:30 a.m. UTC | #4
On Sun, 2023-03-26 at 15:18 +0100, Sam Liddicott wrote:
> Thanks for responding to this.
> 
> Does this method allow symlinking for local clones?
> 
> I have a large git repository checked out using the repo --mirror
> option, which is then cloned for each target workspace.
> 
> I'd like to avoid doing large local clones for each MACHINE workspace
> (there's a good dozen) but instead just symlink git objects as has
> been working previously and according to the patch I submitted.

I don't think we can make everything work for everyone.

As Robert mentioned, using the alternates mechanism will break the
mirror code. I think some users will also have issues if DL_DIR changes
break the checkout from under them.

We are going to have to do something to address the changes made in
upstream git and Robert's patch looks like a reasonable step to me to
fix that.

I'm open to other proposals but we can't regress other workflows. I'm
also reluctant to complicate the code more than we need to, there are
already far too many different codepaths we have to worry about :/.

Cheers,

Richard
diff mbox series

Patch

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index d0d68538e2..2a3c06fe4e 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -367,9 +367,13 @@  class Git(FetchMethod):
 
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
-            # We do this since git will use a "-l" option automatically for local urls where possible
+            # We do this since git will use a "-l" option automatically for local urls where possible,
+            # but it doesn't work when git/objects is a symlink, only works when it is a directory.
             if repourl.startswith("file://"):
-                repourl = repourl[7:]
+                repourl_path = repourl[7:]
+                objects = os.path.join(repourl_path, 'objects')
+                if os.path.isdir(objects) and not os.path.islink(objects):
+                    repourl = repourl_path
             clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd, ud.url)