git-make-shallow: use stdin mode

when there are many refs to delete, using xargs to exec git can take a very long
time. make this faster by only running git update-ref with stdin mode.

for a repo with over 34000 git tags this makes git-make-shallow finish
in 2 seconds instead of 3 minutes for me.

Signed-off-by: Nick Owens <nick.owens@eero.com>
---
 bin/git-make-shallow | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

On Sat, 2025-01-04 at 18:18 -0800, Nick Owens via lists.openembedded.org wrote:
> when there are many refs to delete, using xargs to exec git can take a very long
> time. make this faster by only running git update-ref with stdin mode.
> 
> for a repo with over 34000 git tags this makes git-make-shallow finish
> in 2 seconds instead of 3 minutes for me.
> 
> Signed-off-by: Nick Owens <nick.owens@eero.com>
> ---
>  bin/git-make-shallow | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/bin/git-make-shallow b/bin/git-make-shallow
> index 9de557c10..8ad704fae 100755
> --- a/bin/git-make-shallow
> +++ b/bin/git-make-shallow
> @@ -115,8 +115,8 @@ def filter_refs(refs):
>      all_refs = get_all_refs()
>      to_remove = set(all_refs) - set(refs)
>      if to_remove:
> -        check_output(['xargs', '-0', '-n', '1'] + git_cmd + ['update-ref', '-d', '--no-deref'],
> -                     input=''.join(l + '\0' for l in to_remove))
> +        check_output(git_cmd + ['update-ref', '--no-deref', '--stdin', '-z'],
> +                     input=''.join('delete ' + l + '\0\0' for l in to_remove) + 'commit\0')
>  
>  

This sounded really good and it merged however we're seeing intermittent failures:

https://valkyrie.yoctoproject.org/#/builders/54/builds/725/steps/11/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/23/builds/810/steps/11/logs/stdio

This appears like it might me python or git version specific, there is
a deprecation warning in the log which I also wonder about...

Cheers,

Richard

right. i think this is because the transaction model was only introduced in
git 2.27 whereas ubuntu focal (20.04) only has 2.25. and, i think i bungled
the use of the transaction commands by not issuing 'start'. i think
dropping commit should resolve it.

diff --git a/bitbake/bin/git-make-shallow b/bitbake/bin/git-make-shallow
index d0532c5ab8..bead94a766 100755
--- a/bitbake/bin/git-make-shallow
+++ b/bitbake/bin/git-make-shallow
@@ -113,8 +113,8 @@ def filter_refs(refs):
     all_refs = get_all_refs()
     to_remove = set(all_refs) - set(refs)
     if to_remove:
-        check_output(['xargs', '-0', '-n', '1', 'git', 'update-ref', '-d',
'--no-deref'],
-                     input=''.join(l + '\0' for l in to_remove))
+        check_output(['git', 'update-ref', '--no-deref', '--stdin', '-z'],
+                     input=''.join('delete ' + l + '\0\0' for l in
to_remove))


 def follow_history_intersections(revisions, refs):

On Thu, Jan 9, 2025 at 12:35 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Sat, 2025-01-04 at 18:18 -0800, Nick Owens via lists.openembedded.org
> wrote:
> > when there are many refs to delete, using xargs to exec git can take a
> very long
> > time. make this faster by only running git update-ref with stdin mode.
> >
> > for a repo with over 34000 git tags this makes git-make-shallow finish
> > in 2 seconds instead of 3 minutes for me.
> >
> > Signed-off-by: Nick Owens <nick.owens@eero.com>
> > ---
> >  bin/git-make-shallow | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/bin/git-make-shallow b/bin/git-make-shallow
> > index 9de557c10..8ad704fae 100755
> > --- a/bin/git-make-shallow
> > +++ b/bin/git-make-shallow
> > @@ -115,8 +115,8 @@ def filter_refs(refs):
> >      all_refs = get_all_refs()
> >      to_remove = set(all_refs) - set(refs)
> >      if to_remove:
> > -        check_output(['xargs', '-0', '-n', '1'] + git_cmd +
> ['update-ref', '-d', '--no-deref'],
> > -                     input=''.join(l + '\0' for l in to_remove))
> > +        check_output(git_cmd + ['update-ref', '--no-deref', '--stdin',
> '-z'],
> > +                     input=''.join('delete ' + l + '\0\0' for l in
> to_remove) + 'commit\0')
> >
> >
>
> This sounded really good and it merged however we're seeing intermittent
> failures:
>
>
> https://valkyrie.yoctoproject.org/#/builders/54/builds/725/steps/11/logs/stdio
>
> https://valkyrie.yoctoproject.org/#/builders/23/builds/810/steps/11/logs/stdio
>
> This appears like it might me python or git version specific, there is
> a deprecation warning in the log which I also wonder about...
>
> Cheers,
>
> Richard
>

On Thu, 2025-01-09 at 10:55 -0800, Nick Owens wrote:
> right. i think this is because the transaction model was only
> introduced in git 2.27 whereas ubuntu focal (20.04) only has 2.25.
> and, i think i bungled the use of the transaction commands by not
> issuing 'start'. i think dropping commit should resolve it.
> 
> diff --git a/bitbake/bin/git-make-shallow b/bitbake/bin/git-make-
> shallow
> index d0532c5ab8..bead94a766 100755
> --- a/bitbake/bin/git-make-shallow
> +++ b/bitbake/bin/git-make-shallow
> @@ -113,8 +113,8 @@ def filter_refs(refs):
>      all_refs = get_all_refs()
>      to_remove = set(all_refs) - set(refs)
>      if to_remove:
> -        check_output(['xargs', '-0', '-n', '1', 'git', 'update-ref',
> '-d', '--no-deref'],
> -                     input=''.join(l + '\0' for l in to_remove))
> +        check_output(['git', 'update-ref', '--no-deref', '--stdin',
> '-z'],
> +                     input=''.join('delete ' + l + '\0\0' for l in
> to_remove))
>  
>  
>  def follow_history_intersections(revisions, refs):

Thanks for the hint, I've turned this into a patch, sent and then
merged it since this was breaking builds and there are enough other
issues atm to worry about so I wanted to get it resolved! :)

I did test it on the failing 20.04 worker and it works there now.

Cheers,

Richard