| Message ID | 20250404185911.137153-1-peter.marko@siemens.com |
|---|---|
| State | Accepted, archived |
| Commit | 4e6fb31a88448cdacf4e9f84cf0a8a8035f84d63 |
| Headers | show |
| Series | [1/2] fetch/git: always fetch lfs when creating shallow tarball | expand |
On Fri Apr 4, 2025 at 8:59 PM CEST, Peter Marko via lists.openembedded.org wrote: > From: Peter Marko <peter.marko@siemens.com> > > Before fast shallow tarballs the shallow tarball with lfs worked only > if the original .../downloads/git2/... directory existed. Once it was > not there, do_unpack on shallow tarball without lfs files failed due to > disabled network. > It was fixed for fast shallow tarballs and this commit fixes the other > scenarios, too. > > This can happen when such shallow tarball is put to mirror or if > someone does cleanup to reclaim disk space. > > Signed-off-by: Peter Marko <peter.marko@siemens.com> > --- Hi, Thanks for your patch. I have a failure on the autobuilder, it might come from this series but I'm not fully confident so far. I will try to drop it and see if this error remains: 2025-04-05 18:34:18,526 - oe-selftest - INFO - Starting client DEBUGINFOD_URLS=http://192.168.7.3:36583/ debuginfod-find debuginfo /usr/bin/xz [Sat Apr 5 18:34:18 2025] (1095188/1095201): 192.168.7.4:60184 UA:elfutils/0.192,Linux/aarch64,/ XFF: GET /buildid/d844ffb4600923775c544fbf26e06b42986c9e9f/debuginfo 200 299136 0+7ms ERROR: When reparsing /tmp/selftest-fetchd_wtdpmg/test.bb:do_checkuri, the basehash value changed from 84b9e23a83a764e7a5d8a96ee523199e2b1c604f4280a45030e248de0ff52962 to 4123da244f465feffecb2b4fd2b132b64851981ec2fb70bf9445df8b0ea6a69e. The metadata is not deterministic and this needs to be fixed. ERROR: The following commands may help: ERROR: $ bitbake test -cdo_checkuri -Snone ERROR: Then: ERROR: $ bitbake test -cdo_checkuri -Sprintdiff ERROR: When reparsing /tmp/selftest-fetchd_wtdpmg/test.bb:do_fetch, the basehash value changed from 6d55d6743729b615749dc016857d7e5f9c884a8b92f0d57f68e743f4910333d3 to fa7d6ea0563b60362418d2a6c5e41a6684183d01ec578b65abd0c64a9cfba7cf. The metadata is not deterministic and this needs to be fixed. ERROR: The following commands may help: ERROR: $ bitbake test -cdo_fetch -Snone ERROR: Then: ERROR: $ bitbake test -cdo_fetch -Sprintdiff ERROR: When reparsing /tmp/selftest-fetchd_wtdpmg/test.bb:do_unpack, the basehash value changed from ae9d94aafc721d891f8754df82ed90da8191357178f3db64221a49d5c1de3bcd to 1ac8f87a71db151b8eb2ae8d13f9a6ea1a0c884ef98555d61d2c2fe07a476a8a. The metadata is not deterministic and this needs to be fixed. ERROR: The following commands may help: ERROR: $ bitbake test -cdo_unpack -Snone ERROR: Then: ERROR: $ bitbake test -cdo_unpack -Sprintdiff ERROR: When reparsing /tmp/selftest-fetchd_wtdpmg/test.bb:do_cleanall, the basehash value changed from dada587ade4d28783151e52fcc21d922f3bdad57064c057c23796f4743430427 to cf9daca04d1ff9b60f84b2bed99c89ddb03a867efaf079e92b5532f86c2a0ef5. The metadata is not deterministic and this needs to be fixed. ERROR: The following commands may help: ERROR: $ bitbake test -cdo_cleanall -Snone ERROR: Then: ERROR: $ bitbake test -cdo_cleanall -Sprintdiff ... Similar error on oe-selftests on all machines: https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1389 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/1304 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/1253
On Sun Apr 6, 2025 at 4:42 PM CEST, Mathieu Dubois-Briand wrote: > On Fri Apr 4, 2025 at 8:59 PM CEST, Peter Marko via lists.openembedded.org wrote: >> From: Peter Marko <peter.marko@siemens.com> >> >> Before fast shallow tarballs the shallow tarball with lfs worked only >> if the original .../downloads/git2/... directory existed. Once it was >> not there, do_unpack on shallow tarball without lfs files failed due to >> disabled network. >> It was fixed for fast shallow tarballs and this commit fixes the other >> scenarios, too. >> >> This can happen when such shallow tarball is put to mirror or if >> someone does cleanup to reclaim disk space. >> >> Signed-off-by: Peter Marko <peter.marko@siemens.com> >> --- > > Hi, > > Thanks for your patch. > > I have a failure on the autobuilder, it might come from this series but > I'm not fully confident so far. I will try to drop it and see if this > error remains: > Ok, I still have the same failure without this series. I will put it back in my -next branches.
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 5f4ec6128c..587a16994a 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -634,9 +634,8 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=dest) runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) - # Fetch Git LFS data for fast shallow clones - if not ud.shallow_skip_fast: - self.lfs_fetch(ud, d, dest, ud.revision) + # Fetch Git LFS data + self.lfs_fetch(ud, d, dest, ud.revision) # Apply extra ref wildcards all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \