| Message ID | cover.1719033434.git.liezhi.yang@windriver.com |
|---|---|
| State | Not Applicable, archived |
| Headers | show |
Hello, Any comments on this patch, please? // Robert On 6/22/24 13:20, Robert Yang via lists.openembedded.org wrote: > From: Robert Yang <liezhi.yang@windriver.com> > > * Test info > - bitbake world --runall=fetch works well when BB_GENERATE_MIRROR_TARBALLS > and BB_GIT_SHALLOW are enabled or disabled > > - bitbake-selftest works well > > // Robert > > The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0: > > bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes (2024-06-18 08:45:22 +0100) > > are available in the Git repository at: > > https://github.com/robertlinux/yocto rbt/fetch_git > https://github.com/robertlinux/yocto/tree/rbt/fetch_git > > Robert Yang (1): > fetch2/git: Use basename for gitsrcname when ud.proto is file > > bitbake/lib/bb/fetch2/git.py | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#16371): https://lists.openembedded.org/g/bitbake-devel/message/16371 > Mute This Topic: https://lists.openembedded.org/mt/106812685/7304958 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [liezhi.yang@eng.windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- >
I submitted a bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15856 the bitbake on my env generated a git shallow tar for glibc, but with local mirror path in filename. and linked to it as a symbollink located in git2. when unpacking, the bitbake can't find the shallow with official shallow (which file name should contail sourceware.xxx), can can't find bare repo (which linked to the shallow file). I think the latest patch for git shallow result it: fetch2/git: Add support for fast initial shallow fetch 457288b2fda86fd00cdcaefac616129b0029e1f9 the patch author said that the bitbake don't want to generate shallow for local mirror, but why I got it? If so, maybe we should set shallow_skip_fast as True if protocol is "file": - ud.shallow_skip_fast = False + ud.shallow_skip_fast = (ud.proto == "file")
Hi Definition of ud.shallow_skip_fast = False is: Prefer an initial shallow clone over an initial full bare clone, while still utilizing any already existing full bare clones. Before BitBake 2.12: - Create a bare clone of the repo - Create a shallow tarball of that bare clone Beginning with BitBake 2.12 - Do not create a bare clone of the repo (but use it when it already exists) - Instead, use origin directly to create the tarball - Only in case of failure, switch to the old behaviour by setting ud.shallow_skip_fast = True and retry General example using raw GIT commands with the Linux-CIP repo `git://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git`: - `depth=1` (shallow) data transfer for Linux-CIP via git/ssh is about 148 MiB (default branch) - bare mirror clone data transfer is about 3.91 GiB => So the bare clone needs 27 times more data transfer than one shallow clone. => The shallow clone size is similar to the size of usual kernel tarballs. => Fast shallow archieves that small clone about 148 MiB. To fix that issue here: - EITHER adapt the mentioned patch from Robert "Use basename for gitsrcname when ud.proto is file" - OR set ud.shallow_skip_fast = True in case of PREMIRROR/file proto (are there other cases, too?) Maybe it was only a bit of luck that the PREMIRROR has worked, because BitBake has always created a bare clone, even when using shallow. Regards Stefan On Mon, 2025-05-12 at 02:26 -0700, Yi Qingliang via lists.openembedded.org wrote: > I submitted a bug: > https://bugzilla.yoctoproject.org/show_bug.cgi?id=15856 > > the bitbake on my env generated a git shallow tar for glibc, but with > local mirror path in filename. > and linked to it as a symbollink located in git2. > > when unpacking, the bitbake can't find the shallow with official > shallow (which file name should contail sourceware.xxx), can can't > find bare repo (which linked to the shallow file). > > I think the latest patch for git shallow result it: > fetch2/git: Add support for fast initial shallow fetch > 457288b2fda86fd00cdcaefac616129b0029e1f9 > > the patch author said that the bitbake don't want to generate shallow > for local mirror, but why I got it? If so, maybe we should set > shallow_skip_fast as True if protocol is "file": > > - ud.shallow_skip_fast = False > + ud.shallow_skip_fast = (ud.proto == "file") > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#17600): https://lists.openembedded.org/g/bitbake- > devel/message/17600 > Mute This Topic: https://lists.openembedded.org/mt/106812685/7767788 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake- > devel/unsub [stefan-koch@siemens.com] > -=-=-=-=-=-=-=-=-=-=-=- > -- Stefan Koch Siemens AG www.siemens.com
upload my glic recipe as attachement. my download directory is "/mnt/datum/downloads/". premirror of glibc is: /mnt/datum/repositories/sourceware.org/git/glibc.git the file system structure generated by bitbake in download is: lrwxrwxrwx 1 ll ll 120 May 7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git -> /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 6 May 7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.done -rw-r--r-- 1 ll ll 0 May 7 17:09 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.lock -rw-r--r-- 1 ll ll 24413461 May 7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 0 May 7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz.done when unpacking, the bitbake expect a bare repo at /mnt/datum/downloads/git2/sourceware.org.git.glibc.git, but indeed, it is a symbollink to gitshallow file. and then bitbake expect a git shallow file with name "gitshallow_sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz", also no luck. so bitbake report error finally.
Tthe patch Robert provided not work, shallow file's name changed to gitshallow_glibc.git_0557361-1_release.2.17.master.tar.gz , which still result unpacking error. no bare repo, and shallow file's name is not expected (no sourceware.org).
indeed, what's I want is to compile program, which is expected running on OS with lower version glibc. now, I'm using my older glibc to compile library, and use official latest one to compile my application, the library can linked by user directly, at the same time the application is provided to user with a full root file system (running with chroot). I checked the static link glibc solution, which is not recommended. and my library link to external 3rd binary library, so I can't use musl libc solution. for library: now I have ported the 2.17 glibc to the latest bitbake/oe-core, static linking gcc/c++ related library, dynamic linking glibc, to generate my library. for application: The Appimage looks like is too complicated, and need to check every so on all OS, to simplify it, I selected chroot+root filesystem (with mounting some directory, dev, var, proc, ...) as solution for my application. maybe there is a better solution?
From: Robert Yang <liezhi.yang@windriver.com> * Test info - bitbake world --runall=fetch works well when BB_GENERATE_MIRROR_TARBALLS and BB_GIT_SHALLOW are enabled or disabled - bitbake-selftest works well // Robert The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0: bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes (2024-06-18 08:45:22 +0100) are available in the Git repository at: https://github.com/robertlinux/yocto rbt/fetch_git https://github.com/robertlinux/yocto/tree/rbt/fetch_git Robert Yang (1): fetch2/git: Use basename for gitsrcname when ud.proto is file bitbake/lib/bb/fetch2/git.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)