From patchwork Mon Mar 3 13:49:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Koch X-Patchwork-Id: 58199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75120C282CD for ; Mon, 3 Mar 2025 13:49:25 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.85]) by mx.groups.io with SMTP id smtpd.web11.53205.1741009758511073424 for ; Mon, 03 Mar 2025 05:49:19 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@siemens.com header.s=selector2 header.b=FKWvl4qC; spf=pass (domain: siemens.com, ip: 40.107.22.85, mailfrom: stefan-koch@siemens.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EhPfPy4MyyMi+T+YoXj//CMvZBzGfXFJ2QkzNtYNvecCPYvTRGRzcnQRv+y+2WLZntYdl2VviJyOkX4qRlmkR/AYRSU5WqOVd8WJOaJIKUQ2YtyHl+mt89o6argZMV305PjOoUqmrLYtzh17Zokm/XkRIa2jiASwso931Zp5C2BAKs/FZIeHV7cy1uU7hSXYl8gjwrzM4U6mY7TaWli6v/QC/T9/pB8ndOHIN5IG6OvgefOswMQJ8xAgEPOt7wneUzuxOrtLVLG8mH3MnLSgUnRhLxNKJqNgcrT/hyAbx+lMliqwA9xjom2YaXS9oX0vVuF7V70/5VQfW2HmoBlaqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mI82u8jkX7pTDB+xqh/wuiDeKiqh2em8AouR4ZZOgDA=; b=BmROOSXXJouBhPiVpYLF00ut8Pg3SHQjUC5gH1oledOwrg/UMvIgXfCU7sTgW3FvDm6TDfGb6vphsIOXbf1/nfurMSAwoDqXewamnEPNZDUX5HEAUYRx8ciAobC7ARk5MSzoJ5IalU/4zB2gMWBSmAijA9ko3+uQrCvd2PhVV0uO/AjsClt3MokiX196QinD7/tmpOgIFQJbOQY2K1+/3BVyjyAfsieE+qdBswe/h7Qg8ImBgNGQ4GI9SERqbrgUID0fyfjRFUbW98jAXCjyJbmdoZMD51LyXHJlgxpUH4SrjywdC4BPjQcRTH4i72CE83bnMK+2tibOpOtJbCHpSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mI82u8jkX7pTDB+xqh/wuiDeKiqh2em8AouR4ZZOgDA=; b=FKWvl4qCuIxCXt4VdAbjd2CkyXxjJ7PqJzAWyqnts1f43vzFeaJZwxkD35kjIDPBcwW1Lq2Lc6zboL3NKF/juNqoInE7XLxKnceQMequil5vLsfWd33ymyI5XHOG79lA+CM0ZSbFkF1b4Vin2s0+muYIhzx7JnBZe1i/Yul0MTBQsBIMGXxMadNT7df44Ijkl+/PIknEMXg/PrzFzJWyPGhtUSQPwu5UvpgPl9ieAa/zXVnVUOUmaPs1Vp8D2gj2rXatdQYdamPjiGhLhKf3SQmSsME/oKEMuAXGkbn3ZGN5nhcabVywwzlmF+8u7TabkqLLR10UVpFYBXiN4EyYFg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) by VI1PR10MB3165.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:803:135::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.28; Mon, 3 Mar 2025 13:49:15 +0000 Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec]) by AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec%6]) with mapi id 15.20.8489.028; Mon, 3 Mar 2025 13:49:15 +0000 From: Stefan Koch To: bitbake-devel@lists.openembedded.org CC: docs@lists.yoctoproject.org, stefan-koch@siemens.com, simon.sudler@siemens.com, jan.kiszka@siemens.com, alex.kanavin@gmail.com, richard.purdie@linuxfoundation.org, quentin.schulz@cherry.de, mathieu.dubois-briand@bootlin.com, Ross.Burton@arm.com Subject: [PATCH v5 1/5] fetch2/git: Add support for fast initial shallow fetch Date: Mon, 3 Mar 2025 14:49:09 +0100 Message-ID: <20250303134913.356778-1-stefan-koch@siemens.com> X-Mailer: git-send-email 2.39.5 X-ClientProxiedBy: FR0P281CA0040.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:48::11) To AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR10MB4959:EE_|VI1PR10MB3165:EE_ X-MS-Office365-Filtering-Correlation-Id: 2a86ffe1-8885-4780-885c-08dd5a5a2fde X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: x1OfpewKTUiSNYFH8aHPU17jZfr5Rk5LnmwOV3iX9znEaUCK7DtDdzfax4rYuOXk/TcAUgyI8ukxmbQg0OtOO8zC4+NWFwSsV/flr35ZrJVO/t350sD7calJclFAcSsM3MGA4b8BLzvtCvh0r8OYYc7+BY1+hPrfd59jb9c7yhFsBqAb0oWvVSSBQ0RR8EHUK/yb3JiSqNFTm7mBb5MFW5FsP3B0AORS5cT7lsY4znOCox/C/JWAnSkyXf08C7EA1BGR+IUHEdWahHovvVC75VLkV+TLCqJM0F9/K4s3TA1hYh2JOxanOMvEWoNU/9NO8bqpwbaqWeiK2uhE1bCwfDujmDKbS6IyBocttssQa2OnbhhTWkIUR/1HewZ5dac6zsi3IGUALy1rpo8e2R9Okut8OCz1dELre7iH9gIHDn+8xVWuJ48v/rpnyZ+XD6fj433VFQsngqeaOIWzT8Zehb36DhBZCF4rDprdAWG0eTZEuhFSnj7r3Hx5LcQJw/ME1qT67CaQ/JmVKvn5izux+M6m2rmtimeBBJunflGo7tAD+UqgDJnrqYv/trBwxDn5Ss09ibL4P6fRy39Fg5G+TwzYFGmdK9h703LOCeC2ZxpSkpSxXA62H9/2V3alnbLqAaZ4pljYU9b9E4R22HJ4PhsGHBRHtQy2XAxZGfiVdMIa1v/s7O6mt8mVjyQhSN3g8RMFtdU+Nm64FJvjTOPUy16E1we23ARRKqgpVQH8Mwr5U5cSmrqoX0tebOHbft+aWjCtZvI+tVSAXMOcENjjpogRfWY835xjAYs7WcJEM5qaGkGaeNzWXrYoTNDYj8k+SCNKWeRTDYnAT2YfMX5MOxmVXG+yxzv2/PppXG1VVfRFY4SdbF8h4aOtrSvL8H+1Z6dDbe8sg58/lHiimnDmsB+T+2s5bZZWKfAkbW4hDcLmpcBeZ+VN/UPRwobcMBK0AKeLu67649MCRM4FilsOmdMRwrufoD9i5ktknI6N2BDSoS5b3rRu/O7z1Jo+wsuwSBHDrufITZNyMVNRe6ovmtvGY3oZU/ZT9WJ07iz/DI6brJhDSHtkKeC0iP90RWSraZjPSct53c7ZEFQAViZreb8VX5bFijgr0er5+Ou736lXGSZPIrMj9YxGZoi902TUrj1ertuTpDEHzg1ow39PWujw2w0bbMrqqRCa4TFF/qcKe+CyM4dMZgFvTT01ZWWqv/RAiHv4TNt5EQRG84Oa8Mkp7tUeL7wkP+EJLJwZ3LFdgj/bJH9mCirpUO1nNXPEh3Mf13PxxZWJl4dGrN5lFOacf76rzDjz/9yeNtpOez1k9UtLy/14GvK2W793abFds8i3Nh/41rfdlRUctmQg6NkJLPmnrZ/BvINM0V+lGvDW42PSCkbu/riejXd/U0K4 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qD5QWtGYb8TnAs8Zcio9KfKz+u3EU6bE8ekNoNWVT7CGKfw3pvg8jXdOnCBSeBHkHhGlKK5j9LJ1h95sD37f868KATlTTlUQuTfCy/qMA/o9ZpXg8jjI2n7AxLiFiRTqVg3aqmNbb/79o7diGIkyjIFCtHZt/PURAISGRKxa0H0MwGW8w0x7ZD+9/cCPuiyceUq+kXyLAqroDicZsCfXyHr6/VJa4728G6Mrl+b6mzzpKcGSI1jyaEJSui0w1R5e51fk7wUQFEIwfXcz09llqSpgQ2bm8XwWfdym74VrSj1+2dS0xuQUtrldKg+o9bLUrqHL7vkhmjbOidcIM7oCJ1kWBBJD/om8Mt4Yp+0YOCAKa6c062OVp2MF0cWtbZpcFUaB/iWD/vwsMwM8cGE5imKyrUVxRZ7SpHe7pYv05Hu1dzdXgEG1r+aPgr1WoyI8k5y6O8TJBpuSA6h3uOZYilqXcBzsvgApIVC7GSs/mxsLhTye8FWSAy+ehWwCuUK5N/Yd+/DFjhpbmbmnf85H4W1O72sresFwDHRlJD6rQwETQ2170A/BPhnZoDF9vsQdrrhBMHZeFbLXQsR98tMNIAydEsED8ZW1ITmOAUcWLtZvyArbHNbdBslqPGqtLG1nT6oELvb1DoVCaJYKNVRHF8dtGNTGNBKOvv7qHABpCWHJY7sF8owNwMx5tqGX4iy3tgTOTA2FsYPZGKTNdzuvUlO58xb8Mcj0WvgJtQW8xAW8oTVaeRCuE+4AccFD+jZQ0YXJesIaUEcU2kHVQ85rwDWn+IKlWLo/bsWXa+o5N7Wni4mntlimL0zJ+P3dnUcZgAiWjK6+xjayjsE8GKnMtGxKYUkPfS9lSnXWr7jhltl3CGWfeFg+s0iejlB+mcO4m3T9oKQozWdWhc78OjhrBX6Zbf5akNxhL0xlvWh0q+vDfhqN+i4nl7dzR06SoeuL+QOX5GwmxvkptJjbqn3VvjIyq36putCJ0irs7q24rHjbYhqo6ro7YbKY2nWxcyTXGdqgrbLMI//PoWyrByMq4tOwFxLC+noLXj1lmEMyIpVYf9euaD0jRfilOCCaRcPSnGemwWfuhOM3iVIrKKj3zZn44kup3oQyqhl/DCrzfKTanZqRzH+67jr5Kz3DwLlP2Z/d7zoywLyYNXkhKQKLDyJdIbcD3RMAFwNi3U6QFyjYM2s5lYYZTXfGW7gyWdN5tbQVr+hQiR5aBunhJRK/AoO7R1Vgu7nlez78i2XS6UgzHpLqoJoM4BHd6qM6gRSFOAGWt5ZxgtPiBA92STNIdBhA2ywylvwogbUH8SIcJlvixNKxxCOIvAf5cSfcdDB+aujYHEgrHE+RNVx4WFwKh3/3umFSa/AfCvKDEis4gTZ2109r5pBlRa5COpMm4LQ+nn8DTKUAvdjZ7NtTmK7IBMgdDoIfGzHteQJhENehDGmeXm0rgW+6mkYVpRcOQNzpv2bmYhn1u3dx1vUICDOgfAkQVj8HTljBiZyHd8GWDLyd77H3V1NwjAEzgxno3G1brNh4OOyPCuKL+noX1x82LHNC6QLoRZcgsUJKv20DpzNWS7DEbPO0g7O5ZAPjlMLR2XdI5mIFPgVFze8WhgQfAQ== X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2a86ffe1-8885-4780-885c-08dd5a5a2fde X-MS-Exchange-CrossTenant-AuthSource: AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2025 13:49:15.4489 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TKk/wHAjajGqC941HX7qDla0Im40h7P9FdV8KtVlWOithGzXddnVLhnbmznwh8D5VrCFtG3iL/chDOnNbdgsOA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR10MB3165 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 03 Mar 2025 13:49:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17375 When `ud.shallow == 1`: - Prefer an initial shallow clone over an initial full bare clone, while still utilizing any already existing full bare clones. - If the Git error "Server does not allow request for unadvertised object" occurs, the initial full bare clone is fetched automatically. This may happen if the Git server does not allow the request or if the Git client has issues with this functionality, especially with the Git client from Ubuntu 20.04. This improves: - Resolve timeout issues during initial clones on slow internet connections by reducing the amount of data transferred. - Eliminate the need to use an HTTPS tarball `SRC_URI` to reduce data transfer. - Allow SSH-based authentication (e.g. cert and agent-based) when using non-public repos, so additional HTTPS tokens may not be required. Signed-off-by: Stefan Koch --- lib/bb/fetch2/git.py | 114 ++++++++++++++++++++++++++++++++----------- 1 file changed, 85 insertions(+), 29 deletions(-) diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 168f14d0c..9a15abaa7 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -207,6 +207,7 @@ class Git(FetchMethod): if ud.bareclone: ud.cloneflags += " --mirror" + ud.shallow_skip_fast = False ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1" ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or "").split() @@ -446,6 +447,24 @@ class Git(FetchMethod): if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url) progresshandler = GitProgressHandler(d) + + # Try creating a fast initial shallow clone + # Enabling ud.shallow_skip_fast will skip this + # If the Git error "Server does not allow request for unadvertised object" + # occurs, shallow_skip_fast is enabled automatically. + # This may happen if the Git server does not allow the request + # or if the Git client has issues with this functionality. + if ud.shallow and not ud.shallow_skip_fast: + try: + self.clone_shallow_with_tarball(ud, d) + # When the shallow clone has succeeded, use the shallow tarball + ud.localpath = ud.fullshallow + return + except: + logger.warning("Creating fast initial shallow clone failed, try initial regular clone now.") + + # When skipping fast initial shallow or the fast inital shallow clone failed: + # Try again with an initial regular clone runfetchcmd(clone_cmd, d, log=progresshandler) # Update the checkout if needed @@ -508,48 +527,74 @@ class Git(FetchMethod): if os.path.exists(os.path.join(ud.destdir, ".git", "lfs")): runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clonedir, d, workdir="%s/.git" % ud.destdir) - def build_mirror_data(self, ud, d): - - # Create as a temp file and move atomically into position to avoid races - @contextmanager - def create_atomic(filename): - fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) - try: - yield tfile - umask = os.umask(0o666) - os.umask(umask) - os.chmod(tfile, (0o666 & ~umask)) - os.rename(tfile, filename) - finally: - os.close(fd) + def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None): + """Helper method for fetching Git LFS data""" + try: + if self._need_lfs(ud) and self._contains_lfs(ud, d, clonedir) and self._find_git_lfs(d) and len(revision): + # Using worktree with the revision because .lfsconfig may exists + worktree_add_cmd = "%s worktree add wt %s" % (ud.basecmd, revision) + runfetchcmd(worktree_add_cmd, d, log=progresshandler, workdir=clonedir) + lfs_fetch_cmd = "%s lfs fetch %s" % (ud.basecmd, "--all" if fetchall else "") + runfetchcmd(lfs_fetch_cmd, d, log=progresshandler, workdir=(clonedir + "/wt")) + worktree_rem_cmd = "%s worktree remove -f wt" % ud.basecmd + runfetchcmd(worktree_rem_cmd, d, log=progresshandler, workdir=clonedir) + except: + logger.warning("Fetching LFS did not succeed.") + + @contextmanager + def create_atomic(self, filename): + """Create as a temp file and move atomically into position to avoid races""" + fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) + try: + yield tfile + umask = os.umask(0o666) + os.umask(umask) + os.chmod(tfile, (0o666 & ~umask)) + os.rename(tfile, filename) + finally: + os.close(fd) + def build_mirror_data(self, ud, d): if ud.shallow and ud.write_shallow_tarballs: if not os.path.exists(ud.fullshallow): if os.path.islink(ud.fullshallow): os.unlink(ud.fullshallow) - tempdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR')) - shallowclone = os.path.join(tempdir, 'git') - try: - self.clone_shallow_local(ud, shallowclone, d) - - logger.info("Creating tarball of git repository") - with create_atomic(ud.fullshallow) as tfile: - runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) - runfetchcmd("touch %s.done" % ud.fullshallow, d) - finally: - bb.utils.remove(tempdir, recurse=True) + self.clone_shallow_with_tarball(ud, d) elif ud.write_tarballs and not os.path.exists(ud.fullmirror): if os.path.islink(ud.fullmirror): os.unlink(ud.fullmirror) logger.info("Creating tarball of git repository") - with create_atomic(ud.fullmirror) as tfile: + with self.create_atomic(ud.fullmirror) as tfile: mtime = runfetchcmd("{} log --all -1 --format=%cD".format(ud.basecmd), d, quiet=True, workdir=ud.clonedir) runfetchcmd("tar -czf %s --owner oe:0 --group oe:0 --mtime \"%s\" ." % (tfile, mtime), d, workdir=ud.clonedir) runfetchcmd("touch %s.done" % ud.fullmirror, d) + def clone_shallow_with_tarball(self, ud, d): + ret = False + tempdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR')) + shallowclone = os.path.join(tempdir, 'git') + try: + try: + self.clone_shallow_local(ud, shallowclone, d) + except: + logger.warning("Fash shallow clone failed, try to skip fast mode now.") + bb.utils.remove(tempdir, recurse=True) + os.mkdir(tempdir) + ud.shallow_skip_fast = True + self.clone_shallow_local(ud, shallowclone, d) + logger.info("Creating tarball of git repository") + with self.create_atomic(ud.fullshallow) as tfile: + runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) + runfetchcmd("touch %s.done" % ud.fullshallow, d) + ret = True + finally: + bb.utils.remove(tempdir, recurse=True) + + return ret + def clone_shallow_local(self, ud, dest, d): """ Shallow fetch from ud.clonedir (${DL_DIR}/git2/ by default): @@ -557,12 +602,20 @@ class Git(FetchMethod): - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude= rev """ + progresshandler = GitProgressHandler(d) + repourl = self._get_repo_url(ud) bb.utils.mkdirhier(dest) init_cmd = "%s init -q" % ud.basecmd if ud.bareclone: init_cmd += " --bare" runfetchcmd(init_cmd, d, workdir=dest) - runfetchcmd("%s remote add origin %s" % (ud.basecmd, ud.clonedir), d, workdir=dest) + # Use repourl when creating a fast initial shallow clone + # Prefer already existing full bare clones if available + if not ud.shallow_skip_fast and not os.path.exists(ud.clonedir): + remote = shlex.quote(repourl) + else: + remote = ud.clonedir + runfetchcmd("%s remote add origin %s" % (ud.basecmd, remote), d, workdir=dest) # Check the histories which should be excluded shallow_exclude = '' @@ -600,10 +653,14 @@ class Git(FetchMethod): # The ud.clonedir is a local temporary dir, will be removed when # fetch is done, so we can do anything on it. adv_cmd = 'git branch -f advertise-%s %s' % (revision, revision) - runfetchcmd(adv_cmd, d, workdir=ud.clonedir) + if ud.shallow_skip_fast: + runfetchcmd(adv_cmd, d, workdir=ud.clonedir) runfetchcmd(fetch_cmd, d, workdir=dest) runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) + # Fetch Git LFS data for fast shallow clones + if not ud.shallow_skip_fast: + self.lfs_fetch(ud, d, dest, ud.revisions[ud.names[0]]) # Apply extra ref wildcards all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \ @@ -629,7 +686,6 @@ class Git(FetchMethod): runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) # The url is local ud.clonedir, set it to upstream one - repourl = self._get_repo_url(ud) runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=dest) def unpack(self, ud, destdir, d): From patchwork Mon Mar 3 13:49:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Koch X-Patchwork-Id: 58201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 851F6C282D4 for ; Mon, 3 Mar 2025 13:49:25 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.85]) by mx.groups.io with SMTP id smtpd.web11.53205.1741009758511073424 for ; Mon, 03 Mar 2025 05:49:19 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@siemens.com header.s=selector2 header.b=R0a+yRz/; spf=pass (domain: siemens.com, ip: 40.107.22.85, mailfrom: stefan-koch@siemens.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tp823018D+0Ap7kh0BVfJCglQoWwD3zVK9QXSq0fPGH21FBRuty9WcNlrb442KLuG7YVWo0JSq0L57a9hG6B0RzVgWRmFXmMACzZVcaDqV3hsdWvjwookmiQleiTRA5tAqpfIjyV4Rk7riPOjf5NYhZFBNyapQCrVxuXRQ489NJmzZDNv9FJ9N7lWuqEErCZcAFYLwjF+0JLnKErR5KJwCPvAgnMJ0hen5FeYulKQBVkLSk9uTus1zv+e0AkW+ejRNy7XZhk1GKs9K/dIAWvSf/NEK6U26Ga8K//tr1o4uS/98looOQUybmoV8WnIdV4PZc/hROOuLco6Okp2po/EA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7L1SrloaMuoXa8iLbuu7Qf3jo26EOmH8Q2bZIPiVjyw=; b=beNdGe3vnTjDlWhijvTabIlvsx4i/glUfTdfahOv87kusXPMgow0s7tsKIn85auHiCq6DFjdWNMIRgJIIwAp+dhIgc54e0Q0nSdkYXc0GYzykluqo6NLb/r98tzVAkgYLh4gx+KcW7Z3DSMWXRrXVgPbX8COw/aAkL45+EyG4bAOJx8emHoZHP7DKYAEIpmEGGDEXA/EwobCq72Ii8mzP/d/UWYBIHeL8Ss5zh47cpGOfbCNafTR/dubyTs1bERhgmvHPpI1maziRVpxAvCIHk+suXJloclaGbFBAOZa4T3WNkX+3DCFvLS7KQOa11HIOwbBfKak9zrdu8UIUQ1Ikw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7L1SrloaMuoXa8iLbuu7Qf3jo26EOmH8Q2bZIPiVjyw=; b=R0a+yRz/a1kJ64L/5K8M2sX7n8PWvXebEqOfHovx8xKX6lYwNsGELd8jOZgTqRLv2oKoNkkUzNTNjgtW8cnqFFfIDMERKhULzNpkVOyyhaPdOs4erF4HlBnT9x00zwKDBQu/Bj+1FvYglfIIrWZBcE82iOwBy1u5Jvb+NrBrPUpVW6SiBrEopLCXFuzi9IfF3K9hn5YcbU1Vi+00LSVexJ0ZL0jOczralSN8xrfuYfYi7BtaEU9Kcu28qVF4Zv+ib+5CQbQUhn9tZieD21Onq8zeejj4aAWWlZs+LfRVBNLmW+DYQJ51mP2OpwNkFlztBuawyeoTTNkq2qfisxc4xg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) by VI1PR10MB3165.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:803:135::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.28; Mon, 3 Mar 2025 13:49:16 +0000 Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec]) by AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec%6]) with mapi id 15.20.8489.028; Mon, 3 Mar 2025 13:49:16 +0000 From: Stefan Koch To: bitbake-devel@lists.openembedded.org CC: docs@lists.yoctoproject.org, stefan-koch@siemens.com, simon.sudler@siemens.com, jan.kiszka@siemens.com, alex.kanavin@gmail.com, richard.purdie@linuxfoundation.org, quentin.schulz@cherry.de, mathieu.dubois-briand@bootlin.com, Ross.Burton@arm.com Subject: [PATCH v5 2/5] bitbake-user-manual: Update documentation for fast `BB_GIT_SHALLOW` Date: Mon, 3 Mar 2025 14:49:10 +0100 Message-ID: <20250303134913.356778-2-stefan-koch@siemens.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250303134913.356778-1-stefan-koch@siemens.com> References: <20250303134913.356778-1-stefan-koch@siemens.com> X-ClientProxiedBy: FR3P281CA0097.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a1::15) To AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR10MB4959:EE_|VI1PR10MB3165:EE_ X-MS-Office365-Filtering-Correlation-Id: 6e513e88-2904-49ef-3a4d-08dd5a5a3072 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: Yw+jsUjrMK+rOsTYfViaxeMaHDNaU+Ehx1BzH7ETSS4xkK8zCN+L8csYUByTOnaJJkY/V6WyO180L2hAEnl+i317yMlpISMN3fGbQVdNdvqgjd+wg852DW6taVYShpJfNyLHgMHT0F+85HJS+/o3YIOJRFZTKBtSD+FfTZswCgIxPqYPVZ16KnbXfeAL6EAOb5uzhggPTZUWvUhUBO4m4iybpFWtumVgkoDzHjPfeYqc6luGLpE2KXSPrYbN6ZdLH9ur9eBEvL+06Bx+4r7uuBmjJ687GobQmcjf14byej73PtKpVrk5xjgkZY5VQM5gIL+rE/tesKIyxZQ0+J1cV0lCjx7O0tFmQGGf4OE+L/ObQ2kNUWvIuuoMac68qlWwkbQnSxK8Q8hckQMNmPdtIWNCX4kVb6v/6RKoJFI/mmjfBRFYSIDc00Uy1TVH0vOUvc9YTuUr//kys4Xx3gwPRfMg4M4K9cYuCni5d7qjuxXF8EeUJMzzO64eLK5YkevKeEKNvyYnV0i6mYpm/WiQfxngmqJrbvKzwFosjEJk8zgG8IxPjLGfeiims0Raf7UlkwzWp6Ww9k+5SHqTW42IwadC4LVikAqxKvP0UZQvQH98R6nqAihiZcpOz4DasU1xKNZGPWDfrV7B2ix1/bqeI1XF/CT6m/ydL7ac42Y2taDNQjPWAV8Sciw+U2ks6LP+/2pkiFBsFBTn5KJhI9/GNFWibcKiglGYeBb6OohF3CMsGT0VWbmkzcjZQYTpDAmzeHI9oQkQkG508xiNfMMFCDGPQcIN1EKgzmz0EGX9l0wJnYrJRZMhwT5tVrrvAOw6OGmfuXCIysgr9apaLDctLjG2u/WkkcsdelBD5vyzM+Ain48Eg9dFX/9ZveOKA1S503rs0pMi23AR9eUoV2S/TSpTV2wv5PTDz/w75CzRwuoXsvslLoJ7behkltXgiFXuKorCNI7cTsziBAkNkDA48MlJVuMSxZwOysWiRack2Z8a1fs/qfonRTulZl25M+IIuDgAbRwpQDD8ugNlSNdxigGse0mR0tDED8v+rpFUI518mwVMjfqizjRQGlTe9+KNqlwL2IYrzsCX+V6tUYmthd5FW3bYbANQXdHAqq75UBIvdFrfrzMieuKpbyqc2i3ix3vzR8I+UfJYgBXWD1KlIxnkLwrGl+i8dho/GIEH54wucB7IDGHnY9qYAkWwazIBsD/AfYkDXg9CZqyjIaBQaOouXeG36l8tiYEGtu70xNQFr/fIKDUPrgLO/8TqEH7tCG14CFwtED46dAn0a72W/JJW3IvRhswmnX2JHwwWPf6icwtAsunJHFR3NCvmB83ps/bpZx+BoaeV0TNdNh/HV12AoDTIxJnm6atIybJy76mgqRejvjWjBUhrLfSL/7Wj X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: eD2/x1bfMTr2ouYtIKD//33vVT1NJAeQTavSwYrzeJ7qRPO7ONf9GmAyihwtcJFgFqj93lLoc8DJ2WTUzyWGO0alh3e1azrqbez9DuT2Qg320fMCBNuhu6/I2Z8ub1+YteVkTwGx24FUVfGHcNNLRk08AzfXzLJ3PtL/g9s2BOIR+NyPWrE3SBP6xjpUVgBKPUSQaeoTYgbAJd8sNHAmLv0vi6Np2+aPk59wcQdQs9GmoFXlco4dRNZmG22aKrqn8LKcarjUtI8pcPxtqoT6dkbFEqswDJ4RDUtGhIXPhyaSjQAOsGCMibX0ampbe8+pYLd8GpSE3/t0WMqS2kajHWOD5qmSevl9OxKUTBICZqTC+0QpZq3e9qJ8chwdkF2SDxUxG6o0CMwqITWd+sOkp/iPcLfDqhreJ/K+/3Q58kijOc9oSsySLeRBSrlsAZ/FqSL7F81p/IxF7nfwhSQSMcooe+fwrbfiENbSjiZXDnyz/sxE9RA3Z+Q/nEaq6WOcIR9zuLL7wxdEONLMn8JpjBt7smDinAeyv1WDEXVg3W76jUI7O/GFFhgZL1xqmqc/5HT6KACHqPxZZKDhRHFl3hu7aD5fmJGg9eCW8jE3kyEudzOThMpFONHGurvDNdN9QZgxBk0vecpmIHNu46DDcW/zQdTqbgZwOqFjwqN9Y9P2LMNjpngyI4qJZK1OJgSjdvG1+C/0lkUXtDcAVXZ3xRS6N85p/3gwlLTqBo0T9AQGPHk4llu8oJSGFqTZOhNLhL+61nW1k+wWI2GkD66wC08LEOhonsk4QiGriNdKyayWdh0RF5rpUlESV6zBxCHSgJTfBG9CrwZgYVgVrPgfn8Ed7Oc0DDXNsc79HBShfLrAW4C2A+wlZYi7qUVw5+v2N89leNmjwxhoBAJzuSFcRMs9mXTR/i+JiA+0AyBhQqs71zEqaw9Ul8V+qBWsQ1s1tZ3nyQbb7YSFF11YY6LnRuTVx40k8q6znccYmz4OjtLTg3Z7bjE/WfcWJCw7vDGfPV6E4n9hQ/W5i+L+Ye8gio2Oq8zlUi0C67DXOK6ozCpqNo/Zk8P5YqPvsJnnBQiBQFf7CP16FI9KNurC68ZJH54Pq8W8vDwgC+iVTD4V7VhNvWxMcqhpCEm3JlBOCHKJkwio6YT2rWzsuIQRsynZcueRG09388KXj5pmY4BRroSbFSdRrF7M2o8slUHLKjCVxgj3AguHQrXf6kYck+rXGNat7/RyBvsYbooamyp77PVw1tl7h2jVe0+IsJ0rb+5puK/51cwe5rckrEcm2P6dXrcMJFieDHmVdU9gV3UDaMPMKh6EEZv5kYW60qRLBBGblcXFtDRRGJRDy1a36jCLrBKKq18nuUD85BZ9TFyZSKZYLFKpX3n9ozVD3D5gqVyQY3NE4hJ2xHR6r07otvfahftGSb09k/sPWKoRpnTo7GdGJz5gz18i0gOXALLwSrBt45PwIIlKpC4IbVszO0/J0sY9jYF5J8F30wrPcKKQoN/YS9HXODbwQN323PwFRNE5jlxJbnEw4Tz9hN32WXhWdt/QdYocfo3eYEkuVG+orR58EhEj5w1/Sfo271PpxRV7qpKy3nTQLo6HMBPhDTypMw== X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6e513e88-2904-49ef-3a4d-08dd5a5a3072 X-MS-Exchange-CrossTenant-AuthSource: AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2025 13:49:16.3990 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8drx4IPsjeXTopPa+eloCzYF+ZKY9MqIsKVCDft5ER4jWbCaSu+syAProwzYwtNeQPhHsxJhRtu5SLCsj788+A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR10MB3165 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 03 Mar 2025 13:49:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17376 Signed-off-by: Stefan Koch --- .../bitbake-user-manual-ref-variables.rst | 25 +++++++++++++++---- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst index 858de6347..dd8cf2409 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -320,11 +320,26 @@ overview of their function and contents. mirror tarball. If the shallow mirror tarball cannot be fetched, it will try to fetch the full mirror tarball and use that. - When a mirror tarball is not available, a full git clone will be performed - regardless of whether this variable is set or not. Support for shallow - clones is not currently implemented as git does not directly support - shallow cloning a particular git commit hash (it only supports cloning - from a tag or branch reference). + This setting causes an initial shallow clone instead of an initial full bare clone. + The amount of data transferred during the initial clone will be significantly reduced. + + However, every time the source revision (referenced in :term:`SRCREV`) + changes, regardless of whether the cache within the download directory + (defined by :term:`DL_DIR`) has been cleaned up or not, + the data transfer may be significantly higher because entirely + new shallow clones are required for each source revision change. + + Over time, numerous shallow clones may cumulatively transfer + the same amount of data as an initial full bare clone. + This is especially the case with very large repositories. + + Existing initial full bare clones, created without this setting, + will still be utilized. + + If the Git error "Server does not allow request for unadvertised object" + occurs, an initial full bare clone is fetched automatically. + This may happen if the Git server does not allow the request + or if the Git client has issues with this functionality. See also :term:`BB_GIT_SHALLOW_DEPTH` and :term:`BB_GENERATE_SHALLOW_TARBALLS`. From patchwork Mon Mar 3 13:49:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Koch X-Patchwork-Id: 58202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 989A4C282D5 for ; Mon, 3 Mar 2025 13:49:25 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.85]) by mx.groups.io with SMTP id smtpd.web11.53205.1741009758511073424 for ; Mon, 03 Mar 2025 05:49:20 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@siemens.com header.s=selector2 header.b=owQveZLy; spf=pass (domain: siemens.com, ip: 40.107.22.85, mailfrom: stefan-koch@siemens.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=AtTD0NWCvyxbkQULQPSdmzhCJzqIYl5gmSlqhmk+eLV3vix0EHrNsKVZQd5tIM2Kfq4Hjk5J8hX/reJfNeejH50t42QWNZI4SjvgHUoE9ysV9ziH/BxLVk/I9WGalyXJlw4lxB+z68JHjuRT/uIMxvR7OBLgtOVKcPlom0KwkbqJBbIsXALFIBd9LApZvXIWr7Fdq6qhiDi+tkqY0TlmrTfssZ45j+tcqwfz47xVGPJ4Esj+ZwonIpjyAzD56AgsaSgeZougfFqMtxBDlD55VIoWYqnDAyGpvmT2oGr0L+AvYnMtBYBvZ6sP38eZ8eD6gM1jbKgzIr3PaSb2Sy8qPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ov/xZWl5o5VTKtokglzYlj+loidP180y7kDoa7gJA5s=; b=Mx3g1gBWwNsb+9TMGTWAVjaEY97JSeMCBnfbllemKBtfMrMuLoE9BJFycPP8CLP4+Ko2uMUsd4cu/GnH/9CuOQ/JAeBHCdtTq1IPZylX+QzilJmLTSQVgo94mPdbaXL/x2L35s455KrlnZuNXrHV7gyBxAZAdM3yTM1OM0jrZuXeFjoR3+ffaxbUzKYg/9V/dpzx3taTOCay4mUeXiBcL3pXIVFotqOZf43Mf1at/6nW0Fv4X0acX2rI/BJom1G6FDQjyJFtwN3xcdlD1/mbyhGl3jxmtSmryn2SHwPWHVblWWw7TFlATswq3xqMuurCXWR7p+c9fSx/S9OkGFAEAg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ov/xZWl5o5VTKtokglzYlj+loidP180y7kDoa7gJA5s=; b=owQveZLygccSC8z03Q6ZYJEFHnCUJnGCzXTZdBSkh63RhPoLLaZ5xW+K8c0Dm1llywZEdLYNmtJx4N0sbgi+LbDcssZzzCSEe7AKejEctWgGQFAw5+kzeXw2VQNRKpR6EsXtomP8nx0uYqAhNJCpbnd9fmePdDJAn+48hcDsEP6i1XOQviX2Ju7RRCrSN+bIKTQPnvUAPeB1u88QhCdHGuZQC9fdgo+bXGLPilw4J2xSM2vVny78xEAyjwxR/21bdLH76RZ1TlChnBKeEf1qoRHDJ+uYdRNxX19YKgsuXBuEwX98gjtGA4tVVNoek/FyM6NAq+p6JxDRTXsSsnZqOQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) by VI1PR10MB3165.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:803:135::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.28; Mon, 3 Mar 2025 13:49:18 +0000 Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec]) by AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec%6]) with mapi id 15.20.8489.028; Mon, 3 Mar 2025 13:49:17 +0000 From: Stefan Koch To: bitbake-devel@lists.openembedded.org CC: docs@lists.yoctoproject.org, stefan-koch@siemens.com, simon.sudler@siemens.com, jan.kiszka@siemens.com, alex.kanavin@gmail.com, richard.purdie@linuxfoundation.org, quentin.schulz@cherry.de, mathieu.dubois-briand@bootlin.com, Ross.Burton@arm.com Subject: [PATCH v5 3/5] fetch2/gitsm: Unpack even when `ud.clonedir` is not available Date: Mon, 3 Mar 2025 14:49:11 +0100 Message-ID: <20250303134913.356778-3-stefan-koch@siemens.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250303134913.356778-1-stefan-koch@siemens.com> References: <20250303134913.356778-1-stefan-koch@siemens.com> X-ClientProxiedBy: FR3P281CA0099.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a1::16) To AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR10MB4959:EE_|VI1PR10MB3165:EE_ X-MS-Office365-Filtering-Correlation-Id: d52456b1-98fc-4dc0-2869-08dd5a5a30e3 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: IwbfXZWNlUx4km0zVrjI5BwjqwCdHpSpISiHTGpVPM5C076Iia7VxTHOecTb0IbJAU5qqd/XnNcSB9Z1YqcvC8zX98ueZMqBe5yauXZtaJw35yhIDP4S+XQm/kcO3HquhmubH0b0+DpMpTehGHe54k6DbXcw4AQpAp1kKFmuKUV5mKkDX9tXdOTQ/V/DM/r6Ymfx5FFYxdfKfZPQtA/kJMZLBej196T6E2gsibXfYA+6rSuLrvSyqGGL/8hEUTl8cApLQDqEGDmIP8WYOYo+zHSFUwqDuIYrqlFAwXsiXa9DmJW73KRO8Uj920/s+2TbB34GUt60BHDReSVQGqpbK+k59WRFmtGxCzGEX4AfsPLbsHrUK+BqG3hpRr5jdXHuPjHcVE9sYJexo6aYKY7TIrd2ZCI9h8GTjB8lJbFtkVk5d1JJ5W5NJ2rpSmXIemrAOvcRNK1rJhQ8VK0seYga/pQ1m3DDP+TYyI+dNAdz5z2IzbmMxGjBz7my3kYnGxgvFLrM71RK4U6rCCNIfyfMD80Iztt3FuDJk4y47KoAU/Z6m1eWzzkVnoQ/O2QYtvAsrE1eJPniP6uJn1oH/rbkFPUW00Ll4kLVk7GsMZA3Qs6tilScrDkYfzgePUVgn1hO6kllEuLSynA84FP/DSyBfGqsZNoOiBQEdbuuQGCy1/Uvnw/+27Std0Heh0Rw4wwH7MR3A2eYFLDMn7nK0FtsFCUd8PDZ8zg5CICGo6l6be7NHoyCBEz8011a20avXBTkOqNyQHxZ05Sc+90cou1ceUCZohWoD7BzovbDNSfte1aFnhalhe+zX+TmY1J3s2wkqc0c8X+rkY3dsvTO70IaeF7pYH5WkwAuu1LPrCG8w7JM3ezG4MoBWhjxbNe3Ekx1jJb3MFadFdjvu7DzmTC6797wQV6nDgzqLxL0ruKYtPkk4B82u7R2OlpIjyPl93SpxYNpr0qy7yX2WIT72GJ/oE3sbT/Zm39scmeP8XnzmA6fhcu8GM8kBjFeZ2HOH7YsaFXWRu1o5/zQhJgPbEacwUgafZ/FMJIiVkneZJDr8xej4HQPMUajhQBFABc9pSEw3ePo0w3ls5ONl4pxrG56lkkE+FJcxTLu7y/unz7mQwoCIIcicWI0MROl8Vzrw2nZQK0ZiclZEKC0pJgIKmpp+0t7RS1kvrddV6qmGOpdkLecEwPrxxHctm9qjJoY8hOHIh89mRyvvN+SkOALdzhEQYLy3hmmDCXk6TLuv/b0cTdk8CQpTFAqDLuxDiwS7RfD20a6Bq7vPLDpudbaMha8KaidbQm6UcSqL4XRqiDq6EL9xz4zugiHZjwsMavrI4thcNgLhLjwCufZ+cm9/5GM2zSMRBWKWrHtrrMqb9PJwZv7Mu83oVTV4B6VCnQS9h/m X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: iL/L0kQNV51GnGbl3Rq9F3JnRAPVIJ+9ElLjmAR8WHs0bH+yUdlwgC9kltAOPOzT4fzGGymq/gb8dJV/tZyGV7VZg96a7jvjGjuQeVtpu1+NDTLI2WpPZvMmTF3vnnroBCP+eSNSaWu8yx/Jez8ohET2b3Qu6LLOMORg5lfcYaJ1gLkLS0ZZAXRd+8qLjclmDO+TF8KWwBkVQBn2Hjc9ntCQXMf7FYFKlEapTIupEEq+IK8YEVKuPGKJ6JQy/mcWg3eKXIWBhDpPdJRXVurCFQvhMQw4ZPjpqjln0qzgN1kPmSuKgPJK9ondmzv34XZnaLEPDFbkRDrzDtMyE5uC1Hb8f9crv5uZVTbO24TrOK4056UtT974inknUwrPOPOfZKX20pbY63vj8jzht3VJb3tqwFXkHtba/OW2vJln2ebMr6Vqg69OhxyvVlobq8Cg7Qfg/m6sC6r/AQQUxSYQKpEuUdKY9ODojeu8FUe31qsJJ9Btr1da6l8vFas4tPpzQS2jqesVFlTPj+INBsntU6L4w7z/NTc4XqHQeNSqTZay77Tn+I7sTLAHQT73VbGgus4dOwFpF3m7DEQ4HRAkGDM3MH59c/bJZiOE0VwtJ5IHkLNy3gs92+Rx2WJ/1B3mvNoQLzfK6u7f/kEFH/K09ai+zglDl3A9br29nJ47XJgun985Qr20Tndbf9rZfvgtsq8ffBVQ5U4Oy0goTLuR0AkDfhtYJm60moR5RfVoK03LN8hGgk2MUOP35us3Vhetnupu7VbtKTfYNbX8+SvyyN3mCNEDPQ7ClmNXFszws3kesJRfTY0B9gk7MijAIwnrY81zjxN007/GG9n81zStiaUnf6s/TSWh8dLa2FBBp4zinVF3VTMOYN60JjlpO5KVT1wSmqLtExLRSEO86tXRtfQd1B1tAZMqcC+kYN8ML/SHiyFmj/7lp+LtHXRiEg6Qgc3fXjuPLQSnr00yIghajj3GymebTD9qXATCZsVdYaLccXmifUi7SOgwxRZRjp+hO348jZNpXBQ9BS8R0bzckiiHaTqZ7LBIaWHZVUJ3b9TS4um3zniyTJljGiv9w0p/FTNJvM81ox1UYAdbiD9JMZTkih0ZJC3cjjOqRZi6YlRQewhEj4lNpnagwrB3Akz1b92w5nL3OqvuwfElpKES2J0VsoweyUnPXtG1k49D54/NZjCipsye4/1YdbPs8DfMiIHCjGYlfRNPCwiQ25xtnEQlcOAWd6eZw72CU/fzU/JLpbziYxBgjY0D8+DTHwxarm8hXGEpzYUg91UsnqzWEBLSLJUTQnjsplpJLQZUNiAb0ixzH4OMx+kJEAObtZRPlsng10zVLrO4B86ayvw5doFYM90+vtS/1h1cldCJYZJSKQH3PV0vnp2n+pM/czjdip7kgYTCyFY855dtminLVklSN1urZkziERU7gr7sKZ+VAwqmAGCQgNjThnBe8vaybTZovaU18TBWs9bfGNpA6iUExxN18M+NShPJlTQSIueMptzRCN6HfXESX5UZeogAXpwHL5X561Q7ZojGtI40CWt5D9ADJsSxQtsVldxRWW3gEUgJ20sRAFxj1/WpF+/CLzUBjGl/HmvG1kIzofSVwA== X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: d52456b1-98fc-4dc0-2869-08dd5a5a30e3 X-MS-Exchange-CrossTenant-AuthSource: AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2025 13:49:17.1917 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jubodUdDpT5N0x/srIl9/9zxC40tt60Ru7cCC5kXcS+3QpBG1eIA4ppERdm2cWzdcvMroFYzRkhYr2D+StPZyA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR10MB3165 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 03 Mar 2025 13:49:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17377 Signed-off-by: Stefan Koch --- lib/bb/fetch2/gitsm.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py index bab1bf258..1c58230ea 100644 --- a/lib/bb/fetch2/gitsm.py +++ b/lib/bb/fetch2/gitsm.py @@ -150,7 +150,10 @@ class GitSM(Git): def call_process_submodules(self, ud, d, extra_check, subfunc): # If we're using a shallow mirror tarball it needs to be # unpacked temporarily so that we can examine the .gitmodules file - if ud.shallow and os.path.exists(ud.fullshallow) and extra_check: + # Unpack even when ud.clonedir is not available, + # which may occur during a fast shallow clone + unpack = extra_check or not os.path.exists(ud.clonedir) + if ud.shallow and os.path.exists(ud.fullshallow) and unpack: tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR")) try: runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir) From patchwork Mon Mar 3 13:49:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Koch X-Patchwork-Id: 58203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95131C282D3 for ; Mon, 3 Mar 2025 13:49:25 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.85]) by mx.groups.io with SMTP id smtpd.web11.53205.1741009758511073424 for ; Mon, 03 Mar 2025 05:49:21 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@siemens.com header.s=selector2 header.b=Dz6oH/Ol; spf=pass (domain: siemens.com, ip: 40.107.22.85, mailfrom: stefan-koch@siemens.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nsy8eOO76EkH517Q+mKglfCCOmm6imBX5D6SoGx0Tq/QyJEQPdj+YlFVCddGF/SaoXO0zgUWVX9PsHtQBHTOMf9LmwI8e1dm1CMOPLvKwDul3Aq4a4Pw6YIbk/xj2FG6U7rihcM/mRNVCH2avcJPbhgtcFchW00MEg9zrL5TQJWuJTtKCBqQp18ADc2/3UUPD7lCKzCh4Hmr3BNJAkiay1cqDOrjc/iQLAq/BTkytbMqp+Pvau2oYxAhDwdx0UK4u1hfyv8R2iodsZJp34iFdtNOLo/Iqp8gD3/NPsDdQ4fja1eD7T7/AlqcQnjEQVD0RTkMI/Zek/vsSKfvWpy7oA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ClpcmoPFdfrlx+od8n/luzQpiyHPp/0x5T+gG6ywBsk=; b=jqmek/kPlj0Lnc0i8g20U1WURAlvQSyB0R3S8PNZ2K7LqUOEwRkWOhiWdVRcHeD9xLzvlUjJWQJj9HQc18b9XDlgdQiT4AMHdTcy7aFnC9E9fOxejrFWN1RIrsT+i5e1ax35cHjMi+HwIiJNoN/+SkESO9LoFvjrK2nVn3Q1hwqUKtCv3DttheqUdM+UMicToc7ahJuISgfIM75KSSjHieWF08MaB+I0j4WUcETgNAm57Npk5K+y6scaDzdgPm8L0+qoAh4vpv/8L/rp33yoxceFKy5nLmkon5OWFO3LSPl6TAQmWCoWTaKeHvOdKW/g+WyuwnP4d/CwIBQucY6G2g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ClpcmoPFdfrlx+od8n/luzQpiyHPp/0x5T+gG6ywBsk=; b=Dz6oH/OliEhWSiP+zFIdNsP9xIGVaIWEdUYdaC6zFySgVYGXP19dt/n/kYaoTqPIv3Ifx7olLWs+L0TVLI4dyLi4rf0Cm4vBvwRZWP9o33x7i7Fizh7WzOsNXzxpNsRN44D+IjpA+4w857LmrHFOfvRYTDYSpg3nOoXcPH0zyURK3dMDK4NgBPHgY6FY+7deqNxEmC9dxs6kaVkIYdHcHuZnpHl4yhw3sHSRJuKjhVJmbpIX2SWgwNAWG76BcjU4nwk7WT88RhzlnzPWaH83UlioTXSm/wF3qwo7Sb6RfrwqJ5u971G402jeIbJgoPbrI95T6u+B0jzrOULRgg5GnA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) by VI1PR10MB3165.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:803:135::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.28; Mon, 3 Mar 2025 13:49:18 +0000 Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec]) by AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec%6]) with mapi id 15.20.8489.028; Mon, 3 Mar 2025 13:49:18 +0000 From: Stefan Koch To: bitbake-devel@lists.openembedded.org CC: docs@lists.yoctoproject.org, stefan-koch@siemens.com, simon.sudler@siemens.com, jan.kiszka@siemens.com, alex.kanavin@gmail.com, richard.purdie@linuxfoundation.org, quentin.schulz@cherry.de, mathieu.dubois-briand@bootlin.com, Ross.Burton@arm.com Subject: [PATCH v5 4/5] tests/fetch: Adapt test cases for fast shallow fetches Date: Mon, 3 Mar 2025 14:49:12 +0100 Message-ID: <20250303134913.356778-4-stefan-koch@siemens.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250303134913.356778-1-stefan-koch@siemens.com> References: <20250303134913.356778-1-stefan-koch@siemens.com> X-ClientProxiedBy: FR0P281CA0113.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a8::13) To AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR10MB4959:EE_|VI1PR10MB3165:EE_ X-MS-Office365-Filtering-Correlation-Id: 4bd0525d-b636-498c-0f7b-08dd5a5a3169 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: ELcnnwPV+VKPqWP5sS9wvseGLjev5kuOllRrrqRPVG+blGzlIkq6ObAHl5henoJqdl+AlX40BOpqF6hD1voPff3WNL4tTq6DHTH+LhQsWrIUDqlvJlYbbdMixr65/5mRQopGkyruc8qhuH/XqdBzQyQ4038D3ok8MARvSpggO6fyJPxmf+K8x9LzyNOMCPiFFKdp5P7NeSPVZNrTcg2Sula/H11dfYQR0msLwP3bWEhYrlGGoRusMKGxe9fhHyuS+sRQKnpxZVzcJ8v94QghBAMxSbIu+W0XoSxyFwrvLcr2WhuFXv8jfD5VlcBtyNaFk0is5l88UEXqNFguiWklAyzT0ER6389/OEDewKC8MbadTz/GI1moaY+J3PRYG/KsxeUF0wVSzB+tNvD6n6pwsDgPika2QUU1jqh46C3MR2uPfaL+7ulIAcRshr2hVlKAfH/hTWNv2LxQLDkmm6qeHs1jO3lP53OrwCPOtv1vG6C8KQ2SMzIzBLPDliwzJ/dCcF8I5XF/x3KFdf+dj0qypWjuVgIZg5XN6fg5jM578FEioGI0HGgr3qUHuUtGE3EQAl7DvqpafZXo0JEdnxhxhXeLG5dcZdjnTJG25HDBdCyHzMbPmt7NUfIrKea0hfKMnRfcPmHlI+YXp+yHP+Bnbn0dGfICds0fXx6CYwY8smplSH6+Qw14Ed2RHAlL5kkPnr0KvaRELtqLLPBm8pmf8DkbJq1K5YVmmxgSpZz96bj+afXItGR3E9zNpQjPCgb+oTMZykeeyBwNohwmSr3jigLCahWB39sqciOmcWwC9h9fbN9N34thqwCtdWRLZSyskC/mqCRhU9UIGEEUFhjCljso9YmNB5RErApSLK9SdGxENEZsO6dRgztjMvXJGfHMQC9GrMYTp7ck6rhNRXKM0GxMjgIENz1ifQ39vXSRnS740eco4rDqzKeVeabrgv2lvJM9OaxQlLuYDXFngcneR1QIpHpdsv9lkGwChKmrMouenJBSOmFNQOY7rRkVXwOBsgDKO85tGlHQJEs8/wnC9QTAm+N3QjGQfravOa9bnbqUNnkadRc85+DZgrIEMXU2dTooVGpsnw90cpHVBajyPj2GlKwkGb892WQUB7FLgheTBm9VYJOklfodYIJr901Tb83MowJY/aIe4GgMeZ8VUQ46vjqgWq6KKJJSyHM+PmofERlOYkVy9iFrUkBzSwA7haC95Yx3nrDK9ugxGB7SEXoJz0wFPsc3Couh6M30ZyTZGW+E6z6xTvfSG/sN2bBvY/Y45XQe7Z44nfELRnXWd3mD33JN4WTiT1I4r/FQJhM6QkGKAEItIvT46sZxf+sEE5b6tl08hYT6S42P7QTpUiATh/yl+2My8nHlyiinBYycvJmEYNZrvwIsIDS64Kvc X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: XXn2sR62KSwxisLSFVR3k2H8BBTfbIdGmzxOHevuawMqaRjydQ9UrZXpeh3GTURJpvcE6RCdnzESUdog/0AlBPRoQ9fd0GKKf8m2fPyXimJ7buYnR7526kkV+R1G3vCOI/Zw8YiyZ0Ja04oMR2HEdOVuqk/qoQUmsCy2gzbTcZKE/kJ81SGYzJhnmhuzy1L9Im+USvjnRSkSS9WtrNgtzuqey05ma9mecd3qAk8xGaGuKzwdZ3+SMl+/E00WeO7oLl0llIRNz/k/k+qESvb6cw+M/gXinhBF5B5O+g16ixZlDVvMwsKjClil3TEsZmvjf4S09J5vPa+BqrSoytVF/H36PpoQ/hwPsNYcG1Lf1EjYwknaiCHgqUNWe3qgJPqLz+wWHu+xf/Q1yXXPOmsITGA8Q49q4/iInatHTpCjBlTESlZc5XeRrHJ8YpgOmgpRHwsxDj2yq4KG51k+18qaVHc8PPyNJcM7NqtEtrVSYNOm68DsU73onyhgAMwsdcnE/yaGReBtSMHa6xW/MXlv1Ul3HQvNhDDA+WzT4F7mzlElzFCa8jNUFW/bEwux6bxymjInPHOLhPgeE7YlXl/5+IyX7b51rw+WjTMygrHSsW4JvXGJ/EPUC93yShQeM24H7jbT2ZjVUO/4Uk8P+jrCA54YqqWGycVQPMuUxlO7LomhuxHqqb5O401I2+36RB/qjTl8bGxU4hsB1hfkRFZlOEmYyz9P6BamawiyxW+6oyIa01drAM3Mb5TdTXbrGnvrSSowS4VQYTQei6YwQjiM1EprtKN7Zqo9RDEmede7273pEyA6HkaNkupy86Fr2rVYX7qdWuwS3rL+xJCXK+TEyET3PFcwOdh7hC+sXV+2Rvu6jsomwWGd/o71+dXmlnc+/2JTebChAtAz4I4RaQkmP7FLoZjuDThUawAeghKnSm5HFBU5LUvsB/+5lb9HTJakGoHs3XpM6s5jbLzrkSQFu0sLzDM8P0ZnsjkWWnUYWruChQS07Xj7RJ+i1DeX3+x2TtmZAgAgicvnNz87XOSiTkk5NzlQNS1ZR91hVZNGEv15vdIR0zl2NZDU7J6AOKbPyu6qccteYkU/jtzRK/hCtSzp/rXZbih0pj5DM4SlzY8a7STu6tbVzhDOoyMbMbJr3q0Jpg54KsKXg/Nui7mzMchCa1+9yJ0BpuRtQvw6GlCjOPKdeWbIi+xDHhIcxRg/r3CrFS/LDNvShuaPqYhNuAHgqbnEzHWD12DGmwsQcW54rgoFrJYPpOUwoi4jvHEWKFxoNAGFla4rdMPvKrIEAaS3W2HICZd7wAagrMQ6x7g9ZHtsPQoPSbmm2rROGWLjitw2UbaxHjlt79GN2URLiTaroRvXfCBl2eCRroy2CHpG6XIa4oSrmjTMmByWp+Ar3CkwmvfHEZJECdvAdn3SjcnXxo/Qc/ugWoz1yY1Azaav3z/4o/iQ4cHptxrCuIo1IR5R+/NtdvRkQMgwAKlR33ViCNwmjfqEI0MCieI8naetMpqAkVikRL3MSy3WQfe5G43krQAlofhbyAyfDzzQUtWij6zIAhdLy8yJeZxWI4BbfXpf3kjOVGRsU1mY81n6MYz2o5sP7WiBoDs2rY9RYQ== X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4bd0525d-b636-498c-0f7b-08dd5a5a3169 X-MS-Exchange-CrossTenant-AuthSource: AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2025 13:49:18.0129 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: f+ZHHUgmpcJAbiFBfao7ZpJBrytwqupG7Frvlp84SDgmmKmE4Y5gX04pUcnWM5SftJ3jil4+cAhDRX99rZtSIw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR10MB3165 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 03 Mar 2025 13:49:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17378 - Address the absence of an initial full bare clone - Utilize the initial shallow clone - Modify existing test cases for this behavior - Remove incompatible test cases Signed-off-by: Stefan Koch --- lib/bb/tests/fetch.py | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 48b9e4af1..4a5af7106 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -1814,7 +1814,6 @@ class GitShallowTest(FetcherTest): def fetch_shallow(self, uri=None, disabled=False, keepclone=False): """Fetch a uri, generating a shallow tarball, then unpack using it""" fetcher, ud = self.fetch_and_unpack(uri) - assert os.path.exists(ud.clonedir), 'Git clone in DLDIR (%s) does not exist for uri %s' % (ud.clonedir, uri) # Confirm that the unpacked repo is unshallow if not disabled: @@ -1822,9 +1821,10 @@ class GitShallowTest(FetcherTest): # fetch and unpack, from the shallow tarball bb.utils.remove(self.gitdir, recurse=True) - bb.process.run('chmod u+w -R "%s"' % ud.clonedir) - bb.utils.remove(ud.clonedir, recurse=True) - bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True) + if os.path.exists(ud.clonedir): + bb.process.run('chmod u+w -R "%s"' % ud.clonedir) + bb.utils.remove(ud.clonedir, recurse=True) + bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True) # confirm that the unpacked repo is used when no git clone or git # mirror tarball is available @@ -1907,7 +1907,12 @@ class GitShallowTest(FetcherTest): self.add_empty_file('c') self.assertRevCount(3, cwd=self.srcdir) + # Clone without tarball + self.d.setVar('BB_GIT_SHALLOW', '0') + fetcher, ud = self.fetch() + # Clone and generate mirror tarball + self.d.setVar('BB_GIT_SHALLOW', '1') fetcher, ud = self.fetch() # Ensure we have a current mirror tarball, but an out of date clone @@ -1919,6 +1924,7 @@ class GitShallowTest(FetcherTest): fetcher, ud = self.fetch() fetcher.unpack(self.d.getVar('WORKDIR')) self.assertRevCount(1) + assert os.path.exists(os.path.join(self.d.getVar('WORKDIR'), 'git', 'c')) def test_shallow_single_branch_no_merge(self): self.add_empty_file('a') @@ -2116,11 +2122,12 @@ class GitShallowTest(FetcherTest): self.add_empty_file('b') # Fetch once to generate the shallow tarball + self.d.setVar('BB_GIT_SHALLOW', '0') fetcher, ud = self.fetch() - assert os.path.exists(os.path.join(self.dldir, ud.mirrortarballs[0])) # Fetch and unpack with both the clonedir and shallow tarball available bb.utils.remove(self.gitdir, recurse=True) + self.d.setVar('BB_GIT_SHALLOW', '1') fetcher, ud = self.fetch_and_unpack() # The unpacked tree should *not* be shallow @@ -2295,20 +2302,6 @@ class GitShallowTest(FetcherTest): self.assertIn("No up to date source found", context.exception.msg) self.assertIn("clone directory not available or not up to date", context.exception.msg) - @skipIfNoNetwork() - def test_that_unpack_does_work_when_using_git_shallow_tarball_but_tarball_is_not_available(self): - self.d.setVar('SRCREV', 'e5939ff608b95cdd4d0ab0e1935781ab9a276ac0') - self.d.setVar('BB_GIT_SHALLOW', '1') - self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') - fetcher = bb.fetch.Fetch(["git://git.yoctoproject.org/fstests;branch=master;protocol=https"], self.d) - fetcher.download() - - bb.utils.remove(self.dldir + "/*.tar.gz") - fetcher.unpack(self.unpackdir) - - dir = os.listdir(self.unpackdir + "/git/") - self.assertIn("fstests.doap", dir) - class GitLfsTest(FetcherTest): def skipIfNoGitLFS(): import shutil From patchwork Mon Mar 3 13:49:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Koch X-Patchwork-Id: 58200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8728FC3DA4A for ; Mon, 3 Mar 2025 13:49:25 +0000 (UTC) Received: from DUZPR83CU001.outbound.protection.outlook.com (DUZPR83CU001.outbound.protection.outlook.com [52.101.67.60]) by mx.groups.io with SMTP id smtpd.web10.53492.1741009761896938486 for ; Mon, 03 Mar 2025 05:49:22 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@siemens.com header.s=selector2 header.b=TmU8auUW; spf=pass (domain: siemens.com, ip: 52.101.67.60, mailfrom: stefan-koch@siemens.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gJ11zBm1fU22ni0dHxQEoYQe8VVSF7B5mVZmI5K8BJ/aM1pbLX8M3pax63kcr2nF075uXV6M7MQ/SvSV/X1Xr/utT6NsrJG0DLGg/J/bmu74ueZg46YXDRTUkNUs1LtZ8N4df4wExR1vQ/O+vkenLyj6klEPYq/rjghgrYdzP/rPLFE3pkMVR6LoZfE67kfxW6BT9JJecfDgu+8qxq5usEk3wcO0F5i17iJfwT0l4vmiWMNI8gIeMoNhOU2u3PmXCXfdFXMli30wFRsElupL/rAsjFU5yrsq1v3b+EagCDNgLZj6uFmpR/6wZozSswM3yDaG3o/ME2DgTqFOCu3owA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sw2M6g8OJV+P//g6oWmEfSnUlcSTel354RpQHn6B5oA=; b=aE6CfnV3Bl873ZrO5p6LbiNyhLEOVZwxOGX0QIee2un6R5w/jGBMkwYgVDbWt+TJh48KhTeGVJfQDJmNZ5pvJuO0xQHJxp1/T3SBCy/ldopZA/4omxc/2B2/k9UCZC1TBg1id0by/Q8iNujMiJh8iyWet6xiWKK7KIv+T7nmiTtTcRh339jIdgScTrnYFJfypJZ+FVoIJC56EEMWzzLzd2qNUSi1UUIOeSuSAVWcWsyODsCwr/wF89dqNvp3AYqCItVCS7x11uF/JAcuhY1mKTrauoTeQ+vy9Hr0hSeXJiWfQWUqInlpKWd2Oxxk9JhGAPh7RcboyWlFKUHH3rJx1w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sw2M6g8OJV+P//g6oWmEfSnUlcSTel354RpQHn6B5oA=; b=TmU8auUWAeIXYnRkma7TEeialkp0OURle5y/jlGLxI5+g1z35iYc9L3LWAaqtGA2y6UNyin3Qi97lP5qICq46v85IWppsbmZN1nK0Uli1hVaTiqQn2tu279K8l/vBAKnIEMRAPcqwjU5MifVQbgh1JXy1czmQxOfnmoPB1qEIHc1ydf4l45+0TDRHOR5tHkXQSnQcb+SfUwGtqA6qOWMudWcduXbnUKi80uvlawtBr6JZe3dC93+QmskKgjEZATxKE06feS2krHSC4ff7rcbwY5s7rVqwh6YxJ1bAk6lVJBxK8PX3uP8TcJOLtmzyHiyRVmiWwwwO9UJsjbziBST/w== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) by GV1PR10MB6121.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:150:9b::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.28; Mon, 3 Mar 2025 13:49:18 +0000 Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec]) by AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec%6]) with mapi id 15.20.8489.028; Mon, 3 Mar 2025 13:49:18 +0000 From: Stefan Koch To: bitbake-devel@lists.openembedded.org CC: docs@lists.yoctoproject.org, stefan-koch@siemens.com, simon.sudler@siemens.com, jan.kiszka@siemens.com, alex.kanavin@gmail.com, richard.purdie@linuxfoundation.org, quentin.schulz@cherry.de, mathieu.dubois-briand@bootlin.com, Ross.Burton@arm.com Subject: [PATCH v5 5/5] tests/fetch: Add an additional test case to check whether the fast fetch is shallow Date: Mon, 3 Mar 2025 14:49:13 +0100 Message-ID: <20250303134913.356778-5-stefan-koch@siemens.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250303134913.356778-1-stefan-koch@siemens.com> References: <20250303134913.356778-1-stefan-koch@siemens.com> X-ClientProxiedBy: FR3P281CA0097.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a1::15) To AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR10MB4959:EE_|GV1PR10MB6121:EE_ X-MS-Office365-Filtering-Correlation-Id: e87132e4-53bf-4894-606c-08dd5a5a31dd X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014; X-Microsoft-Antispam-Message-Info: jNdr0y64fQanuKV0H41xrzEH0LdpyMgUK9d1nm/6v0K0D7/PEYiJSohDMIqS1U/9GUnkxOEROGMDgNvOJBLeG4ZWOBXBlv6YVWvMLFBwdoxwjYvHRMYKLglKk1fyWe5FHhna1emn7yZhRKVCTdEdKpbBJNQeL9Z+g+7XWMLn69J9/YFOYaj5EHVh/ZRDpYO1hbKELk+CKmgqNyZUfYrkfU1r0+ZBXT1kRclFXGSXfT3HZTNcjyGB4iwgY1cGCl+9SKDHxvP/CyBmieV7fRgApISXgpzMt71wiBEAzgOIpQxBcNP675xq1AR3j2O/UKk6bNHdmZIL8AzYTG8sVKIFbM3pDjMROu1rc8oH+c2501N24rNtlDAWlhcS1YB4EwP4KRFOwpfAhtLyyraGKP3JVjZpe2s8Y2ybCht6KIHjDJKLT2Z/FTgW9iMyaKbKLK7xSYZN57PUC6AxrfJhE4LF5ij6ykW1n9SwRv5gXsOH4Wr0wGcpPeaPUz86rJJ/AtU7MxFnn0maZ5UVuS3q9KzSZ/W/y3EqPQ3QYgII8krpmiU7l3ilYwG9WxyEgx1+IKC/FoebblROQlav28sk+QkQm/y/E5F2S++7cBRX+AW1XEA+7N4+EdYEI7AVtbuT910a6iq4aeskv09BuHfrF8Uxm+6+JiMagmYbN2hanpYiy/3vDGechBey0oiFjNBDOWk9poh031XIZPymOybHOz3pnK9wSbpk+nEwsnRzJOsOIVbH/yqmP4V1r976j0mxSmpTAamOKgFfkv8FugHbKXfEG5Fdd0ZPiAaqyEJ2Y2MxnK04WYRS/3BTHxIaw+cf8tDHJEod+41Hgx+19g9/HgWxFnSAzDPbx0uTt87tQz+nDmGzTXHet6Ts3KEPLa0o+NoQg7h3t4F5L2qxMQPMywC9wVUukDz35HQNuzXrJ6Ih120jAQJokujPslxrqeJLBipFyaQv/ZYKzEzDjQEFIGJp49k0snndTYtRhNDLrKKyMcLLQGtkp6FrK4L0MfEXmLity4Gt0rb7/JKaDB9yyaC050tJTHavBiNEO9BrrfgyB2hIsz5zypxXymfpj+yiprnrzdCnULYXkhsCc3FFOkEkjS66WLKWJMz63dPUDUbtGZTuIzFqqVd6lJUIwBjPEiuYUYfDoR0S+SqB++vz75FkNDgCR9/MgpLDwzODu7k4Uvqi2nleR2Lo6YCfVAgJwQWDpf0olb/rtOlJQv4YnNBncfVIU5WUgHBenXUk8Ox7ESuTp/sY88G9qTep8QlXAI55m49tQnAnVBPQTSfilEjBH86MT7/bEFYd28gzNgvfCnHJ9rbbmmXlzLaIz3iOfnJa+kO7GFeFUTuf+eZ4KLx0N3N+lG6u40yQSMCxZCFwBneiaEGrpi7pSG37mrjBfTuN X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qljkU6ScInCsSYPYZEEOeNRpCMFslwcwykOHpx7KygrpR+ZS7sG3lGAqRay2pH2EAxVEI+a570qERsoATpRowk4UJPtXxLoFCyZOuyd/sQgRUfUU5LJ51le2Bb8tznPv6LpxZ1OjH7rEb+bOTm415qoJr76sThB8UBenGMnazIhvtbit86Ole5+h4pHfZQnPAAs3z5ZK9MNXNwe8zV6fF+NRbOTF9TLh8mLYeVx3p4qebIuTHZqdMmJgF6bCqOwuJH/G7z6v9/NtqUmLMHoottqbgCcVFw48wImsTpd2iLLczQ9ZaA+6EQQG2KZldM4YONWvXqkNym65x0oyyaSphZEBCv9HzgGZ6CJvotUatrboaAuVhVDMAs+OjzV8QzEVzTmGLpmzBIFZCB5M0gBmW5RJ+3jVoqp5Zsq0kqSo4WgiNqkAdpOhtNwKMkuZsHY/K26fHDS4w+8iMkHBH+hgeS4RLMHX9JOcYbZKoECeij0amWozF75upVx1Z9nlli5XL6jyY9Oydp8QXuFqkTeYQd1PEocVZJdRcqT8B/nmYmzJg16OhCYZOmVS+pktZQLhY1aTWQPX6twhUCjeTTn4VV0EaDzEp55L8RYkaQ6Gz7RcfAcyKZeZO3S/uiXPkP7uwBe6s1xj3eLxAAKsHtSRaHVBPZZ/1NI6JpOTNTKxegDGmksoAUwlu9HT7OvWS+LwKU8RC4YocmG11obNtX7iP12QK97UfhB/P+xxXOHEGpdxOgtDs9SfSFKuj6pIbHsbLdkh0uJwumiq1clLcFSYB1j3t3RnxecyvCpsBehmMHoFU6EiqGMVul+CCfgGKj0+i/FipwAd0rusxSF+WpvKAfcOCs+gF3kWCWErkTqQvL+v/deCwljtD5nTmTW1XxSc1N5YT1KQWa3lIUSz4/jldiRP60TabvqGxjisxrRxeAfeDK8YM7tWU0JK8sOicujscO/RWXbHpAIThnlZO41Efmx7CWZUsDLpdrfpMVNk6jiCEcHxpf9X8GR6e/EGVXdQOZ/ZP24PISb3oSJS/1UgKRdeDaftHifh1uSWxmK5ouHWAjYRvzc0wdKyHNtiSidRLdOry4NZEBupMuigL19T/nPR34e0KoduKLqYo95K3udDe8hYoYsFITEtVAodd0CYiFXGuoNAuz07wG9c6ZCfPzwv/NAc1n6umDAcr3d7NJEXuHQsEvs5ctPupE5SojFddQhdx66lURtZfB4Psq9lDExnZf7j3cJa1wBpyc1fjeq8vvQkimZ4InDbE4JeTyNk8O5EVe6CORa6f0uXMKngCwB8m9lM6/nSPCq7TMxOoAJqseHauRvi8fhi27DXtQ/qTohEWw872grB4i1jyLEfVQrrs/ez7A2Lv6alMuGSFdW88daGr10NI/gdNf3oPwPuBHzqFn08oWU2fUtLYSW4VqQaZipoDWHd6/U4tqw9Z4GMZO4yrGZCZqz91W1GQPsBFMser2iqFMMPbB4oHI1zYw9MYBV1LB730p53DuS/LBNnb9dNP3LFjzAi9EEG7DMhZiZyi28R90oaBVGaVP9Q5nOBo3lkXVBmqszc5WXXLwFbuBYh9vhKuTMOy2C5ifprSkEmFz08UjfzSoqrVVQxdg== X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: e87132e4-53bf-4894-606c-08dd5a5a31dd X-MS-Exchange-CrossTenant-AuthSource: AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2025 13:49:18.7647 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6199MfRC2h0PDYuqPJiNVDmMgXh+Z/7IwPcI+mqkc1NpJ/ZrMVcjvSl8SQClbgEjxu7wj30WqyUaTkXzSRKNtw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR10MB6121 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 03 Mar 2025 13:49:25 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17379 Signed-off-by: Stefan Koch --- lib/bb/tests/fetch.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 4a5af7106..c19180aaf 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -2302,6 +2302,18 @@ class GitShallowTest(FetcherTest): self.assertIn("No up to date source found", context.exception.msg) self.assertIn("clone directory not available or not up to date", context.exception.msg) + def test_shallow_check_is_shallow(self): + self.add_empty_file('a') + self.add_empty_file('b') + + # Fetch and unpack without the clonedir and *only* shallow tarball available + bb.utils.remove(self.gitdir, recurse=True) + fetcher, ud = self.fetch_and_unpack() + + # The unpacked tree *should* be shallow + self.assertRevCount(1) + assert os.path.exists(os.path.join(self.gitdir, '.git', 'shallow')) + class GitLfsTest(FetcherTest): def skipIfNoGitLFS(): import shutil