From patchwork Thu Feb 20 17:27:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Koch X-Patchwork-Id: 57663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4152C021B1 for ; Thu, 20 Feb 2025 17:27:27 +0000 (UTC) Received: from EUR03-DBA-obe.outbound.protection.outlook.com (EUR03-DBA-obe.outbound.protection.outlook.com [40.107.104.56]) by mx.groups.io with SMTP id smtpd.web11.2757.1740072443352419987 for ; Thu, 20 Feb 2025 09:27:25 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@siemens.com header.s=selector2 header.b=eHON2IHM; spf=pass (domain: siemens.com, ip: 40.107.104.56, mailfrom: stefan-koch@siemens.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=f/fXeOv4hXb3GjCkFLOmU6QRZZrUWWdpfukQV5MK1KW1ry7Gi5gDQ5f6dB+LYgIUfKWsiBT9ArzClxPz6ObpKnjX5GsE/zZb0eH32XSmXVpR1PuG6DO4Tgrz7DlKwpmo1B6ga33fvk9W9UCVXWhBvFSZ4FP6BqbN7ln5hcS/1JWe3agL6XOTzXQG4x71NbJYY0LbV3RI9sEPuE6V284XAmXi3COCeqhBDVMbPe5390rbzV3HJROJ0FHYJcaWxcGwP8W1+0Kh+PXell4Twvk13Hug0KamH6SiZydp/GExPZOswvJH9mtuR09WptCa5hVpKBnbwCStFkaKWnfFtPrB7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UsxrczEO4YFetLHXnXNy5nZF3OU81aT1xLFybME5Ldo=; b=Bet/XgzUp4UyA+KisszW+9DeeJBh9e60BSB7aAxGrXOXMLUy6jYWRiWTKKDleIB4W47rvRowMM6oidsIycmHCcUUh9UOpl5HfSIJgcX2fKLGhNlO/SXkWxE71utziWZA0Ul09kPxV9xi05REQWbSkSl1JAOqN97zhfHfCVT7d7li04hg4U9YR1Vtr1vgexJWhmSWM11Rn5XXJBsNJLrxwPejWa6pICfl8zBJhjEjZ01Xf3aES3FFAaGkExGTP1aOKfBR0+1ue8UmHO+pG0X5/4b2pW7jTmcuTiXNFVCKAk02T+MFBPHJu0K1UuJgiH+309LNmBa2JiPwFfaBsOJ45g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UsxrczEO4YFetLHXnXNy5nZF3OU81aT1xLFybME5Ldo=; b=eHON2IHM1XZNzuvtV8KQku5+QEX6Pz4jMCB2pcuE9BDCjhV/XmSrJGQMH9MQW8r9P3fEkO+hP1sWJpx56UPZncHpDwnnxlB6XWhxlYvRw7yrhatrxqbIinvR+lW/kIKN06BL8ABOcA77l0YKYABjKcmTb/w+yBRJw4v2Pg2QSLZMTbK1ZvVmz3L2VJSOo6K+Ja5w/Jv0OyBhUnCCra7Ztjfi0x7gRVzRLmmBOmcXOLW18SPztWqEf6aE7FVvwKzFq0+rkGR6QBWA6eoFKtkgbbnj8AsqM3ulAcV4HGkSjsUu/LEEkp05ipAw4IO8UAjD8g8cHDpGB5F7L8p4lfqHIA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) by DB9PR10MB5551.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:30a::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.16; Thu, 20 Feb 2025 17:27:20 +0000 Received: from AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec]) by AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM ([fe80::71d7:e998:3abf:a1ec%6]) with mapi id 15.20.8466.015; Thu, 20 Feb 2025 17:27:20 +0000 From: Stefan Koch To: bitbake-devel@lists.openembedded.org CC: stefan-koch@siemens.com, simon.sudler@siemens.com, jan.kiszka@siemens.com, alex.kanavin@gmail.com, richard.purdie@linuxfoundation.org Subject: [PATCH v3 3/4] tests/fetch: Adapt test cases for fast shallow fetches Date: Thu, 20 Feb 2025 18:27:05 +0100 Message-ID: <20250220172706.3850722-3-stefan-koch@siemens.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250220172706.3850722-1-stefan-koch@siemens.com> References: <20250220172706.3850722-1-stefan-koch@siemens.com> X-ClientProxiedBy: FR0P281CA0165.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:b3::13) To AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:41e::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR10MB4959:EE_|DB9PR10MB5551:EE_ X-MS-Office365-Filtering-Correlation-Id: 0df2251f-a017-466d-91a7-08dd51d3d430 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: P68INBlnrXj5kZnNzEJaLrD7hiONf+kAJym/FedJpahQl2A33irtuH2YRX9l+zXLZgdEJNSMqYogCj4zgMVcFGV3hfK6PO/kE7HwoGYtN91uoIZ7Zssyvhp+YB1Hcv1QVzhy6WXe+DSXrY8swzjdD3wL77mf4JMm3FpQxzG6qNmSc8Uu7zp8Q/fi0TJzHTAaO4ylNaD5jEqVwddyuyOOIjaVG/SqJ98GvE6smRaUJi8lDq6er2IrOvLxsiPe8yNNv8YhhRibCphYzujxkmyJHAxORfzKkVFOtzsSQL1kMQkIOKXgPkt+XIggwZwsQusC3kvzeUrHG56FoHDxCX0iAZt1MXPUCpkfbQg5RDjw4dV61yhLzbB0kbDAneC5XzB5/H7bTkrGfis07FT1Lj3w+iGtf/2jaT+NMz8xKRNLrFStKAxG7hz+S0C22mROcaPGjTpr50TUg18G5NxAVjkv5HyseNqYgBKLEIs2yixJ/03z1gOPefcnmrmORTPcVXLKpDwLClVSf5d7dGPHlElv4OMA/ghmdR6YSbfxAlf9nw8hN2sX3WugvP6K0xb/iUNdXCj8ltGnqrrUq4wrRMppXqcOdccEUHlQKcOE19ajslKUZgfXqoj4O6hJEflTczlri1zsCBsiTX22Ggd2fDAEh/u8DYDugWRY2B+wi8s9pqozCwe7PEGlD4If7/GLsP2psO3GAhPbPRQ6UapionlWO3jkgVz61U8SqDUgHg0tV1kCjiPCx/XDZpLxlS6dJhqbuys6JN08MIpHt6YColPbceCwXmUIJ9ohiPTBFhPSopOLZsDbpAeyWtX7bLqyuBWM+lBZk4EnAL95NcAYjDM10KdbHcCAnzQFbzp+/WFEoTbtOh8/RLGgDDoTImAcJIS+iERGkZN37hPrNBRLOECsqHJvgs0qYnIE+0fs59Z3f7L/LSHssAyZ5je5ggdBYhHYjDVGg1m7tEIctpukkKlFWktsSNPRHZbK3ix9UMqYuRzxtzjdZWdwljS05hBq+DxMPkMaGt4KDbtmoJADjfMYM9XqZpitj3tb//MrcvUnGdaKtRfc5hizB4F6KztLEanwfjHDh5sGYihW/J7cEiQTTmkVaaZD7I3141UI2vlm6VAzRcZD054fy2wE26PJ21H02j9bdxQzmRzT3PWtBK+zanoqnWofYmBvRphtJbKTZqHxGcBepylbXI6ccYkvqFGzAybxhL1BdU6ifMjPj6yTN+eprV4EgFJrKbf7JOT3C55pLXMvCcvqtC6N0zTyAOZkh2pJkpEe7rHvpXbrmbj1J40G5MvPIZwEuSgSoE17mK86TX/JZ0//TY0+CMryQahU5OetAcIDgODNTHrOdQH+StI0qSxIvt0TTF6GeAD/agfjXJ3FwdTdQfKxyAjETPdR X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: oQq6s5ozXYf7MROHnOAkPzyYJ/emxkzkIuF/gLVtdBsZr+9TYvreVhOkyn6oP+UiESVeC99AZBLUvcFzZv4ALk4yd02xA1mw9lzlG73TWqlNtVQwOM33pcs+E1+hrvL+rm6e23ITukwib+CcDpgQVfBrTXtpe97yJD9CmoPhyVa8OyjvvuKFsAw+eDyzaQxEYvTVStOiLstSUI0WF3HMv79T6z2rvB9+8PmtM+s5p1ciCDBt3ZejI1qRCz6yn/6yw325UUWOTD2sIp81NzbI1EsFyQj2NposLbhj1smG7e1uA32DGawhhoIew3vmOhffsd3nWcWyWbpj0u1WA36/mU43UI8lp+qLwtCIB6r3uxHZQFThbihfJFRVV038BGLleQ3d+7yqOW55BtB0+4Seb+iPiQPriv6eZ9rADzvNVWPE621nxXN5IlNduEtR6uEr59dREzFq3GzwkRegRH4vuLhdtI+V4ZfryCJ5m7xJ80IlzTqqAzw5jq02UtyGaF46NBen0uzJNvBLshs9Gv6ennl7wdg9r+8jC0Biu88jUrIb0daFLkfeawW/dmQK7O1+pDthjAICWLjbqxrtBgCUQAe7Q/EcxgPds3cpm6kse1i1x/LdXpgozglTkjXM2Wphn6lL2rVombXvQEUQwl1f0tg9DsO1OmQf+KNJ1x1pcZsZYwQ3LjCua03GX3+qIADGUwYg4o3Qi2UJIjVLoRSNjgoDnKcCx5AKNIHT0amnhxDW0yPuNorG5W6jq2CBYe3oT9r6RBS/CRL+gkZ3yDxYoxuq1d2wIdeBnCiPXUVSqXHN9vPtaJX5xEf3mlY3/fCQfxPReA4dr0cnRmtz+vi1/FIcw2kw0STJ11cTGUhP7LZYk0w3fPJcyM3QtzDNAjRHEHyt1AkIf1cqz8OKmJpENoyc08lbhCV0bdCNvmRg/ilnGXxedxZiqNQbSb3PatwQ5mKbt+Emuq9icmC+mL5Kx95OuF3/h/Cj/xhrDT0ZrVosuwWOH5DXKADAkN4CJJx3Zp1i4eyzzMUM/OY73Jx1OOj1TtFUt2p5S6j1NEUFZFdpR7fOkS6r89Tx19aaGsfFNmZO67gGYLOksl/OeRv0h/aRo+/HvQmcWLk2DuUCvvOQAnIniUkPKWA1zj63NLoIZeSBAL9Ff7xTv6uQbMSm29rfIayG+lYNdBLK3bD+bHxBqaZV7leyAk9x6+lEyc72Zp+xpSUx1lfX9Jjpg8Ii5NFppR0dHMCLBlQbRLKEL775dFWgogYjsCoZnJ3h/i+hM2KZn6bP9uLXbj2I/DZuHpMekel5G5Bpu0NCif4sEsfp7kDbx+DWJYGVIpbFwzBu/gxCpBlNO2ks45Ee+ziR+yDgcFlL2zOFSoPdSYvDZMbrDhBbiS6jASye/Bc2KnrwoNCgKjUkCjYAFCpyZBoy3aOclLBl1OmDj/CtJwr2NFcSo51gZWUiOU+xYxcNdVztMhCI5xemDIn5x+T6BiaRttF+gxUmpjPi0T3NNhs081OmYQAbyIqbDm1J7tRVxZo6vidWNHamOOBlUpf9UgIXAYx0/fPJ4YebQKvQUnYl0FNE9TdhB4aTKP21J6wXwd4TBJUf51KG1261sMeE5SyToA== X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0df2251f-a017-466d-91a7-08dd51d3d430 X-MS-Exchange-CrossTenant-AuthSource: AM9PR10MB4959.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 17:27:19.9392 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +jRvY70RiLUJXyGltDvJQratsTHyoYi3ozYzS0t/RIAH+nWKSQboiPI0rGhz0PzgyYwShFQ86Kq6x6C2PI8XBw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR10MB5551 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 20 Feb 2025 17:27:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17264 - Address the absence of an initial full bare clone - Utilize the initial shallow clone - Modify existing test cases for this behavior - Remove incompatible test cases Signed-off-by: Stefan Koch --- lib/bb/tests/fetch.py | 25 ++++++++----------------- 1 file changed, 8 insertions(+), 17 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 48b9e4af1..c0b53da31 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -1814,7 +1814,6 @@ class GitShallowTest(FetcherTest): def fetch_shallow(self, uri=None, disabled=False, keepclone=False): """Fetch a uri, generating a shallow tarball, then unpack using it""" fetcher, ud = self.fetch_and_unpack(uri) - assert os.path.exists(ud.clonedir), 'Git clone in DLDIR (%s) does not exist for uri %s' % (ud.clonedir, uri) # Confirm that the unpacked repo is unshallow if not disabled: @@ -1822,9 +1821,6 @@ class GitShallowTest(FetcherTest): # fetch and unpack, from the shallow tarball bb.utils.remove(self.gitdir, recurse=True) - bb.process.run('chmod u+w -R "%s"' % ud.clonedir) - bb.utils.remove(ud.clonedir, recurse=True) - bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True) # confirm that the unpacked repo is used when no git clone or git # mirror tarball is available @@ -1907,7 +1903,12 @@ class GitShallowTest(FetcherTest): self.add_empty_file('c') self.assertRevCount(3, cwd=self.srcdir) + # Clone without tarball + self.d.setVar('BB_GIT_SHALLOW', '0') + fetcher, ud = self.fetch() + # Clone and generate mirror tarball + self.d.setVar('BB_GIT_SHALLOW', '1') fetcher, ud = self.fetch() # Ensure we have a current mirror tarball, but an out of date clone @@ -1919,6 +1920,7 @@ class GitShallowTest(FetcherTest): fetcher, ud = self.fetch() fetcher.unpack(self.d.getVar('WORKDIR')) self.assertRevCount(1) + assert os.path.exists(os.path.join(self.d.getVar('WORKDIR'), 'git', 'c')) def test_shallow_single_branch_no_merge(self): self.add_empty_file('a') @@ -2116,11 +2118,12 @@ class GitShallowTest(FetcherTest): self.add_empty_file('b') # Fetch once to generate the shallow tarball + self.d.setVar('BB_GIT_SHALLOW', '0') fetcher, ud = self.fetch() - assert os.path.exists(os.path.join(self.dldir, ud.mirrortarballs[0])) # Fetch and unpack with both the clonedir and shallow tarball available bb.utils.remove(self.gitdir, recurse=True) + self.d.setVar('BB_GIT_SHALLOW', '1') fetcher, ud = self.fetch_and_unpack() # The unpacked tree should *not* be shallow @@ -2295,19 +2298,7 @@ class GitShallowTest(FetcherTest): self.assertIn("No up to date source found", context.exception.msg) self.assertIn("clone directory not available or not up to date", context.exception.msg) - @skipIfNoNetwork() - def test_that_unpack_does_work_when_using_git_shallow_tarball_but_tarball_is_not_available(self): - self.d.setVar('SRCREV', 'e5939ff608b95cdd4d0ab0e1935781ab9a276ac0') - self.d.setVar('BB_GIT_SHALLOW', '1') - self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') - fetcher = bb.fetch.Fetch(["git://git.yoctoproject.org/fstests;branch=master;protocol=https"], self.d) - fetcher.download() - - bb.utils.remove(self.dldir + "/*.tar.gz") - fetcher.unpack(self.unpackdir) - dir = os.listdir(self.unpackdir + "/git/") - self.assertIn("fstests.doap", dir) class GitLfsTest(FetcherTest): def skipIfNoGitLFS():