From patchwork Mon Sep 14 11:00:44 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fredrik Svensson (svsvenss)" X-Patchwork-Id: 98179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F8ABC88E65 for ; Mon, 14 Sep 2026 11:01:14 +0000 (UTC) Received: from rcdn-iport-8.cisco.com (rcdn-iport-8.cisco.com [173.37.86.79]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.17718.1789383664290278839 for ; Mon, 14 Sep 2026 04:01:04 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: message contains an insecure body length tag" header.i=@cisco.com header.s=iport01 header.b=SNnZ9E1R; spf=pass (domain: cisco.com, ip: 173.37.86.79, mailfrom: svsvenss@cisco.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=14073; q=dns/txt; s=iport01; t=1789383664; x=1790593264; h=from:to:subject:date:message-id: content-transfer-encoding:mime-version; bh=15/qNEx/pYhmCXkaYzBXI0Ln2U/ug9iehx03AYdjg9M=; b=SNnZ9E1RjSB1e9bTbq6z0nyAbBLpJqv1mi86OQswTZTEdjX8IUSNKb0w 3NSOuwkjcSMswIR3tWUQOu6UYT1w47ve0UImiDOB2LGRQlEo720ePtHjl T7rn416mySFn62pp5ZN0X7SMTdhuuE0nqtdBc6y7NCpa9SgPT9VJ4Vq3b 7Ld4hGZ4I66+ELJAHYzQI+6lFvrA1bB1v0Nrt/5csSOZ3mAuQS7GyLneI TBgZ+yblyWPTxgrUb9m312tAul4ZGfqpxtgsl0/i6jMaFmmV+M+aVmaFf oJNuwIVfFLAlNE8iKnEGu3iYWuncxAhfqvv191MVT0zFhwxEwzYMjLtCr Q==; X-CSE-ConnectionGUID: XcmzGl2gTLiPJ+NAPL8fkA== X-CSE-MsgGUID: lXC5b1kOTVa8vZcY3NC8Jw== X-IPAS-Result: A0ApAAB006dq/4//Ja1aHAEBAQEBAQcBARIBAQQEAQFlgRgGAQELAYFtUxZzAoEjSYgjA4UrhliCIYEWnQiBfg8BAQENAjEgBAEBhQUCjgUCJjUIDgECBAMCAwEBAQEBAQEBAQEBAQoBAQUBAQECAQcFgQ4Thk8NkSQBfycEARIIgnuCdAMBArk4AolLGzd4gTSBAeAwAQsUAYE4AYhdAYV4O4RBJxuCDYEVQoIxdoRFhBOCMASCIoEMgVqBDHEZfoVVA4lLgUocA1ksAVUTFwsHBYFmAyovLTI8Mh2BIxIsFy9YGwYFgR2BJ4JiIxk2eoEJXoErKWESF4EJgggCglSCAQIBSUMOB0VTCSdBAwcSTCkiCBIGC209NxUZAwSBNQWPBh+CEEYHAXMQCgErFzlBBwhMEUIPGBSSaQiQFoE1jGSVFwqEHowilXAXhASBV6USmQgjgjahE4UsAgQCBAUCEAEBBoFqATmBWXAVO4JnCUoZD44vFYlExjV5PQIHAgcOAwuTZgEB IronPort-PHdr: A9a23:Wzto7h+JfZT6kP9uWBDoyV9kXcBvk6//MghQ7YIolPcUNK+i5J/le kfY4KYlgFzIWNDD4ulfw6rNsq/mUHAd+5vJrn0YcZJNWhNEwcUblgAtGoiEXGXwLeXhaGoxG 8EqaQ== IronPort-Data: A9a23:U4EWX61ke42SoPaosPbD5Yxwkn2cJEfYwER7XKvMYLTBsI5bp2AGy GseX2yEPK6JMWCjL4tyO46+pE8O75fUmN8xTFdt3Hw8FHgiRegpqji6wuYcGwvIc6UvmWo+t 512huHodZ5yEzmE4Ej9atANlFEkvYmQXL3wFeXYDS54QA5gWU8JhAlq8wIDqtYAbeORXUXX5 rsen+WFYAX7g28tbjpNg06+gEoHUMra6WtwUmMWPZinjHeG/1EJAZQWI72GLneQauF8Au6gS u/f+6qy92Xf8g1FIovNfmHTKxBirhb6ZGBiu1IOM0SQqkEqSh8ajs7XAMEhhXJ/0F1lqTzeJ OJl7vRcQS9xVkHFdX90vxNwS0mSNoUekFPLzOTWXcG7lyX7n3XQL/pGPEozL6IcoOlLJWB21 78+AQgpPh6ArrfjqF67YrEEasULNsLnOsYb/3pn1zycVahgSpHYSKKM7thdtNsyrpkRRrCFO IxDNGcpNUibC/FMEg9/5JYWh/mzmHT5aCZwo1OOrq1x6G/WpOB0+OW0bYeOIIzQGa25mG6cl kmb9T36JS0kPYyB+zG0qVSH1s3QyHaTtIU6UefQGuRRqFeS2mYaCxBTWValrP2Rjk+lR8kZL FQZ/Ccrp6U++EGnCN7nUHWFTGWspBUQXZ9UVuY98gzIkvSS6AeCDW9CRTlEADA7iPILqfUR/ gbht/viBCdkt/ueTnf1y1tehWja1fQ9RYPaWRI5cA== IronPort-HdrOrdr: A9a23:30NqwalWsZgUSwuKPrCEFLZckQzpDfM/iWdD5ihNYBxZY6Wkfp +V7ZcmPE7P6Ar5BktApTnZAtj8fZq9z/JICPoqTMuftW7dySaVxeBZnMHfKlLbalbDH4JmpM NdmstFeZHN5DpB/LzHCWCDer5KrqjkgcWVbIzlvhFQpHRRGtldBnBCe3+m+yNNNW57LKt8Pq CxouBAoD2tc2kWaMOUOlkpNtKom/T70LjdTVojHRAI1Cmi5AnE1JfKVzSI1BYXVD1ChYwl7X XElALB4KCiueGQyxPX1WPfhq4m1+cJzOEtOOW8zuwubhn8gAehY4psH5eYuioune2p4FE21P HRvhYJJa1ImjLsV1DwhSGo9xjr0T4o5XOn40Sfm2HfrcvwQy9/I9ZdhLhebgDS5yMbzZFBOe NwriGkXqhsfFb9dRfGlpz1vtZR5w2JSE8Z4KwuZrpkIM0jgfFq3NUiFQhuYeg99WrBmf4a+a 9VfY/hzccTV0+GZHbEuWQq6tmtUnMvWiqieCE5y5SoO/w8pgEi86Per/ZvxUvp+P8GOuh5zv WBPaJymL5USMgKKap7GecaWMOyTnfAWBTWLQupUB3a/Yw8SgXwQqTMkcMIzfDvfIZNwIo5mZ zHXl8dvWkue1j2AcnL2JFQ6BjCTGi0QDyok6hlltNEk6y5QKCuPTyISVgoncflq/IDAtfDU/ L2PJ5NGffsIWbnBI4M1QzjXJtZL2UYTaQuy54GckPLptiOJpzht+TdfvqWLL3xESw8Ume6GX cHVCibHrQ10qlqYA6MvPH8YQKeRqWkx+MBLEHzxZlh9LQw X-Talos-CUID: 9a23:2NpAFG7aLLTDVvqtxtssxEURCO8hKHLh6HboAQyeIF5NGLOWYArF X-Talos-MUID: 9a23:p6oK9g779IMN4aOEw2MNZNhaxoxs04/zAhkOtqk4spmaBydBYBG23BaoF9o= X-IronPort-Anti-Spam-Filtered: true Received: from rcdn-l-core-06.cisco.com ([173.37.255.143]) by rcdn-iport-8.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 14 Sep 2026 11:00:47 +0000 Received: from rcdn-opgw-2.cisco.com (rcdn-opgw-2.cisco.com [72.163.7.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by rcdn-l-core-06.cisco.com (Postfix) with ESMTPS id 56AAD1800048C for ; Mon, 14 Sep 2026 11:00:47 +0000 (GMT) X-CSE-ConnectionGUID: JNgDpe21TBiaNt7dLxV/GQ== X-CSE-MsgGUID: ubjoz9p1T7WMzozJfDU1TA== Authentication-Results: rcdn-opgw-2.cisco.com; dkim=pass (signature verified) header.i=@cisco.com X-IronPort-AV: E=Sophos;i="6.27,102,1787011200"; d="scan'208";a="73699261" Received: from mail-centralusazon11011054.outbound.protection.outlook.com (HELO DM5PR21CU001.outbound.protection.outlook.com) ([52.101.62.54]) by rcdn-opgw-2.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 14 Sep 2026 11:00:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Oi0whcnCg/5f1kuIglL/5ETWMWsWZaDRQHCFpRpoVddQnBOP3uTKBxAXVp7hz7dRjrMGv/TEypXKIVz7mG+WfPYWOOSc2NVdakNoGbFBdsH+qjayXe8bkY4baT1DrmYtZvptbnFUsJicgfmU4LqsO/ATjJCkf9YiBjsrmh7enICxiIYk3nnjb6/6gZkYNBYFaU7Wbw9qenVGMmdafq2V5pjQauLow1oipg+gHsbYEk6KiUrdGdKqTJGg+dCbIwj+7nSnDNRxhknBSFsJsT3yAsnWHk1It+wrTv9rI09W46A5O4emjsOyJfxAlhBHwrG1tXoZlZTXxNDjFoaSS7Fihw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=15/qNEx/pYhmCXkaYzBXI0Ln2U/ug9iehx03AYdjg9M=; b=IqndABryLuNZhTZayyL6ly083ZRS/+4h2bHKKxafmLFWfr5lfiT/CGrHKzKuP8SKUMacRItabO78GWHp6rWq+WAAXZ5jv9R5e4/OCb+kXAdoxGzzw7fnp4MU5nsqRdhTHVkNkycvto4qig2ZJpPUKi6Ridt4zn4tnD5+Yp2utV9pTV77W2DlRJpaXQeuPso5RznzqH69Za5Mnv/Qz7el9DYrHgoGO+agUqVhybHQYexr0TeoT1LT44CWy7oAO8EdKCy4YTvuZZ5J7sG8c4kQ1Y8ke+FcYkG5EqQnguANwP5kWdfBGjS2ZRs3p7RA82tNG0lBhhQaweV+z9eGd20tzg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none Received: from DM6PR11MB3962.namprd11.prod.outlook.com (2603:10b6:5:192::26) by PH3PPF9E162731D.namprd11.prod.outlook.com (2603:10b6:518:1::d3c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.11; Mon, 14 Sep 2026 11:00:44 +0000 Received: from DM6PR11MB3962.namprd11.prod.outlook.com ([fe80::47f3:254:87ca:5161]) by DM6PR11MB3962.namprd11.prod.outlook.com ([fe80::47f3:254:87ca:5161%5]) with mapi id 15.21.0406.007; Mon, 14 Sep 2026 11:00:44 +0000 From: "Fredrik Svensson (svsvenss)" To: "bitbake-devel@lists.openembedded.org" , Richard Purdie Subject: [PATCH v3] fetch/git: unpack from premirror when cached clone is stale Thread-Topic: [PATCH v3] fetch/git: unpack from premirror when cached clone is stale Thread-Index: AQHdRDXrwr9uhxYuSEeZA19OEZk54Q== Date: Mon, 14 Sep 2026 11:00:44 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM6PR11MB3962:EE_|PH3PPF9E162731D:EE_ x-ms-office365-filtering-correlation-id: e8f13483-5ce6-414f-890f-08df124f6c91 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230040|366016|376014|23010399003|1800799024|38070700021|5023799004|11063799006|56012099006|10067099003|18002099003|3023799007; x-microsoft-antispam-message-info: Kdc2V0DyfEbgiySxMFbfwugHbAktUIV1vy0iabi3fuE7CWqsZkFfIcxVNkt9/jNx5RQXVuieNpDlrmsknwVNGaTLbOd4aQcGJa8Zz2yHZ+H5+PCJrmpkWAI9Nrt4GOgZzmDNptZpW363evB7f5bspZa6xKPNVKqUK2koKYmxacFCtbiY0Zm49+TLmss8ae44i+Kpm/poCbGayRup+fjrKFp+4FVT7YZj6VT9g+wLcuayXXBt447M2e5vCdXtJHJ2n687NHns72tYL9jagYK9CUAyVkVqWj43WzL3MwBRNbdcnUx5S3aOKfvHT/fa3G8rYCbhMRbbcSjW25B/0L/vlzPIyNGcxFzAWpIU+GuMdH6r42LjpUKhH4kEFL+4losoNEuW4BwZ+OVp4dDN5hAEHOcXxozteAXoZ7W1DLCQV1e6QQWBUnBuYUvrzPyUPpUwvhdkGRdmWo/2qgH7EllzAc4/rUSBS6u+iRBJ9QOz/g2Ymw98jk4GUUSV/ySe1F5CU/RFLD4FrNbyIVdaOHLnWlpie2VT7/HqyPbT2i2vAFh2Yn/WIJwNhimBKplmMZb/xDOjjTeLAljZ0CDv5PvaOKIXlGqyazNddEun2e1b+RZeACz0Pry985siuFUD45lkxfh6L8ZyC3LnJnna6GLpcO1ghiNIGrcPrQpOFIZ7U9h3lL0a5Vn9uvxfJJO/L4D5XK5ummQNmt6ofc51h2Z5CauL4hcsRcmC2lrkWCDE6CY= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR11MB3962.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(23010399003)(1800799024)(38070700021)(5023799004)(11063799006)(56012099006)(10067099003)(18002099003)(3023799007);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?X3jfX/HJajmuUzwf24GsHXS?= =?iso-8859-1?q?BSWtQRGXl8Qs+jZuEtotFPWx+9QD8rJKHrYxu/57S8E6zKDQpVtSqkm7FR2i?= =?iso-8859-1?q?y2OcNEzv1ahhUdJgy3rgdeMwNrlVcHT4nW1nU7Dh6zdPgwpHrZFIwFvMqif6?= =?iso-8859-1?q?iVHt0vhVkpJAdHbK5Bt82Nm8InMgJZvWTHbcgQWyPze22guZBHzQVc35HFfK?= =?iso-8859-1?q?G1g2vX+k1pgDtkeaT5NLg2XZ0FpsHeq/rc3/d7pivgm4q+sDmrwTLiIsKSuF?= =?iso-8859-1?q?sW1E77qGJxiRnmTEHnxgLEJk3xCH32dWmPndck1wE3cL00MTX5MvT5TlUHOu?= =?iso-8859-1?q?MqirVUJnWH4//xE5NwpHVfsGugoQa+tu/vXJ5HD3M8z9BGRGXa+RLUmRsvMg?= =?iso-8859-1?q?MU9qK5ATtTNPu80FKVDV4zl2Uno8k+p/ILCUVOPVBBhcNXk5ccQe+U6GFIse?= =?iso-8859-1?q?Cj3tUe4ysyeNjrNNzhK7hV8ud72omYTS5KaPXoQWI1D5WHW1/e4scrfQfKsG?= =?iso-8859-1?q?nrL3qxoFqb/nvwdx32OShnJ5zacnrS7ywswiOy0jEb2f6LNsoI0Oq6K6iHQb?= =?iso-8859-1?q?rSt+0TndDaUfBfJjmQcAGfxApEpqoYuV6SlzjMeFpMSsKoS1syuSeW0aLuHk?= =?iso-8859-1?q?WalklWMETPLM4U7xsVWDR1zVJ6faIry4HEkKc2kU/1Q5M3ARsSpBYejiZ9ql?= =?iso-8859-1?q?/NsUhkDFrJHQYDcJRoe0dNWjEja//yPFyvunrHpIVk97IqLKRMerOvseuSyg?= =?iso-8859-1?q?JW4zXD/Nl5+htyOWlvNUmCLsu4jqXk0q8C8klo3i58BlPlCw0Nkrd4UV3Tvc?= =?iso-8859-1?q?xhrDIWFN310Ppuk3xzVIszPbY1e4QGQ2W4gUd7z1fZDM66oDR5P6rnfZdJXG?= =?iso-8859-1?q?JgpJ8ysiaplTBGWEJWNAairMUiQOKVmMXoLzh6K7ZCHkc+DA3RoBh5VErjsv?= =?iso-8859-1?q?fCj0feMQXIYAmeNMBsVQuxblvW997Ctat8F1XzToeuojA9Ig/x1pHMCFLJRl?= =?iso-8859-1?q?c5dXoAJrhz1KmGiuDoKF0fJjd7YrIdFF7EaiZQDbTK54j3AoLBBkQMG5QiTo?= =?iso-8859-1?q?KP+yXApgKQdeOvb1zw4VuU2N+epNznmitXh6we27mAT4wa8pYXny7OpEFUu5?= =?iso-8859-1?q?xfYyHo03z24RBo2T58NOUEkqHUEzN8kzhzPnVNHppfV/2kJ2UZRZU5kpdnMi?= =?iso-8859-1?q?AYFHUfln+jC65fQZnYqNSuo6/x7xfWnAXfxoRC/71Q/LN/vehNVO690Qq4V2?= =?iso-8859-1?q?DKyZKnlceKgdm5jndxBDXMw6BRl6Odcubz+nxnqPU46o/KbVu3ft7+hQCAu2?= =?iso-8859-1?q?1W3CoNReui2LBp+W9iCmm49QxQvtpQ+TDoTqdANdh15a09GFsD+EdEUtMDv2?= =?iso-8859-1?q?jT6MEfUo8J94ABN1F5bIptwTx/DDXPul1NG/DiP3RcRpYpaDWIzfCDxQD0Ts?= =?iso-8859-1?q?otrHtKBAJfJRpzMBB1F04tueMWMMU1ZfQYZsKchHcQDEQ2ozdI8O76rBQlOk?= =?iso-8859-1?q?K94stNG8SNuuMSb/pCxCy1KoltxQlbL8z6vMD40SnNuza8CMMl21DEaMzsha?= =?iso-8859-1?q?bYWa4j8Bs52cpFYwLJyPzIvt6PJiIsflenNsJggRWwe3naZcPz+nBhB8tKpZ?= =?iso-8859-1?q?QUot7SoIqRS2y7nVIIJUtSJrq+6yptQ+P2bmu0iT3Uw2DORqzfXslDDobleb?= =?iso-8859-1?q?n5GH0f1s5ybfvZsWYtbpGabQvKWJ+zq+eu3fIRt/EREKFlZKwcWX2TTLWUJ8?= =?iso-8859-1?q?Tmo8ZBXv1xO3ZfPxFVCgMe57I?= MIME-Version: 1.0 X-Exchange-RoutingPolicyChecked: 5mLuFxpqV7rPsn73PaorIy0lzvOb8jqFyVy3O1aVa1xE/CykR29U3Huqpt5F4bpVvM3MV6QSeShuZCFX9P4K8tJ3OQJ99r5hIx2/uicLg/uaW7NfwZCmt1/VvMW/p5viRTPyvftTbVG0xm6jHzpl/vWceUe4Mft7TFiR1NFCA56oMtl0PH+qnyFMwy9jFjeAP/WMlXJ960lY+7QSR+vA/GADailE1onqnWyPCge3Kd4nXQ3ot1FsDVBImn/betc8Z4gh49u+Xd3ltwV+GrlIWMXhHayoU97MVeTvZqdpGAxAcZ9tTXlcl7HZ6Ob1Eh3pDD5dhAAMg0BQPcYTaVKSdQ== X-OriginatorOrg: cisco.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3962.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e8f13483-5ce6-414f-890f-08df124f6c91 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Sep 2026 11:00:44.2754 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 5ae1af62-9505-4097-a69a-c1553ef7840e X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 7dL1p2lFxlgppK0OgAHmhrhHCKHw87E4VcdjJngxA4gR+qXxEvSDhwV475pVgXPpYonpRNaHtAOPnR2fo6n7VA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH3PPF9E162731D X-Outbound-Client-TLS: ANONYMOUS;rcdn-opgw-2.cisco.com [72.163.7.163];TLSv1.3;TLS_AES_256_GCM_SHA384;256 X-Outbound-SMTP-Client: 72.163.7.163, rcdn-opgw-2.cisco.com X-Outbound-Node: rcdn-l-core-06.cisco.com List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 14 Sep 2026 11:01:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/20195 The failure happens as follows: 1. BitBake has an old Git clone for the original URL. 2. That clone does not contain the commit being built. 3. BitBake successfully fetches the commit from a premirror into another clone. 4. BitBake tries to unpack from the old clone and fails. Remember the clone used by the successful premirror fetch and unpack from it. Leave both clones unchanged and preserve the premirror clone during clean. Add a test for this case. Signed-off-by: Fredrik Svensson --- Thanks for the feedback. This version leaves the stale clone untouched. It remembers the clone used by the successful premirror fetch and uses that clone for unpacking. It also removes update_mirror_links. A symlink is still created when the original clone path is absent, but clean no longer follows that symlink, so it does not remove the premirror clone. The updated self-test verifies that unpack uses the premirror revision, the old clone is unchanged, and clean preserves the premirror clone. Changes in v3: - do not delete or replace the stale clone; - use origud to remember the successful premirror path; - remove update_mirror_links; - preserve the premirror clone during clean; and - update the self-test for the new behavior. lib/bb/fetch/__init__.py | 48 +++++++++++++------------------- lib/bb/fetch/git.py | 60 +++++++++++++++++++--------------------- lib/bb/tests/fetch.py | 49 ++++++++++++++++++++++++++++++++ 3 files changed, 97 insertions(+), 60 deletions(-) diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py index 55ab710f..4d5e7b0d 100644 --- a/lib/bb/fetch/__init__.py +++ b/lib/bb/fetch/__init__.py @@ -587,6 +587,11 @@ def mirror_from_string(data): bb.warn('Invalid mirror data %s, should have paired members.' % data) return list(zip(*[iter(mirrors)]*2)) +def downloaded_path(ud): + """Return the path obtained for a URL, including from a mirror.""" + return getattr(ud, "mirrorpath", ud.localpath) + + def verify_checksum(ud, d, precomputed={}, localpath=None, fatal_nochecksum=True): """ verify the MD5 and SHA256 checksum for downloaded src @@ -1147,11 +1152,18 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): if hasattr(origud.method, "build_mirror_data"): origud.method.build_mirror_data(origud, ld) return origud.localpath - # Otherwise the result is a local file:// and we symlink to it - # This may also be a link to a shallow archive - # When using shallow mode, add a symlink to the original fullshallow - # path to ensure a valid symlink even in the `PREMIRRORS` case - origud.method.update_mirror_links(ud, origud) + # Use origud to find the URL which started a mirror chain and record + # the successful result without changing an existing download. + mirrororigud = ud.origud + while hasattr(mirrororigud, "origud"): + mirrororigud = mirrororigud.origud + mirrororigud.mirrorpath = ud.localpath + if not os.path.lexists(mirrororigud.localpath): + bb.utils.mkdirhier(os.path.dirname(mirrororigud.localpath)) + try: + os.symlink(ud.localpath, mirrororigud.localpath) + except FileExistsError: + pass update_stamp(origud, ld) return ud.localpath @@ -1676,28 +1688,6 @@ class FetchMethod(object): """ bb.utils.remove(urldata.localpath) - def ensure_symlink(self, target, link_name): - if not os.path.exists(link_name): - dirname = os.path.dirname(link_name) - bb.utils.mkdirhier(dirname) - if os.path.islink(link_name): - # Broken symbolic link - os.unlink(link_name) - - # In case this is executing without any file locks held (as is - # the case for file:// URLs), two tasks may end up here at the - # same time, in which case we do not want the second task to - # fail when the link has already been created by the first task. - try: - os.symlink(target, link_name) - except FileExistsError: - pass - - def update_mirror_links(self, ud, origud): - # For local file:// results, create a symlink to them - # This may also be a link to a shallow archive - self.ensure_symlink(ud.localpath, origud.localpath) - def try_premirror(self, urldata, d): """ Should premirrors be used? @@ -1869,7 +1859,7 @@ class Fetch(object): self.ud[url] = FetchData(url, self.d) self.ud[url].setup_localpath(self.d) - return self.ud[url].localpath + return downloaded_path(self.ud[url]) def localpaths(self): """ @@ -1880,7 +1870,7 @@ class Fetch(object): for u in self.urls: ud = self.ud[u] ud.setup_localpath(self.d) - local.append(ud.localpath) + local.append(downloaded_path(ud)) return local diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py index dc1b31b5..836def49 100644 --- a/lib/bb/fetch/git.py +++ b/lib/bb/fetch/git.py @@ -77,6 +77,7 @@ import bb import bb.progress from contextlib import contextmanager from bb.fetch import FetchMethod +from bb.fetch import downloaded_path from bb.fetch import runfetchcmd from bb.fetch import logger from bb.fetch import trusted_network @@ -317,14 +318,15 @@ class Git(FetchMethod): or self.tarball_need_update(ud) \ or self.lfs_need_update(ud, d) - def clonedir_need_update(self, ud, d): - if not os.path.exists(ud.clonedir): + def clonedir_need_update(self, ud, d, clonedir=None): + clonedir = clonedir or ud.clonedir + if not os.path.exists(clonedir): return True - if ud.shallow and ud.write_shallow_tarballs and self.clonedir_need_shallow_revs(ud, d): + if ud.shallow and ud.write_shallow_tarballs and self.clonedir_need_shallow_revs(ud, d, clonedir): return True - if not self._contains_ref(ud, d, ud.name, ud.clonedir): + if not self._contains_ref(ud, d, ud.name, clonedir): return True - if 'tag' in ud.parm and not self._contains_ref(ud, d, ud.name, ud.clonedir, tag=True): + if 'tag' in ud.parm and not self._contains_ref(ud, d, ud.name, clonedir, tag=True): return True return False @@ -339,10 +341,11 @@ class Git(FetchMethod): return True return False - def clonedir_need_shallow_revs(self, ud, d): + def clonedir_need_shallow_revs(self, ud, d, clonedir=None): + clonedir = clonedir or ud.clonedir for rev in ud.shallow_revs: try: - runfetchcmd(ud.basecmd + ['rev-parse', '-q', '--verify', rev], d, quiet=True, workdir=ud.clonedir) + runfetchcmd(ud.basecmd + ['rev-parse', '-q', '--verify', rev], d, quiet=True, workdir=clonedir) except bb.fetch.FetchError: return rev return None @@ -353,13 +356,6 @@ class Git(FetchMethod): def tarball_need_update(self, ud): return ud.write_tarballs and not os.path.exists(ud.fullmirror) - def update_mirror_links(self, ud, origud): - super().update_mirror_links(ud, origud) - # When using shallow mode, add a symlink to the original fullshallow - # path to ensure a valid symlink even in the `PREMIRRORS` case - if origud.shallow and not os.path.exists(origud.fullshallow): - self.ensure_symlink(ud.localpath, origud.fullshallow) - def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout with only premirrors # is not possible @@ -695,28 +691,36 @@ class Git(FetchMethod): source_found = False update_mode = False source_error = [] - - clonedir_is_up_to_date = not self.clonedir_need_update(ud, d) + clonedir = ud.clonedir + fullshallow = getattr(ud, "fullshallow", None) + if hasattr(ud, "mirrorpath"): + mirrorpath = downloaded_path(ud) + if os.path.isdir(mirrorpath): + clonedir = mirrorpath + elif ud.shallow: + fullshallow = mirrorpath + + clonedir_is_up_to_date = not self.clonedir_need_update(ud, d, clonedir) if clonedir_is_up_to_date: if update and os.path.exists(destdir): update_mode = True else: - runfetchcmd(ud.basecmd + ['clone'] + ud.cloneflags + [ud.clonedir, destdir], d, extraenv=extraenv) + runfetchcmd(ud.basecmd + ['clone'] + ud.cloneflags + [clonedir, destdir], d, extraenv=extraenv) source_found = True else: - source_error.append("clone directory not available or not up to date: " + ud.clonedir) + source_error.append("clone directory not available or not up to date: " + clonedir) if not source_found: if ud.shallow: - if os.path.exists(ud.fullshallow): + if os.path.exists(fullshallow): if update and os.path.exists(destdir): update_mode = True else: bb.utils.mkdirhier(destdir) - runfetchcmd(['tar', '-xzf', ud.fullshallow], d, workdir=destdir) + runfetchcmd(['tar', '-xzf', fullshallow], d, workdir=destdir) source_found = True else: - source_error.append("shallow clone not available: " + ud.fullshallow) + source_error.append("shallow clone not available: " + fullshallow) else: source_error.append("shallow clone not enabled") @@ -735,8 +739,8 @@ class Git(FetchMethod): try: runfetchcmd(ud.basecmd + ['remote', 'get-url', 'dldir'], d, workdir=destdir) except bb.fetch.FetchError: - if ud.clonedir: - runfetchcmd(ud.basecmd + ['remote', 'add', 'dldir', 'file://' + ud.clonedir], d, workdir=destdir) + if clonedir: + runfetchcmd(ud.basecmd + ['remote', 'add', 'dldir', 'file://' + clonedir], d, workdir=destdir) try: runfetchcmd(ud.basecmd + ['fetch', 'dldir'], d, workdir=destdir, extraenv=extraenv) except bb.fetch.FetchError as e: @@ -767,11 +771,11 @@ class Git(FetchMethod): repourl = self._get_repo_url(ud) runfetchcmd(ud.basecmd + ['remote', 'set-url', 'origin', repourl], d, workdir=destdir) - if ud.clonedir: + if clonedir: try: runfetchcmd(ud.basecmd + ['remote', 'get-url', 'dldir'], d, workdir=destdir) except bb.fetch.FetchError: - runfetchcmd(ud.basecmd + ['remote', 'add', 'dldir', "file://" + ud.clonedir], d, workdir=destdir) + runfetchcmd(ud.basecmd + ['remote', 'add', 'dldir', "file://" + clonedir], d, workdir=destdir) if self._contains_lfs(ud, d, destdir): if not need_lfs: @@ -802,12 +806,6 @@ class Git(FetchMethod): """ clean the git directory """ to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror + ".done"] - # The localpath is a symlink to clonedir when it is cloned from a - # mirror, so remove both of them. - if os.path.islink(ud.localpath): - clonedir = os.path.realpath(ud.localpath) - to_remove.append(clonedir) - # Remove shallow mirror tarball if ud.shallow: to_remove.append(ud.fullshallow) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 71b0a63f..518d219b 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -616,6 +616,55 @@ class GitDownloadDirectoryNamingTest(FetcherTest): self.assertIn(self.recipe_dir, dir) +class GitPremirrorStaleCloneTest(FetcherTest): + + def test_premirror_ignores_stale_clone(self): + source = os.path.join(self.tempdir, "source") + os.mkdir(source) + self.git_init(cwd=source) + + testfile = os.path.join(source, "testfile") + with open(testfile, "w") as f: + f.write("first\n") + self.git(["add", "testfile"], cwd=source) + self.git(["commit", "-m", "first commit"], cwd=source) + stale_revision = self.git(["rev-parse", "HEAD"], cwd=source).strip() + + recipe_url = "git://upstream.invalid/project.git;branch=master;protocol=https" + self.d.setVar("SRCREV", stale_revision) + fetcher = bb.fetch.Fetch([recipe_url], self.d) + ud = fetcher.ud[recipe_url] + ud.setup_localpath(self.d) + os.makedirs(os.path.dirname(ud.clonedir), exist_ok=True) + self.git(["clone", "--bare", "--mirror", source, ud.clonedir], cwd=self.tempdir) + stale_clone = ud.clonedir + + with open(testfile, "a") as f: + f.write("second\n") + self.git(["commit", "-am", "second commit"], cwd=source) + revision = self.git(["rev-parse", "HEAD"], cwd=source).strip() + + self.d.setVar("SRCREV", revision) + self.d.setVar("BB_FETCH_PREMIRRORONLY", "1") + self.d.setVar("BB_NO_NETWORK", "1") + self.d.setVar("PREMIRRORS", "%s git://%s;protocol=file" % (recipe_url, source)) + + fetcher = bb.fetch.Fetch([recipe_url], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + self.assertFalse(os.path.islink(stale_clone)) + self.assertEqual(stale_revision, + self.git(["rev-parse", "HEAD"], cwd=stale_clone).strip()) + unpack_revision = self.git(["rev-parse", "HEAD"], + cwd=os.path.join(self.unpackdir, "git")).strip() + self.assertEqual(revision, unpack_revision) + + mirrorpath = fetcher.ud[recipe_url].mirrorpath + fetcher.clean() + self.assertTrue(os.path.exists(mirrorpath)) + + class TarballNamingTest(FetcherTest): def setUp(self): super(TarballNamingTest, self).setUp()