From patchwork Wed Aug 12 17:03:57 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fredrik Svensson (svsvenss)" X-Patchwork-Id: 95034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81606C5B572 for ; Wed, 12 Aug 2026 17:04:08 +0000 (UTC) Received: from alln-iport-7.cisco.com (alln-iport-7.cisco.com [173.37.142.94]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.294.1786554240792293007 for ; Wed, 12 Aug 2026 10:04:01 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: message contains an insecure body length tag" header.i=@cisco.com header.s=iport01 header.b=ajLA/BL8; spf=pass (domain: cisco.com, ip: 173.37.142.94, mailfrom: svsvenss@cisco.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=11342; q=dns/txt; s=iport01; t=1786554240; x=1787763840; h=from:to:subject:date:message-id: content-transfer-encoding:mime-version; bh=mc0ELW5hfqVYZ+MGeVxFQrh2kWqiNhiHpkwMMxavzPY=; b=ajLA/BL8u0IGcQnx076cBIqTziH0gK6loAmSsZbjgK0Xx1K9sUMEKB3Q mcdF/VAJQBAREXeqQYA5QHDRjBLEIlnH3QKA7UOHHLslv2psJEBRIKmLb +g0XZlgFDTcGz6aYrktOLEDUt1zWZ+w/pvB7Ug4O/fBJ95oi4u459SvKe +GXGEMKv9Gj8zW+a46IY4x9ylYvBHmsVL/8DbD05EEImuFrTgJsIZt03U siJQj8dxCrZ6OW8xDyMocKuqoCSbHHCIR7XgEcrbdVCejNoyGmKM+8+z5 i8cofRF+NZZfgMleno9K459F4FfVLoBFJKZwvKE6v8pQqiHiZ0Biu5va5 g==; X-CSE-ConnectionGUID: ODOUdFENSYCEw7y5iZLtkQ== X-CSE-MsgGUID: UYfC1dMdS7WrbMFgBaCRUQ== X-IPAS-Result: A0AoAADGpXxq/48QJK1aHQEBAQEJARIBBQUBZYEXCAELAYFtUxZyAoEhSYgjA4RMX4ZYgiGBFp0IgX4PAQEBDQIxIAQBAYUFAo1nAiY0CQ4BAgQDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFgQ4Thk8BDIdsAX8nBBMIgnuCdAMBArd9AolLGjd4gTSBAeAsAQsUAYE4AYhcAYV4hHwnG4INgRVCgjF2hCsahBOCMASCIoEMgkiCFIEXjQ2BShwDWSwBVRMXCwcFgWYDKi8tbjIdgSM+FzVYGwYFgR2BKIQ2Ixk2fIEJYYErKmYSF4EJgiQCgnaBKmECAR4LbT03FBkDBIE1BY1+IIJDAQEBWSEUKgGBIIERkwyReaF7CoQejCGVcBeFW6URmQgjo2eFDgIEAgQFAhABAQY2gTI8gVlwFYMiCUoZD5gHxjV5PQEBBwIHDgMLgWiRfgEB IronPort-PHdr: A9a23:M6rTVRZLWHnCr8dAqgiEFib/LTAchN3EVzX9orIuj7ZIN6O78IunY ArU5O5mixnCWoCIo/5Hiu+Dq6n7QiRA+peOtnkebYZBHwEIk8QYngEsQYaFBET3IeSsbnkSF 8VZX1gj9Ha+WXU= IronPort-Data: A9a23:/p5y3q9r6nc4IkVMGWnNDrUD3H+TJUtcMsCJ2f8bNWPcYEJGY0x3n 2VNWW6OaPqPNGGhc9EjaYTi/ENV75+AytNlT1c++3xEQiMRo6IpJzg2wmQcns+2BpeeJK6yx 5xGMrEsFOhtEDmE4EzrauS9xZVF/fngbqLmD+LZMTxGSwZhSSMw4TpugOdRbrRA2bBVOCvT/ 4mvyyHjEAX9gWAsYjtOs/nrRC5H5ZwehhtJ5jTSWtgT1LPuvyF9JI4SI6i3M0z5TuF8dsamR /zOxa2O5WjQ+REgELuNyt4XpWVTH9Y+lSDX4pZnc/DKbipq/0Te4Y5nXBYoUnq7vh3S9zxHJ HqhgrTrIeshFvWkdO3wyHC0GQkmVUFN0OevzXRSLaV/wmWeG0YAzcmCA2kREaol9shaJ1p/3 tEHdyFQbyKnm+GPlefTpulE3qzPLeHiOIcZ/3UlxjbDALN+H9bIQr7B4plT2zJYasJmRKmFI ZFHL2MxKk2cOHWjOX9PYH46tPy2nWfydSZDgFmUvqEwpWPUyWSd1ZC9aIaFJIPRHZU9ckCwm 1PI1D31Jx0ha+OFwx+r/VilrdXypHauMG4VPPjinhJwu3WZx3AZAR4bEFe2v/S9okq/QM5Eb UsM9ywjqKI/+ECmQp/6RRLQnZKflhcYX9wVF6gx7xuAj/KKpQ2YHWMDCDVGbbTKqfMLeNDj7 XfQ9/vBDj10u7rTQnWYnop4ZxvrUcTJBQfuvRM5cDY= IronPort-HdrOrdr: A9a23:hyCuyax3JTA+StmuEV1tKrPxP+gkLtp133Aq2lEZdPULSL36qy n+ppQmPEHP6Qr5AEtQ5OxoWJPtfZquz+8I3WB3B8buYOCGghrjEGgM1/qY/9SNIVybygc/79 YtT0EdMqyLMbESt6+Ti2fIcadE/DDtytHPuQ6q9QYUcegcUdAY0+4WMHf9LmRGACN9Kd4SEp 2a6sBbpzymV0g2Q62AbEUtbqzonfGOsIjpTyInKncchTWmvHeF+bT6Gx+X0lMkVS9TwbAtyG 7BkwbozKSuufO2x3bnpiDuBpJt9+fJ+59mPoihm8IVIjLjhkKDf4J6QYCPuzgzvaWG9EsquM OkmWZgA+1Dr1fqOk2lqxrk3AftlBw07WX59FOeiXz/5eTkWTMBDdZbj44xSGqY16NghqA+7E t45RPdi3NlN2KFoM073amOa/herDv3nZPlq59Us5UQa/pGVFYbl/1hwKocKuZ/IMu90vFoLM BeSOfB+f1RbVSWK1rdvmVp3ZiQe05bJGbffqDH0fbliQS/WxtCvhAlLMBzpAZwyLstD5ZD/O jKKaJuifVHSdIXd7t0AKMbTdKwEXGle2OEDIu+GyWRKEg8AQO6l7fnpLEuoO26cp0By5U/3J zHTVNDrGY3P0bjE9eH0pFH+g3EBDzVZ0Wn9uhOo5xi/rHsTrviNiOODFgojsu7uv0aRsnWQe y6Np5aC+LqaWHuBYFK1QvjXIQ6EwhXbOQF/tIgH16eqMPCLYPn8uTdbfbIPbLoVS0pX2vua0 FzKAQb5P8wm3xDdkWI8iQ5AUmdCnAX1agAY5Tnww== X-Talos-CUID: 9a23:KSlyNWvtu3SCWJfK2cmcyWQ76IsCXlrgwFn9BXSJIkt2F6zISG205KBdxp8= X-Talos-MUID: 9a23:bokDkwmiK//D0t1KM6PudnpfBclDuqiLBHtSro8MnueLHyVbAQqk2WE= X-IronPort-Anti-Spam-Filtered: true Received: from alln-l-core-06.cisco.com ([173.36.16.143]) by alln-iport-7.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 12 Aug 2026 17:03:59 +0000 Received: from alln-opgw-4.cisco.com (alln-opgw-4.cisco.com [173.37.147.252]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by alln-l-core-06.cisco.com (Postfix) with ESMTPS id B12D118000255 for ; Wed, 12 Aug 2026 17:03:59 +0000 (GMT) X-CSE-ConnectionGUID: 3rNRDeo6RUijwDpWfPt+WA== X-CSE-MsgGUID: zAIn70ZkRryqwsFkJ4YgTg== Authentication-Results: alln-opgw-4.cisco.com; dkim=pass (signature verified) header.i=@cisco.com X-IronPort-AV: E=Sophos;i="6.25,219,1779148800"; d="scan'208";a="83503717" Received: from mail-eastus2azon11011042.outbound.protection.outlook.com (HELO BN8PR05CU002.outbound.protection.outlook.com) ([52.101.57.42]) by alln-opgw-4.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 12 Aug 2026 17:03:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=s8q3AChgOtWc/r8oQWbqMMLcICyH6v+M8rTRznFKNX8QYJwlo8JLnC/imn9Oj0Ek7F3g9bFgnk559VmIilbUSzFkXZA6+tiJOZdxdkMfd2+/KnsoxJh64DzzuD9nQvHitpt6tCKv5IPTzRsGXQYIg8C9bLrw84U2Xx81arBFUseP/ckd+skWoIBzLOOhl99VXwcKQTw+/4sSM+euJDRlHYcOV9qo8iLUhwrDAOwe1QtV6uy10NRw7KeUSNF8WeNUeeuYFK5XmD0HbWHF+RjczIaprq9e9KwPS8MZolxp4bKvcJ0K3RIwRe0a95+jMvOSlSy0IMBcDRK3b3oXPTGWog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mc0ELW5hfqVYZ+MGeVxFQrh2kWqiNhiHpkwMMxavzPY=; b=VZlaUyH3tiSPb2+faJO2Gyk1kaS+Aw+GM+mypLFdEJWac8FWhsC+3jDJPGOsR306EMK3WXabNi/pLpjKng1y6NtgvMs+4I7En3YnUxYGEKiZatIPBugKRfgTeA7Fst+vD1vCYeNLmU4uH7DiiAXLhV0PZfAJcc1bToR5lC5Lo5Sk1JiX8tb+/4Pzwc8zxCW3pTkwJrI5T1FeC4cgySRHwpuKiP0+X2YSJHh7n9x5MJTzfFFTK5SZgm/4hvPFme52Sro/8Bzm2UpzA89q0jRRW+putyU6wge2HORoE2SGBb3jlRBot33PdsF8HdHGu63gQeaSuBZUnGZaweE9Whf5UQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none Received: from DM6PR11MB3962.namprd11.prod.outlook.com (2603:10b6:5:192::26) by MN2PR11MB4517.namprd11.prod.outlook.com (2603:10b6:208:24e::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.14; Wed, 12 Aug 2026 17:03:57 +0000 Received: from DM6PR11MB3962.namprd11.prod.outlook.com ([fe80::47f3:254:87ca:5161]) by DM6PR11MB3962.namprd11.prod.outlook.com ([fe80::47f3:254:87ca:5161%5]) with mapi id 15.21.0315.012; Wed, 12 Aug 2026 17:03:57 +0000 From: "Fredrik Svensson (svsvenss)" To: "bitbake-devel@lists.openembedded.org" Subject: =?iso-8859-1?q?=5BPATCH=A0v2=5D=A0fetch2/wget=3A=A0reuse=A0cached?= =?iso-8859-1?q?=A0HTTPS=A0connections?= Thread-Topic: =?iso-8859-1?q?=5BPATCH=A0v2=5D=A0fetch2/wget=3A=A0reuse=A0cac?= =?iso-8859-1?q?hed=A0HTTPS=A0connections?= Thread-Index: AQHdKnxaqgOfbFZ0Nk+1fTBcZToMqw== Date: Wed, 12 Aug 2026 17:03:57 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM6PR11MB3962:EE_|MN2PR11MB4517:EE_ x-ms-office365-filtering-correlation-id: 7601d747-236b-4467-5a5b-08def893b2d4 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230040|23010399003|376014|1800799024|366016|38070700021|10067099003|11063799006|18002099003|56012099006|3023799007; x-microsoft-antispam-message-info: U0CnEkl/iu6EwLSQdA+xxaXOWKafEQ6hVmSiwVdp2E/5oKQae2BbmBvqzsyU6bAOoKxNUHdGHc9zaAUnCGaNitBtw6XPhkSTv76zcyTGXYFrPWRsHgFA1BavSYRlupI0QM/sTvTCDd/RebLmleetDkEIKj6fxeHpvi25BZ2Z5GOOrTe5MOCP4R2T3fxpBQreqjfM+JXYL+tOCi+lOqPbkp/1YpD4DS2xsTICze+Wp4LSdFwEEd/TpOaGQalQCF1HuOTuH9jBYvCCil+i7kCXMc1efCGhezp25mdxJTCrR/oEZ6VJwDq4zxnSshA/U83OQV8gAhaD7S4SgyCkUQXOOApUi8ey4lzdBMhXNxrrHmPE2QCmNjCB6L5hG1YVVe+rr5kM38IyjAW8YoIPo9KWOpvzQfzP1trySGFaGMBmgXWw68NWD3LTH2EvpThFNKO0sfcYonjDpjUa+lTv+czQY4KJh9/Pcp1Sxgu/HU5EcoDJFUYSd6LpfW+VazYfWmlodHvzHd6dMcR93vAmAvDpe8T8hU5XJmO0Fqhl85cHN+reHk/au+iOJcughMRHJZpGlfq1pwkKy6jUGn3o4qmxkF4yIFez7LU4bYkUTZ/aJJmLxHPzTYsW2oqDzD/PDvGqJtRGmc6Jq6inUXhESoymcdALKEJEh7Z9H7JFhHApOsYmRVKmdTKkMy7uWAKUl2YPTgABoByWyNjWbLmqwX9/By2XpwJSRw3RhgsbzEMXG6E= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR11MB3962.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(376014)(1800799024)(366016)(38070700021)(10067099003)(11063799006)(18002099003)(56012099006)(3023799007);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?Mw/D7B4mjvuDfAs7M4LEYtN?= =?iso-8859-1?q?x5YU4/Bs5z1YGBrgf0AhX32srUy26P2sSsttv+VWClxK5vii4Nuh0wUNHt0X?= =?iso-8859-1?q?dmVyyKuc8rXlyet8wO5wKSy7ZaBq60sV2R9XrrpS72F1Yur6HjOMjLh55iW1?= =?iso-8859-1?q?TM1xWRji7ZNQl45Q0Aea2mUt8c8xdsbl063Sd2Vdc/s0angtjaoZxxpHy1hE?= =?iso-8859-1?q?5L5db3ifqrUl+O1kFkgmGNrKiS12SDFVPwyFD3f5gCLvuTdXJ8joSqP1zRxO?= =?iso-8859-1?q?KVrjOIW7a5NPFmvVAaM0UraED1QoldQKgA/0ve7jQJZBOhYy8eiIU5GfMIQ9?= =?iso-8859-1?q?xDjdUaKGNVqR+UGDFTG2ickt/rQI6hXbYZgqTpQ9g+Y7c/c5w9MCVE4BJpX9?= =?iso-8859-1?q?lDlZkY5xikgBtuanGRpk1Eaw+ae3sXabxBZE5qwRGTaR5mkIVGWwwobnKzXq?= =?iso-8859-1?q?RGazTj2kC/eOUwRsRQJsTvmtQzOzLqjBI+PlOgSTcWWuyKIvG+68DUYTDGh6?= =?iso-8859-1?q?1ZtMMN4xsyKhB1D8JDzPbE8gan2GPpmesZgpDI9zbJ9qZfRXVsSPwth0c0i+?= =?iso-8859-1?q?fh6I4jfSH0ULw6OqiMEL98uJ85dHMM7bYOwxRzqBweaDU3PHWUXn05pV3cfY?= =?iso-8859-1?q?qOf/8ktWvEpIgER/YbfR/UQQSqDDNvIdvRklnppPuMUF5m2cG4ei4lgZFcKu?= =?iso-8859-1?q?rrXWqpSPdtiITFVEIU+Bg9VKwbZnAfTjo0nV2RMqxGLtSC1lxAxL+p4fxp8a?= =?iso-8859-1?q?J1u5sO1tKLTBRc3IvKIKGG+iF+vSIv+sG0bSgZCMSKTqjclSCP9QcNiw/c4b?= =?iso-8859-1?q?V18R7L35PQW16ul0USyAaNN9KWsssdeJOcsSHqfD9d/GjfiD4O7FkH7a0HAC?= =?iso-8859-1?q?2OHxcNLL4J1/o0jkjTb0LCgL5WYL/81gxSUHwaAZ8m07l+d/7lazGSKz8dKL?= =?iso-8859-1?q?1v6RwhZpEKy24goxsAW1gSqJ3ggEmaY2hCxwVadYaOoXkKQzA4RjMhZWM7nB?= =?iso-8859-1?q?sNrKCslawJVShWeAEAnmECtNWMXwlLmtXglFGV6NPdmp3yxWGxNNw8w8EyWk?= =?iso-8859-1?q?rnAFIkvSy6LS+zj88/8BpEcJcwF8TWxEzut7MotZE9i1g7UCDC2Iygz0ZbGv?= =?iso-8859-1?q?O/tuCbJLYvFx1Ou6fx6dSTmcgdAjzrppViohYsKGPxDqTJalGoGfDHU/2os0?= =?iso-8859-1?q?FjOLVLuT/TA8535lyiaB44kubERn1AGdMKG1RlFphbNcDIirQnCGvZkprS2w?= =?iso-8859-1?q?DYIzS1FhqKnwW/CtOc7hVksXE2vCbFwwoNLqGtOAdW/eexZUSHndiMivituX?= =?iso-8859-1?q?5UxjOBjAbyabnU2w2maao8VANKHXLfFDQQr1qNoYDe4hD1jsqOPe6wtCJXlS?= =?iso-8859-1?q?ZxejiFVtD7MuNIlMeGn36zkrffBPQDWiNMyzHmnG89F09SaU2KdpQ/9b/uwD?= =?iso-8859-1?q?K6OGX9h7rQAVFAsTFhyANbA70+ZXeaHWG8rntgdnn2BB8UU8qTh664G5FUKL?= =?iso-8859-1?q?AIaNt5miMfu8B0LCGs717DA/tZ2Wv4vQ+4Ulvt1SBRh7V2DmzWRMnYRgFocY?= =?iso-8859-1?q?jJpgL/GzY4HWXBD2LFQO9REEvfyZ6rOo/O3LaTsGFX6IdpUVcRILbPiMCNGs?= =?iso-8859-1?q?LDaCwXwkbfOnGijrRKy3s4Z41c7nYcW/qXR+oN17diQqZXCLjwdStP2BA9bf?= =?iso-8859-1?q?LVKSyh8d6urzPJu5+KsJTtwjeYQ0rhbCu6EpisirCxBCf9O6meJfYwSe7Msq?= =?iso-8859-1?q?8eoc3z4ZgE481O9PqEgNkbatO?= MIME-Version: 1.0 X-Exchange-RoutingPolicyChecked: qtLtm/rVOujDxSriq94RXnc2DwrAVmnr3zrpFiQen8rwpZUoyb6AU+iCftDLu+ET0Rjqp8FdW1BjY4ptKVwvUgULiD5tLI5YvKwH6VcAIYFWts118slQxaHjvZ1TXp010S6+7jIcpUd9F5kM4aY4G/vZFCosoyynD8Gv01wLM/qiDLNrl9UEuI1zN9BfrvqDF/pk8qc+N1FeyAARm8GoSsaeUuMbob7KMo75PtScdeAdDSZ9GBZf6h8gj5cknrwypnt1e1WTjxynEiSn5BAFD2zmSe/q6anMDASwxLisG5lLCCfjJZROm25sYO3DRiy6jjvvNk+esEVN5E61tnHKIA== X-OriginatorOrg: cisco.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3962.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7601d747-236b-4467-5a5b-08def893b2d4 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Aug 2026 17:03:57.6892 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 5ae1af62-9505-4097-a69a-c1553ef7840e X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Tg6AZxeAeVqSTUJlqtiJzVbnFSNam5Qln47Lhs4tajK/giEzDeaKgCqNi+neQLiaDFR6RngbfDNGKUpYPt430Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4517 X-Outbound-Client-TLS: ANONYMOUS;alln-opgw-4.cisco.com [173.37.147.252];TLSv1.3;TLS_AES_256_GCM_SHA384;256 X-Outbound-SMTP-Client: 173.37.147.252, alln-opgw-4.cisco.com X-Outbound-Node: alln-l-core-06.cisco.com List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 12 Aug 2026 17:04:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19918 The checkstatus() path reuses cached connections for HTTP requests, but uses urllib's standard HTTPSHandler for HTTPS. Each HTTPS availability check therefore creates a new TCP connection and performs a new TLS handshake. Add HTTPS equivalents of the existing cached connection and request handler. Preserve the SSL context selected by BB_CHECK_SSL_CERTS and extend FetchConnectionCache keys so HTTPS connections are kept separate from HTTP connections, different certificate policies and CA files, and different proxy tunnels. Add a local TLS self-test which proves two HTTPS HEAD requests use one TCP/TLS connection. The test also verifies that a connection established with certificate checking disabled is not reused after checking is enabled. In an ABBA benchmark against BitBake master, 4,283 remote sstate availability misses averaged 761.639 seconds without this change and 254.742 seconds with it. This reduced the check by 66.55%, a 2.990x speedup. The benchmark had no matching mirror objects, so it isolates availability checks rather than download and unpack time. Signed-off-by: Fredrik Svensson --- Changes in v2: - Avoid urllib.request.HTTPSHandler's private _check_hostname attribute, which is no longer present with Python 3.12 and newer. lib/bb/fetch2/__init__.py | 18 ++++++----- lib/bb/fetch2/wget.py | 56 +++++++++++++++++++++++++++------ lib/bb/tests/fetch.py | 66 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 122 insertions(+), 18 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index a5772f8a..ea007f36 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -2127,26 +2127,28 @@ class FetchConnectionCache(object): def __init__(self): self.cache = {} - def get_connection_name(self, host, port): - return host + ':' + str(port) + def get_connection_name(self, host, port, connection_id=None): + if connection_id is None: + return host + ':' + str(port) + return (host, port, connection_id) - def add_connection(self, host, port, connection): - cn = self.get_connection_name(host, port) + def add_connection(self, host, port, connection, connection_id=None): + cn = self.get_connection_name(host, port, connection_id) if cn not in self.cache: self.cache[cn] = connection - def get_connection(self, host, port): + def get_connection(self, host, port, connection_id=None): connection = None - cn = self.get_connection_name(host, port) + cn = self.get_connection_name(host, port, connection_id) if cn in self.cache: connection = self.cache[cn] return connection - def remove_connection(self, host, port): - cn = self.get_connection_name(host, port) + def remove_connection(self, host, port, connection_id=None): + cn = self.get_connection_name(host, port, connection_id) if cn in self.cache: self.cache[cn].close() del self.cache[cn] diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index 141c2d06..c4534b05 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -151,27 +151,54 @@ class Wget(FetchMethod): return True def checkstatus(self, fetch, ud, d, try_again=True): + check_certs = self.check_certs(d) + newenv = bb.fetch2.get_fetcher_environment(d) + class HTTPConnectionCache(http.client.HTTPConnection): + def cache_id(self): + return None + if fetch.connection_cache: def connect(self): """Connect to the host and port specified in __init__.""" - sock = fetch.connection_cache.get_connection(self.host, self.port) + sock = fetch.connection_cache.get_connection( + self.host, self.port, self.cache_id()) if sock: self.sock = sock else: self.sock = socket.create_connection((self.host, self.port), self.timeout, self.source_address) - fetch.connection_cache.add_connection(self.host, self.port, self.sock) + fetch.connection_cache.add_connection( + self.host, self.port, self.sock, self.cache_id()) if self._tunnel_host: self._tunnel() + class HTTPSConnectionCache(http.client.HTTPSConnection): + def cache_id(self): + return ("https", check_certs, + newenv.get("SSL_CERT_FILE"), + self._tunnel_host, self._tunnel_port) + + if fetch.connection_cache: + def connect(self): + """Reuse an established TLS connection when available.""" + + sock = fetch.connection_cache.get_connection( + self.host, self.port, self.cache_id()) + if sock: + self.sock = sock + else: + super().connect() + fetch.connection_cache.add_connection( + self.host, self.port, self.sock, self.cache_id()) + class CacheHTTPHandler(urllib.request.HTTPHandler): def http_open(self, req): return self.do_open(HTTPConnectionCache, req) - def do_open(self, http_class, req): + def do_open(self, http_class, req, **http_conn_args): """Return an addinfourl object for the request, using http_class. http_class must implement the HTTPConnection API from httplib. @@ -185,7 +212,7 @@ class Wget(FetchMethod): if not host: raise urllib.error.URLError('no host given') - h = http_class(host, timeout=req.timeout) # will parse host:port + h = http_class(host, timeout=req.timeout, **http_conn_args) # will parse host:port h.set_debuglevel(self._debuglevel) headers = dict(req.unredirected_hdrs) @@ -231,7 +258,8 @@ class Wget(FetchMethod): # If it still fails, we give up, which can happen for bad # HTTP proxy settings. if fetch.connection_cache: - fetch.connection_cache.remove_connection(h.host, h.port) + fetch.connection_cache.remove_connection( + h.host, h.port, h.cache_id()) h.close() raise @@ -265,10 +293,20 @@ class Wget(FetchMethod): # Close connection when server request it. if fetch.connection_cache is not None: if 'Connection' in r.msg and r.msg['Connection'] == 'close': - fetch.connection_cache.remove_connection(h.host, h.port) + fetch.connection_cache.remove_connection( + h.host, h.port, h.cache_id()) return resp + class CacheHTTPSHandler(CacheHTTPHandler, urllib.request.HTTPSHandler): + def __init__(self, debuglevel=0, context=None, check_hostname=None): + urllib.request.HTTPSHandler.__init__(self, debuglevel, context, + check_hostname) + + def https_open(self, req): + return self.do_open(HTTPSConnectionCache, req, + context=self._context) + class HTTPMethodFallback(urllib.request.BaseHandler): """ Fallback to GET if HEAD is not allowed (405 HTTP error) @@ -370,12 +408,10 @@ class Wget(FetchMethod): # Avoid tramping the environment too much by using bb.utils.environment # to scope the changes to the build_opener request, which is when the # environment lookups happen. - newenv = bb.fetch2.get_fetcher_environment(d) - with bb.utils.environment(**newenv): import ssl - if self.check_certs(d): + if check_certs: context = ssl.create_default_context() else: context = ssl._create_unverified_context() @@ -384,7 +420,7 @@ class Wget(FetchMethod): HTTPMethodFallback, urllib.request.ProxyHandler(), CacheHTTPHandler(), - urllib.request.HTTPSHandler(context=context)] + CacheHTTPSHandler(context=context)] opener = urllib.request.build_opener(*handlers) try: diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index cd50c37a..fd064743 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -1781,6 +1781,72 @@ class FetchCheckStatusTest(FetcherTest): connection_cache.close_connections() + @unittest.skipUnless(shutil.which("openssl"), "openssl not installed") + def test_wget_checkstatus_https_connection_cache(self): + import ssl + from socketserver import ThreadingMixIn + from bb.fetch2 import FetchConnectionCache + + class HTTPSRequestHandler(http.server.BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def do_HEAD(self): + self.send_response(200) + self.send_header("Content-Length", "0") + self.end_headers() + + def log_message(self, format_str, *args): + pass + + class HTTPSServer(ThreadingMixIn, http.server.HTTPServer): + daemon_threads = True + + def __init__(self, *args, **kwargs): + self.connection_count = 0 + super().__init__(*args, **kwargs) + + def get_request(self): + request, client_address = super().get_request() + self.connection_count += 1 + return request, client_address + + certificate = os.path.join(self.tempdir, "certificate.pem") + private_key = os.path.join(self.tempdir, "private-key.pem") + subprocess.check_call( + ["openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", + "-keyout", private_key, "-out", certificate, "-days", "1", + "-subj", "/CN=127.0.0.1"], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + + server = HTTPSServer(("127.0.0.1", 0), HTTPSRequestHandler) + context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + context.load_cert_chain(certificate, private_key) + server.socket = context.wrap_socket(server.socket, server_side=True) + server_thread = threading.Thread(target=server.serve_forever) + server_thread.daemon = True + server_thread.start() + + connection_cache = FetchConnectionCache() + try: + url = "https://127.0.0.1:%s/test" % server.server_port + self.d.setVar("BB_CHECK_SSL_CERTS", "0") + fetch = bb.fetch2.Fetch([url], self.d, + connection_cache=connection_cache) + ud = fetch.ud[url] + self.assertTrue(ud.method.checkstatus(fetch, ud, self.d)) + self.assertTrue(ud.method.checkstatus(fetch, ud, self.d)) + self.assertEqual(server.connection_count, 1) + + # A connection established without certificate checks must not be + # reused after certificate checking is enabled. + self.d.setVar("BB_CHECK_SSL_CERTS", "1") + self.assertFalse(ud.method.checkstatus(fetch, ud, self.d)) + finally: + connection_cache.close_connections() + server.shutdown() + server_thread.join() + server.server_close() + def test_wget_checkstatus_same_origin_redirect_keeps_auth(self): server = self._start_checkstatus_server() server.redirect_url = "http://127.0.0.1:%s/b" % server.server_port