From patchwork Fri Mar 7 09:43:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 58454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F2E0C19F32 for ; Fri, 7 Mar 2025 09:43:38 +0000 (UTC) Received: from DB3PR0202CU003.outbound.protection.outlook.com (DB3PR0202CU003.outbound.protection.outlook.com [52.101.65.136]) by mx.groups.io with SMTP id smtpd.web10.7629.1741340615886651356 for ; Fri, 07 Mar 2025 01:43:37 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=NqXGzlwJ; spf=pass (domain: weidmueller.com, ip: 52.101.65.136, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=slSfRxl/e/hm/w6iXfRBdfd6U4Ns7vkWxnY85Y3tf40XtCukfxPQK8CRq0kt/OIHEnxr/AQ+pq6f+yRYRvUobinFeqx19akyQV6k3KUmvAWXIHU5kkFqS8WKdlX3+dAlSQWMrLFroDPdCOsLr99ZSxJ7mNH5lHyWvrbkvc0BCHxxA0qV+age5QnPwG6k8P6mPLwP46Zmwa4IAXtiRoSgSrB4LAyZO4JsEXysZDlUNQHGDP4GqGgYSu1BP3roBgQYz7C47CWz38XzIbDn737A5xF8111fD1/qsnomzNKxYCNO64r674fF81ckIp3p8ICnjREzPWRUUvA2PMiP/Ekm8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oILsBLD7F3hGhIc9qZj6R+KE6E5L+cCkkCvcWF9Pfvs=; b=g1Ecip5Hko8ftXYkgd8IVnmzpbd32QqwfWZY23UzI5MtBLWy6g3XQXZEHF6OLfS9cVjxdRjQD6F9JoNZ1k8hnKQheOyssDtypXF1rSSDhyBaWGfLrWYyOvSwiu72ixEmPudgvSnbhtzyAGUqYV6C6kxVjJQJvnAx+5JICfF13DvevrFY3uCOkUsELr2o9HZsGxLIeOd7B6yYm+xaBk0CEHCISP2DN8SfA8LsZKgTrImRLjZttQ4hLubvZ8IcAQvOn0lWtu5T1+7wy8iSZ8BkEdgY3//ajYSvmHD9DYv45nZF4Qt4WtDewru8kBoc5Q2vCihjD2Aq++Gbxo3J7FLJHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oILsBLD7F3hGhIc9qZj6R+KE6E5L+cCkkCvcWF9Pfvs=; b=NqXGzlwJFtzribmnicSz256WrvTbOYPPqatcVV6FZ4CIGvY2NmFwpIHLq7BRkb+MhgOVca2RcbbOZTq6F00sqpYC6GDzhMsNSOdfBm88cCOvDeCz/2mkdgWu0HZoiPG63jdxgVDFu3KFNtWQStnPkmPF1NS3x6OuxN6ZiVWaoiWDcHTWlta5P3ylyU0HxyyxM0q9qOnuCzo76QUKWJSwv2DKAq/VfRwP2FrMbQK6UZHumUXW+K5Us2UGNvU4OA1jWYyHKyzz5S2wARnKXaTcW1mzaEo7VnwswOKj4U0969KIa7hfHLYm0F9GMK3ZTIlsN6OukKiROv7BksgLdu74Hw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from GV1PR08MB8426.eurprd08.prod.outlook.com (2603:10a6:150:8a::17) by PAXPR08MB6511.eurprd08.prod.outlook.com (2603:10a6:102:12d::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.19; Fri, 7 Mar 2025 09:43:32 +0000 Received: from GV1PR08MB8426.eurprd08.prod.outlook.com ([fe80::f9f5:b4bd:9e01:9013]) by GV1PR08MB8426.eurprd08.prod.outlook.com ([fe80::f9f5:b4bd:9e01:9013%4]) with mapi id 15.20.8511.017; Fri, 7 Mar 2025 09:43:31 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Peter Kjellerstedt , Stefan Herbrechtsmeier Subject: [PATCH v2 1/2] fetch2: Partial revert decodeurl() to not use the URI class Date: Fri, 7 Mar 2025 10:43:20 +0100 Message-ID: <20250307094321.8993-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 X-ClientProxiedBy: FR0P281CA0072.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:49::9) To GV1PR08MB8426.eurprd08.prod.outlook.com (2603:10a6:150:8a::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: GV1PR08MB8426:EE_|PAXPR08MB6511:EE_ X-MS-Office365-Filtering-Correlation-Id: ffb2cef4-649c-48cd-70eb-08dd5d5c8579 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|52116014|38350700014; X-Microsoft-Antispam-Message-Info: RQ9cFrzUwKGJf/DbQlVvhYQJFGwFuEdbXRvs/PaG2uB6/3f4CDiIkJyg88h2iWwKAYBeGILbiC7hHav9BNqIORkVIzLjSXUr+JHg8lCnM+v6pjqKuxLkLt44aJzczpFN2QzqHuxtDW3wYojwusjKLKL2WITPfYHBDVF0l6fEmiFqz6JH8+xpzURNwE827GEcHllympSJuB9Ql8gBDExp+fuh2JTuEeB2/bu/DjXRQLJZ4f8kbNafh46TR2ti4BzxFQPhBmQp81mwf+ZMD0AOvPJXFgNtbsa1TJ9wmNL8bk3DEWsP2arWZhrhNIwVaC2BXP5zc4W8Ut3/1ytxUfqGx8MqI4qSpiv6CysW7Sh/X4/sXv+dMCmIZUjlPIHc8nML/8LAaiH79EW25QA2jf97X4/oIvXM3/wGtcsDAG17WBMYoDlZawVaWvydT++CQwloO1rHDxEuPcqnsMjEaDJIs1ZLWZBtuRDz4zPeyc/zeAZIpE/zKEcdL7BOHvLZYNqQJudrV2M4t+upG4hyilmgMCb9/LBCEYB922bABhp8IqPkutR3XFVTTyWAIg2A5y7Z5PbewzPasK5YoifVBG1vSadbRlOLYd9EwTmEL0BNhld9myMrnO5W5LZk0A7vABKwcWmponXWYS/4G1L8HCvpS1bq5hgewCJYeu15I00+P9XZ23MLL1kcxRibz6L9PfO2Kuj4H8mpyNagnKKzG/gO70bxgGNZjTlJwrslaIXxGUnP3MhyvHX0YA9dxR69TYH0s/Gy3fnyKwggclNeCN44RvH04pHrQssWuctmXxTCq3JKtzLAjKDCMbCGDgoUkMobMf18HzNput3LlEjZR2S3KI/skfWB8WDFuxOHLM9AC1fvqsLYbfKFWcZk+5I6h9tBLDDK8sYpEkDbP+HCmOabMaKh0dQBWRqG6u2RCnXwQRfDGDy9baztBzsIM2VJY2U+sNh8yrHzb4gSTzE96154g6NVXwfK/d8wuQXyCCba4kDG6fypuhU95ZchZVFO5lFszWnH/nclpuhHy0TMDMvaMg4hte5EsoyzudvISf94M3dx/WMpfvnSbBl4L2S93+WeJ7biMKQD2IwSXmbfrrLRdUGsPFNp1tpZh3Gy8PwGYMdj3bHJ1d3Qi9yFrDupm6LJXqaDu+ltfzQdYv9Gx8XQsSDIZbVnei7ro49Vc5I3J8eQvfPgiOZXbHHQdvnfXSuElq2Sn7IWWYaA4h0kSgePUZ3ghlKyELZj/V/QHVAjLW8T26lkmsoMgpSN7ICT+xCuOl90BvsMMdpSGPqoYflWItCqe1l1t3BFWjjAaDaPUThoIDDFyv5C2ttfrlUChCxtgnzyZxTm9hl7by/bM8C+Z60SH7vCoSLV5664k2aDfTXa1XbatR+XN0jn/Y8Q2biYaYH+DHz/FlCFZqsjp89sDA0/m4IF+ri7hvdkQ91Cj6GIybzGcQ5hKy8Cz/OZv+zs X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:GV1PR08MB8426.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(52116014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: DjLG39ClyrETkYZ6iCsE3NmCMk66lqiFfJPH5B2k6BKvVUSBTQ/VF/1/JHVUXGvkiT1lnFBIG4BkGrByuOxfm0c04HCeQ4cPxWxPgNClHkvShxeJbJe22mJhKa5dasQZnTTIbwlAhl3MpMppyeVJe1MpwvAFCeCKqihKziW/z6/ieE+HOP0xlvLAm6ivffkadFEM7HSY37WwsCdxwuWp2h0X5oh1LoHBFZgEiTWEeziDaAh3ZhcADWYOU5gCBl0w8SpWFIIY2HAP8nOiJW9hFG07ga3ZCDcKopY9RcVf7ac1HalEakhJJEeNVXrSVXJCq1S4vDgvDeu3VqwXk+OVx/e+uJwB93x4tCgg3d/jN6zqQw68yTB89PrqqH5mdi58s2gFPfNaGjsCkJEVDVsX1OzzttbvNuISSzEf/SEihOZ4DH/TdFkdZY7kkAM0//1Ge2TuB+Tj6nn51cV5eTOPqyb3DUoWmp/NCB/QAEcRj1caE64GkcArrtpTkKlh8csHJFG16eg8Ea3tQ5+V5ES24xD1ogHHGLnPDb7/4XLqp6XQficbJzoJkWxlxWg5KUKBU1ZY7wla/tvzpL2xExtBY77xvVqqwsqtCqUICYEHBVMcCJ3zjrwANNZAjNoyXy35+t7iijhYTTbiUL0mWC0ftfokvAIKPKD6JUNprvcbXJVSDU2EW5wT+oIhFkA9Gbpfx8LoTpS3C316m2B7Z0cl8GRX1Aer2+3m3I0HdkUd442dto4yVZNaugB4trQqMTUPaIT8cvt4pcz0milu6ttdI819DtY5pku+ScjZrEJ+XsrT/UdCskyFeSPYx9xclCcgCFvv2AlrugIP94qVs+TB1oTBICEfLmfBKxBGgqAngT65F0K2ZEadINgO002MzJZbA77YqLqAMh+KSJ9eEO2+tkVvhHMFHnADZ2WI313Fpab0BMnUEX/mdiQmEVINBDn0isPOKETv1GTA9CKPdSbF0pyXRuy1Qx9A658pA7dP8PR0Et/aHhpCMqoQ7zFi16CYpYFdwWmPGl3WasWiE/Ju9eD+WXT+rlMKXnOrCYThaSojTU1CXVhR921eYfp+sq4av5Edkq1Xy/OXS0go2n9/5HTDwG4jnZcpTjyc+PfPfEbN3K6YiE4H2i3UXPFAZgAChyqcAW+5DZ6C+3fUAFZ9M4y6NHU9ZLJd5kRGz/dpdE/jWrVgAD8B4Qqmc3/ZMrdU8rNRr4EGKrgbpXyQYCo7DKGKrXdpNBJAVTvwOv7DgPCPVM8n9yZf8zwNoFomZSJg1JVxEZmqh0jDEyBASZJVZJDi12JkOt/GQJGwO3AHrhoNnnrITm4oi9djoUqKdh0FVEAJ3Bttw6QLTugoAYgWJy/m0eXXC7u6PoUjaqpym9UWw4RWPrIazh6V75tsZiz0pkmdU08ZdEn152LwCm2AWSUu1bkArL5K/yZUeulThtcAs05d4lwTejMrve8yLk3vaWQwQ0tKFEnwgfAEwXZh+5t0qhEpK2z+3i9+6LsOV/1S4RtozmgU2XTax7ctt8TlaxyNfKQiZQUSFL/LkYCEhWgW4jrokpRU50uAKUnTyTjPNa+ia0uSLri9rltmzwS9aCE1ypqUBAXBbk99t9SixA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: ffb2cef4-649c-48cd-70eb-08dd5d5c8579 X-MS-Exchange-CrossTenant-AuthSource: GV1PR08MB8426.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2025 09:43:31.5815 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8nP4pj5VeXGG0LGjss/bWa58kAPceLCMMbwla8T+BVvkBNG3sEzhWo87Biy8+/Outl9rD1FRTEFXuWDiIeymJw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAXPR08MB6511 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 07 Mar 2025 09:43:38 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17398 From: Stefan Herbrechtsmeier This partial reverts commit a5d569c94700f04b8193c6bccae5af619931b00f which changes decodeurl() to use the URI class to parse the URL instead of doing it itself. While reusing code is generally a good idea, using urllib.parse.urlparse() (which the URI class does) to parse the regular expression "URLs" that are used in PREMIRRORS and MIRRORS does not work. A regular expression URL containing https?://... would be silently ignored, while a URL using a negative lookahead such as git://(?!internal\.git\.server).*/.* would result in a cryptic error: Exception: re.error: missing ), unterminated subpattern at position 0 The problem is that urllib.parse.urlparse() treats the ? as the start of URL parameters and thus stops parsing whatever part of the URL it was parsing. Restore the old function and use it in the PREMIRRORS and MIRRORS code. Signed-off-by: Peter Kjellerstedt Signed-off-by: Stefan Herbrechtsmeier --- (no changes since v1) lib/bb/fetch2/__init__.py | 54 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 51 insertions(+), 3 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 93fe012ec..5aa67accc 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -357,6 +357,54 @@ def decodeurl(url): path = uri.path if uri.path else "/" return uri.scheme, uri.hostport, path, uri.username, uri.password, uri.params +def decodemirrorurl(url): + """Decodes a mirror URL into the tokens (scheme, network location, path, + user, password, parameters). + """ + m = re.compile('(?P[^:]*)://((?P[^/;]+)@)?(?P[^;]+)(;(?P.*))?').match(url) + if not m: + raise MalformedUrl(url) + + type = m.group('type') + location = m.group('location') + if not location: + raise MalformedUrl(url) + user = m.group('user') + parm = m.group('parm') + + locidx = location.find('/') + if locidx != -1 and type.lower() != 'file': + host = location[:locidx] + path = location[locidx:] + elif type.lower() == 'file': + host = "" + path = location + if user: + path = user + '@' + path + user = "" + else: + host = location + path = "/" + if user: + m = re.compile('(?P[^:]+)(:?(?P.*))').match(user) + if m: + user = m.group('user') + pswd = m.group('pswd') + else: + user = '' + pswd = '' + + p = collections.OrderedDict() + if parm: + for s in parm.split(';'): + if s: + if not '=' in s: + raise MalformedUrl(url, "The URL: '%s' is invalid: parameter %s does not specify a value (missing '=')" % (url, s)) + s1, s2 = s.split('=', 1) + p[s1] = s2 + + return type, host, urllib.parse.unquote(path), user, pswd, p + def encodeurl(decoded): """Encodes a URL from tokens (scheme, network location, path, user, password, parameters). @@ -391,9 +439,9 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None): if not ud.url or not uri_find or not uri_replace: logger.error("uri_replace: passed an undefined value, not replacing") return None - uri_decoded = list(decodeurl(ud.url)) - uri_find_decoded = list(decodeurl(uri_find)) - uri_replace_decoded = list(decodeurl(uri_replace)) + uri_decoded = list(decodemirrorurl(ud.url)) + uri_find_decoded = list(decodemirrorurl(uri_find)) + uri_replace_decoded = list(decodemirrorurl(uri_replace)) logger.debug2("For url %s comparing %s to %s" % (uri_decoded, uri_find_decoded, uri_replace_decoded)) result_decoded = ['', '', '', '', '', {}] # 0 - type, 1 - host, 2 - path, 3 - user, 4- pswd, 5 - params