From patchwork Wed May 10 13:10:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: CESTONARO Thilo X-Patchwork-Id: 23790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82A50C7EE22 for ; Wed, 10 May 2023 13:10:59 +0000 (UTC) Received: from thsbbfxrt01p.thalesgroup.com (thsbbfxrt01p.thalesgroup.com [192.54.144.131]) by mx.groups.io with SMTP id smtpd.web11.16041.1683724256247861652 for ; Wed, 10 May 2023 06:10:57 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@thalesgroup.com header.s=xrt20181201 header.b=JlMDDxDX; spf=pass (domain: thalesgroup.com, ip: 192.54.144.131, mailfrom: thilo.cestonaro@thalesgroup.com) Received: from thsbbfxrt01p.thalesgroup.com (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 4QGb4Z49nmz45fp; Wed, 10 May 2023 15:10:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=thalesgroup.com; s=xrt20181201; t=1683724254; bh=ncm19NG++TROcjCi2rl31fdpGdcBGRYcLVRMc6w2meI=; h=From:To:Subject:Date:Message-ID:References:In-Reply-To: MIME-Version:From; b=JlMDDxDX4UL87SzcnJSt9l7+2Gf2EVslwiyTSHAsdQASzliC0aujgzX40XkWt/uFQ tI3Q5R8eBfMC6JOF2AlTW+itP7YHKerk0ckXnsXFZZ6q/z0f6iSVy8PP56M7VBNOri c8Gwb5YzFyk0shAFp4YBuCB3s3dpBtHAnBwT8xxJIOIOJS/z1tYcrrgDfJjyvUlI57 PIU+Uwgc77dztDeUVk79kZo/maNTe9EMqpG7Eg4TL67fZidrROfRHblBMIEdGupmBj CMb+JSjeQ8llxq5Q7VNIa847wH3NjxwZUEEhGcTOTyEKZPeBl33UPZJwnKAKk46xZl lWwY5sH/kgP0g== Authentication-Results: thsbbfxss01p.thalesgroup.com; dkim=pass (1024-bit key; unprotected) header.d=thalesgroup.onmicrosoft.com header.i=@thalesgroup.onmicrosoft.com header.a=rsa-sha256 header.s=selector2-thalesgroup-onmicrosoft-com header.b=CiY9rhzk; dkim-atps=neutral ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Rb9hl7IVqjbR1xM8JPEXNHOuXzJlrgx2PfypNoIgcc+ZcxleabkxO75hj3HuwWjlxvpLoPhdPvSBNUQ6Dgm+CNYFZhI6bWIW0Bx7ZDFosO3g3TQfXrvQJ/B9DrQX3N/MSWQILPsO8gvvEUunZpITIi2aDgDM1npq92DhPeNG/KRrxXJxrv4VXfL3zWhL/8IelEB+0BUYW4Qdz4daW99ciwLCHzEfytjmQ0Vik0Am2cm/OD/FKB9vnsGBgzk6UL/fj1w7v1rNKOOK0Dk3CsqEl1UaIM9mOT5ScacyvgpBTD53hgXkpV1bki/ZLF3pGRXPwAYjgYM0sNuSUnJKoFH49Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ncm19NG++TROcjCi2rl31fdpGdcBGRYcLVRMc6w2meI=; b=jiYBPo4qRvpMMnvrBQ0TrWujkEZRkJc7JiTsLnNGsRtFBCGC98GYNNxPc65F3AeGDgy3OCh3eqyBqCpk2yLiDK7DHsTo/mB+GBJfyNPdwIz6vKjS9o1xx5YuL3yfxfxv8QVZj8XEYX4HAD1LYuPmgVw6k7ztAcxbeFVPFgq18n0VJzrm3lqP/AkgN41jTVW8w4ltU4UQAA0OVXwY1Mh6gPoVKiMLdB57p8rzrJN358nce8COkPj99pQRzNmmVpphSw3pNpXCGH6i3RA0vMEhG8gu/RenPeATL567krZx0sfuxN0yfJCaVj6F51t56UJTSPbW10+3xkQpFM0Ft/kU3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=thalesgroup.com; dmarc=pass action=none header.from=thalesgroup.com; dkim=pass header.d=thalesgroup.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=thalesgroup.onmicrosoft.com; s=selector2-thalesgroup-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ncm19NG++TROcjCi2rl31fdpGdcBGRYcLVRMc6w2meI=; b=CiY9rhzkmbE7Bf0JvW3Akjg52Bpe3RzO2rOHA3ZANd1hekU94YdYxpcEXyS7baXROc0+jHn/AznOKoJ6/jW1Cbs3OhXTjFoM5MlmNzg1qb2bMth0buA4q47xzQcivPJXgwZR3/HFHfYpPfG6AH86L4F9K5R4BvBro/Kpd55KV3w= From: CESTONARO Thilo To: Richard Purdie , "bitbake-devel@lists.openembedded.org" Subject: [PATCH v2] bitbake: git fetcher: use urllib quote ... Thread-Topic: [PATCH v2] bitbake: git fetcher: use urllib quote ... Thread-Index: AQHZg0DYRC4+lIjd80eTaxXXaWTz/w== Date: Wed, 10 May 2023 13:10:51 +0000 Message-ID: References: In-Reply-To: Accept-Language: de-CH, en-US Content-Language: de-CH X-MS-Has-Attach: yes X-MS-TNEF-Correlator: msip_labels: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=thalesgroup.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PR0P264MB1882:EE_|PR0P264MB2028:EE_ x-ms-office365-filtering-correlation-id: ae24cfbb-1fa1-4cce-a053-08db5157faf8 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 86CzKxLuY79L/Wvn47ZPDSwWIHgl9rQQE9CWiHTWGCqLnCB/6sZWW/rPsKlIMQ79JwlTymc1xzSMMgyPEyG2fkLGSte9yYNgPufibLe81paPh+YDRtHc5lF9ZnxX9jb3jR71b6mnNlI2pWKh0E5z2cWsYgedlTO/CqzctaQkny4GCkfrA1+eBz6I+7oWzzZ8epyY5E4dI+vNZFf6GdPWNGCoMI6xYVTPsRVZGa0jDaVYIvwSdUtrPjq8xA/W3p0stcnQ9EcpsCWH439+ploohKC3EXxndpIxDbOvwMx/+vQIFsHmmW4114YwaLZzl5Pwygpb+PWibw3D1x3eu6tDSP2kB2GiJlTt+YJckZ1tY5OWSk9IvkvKmA5hIm+zfMy84+fY0Dd3rqIOtR/jgAPxeOisXhDtbuOlhAGNjHrBlU32EbG3GNyP/HJG+jO91vFXn9oYFxopGPVoCdVngazmbmltPoML1qOHzZPQBjVUyuzlq/vAiT3VvEhUVKts9d6ERb9dFuEy7imWkSTJc2TZ9fZVUx59JNyeO+aISB50t8qutMCby+Ja89+YyruSPPaz3atOSheP2T3f6ekn9ZvTc0KAD2nrJFUYdu87Cnb6fzixGfbIhbj89fGdoiKxFBkF x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PR0P264MB1882.FRAP264.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(346002)(396003)(366004)(376002)(39860400002)(451199021)(4744005)(66446008)(8936002)(8676002)(5660300002)(52536014)(41300700001)(316002)(64756008)(91956017)(76116006)(66476007)(66556008)(33656002)(66946007)(26005)(71200400001)(186003)(478600001)(6506007)(9686003)(7696005)(2906002)(82960400001)(38070700005)(38100700002)(122000001)(86362001)(55016003)(110136005)(99936003);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?//bAjXgPRGPHvcTSiRi6d8A?= =?iso-8859-1?q?Y9OkqsYvWTJcXAQiyS3viiC/+pEjpTwscePqdpOD/YqXMmxdhPAPoysZpaMQ?= =?iso-8859-1?q?paNGoTFd6Yw7tJCGZX6FZZwsUutRCYUCMZg2/7ZDG9ZH+tCryU+o+hAh3BPw?= =?iso-8859-1?q?ujSAFPEmrvgPzG2n5WS3Xy6L7U+9ikUDn4rpI5sSV9KSOehsHsq3F84ffxFM?= =?iso-8859-1?q?pZ4L6TUvd+koC4H+ggCRogux3gKQDy5eK9fTDX5uSe67yI5WaTojWIIC3qSn?= =?iso-8859-1?q?7isT4cExtAoH9O73Nulk9BEcqw1tVwlseO6Jx+xVG7YHucftmkGRy+x2PsYC?= =?iso-8859-1?q?137D+IQavcEUZdQZ0IZQ5PgBhRe38I+eYbpplwur43G5nbZvYRVCBj5sDf0d?= =?iso-8859-1?q?XNZntMug7+hyv7Ihc8LGz5QPtUlc6pc4TYkIoTNxhHmFzC1RTbTGS7cjkYGD?= =?iso-8859-1?q?dP0i3Uiv8FFt1r9T+lcPMiOqtlGE71UbB6mCMDGeIjvrpVdJoF0P56mFxUqC?= =?iso-8859-1?q?xXiSg0Xl8P/JMqu8+ou5bBwNm2O6Vi19GYIdqFNQ7r/QdTwHOzv2/5Oszjye?= =?iso-8859-1?q?M8yoxqfsDg/IyMekrUyZO9rlcuY9Zb1Lb/VrKqL2NzlrMwDZODm1Isn9AQJy?= =?iso-8859-1?q?IWo2GmpRYuMnID0Ntjx69W+Q774S5Yg6AMs1ld1Js9Iqdv7CEWh52cChF7D+?= =?iso-8859-1?q?pdf4WOVdqdcyCvQXepF78LO9TPtmrF7D+QVX9uGq1u7VEQjo+WiHlHGikMVQ?= =?iso-8859-1?q?bLr6rm53FF4Nx9c6bKB5PwdtJj2ClpgZtuiMth2wM7bTPBBuVInv3XOeKxHE?= =?iso-8859-1?q?B6lf+w9tmaLuWsGL2FwRflPf44Fi++ez0KVIxvBasdxP4THCxMP6q7NcJ4ms?= =?iso-8859-1?q?Kb0ke0RYjIwJ8+zioJXeH7qxZpUOXdwBsZmIVYyulviDJVWnvdxfBwQ3VhkL?= =?iso-8859-1?q?BUS4M5M29lygzSmIelk5Exjg2ZRAoFB0Kk7D3bAnzLc8q640G/9n2jJkijkW?= =?iso-8859-1?q?c/XFtQ6jIxD39PqTINC5T9zXhC/kK1U3u6Ja4fFLLDUXkLaTTN8sQRzcIe9a?= =?iso-8859-1?q?SJX8Hn4CKpm4D5dINTzfxhVutf4ulDkNZqDpLwwrMYvI6XZd3oyMI+UUWcFW?= =?iso-8859-1?q?EhV8oEwsanZvktAnSDyawnOF2me67QSwN7ZP4/uLII9Y9HOUa12PpLkjU9om?= =?iso-8859-1?q?cVsCrQMCIULXGR/W9tnHpUCd6tQRMmT6iBDHRMCVeVXbfHBmqHTDTQlSd6Q4?= =?iso-8859-1?q?ZXbk9aP9flAB2agrzGgbqrSw7G87sbHOIeDT1N4pjrX1FfQ8OLFGz/hvGW7w?= =?iso-8859-1?q?0rTPrm541GCkeXVcLXmNlW3GD5lzVzf9/ITm2qcAQXmVNHFfaMn21zCGQ/De?= =?iso-8859-1?q?M2M5tG8CXyAmlO5o+ku2jGN6q6shwIUjhf8TMGm785y9F/OWKNn21/z+1iXL?= =?iso-8859-1?q?L6+8emPdHs3KbWEOrxyeFCwuW/3XzSqotISHDgWOUmAIqPyJLHdRUou/nFlQ?= =?iso-8859-1?q?ePsJF/NQ55RhCipW2bq6QSas0zOQdbX44I4GOPXJaJJrpaqFlcUlVVUFbyeb?= =?iso-8859-1?q?oQ+V52Eczxwg9BX8DKTFJwu2FV82aaPyw1MKPkB1zMcUhmraslP6PL8vC6sg?= =?iso-8859-1?q?L7/JmsoCygqoF7jwRhzGwQ3GSAjAza5Oftz/kmQ=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: Thalesgroup.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PR0P264MB1882.FRAP264.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: ae24cfbb-1fa1-4cce-a053-08db5157faf8 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 May 2023 13:10:51.7160 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 6e603289-5e46-4e26-ac7c-03a85420a9a5 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: v86hLWZ9OWRYcB2YWBiSTRe1eJtQmG3/eHFJqSJcJtJgIK6p7Q0OORdWI3HC/6jyalTYY6oUmIYq3+2epYlYWiwUZFl+lP2PvpJiyLus+Ro= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR0P264MB2028 X-PMX-Version: 6.4.8.2820816, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2023.5.10.130316, AntiVirus-Engine: 6.0.0, AntiVirus-Data: 2023.5.10.600000 X-Sophos-SenderHistory: ip=104.47.25.109,fs=392995,da=132438784,mc=9881,sc=3,hc=9878,sp=0,fso=131687209,re=222,sd=0,hd=30 X-PMX-Spam: Gauge=IIIIIIIII, Probability=9%, Report=' INVOICE_ATTACHMENT 0.1, HTML_00_01 0.05, HTML_00_10 0.05, MIME_TEXT_ONLY_MP_MIXED 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_4000_4999 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, DKIM_SIGNATURE 0, DQ_S_H 0, IN_REP_TO 0, KNOWN_MTA_TFX 0, LEGITIMATE_SIGNS 0, MSG_THREAD 0, NO_CTA_URI_FOUND 0, NO_URI_FOUND 0, NO_URI_HTTPS 0, REFERENCES 0, SPF_SOFTFAIL 0, SUSP_DH_NEG 0, SXL_IP_TFX_WM 0, WEBMAIL_SOURCE 0, __ARCAUTH_DKIM_PASSED 0, __ARCAUTH_DMARC_PASSED 0, __ARCAUTH_PASSED 0, __ARC_SEAL_MICROSOFT 0, __ARC_SIGNATURE_MICROSOFT 0, __ATTACHMENT_NOT_IMG 0, __ATTACH_CTE_BASE64 0, __AUTH_RES_DKIM_PASS 0, __BODY_NO_MAILTO 0, __CT 0, __CTYPE_HAS_BOUNDARY 0, __CTYPE_MULTIPART 0, __CTYPE_MULTIPART_MIXED 0, __DQ_IP_FSO_LARGE 0, __DQ_NEG_DOMAIN 0, __DQ_NEG_HEUR 0, __DQ_NEG_IP 0, __DQ_S_HIST_1 0, __DQ_S_IP_1MO 0, __DQ_S_IP_HD_10_P 0, __DQ_S_IP_MC_100_P 0, __DQ_S_IP_MC_10_P 0, __DQ_S_IP_MC_1K_P 0, __DQ_S_IP_MC_1_P 0, __DQ_S_IP_MC_5_P 0, __DQ_S_IP_RE_100_P 0, __DQ_S_IP_SC_1_P 0, __DQ_S_IP_SP_0_P 0, __FORWARDED_MSG 0, __FROM_DOMAIN_NOT_IN_BODY 0, __FROM_NAME_NOT_IN_BODY 0, __FUR_HEADER 0, __FUR_RDNS_OUTLOOK 0, __HAS_ATTACHMENT 0, __HAS_ATTACHMENT1 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_REFERENCES 0, __HAS_X_FF_ASR 0, __HAS_X_FF_ASR_CAT 0, __HAS_X_FF_ASR_SFV 0, __INVOICE_MULTILINGUAL 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_TEXT_P 0, __MIME_TEXT_P1 0, __MIME_TEXT_P2 0, __MIME_VERSION 0, __MSGID_32_64_CAPS 0, __MULTIPLE_RCPTS_TO_X2 0, __NO_HTML_TAG_RAW 0, __RDNS_WEBMAIL 0, __REFERENCES 0, __SANE_MSGID 0, __SCAN_D_NEG 0, __SCAN_D_NEG_HEUR 0, __SUBJ_ALPHA_NEGATE 0, __SUBJ_STARTS_S_BRACKETS 0, __TO_MALFORMED_2 0, __TO_NAME 0, __TO_NAME_DIFF_FROM_ACC 0, __TO_REAL_NAMES 0, __URI_NO_MAILTO 0, __X_FF_ASR_SCL_NSP 0, __X_FF_ASR_SFV_NSPM 0' List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 10 May 2023 13:10:59 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14777 Hi Richard! > I'd like to ensure that we add test coverage to "bitbake-selftest" to > cover this situation (see lib/bb/tests/fetch.py). > > We tend not to accept fixes like this unless we have test coverage to > prevent future regressions. Next version of my patch. I tried to have a as simple as possible test which exactly does show what goes wrong without my patch. Thank you! Cheers, Thilo From 1bc898b5e1abd98d2932dce053dc82b3f3d6b3f8 Mon Sep 17 00:00:00 2001 From: Thilo Cestonaro Date: Tue, 7 Feb 2023 13:44:02 +0100 Subject: [PATCH] bitbake: git fetcher: use urllib quote ... to use the path url-compatible. This needs to happen before the shell quotation happens. Without this commit, spaces in the clone URL will be used as " " and not as "%20" which will fail. This commit changes the " " in the URL to "%20" when it is a http or https url. Signed-off-by: Thilo Cestonaro --- lib/bb/fetch2/git.py | 8 +++++++- lib/bb/tests/fetch.py | 13 +++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 2a3c06fe..28729241 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -67,6 +67,7 @@ import re import shlex import subprocess import tempfile +import urllib import bb import bb.progress from contextlib import contextmanager @@ -698,7 +699,12 @@ class Git(FetchMethod): username = ud.user + '@' else: username = "" - return "%s://%s%s%s" % (ud.proto, username, ud.host, ud.path) + + path = ud.path + if ud.proto in [ 'http', 'https' ]: + path = urllib.parse.quote(ud.path) + + return "%s://%s%s%s" % (ud.proto, username, ud.host, path) def _revision_key(self, ud, d, name): """ diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 6ef0836f..db798f0d 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -1697,6 +1697,19 @@ class GitShallowTest(FetcherTest): ud = fetcher.ud[uri] return fetcher, ud + def assert_uri_with_spaces(self): + class FetchDataFake(): + proto = "https" + user = "" + host = "example.org" + path = "/git/url with spaces/imaginary.git" + + m = bb.fetch2.git.Git(self.d) + ud = FetchDataFake() + urlgenerated = m._get_repo_url(ud) + + self.assertEqual("https://example.org/git/url%20with%20spaces/imaginary.git", urlgenerated) + def fetch_and_unpack(self, uri=None): fetcher, ud = self.fetch(uri) fetcher.unpack(self.d.getVar('WORKDIR')) -- 2.39.2