From patchwork Fri Sep 6 09:27:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Lindeberg X-Patchwork-Id: 48724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D6D1CE7AF8 for ; Fri, 6 Sep 2024 09:27:54 +0000 (UTC) Received: from AM0PR83CU005.outbound.protection.outlook.com (AM0PR83CU005.outbound.protection.outlook.com [52.101.69.52]) by mx.groups.io with SMTP id smtpd.web11.30778.1725614868745324130 for ; Fri, 06 Sep 2024 02:27:49 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@axis.com header.s=selector1 header.b=OYikjzvo; spf=pass (domain: axis.com, ip: 52.101.69.52, mailfrom: christian.lindeberg@axis.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=IoK4CBZ0MqXiIEjYEfyWlUPEpKF7nXW2KxPELWc4Ae3818sCiXuSJmzivKoPC4VowRnSZ8zh4c6wNEKK8M/zYJUQdUsfUfYDsXO4VIY3czIC97OVDaUWqgbtr4lNSKHERISezA9RVzU2eEGArAXb8zU+UMET90uqRM1TVHIHrdE6vBxp2I8vNHGo6XXpq/zES551sxaGQJQoVXMW+j5/l1+lI9oQ+m0uUS6LRpqqzDN+fztNMcy357hrUDumyQS+h2wg0smPxi6KgrLHGIMG0eNi9xwfTwA8HCYJ9h+Ou5EiHNMnuO8oGNaiVK6Vr++a6LYFwEV5fve828VvherLMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qUgbWR0u1UXHJQAszdXZ/ZTGdpXb57jf0sX13f/7EHQ=; b=n6NCUBl/RbCQbXwFTdlmyM2/VnuEdg4MLlUR/8YjvtzbZwL948hKf0ToDzqq1QdlBSUFyaL16ImKM5qK0yeR0B3Xaeat7/k9dpvjcWdrmQxcfp4TMVDDijciFKwGgiKSDQbTpU+HETIg6MbtwV/GrldYozHBV7cCpHS7AkJOBojYU0ksJR0fr1+nfN2cVdNUhcW71cWlcH0y7vJPD4aOkv5mVXet7uME68e5fzsc8SuAlXiHHmUKOOnIOSCmp4/5Nag83LLpAI5BKFZCescOl8lM+4LuVzgp+aW+986EiO7ZQaBq4XWNwTIleQvP1PAqkFdZlGJwF+NLgtn42bAyGg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 195.60.68.100) smtp.rcpttodomain=lists.openembedded.org smtp.mailfrom=axis.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=axis.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axis.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qUgbWR0u1UXHJQAszdXZ/ZTGdpXb57jf0sX13f/7EHQ=; b=OYikjzvo7Tfurj1ZicYqZ2+Ab6MvGanN650mRZ+prY99DS8qgy/AW+GRCDYbPKgqMUMOE8vyildQ+jPHv2EOLL5WFpb/FHg1KIrRIWCKlkSdBdEOEGlfh8xmhGM4dlOzkJXS3LDqqWzAvDWx4fLhGzmifjasnDP+gu1UXvy/B0Q= Received: from AS8PR04CA0076.eurprd04.prod.outlook.com (2603:10a6:20b:313::21) by DB9PR02MB9828.eurprd02.prod.outlook.com (2603:10a6:10:456::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.27; Fri, 6 Sep 2024 09:27:40 +0000 Received: from AMS0EPF000001A9.eurprd05.prod.outlook.com (2603:10a6:20b:313:cafe::9a) by AS8PR04CA0076.outlook.office365.com (2603:10a6:20b:313::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.17 via Frontend Transport; Fri, 6 Sep 2024 09:27:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 195.60.68.100) smtp.mailfrom=axis.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=axis.com; Received-SPF: Pass (protection.outlook.com: domain of axis.com designates 195.60.68.100 as permitted sender) receiver=protection.outlook.com; client-ip=195.60.68.100; helo=mail.axis.com; pr=C Received: from mail.axis.com (195.60.68.100) by AMS0EPF000001A9.mail.protection.outlook.com (10.167.16.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7918.13 via Frontend Transport; Fri, 6 Sep 2024 09:27:40 +0000 Received: from SE-MAILARCH01W.axis.com (10.20.40.15) by se-mail01w.axis.com (10.20.40.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 6 Sep 2024 11:27:39 +0200 Received: from se-mail01w.axis.com (10.20.40.7) by SE-MAILARCH01W.axis.com (10.20.40.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 6 Sep 2024 11:27:39 +0200 Received: from se-intmail01x.se.axis.com (10.0.5.60) by se-mail01w.axis.com (10.20.40.7) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 6 Sep 2024 11:27:39 +0200 Received: from pc50740-2234.se.axis.com (pc50740-2234.se.axis.com [10.94.131.4]) by se-intmail01x.se.axis.com (Postfix) with ESMTP id 704801CE for ; Fri, 6 Sep 2024 11:27:39 +0200 (CEST) Received: by pc50740-2234.se.axis.com (Postfix, from userid 21184) id 694BA26B985E; Fri, 6 Sep 2024 11:27:39 +0200 (CEST) From: Christian Lindeberg To: Subject: [PATCH v2 1/2] fetch2: Add gomod fetcher Date: Fri, 6 Sep 2024 11:27:38 +0200 Message-ID: <20240906092739.930765-1-christli@axis.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AMS0EPF000001A9:EE_|DB9PR02MB9828:EE_ X-MS-Office365-Filtering-Correlation-Id: f8850301-50e9-41fb-b882-08dcce56273b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: fCj8hnrWxCBFZZEAk5HhjIr/JbpKJXXPCeAexeO4TGlE3/tlkSf+rnUqtLGbcNRP+u6M5wVUMQxNT8YgOq/KCIRMCs1WlvyCqv0e9hoGO4qERtRugP3CWGI7yiWQeo5fjwBd1XwyftqNqbDBhuAxiW6WCjNJ8+01hWHGDLuK9n3FbH4xTDvQFuCLsXkXz8VUX+W6kYh+M3qm3DF2BKUJ30vqo6oOJ79drDIYROieOKRiJu8UHMayk7/LY5Z6a/d3LaV0pBUpTJJi94Hygfo+1+ajooG0KwnDgKCYBQiBHeaODLrVuWL3EVfpc2AQd4pjWK4LRnXOkM4qAlNBicAVtJyCCIGHLXeBpUUza0/JYtVOF59R2rg8cEel9pUXnQaz6uc5yThPcFPtuebR9cs9+evFHJlHNvO49zb4w/CHLmMPI38jx1ACEH6vI8L/9UFDlPGxBLsVS15S69x01yW1D3z4BR9PM/qd+gUMw8R0UXUutasHyQeAxZ9Qir+C+a5sb5CtfCF/CUO1HBZZCGFK35dCJWiE0K/zH3y7qLKHUNeKe52g8eVF33BYPDQrryyd/dkXALq6EmLbWWrDLNYLBN8Wv3QZl7OE5L6PO/vxLxM8GWM5JOVp48qqGphoppL/VieGkGYZjHtHWayPHUNvSO2TdiR58enl0Lu97CMTfU9wp6weS+Untza+EC0oZsD84/oXzCTABkY7OAMjq1i/OJetOS1TX2iEX68PkswGsknXBxTPC0cfy+x95Lfjf0fmew3AL2AV2eGNUp1vAjMd/X/MsE3YZLUXkTWVDofoyAxLv09JgyRjzz2ZISembZaVRM+K8dkqsBD8kuCuPZNYBlYoEam6Z+ID2XtsyM2aJ0A7aJDcAiri4VNZNhW84NvuDk/OdDF6Fse3LL+DzNXIN9fEO0xRJ/ww5JVhQ7WJWmjd0DPryEbsZMwAUP3iwjHQRdoCa21kcsUsPhC/O5hZw9LYPz1y4plE+hofVEhP+m50jNWJ8QqbhfnC6G/04SsPKjlJc3hWDjaotD1oN7uMbsyGNo4MVrb/qGfNxvVQgVIcBk9Rc6ohpAwYebN1HSQ5Rdz18Oq296/QuQvT8VF0DpQBTU84TTyhvoZmzlYI99hJTdwucbXMuKDq9OZwT2MeH7ZsNVDWhrHw1I3qqasSQIZSwbYSaG2/Dy3OGJHHySBDLSM7HEF44ONYUSD2jpCIKBvW6r8KBF39R0Bty90m0AF1nrY9KBEK9n3+TvvJBVBv57OPEWliqXY1FKqsCb59QBGlUEaVv3kHB9orrxnAc11MFqfb7eIF8Bflp0OBRV+Yc2sSMqaDQ8wDxNSHv4X56lvw5dX5gvcDqWxa6DXCiMsHAHXFO2oCeNNC6FEZbh74dyTnde8GgXoG0c6zqG0T X-Forefront-Antispam-Report: CIP:195.60.68.100;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.axis.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: axis.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Sep 2024 09:27:40.0033 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f8850301-50e9-41fb-b882-08dcce56273b X-MS-Exchange-CrossTenant-Id: 78703d3c-b907-432f-b066-88f7af9ca3af X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=78703d3c-b907-432f-b066-88f7af9ca3af;Ip=[195.60.68.100];Helo=[mail.axis.com] X-MS-Exchange-CrossTenant-AuthSource: AMS0EPF000001A9.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR02MB9828 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 06 Sep 2024 09:27:54 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16545 From: Christian Lindeberg Add a go module fetcher for downloading module dependencies to the module cache from a module proxy. The fetcher can be used with the go-mod class in OE-Core. A module dependency can be specified with: SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..." Signed-off-by: Christian Lindeberg --- Changes in V2: - Use GO_MOD_CACHE_DIR variable or default instead of requiring the GOMODCACHE environment variable to be set. - Add more documentation to make it cleared what is downloaded and what is unpacked. lib/bb/fetch2/__init__.py | 4 +- lib/bb/fetch2/gomod.py | 128 ++++++++++++++++++++++++++++++++++++++ lib/bb/tests/fetch.py | 65 +++++++++++++++++++ 3 files changed, 196 insertions(+), 1 deletion(-) create mode 100644 lib/bb/fetch2/gomod.py diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 5bf2c4b8c..f84ce5999 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1317,7 +1317,7 @@ class FetchData(object): if checksum_name in self.parm: checksum_expected = self.parm[checksum_name] - elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs"]: + elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs", "gomod"]: checksum_expected = None else: checksum_expected = d.getVarFlag("SRC_URI", checksum_name) @@ -2088,6 +2088,7 @@ from . import npmsw from . import az from . import crate from . import gcp +from . import gomod methods.append(local.Local()) methods.append(wget.Wget()) @@ -2110,3 +2111,4 @@ methods.append(npmsw.NpmShrinkWrap()) methods.append(az.Az()) methods.append(crate.Crate()) methods.append(gcp.GCP()) +methods.append(gomod.GoMod()) diff --git a/lib/bb/fetch2/gomod.py b/lib/bb/fetch2/gomod.py new file mode 100644 index 000000000..fe025e367 --- /dev/null +++ b/lib/bb/fetch2/gomod.py @@ -0,0 +1,128 @@ +""" +BitBake 'Fetch' implementation for Go modules + +The gomod fetcher is used to download Go modules to the module cache from a +module proxy. + +Example SRC_URI: + +SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..." + +Required SRC_URI parameters: + +- version + The version of the module. + +Optional SRC_URI parameters: + +- mod + Fetch and unpack the go.mod file only instead of the complete module. + The go command may need to download go.mod files for many different modules + when computing the build list, and go.mod files are much smaller than + module zip files. + The default is "0", set mod=1 for the go.mod file only. + +- sha256sum + The checksum of the module zip file, or the go.mod file in case of fetching + only the go.mod file. Alternatively, set the SRC_URI varible flag for + "module@version.sha256sum". + +Related variables: + +- GO_MOD_PROXY + The module proxy used by the fetcher. + +- GO_MOD_CACHE_DIR + The directory where the module cache is located. + This must match the exported GOMODCACHE variable for the go command to find + the downloaded modules. + +See the Go modules reference, https://go.dev/ref/mod, for more information +about the module cache, module proxies and version control systems. +""" + +import os +import re +import shutil +import zipfile + +import bb +from bb.fetch2 import FetchError +from bb.fetch2 import MissingParameterError +from bb.fetch2.wget import Wget + + +def escape(path): + """Escape capital letters using exclamation points.""" + return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path) + + +class GoMod(Wget): + """Class to fetch Go modules from a Go module proxy via wget""" + + def supports(self, ud, d): + """Check to see if a given URL is for this fetcher.""" + return ud.type == 'gomod' + + def urldata_init(self, ud, d): + """Set up to download the module from the module proxy. + + Set up to download the module zip file to the module cache directory + and unpack the go.mod file (unless downloading only the go.mod file): + + cache/download//@v/.zip: The module zip file. + cache/download//@v/.mod: The go.mod file. + """ + + proxy = d.getVar('GO_MOD_PROXY') or 'proxy.golang.org' + moddir = d.getVar('GO_MOD_CACHE_DIR') or 'pkg/mod' + + if 'version' not in ud.parm: + raise MissingParameterError('version', ud.url) + + module = ud.host + ud.path + ud.parm['module'] = module + + # Set URL and filename for wget download + path = escape(module + '/@v/' + ud.parm['version']) + if ud.parm.get('mod', '0') == '1': + path += '.mod' + else: + path += '.zip' + ud.parm['unpack'] = '0' + ud.url = bb.fetch2.encodeurl( + ('https', proxy, '/' + path, None, None, None)) + ud.parm['downloadfilename'] = path + + # Set name parameter if sha256sum is set in recipe + name = f"{module}@{ud.parm['version']}" + if d.getVarFlag('SRC_URI', name + '.sha256sum'): + ud.parm['name'] = name + + # Set subdir for unpack + ud.parm['subdir'] = os.path.join(moddir, 'cache/download', + os.path.dirname(path)) + + super().urldata_init(ud, d) + + def unpack(self, ud, rootdir, d): + """Unpack the module in the module cache.""" + + # Unpack the module zip file or go.mod file + super().unpack(ud, rootdir, d) + + if ud.localpath.endswith('.zip'): + # Unpack the go.mod file from the zip file + module = ud.parm['module'] + unpackdir = os.path.join(rootdir, ud.parm['subdir']) + name = os.path.basename(ud.localpath).rsplit('.', 1)[0] + '.mod' + bb.note(f"Unpacking {name} to {unpackdir}/") + with zipfile.ZipFile(ud.localpath) as zf: + with open(os.path.join(unpackdir, name), mode='wb') as mf: + try: + f = module + '@' + ud.parm['version'] + '/go.mod' + shutil.copyfileobj(zf.open(f), mf) + except KeyError: + # If the module does not have a go.mod file, synthesize + # one containing only a module statement. + mf.write(f'module {module}\n'.encode()) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 2ef206343..2365a5096 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -3390,3 +3390,68 @@ class FetchPremirroronlyBrokenTarball(FetcherTest): fetcher.download() output = "".join(logs.output) self.assertFalse(" not a git repository (or any parent up to mount point /)" in output) + +class GoModTest(FetcherTest): + + @skipIfNoNetwork() + def test_gomod_url(self): + urls = ['gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;' + 'sha256sum=9bb69aea32f1d59711701f9562d66432c9c0374205e5009d1d1a62f03fb4fdad'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.zip') + self.assertNotIn('name', ud.parm) + + fetcher.download() + fetcher.unpack(self.unpackdir) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod'))) + + @skipIfNoNetwork() + def test_gomod_url_go_mod_only(self): + urls = ['gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;mod=1;' + 'sha256sum=7873b8544842329b4f385a3aa6cf82cc2bc8defb41a04fa5291c35fd5900e873'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.mod') + self.assertNotIn('name', ud.parm) + + fetcher.download() + fetcher.unpack(self.unpackdir) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod'))) + + @skipIfNoNetwork() + def test_gomod_url_sha256sum_varflag(self): + urls = ['gomod://gopkg.in/ini.v1;version=v1.67.0'] + self.d.setVarFlag('SRC_URI', 'gopkg.in/ini.v1@v1.67.0.sha256sum', 'bd845dfc762a87a56e5a32a07770dc83e86976db7705d7f89c5dbafdc60b06c6') + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.url, 'https://proxy.golang.org/gopkg.in/ini.v1/%40v/v1.67.0.zip') + self.assertEqual(ud.parm['name'], 'gopkg.in/ini.v1@v1.67.0') + + fetcher.download() + fetcher.unpack(self.unpackdir) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod'))) + + @skipIfNoNetwork() + def test_gomod_url_no_go_mod_in_module(self): + urls = ['gomod://gopkg.in/ini.v1;version=v1.67.0;' + 'sha256sum=bd845dfc762a87a56e5a32a07770dc83e86976db7705d7f89c5dbafdc60b06c6'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.url, 'https://proxy.golang.org/gopkg.in/ini.v1/%40v/v1.67.0.zip') + self.assertNotIn('name', ud.parm) + + fetcher.download() + fetcher.unpack(self.unpackdir) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod'))) From patchwork Fri Sep 6 09:27:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Lindeberg X-Patchwork-Id: 48725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EDE8CE7AF9 for ; Fri, 6 Sep 2024 09:27:54 +0000 (UTC) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (EUR05-AM6-obe.outbound.protection.outlook.com [40.107.22.84]) by mx.groups.io with SMTP id smtpd.web10.31069.1725614868331835284 for ; Fri, 06 Sep 2024 02:27:49 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@axis.com header.s=selector1 header.b=CD1+zOJT; spf=pass (domain: axis.com, ip: 40.107.22.84, mailfrom: christian.lindeberg@axis.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lld813aV9bMGw7WlWGXlbn6AQOH2YkvOrSFWmxBOxHsmoTw1B7jhkRGsudLRLUJ95w9jJCvhhD1TiHkvzPC9jegXuUeyI63PQ409L6ZIFaTDSCSAW6G9p7uWGvpjfDy00vgqSOLMj0Y9YKhiGsLdBOPUpn01yh4GsWrGebJGr7pz/nZry6XOWxGMkLZ/F64kQdxw7t4g4reA6EkVc2huQ8FnrBqSs3DN7sxTmhM6yCyYUN+xYtyNHL3zFYe7Dn/BLe/ZSDREHQNYWqYv//imqoReigvAsSCws1XLhiC2QtXpMRAUEzN+rOHHZIGozpO0rSgzO0Vmu/0BMxYBCOdftQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Kfy1qE/o6leAYcn3EQhIWgZPOGGyaRYAZvSpekzUFJY=; b=Wp1wHAyKYrdIl1YbJOdl0Y1TJEIRwQU+i8xCtxxrqAO93tNSyhDYfs0GoT5M1j1ujD3Vc71kmkaad/OXTFJ4gDqRO+TzmzkkEuFYO8LYe8Tjn4RpcO7uxQ8m6TrwC4W/J9FvajJdgEysRFGYpcLFFR+OWNaodKfiVdYOv1YZGry4TavlfOce2uwEFYz4K6y8Y7RCx/p23KfvPX07A4raM1HgjHiIExbGrGYTQtAEEBdCPZVcoQIlsnPP+NJ/xQTKsy/hUtDmRs3GGgM4krPkMaZq2B9+2auL/orvRpApTqkrnfNzG3QCqnMt9O4JW8NkMTIbxT3PLGHvL4YZssrzPw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 195.60.68.100) smtp.rcpttodomain=lists.openembedded.org smtp.mailfrom=axis.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=axis.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axis.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Kfy1qE/o6leAYcn3EQhIWgZPOGGyaRYAZvSpekzUFJY=; b=CD1+zOJTkTV/163JwwifWe74Suh4S8AintwP+3k5XRPbyA/BLCQtWKj/vf2483w8KuteDrvHdhXqp/uQZVZurap3Ev3oY5WB+txdwbTH1PxXqRbOnqTnca136lCKn4FmZPQ2aA4auYEPAKK3CgaLsoU65F9cLJtxfRzboAz2suM= Received: from AS4P190CA0051.EURP190.PROD.OUTLOOK.COM (2603:10a6:20b:656::25) by AM0PR02MB5905.eurprd02.prod.outlook.com (2603:10a6:208:188::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.17; Fri, 6 Sep 2024 09:27:41 +0000 Received: from AMS1EPF00000042.eurprd04.prod.outlook.com (2603:10a6:20b:656:cafe::6) by AS4P190CA0051.outlook.office365.com (2603:10a6:20b:656::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.17 via Frontend Transport; Fri, 6 Sep 2024 09:27:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 195.60.68.100) smtp.mailfrom=axis.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=axis.com; Received-SPF: Pass (protection.outlook.com: domain of axis.com designates 195.60.68.100 as permitted sender) receiver=protection.outlook.com; client-ip=195.60.68.100; helo=mail.axis.com; pr=C Received: from mail.axis.com (195.60.68.100) by AMS1EPF00000042.mail.protection.outlook.com (10.167.16.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7918.13 via Frontend Transport; Fri, 6 Sep 2024 09:27:39 +0000 Received: from se-mail01w.axis.com (10.20.40.7) by se-mail02w.axis.com (10.20.40.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 6 Sep 2024 11:27:39 +0200 Received: from se-intmail01x.se.axis.com (10.0.5.60) by se-mail01w.axis.com (10.20.40.7) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 6 Sep 2024 11:27:39 +0200 Received: from pc50740-2234.se.axis.com (pc50740-2234.se.axis.com [10.94.131.4]) by se-intmail01x.se.axis.com (Postfix) with ESMTP id 712093CD for ; Fri, 6 Sep 2024 11:27:39 +0200 (CEST) Received: by pc50740-2234.se.axis.com (Postfix, from userid 21184) id 6AE6B26B985F; Fri, 6 Sep 2024 11:27:39 +0200 (CEST) From: Christian Lindeberg To: Subject: [PATCH v2 2/2] fetch2: Add gomodgit fetcher Date: Fri, 6 Sep 2024 11:27:39 +0200 Message-ID: <20240906092739.930765-2-christli@axis.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240906092739.930765-1-christli@axis.com> References: <20240906092739.930765-1-christli@axis.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AMS1EPF00000042:EE_|AM0PR02MB5905:EE_ X-MS-Office365-Filtering-Correlation-Id: bf26b752-21d7-4d65-8f78-08dcce562736 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|82310400026; X-Microsoft-Antispam-Message-Info: pupOr6iRlszddrgKmIwt+3Wzo0dYWWLchkda4RX+21FJuEY3Bj/aPpwy9VPoPe3pCIgJaF8fJ49K1cGMtFrEkRuL/mfHy7Ot1l1pIHExd/qy8GYgY00F/Map1dgVLJqF4cYDABIOcLZ2WjTeh+DxPzgjCNrMF0yuUZvZKxAn0hwnKOJILWaz3yDTfipwo1APQVMYFx9qV4pY8kmr7H9V+ICZXV4TXhgUIHCNTAHth9e7jgDgtVdUw6lXoUN/IhU/q8A2Ds21k/jDu2voaUEi1HQK1A0VC1iqxn8fhHazGr52O02nJL7rm1p1q5DteUw1z4QlHD5mued0GzBMJBPO8eM4VnQJDRtv7r43BgOaQ/MNuGnorllV5ByRROX2lVa3bBpbuJdpiK9fGw9BhR61Y9aixf3I6GfvKKG5a0apzSZIjBXeML4M74ZVyp/LX4ttvnhOfCAOO8Re4IjN2F9dQ3EZ3KrLAeRsR+3Pi96loxHiaKXJGq1xr0cRXh7qyl6td9UylJK1N+ckH5p3C8E9OisSUuJBzbVA43KyuMKAlBYUi14rj7k7n4DNAT4yf9M8gajJJKhsgK8/HGey3YrBJZua26/LXQINuyVmLsCBQNOqRFENOfnUaQcKD1mJgvRRaqh7/LDzktiXM7dlN+BFWNk3mzc3wk3D3OdSnbd/DfS0z9Odwb9J74PUL94ztdBdGHJz3BgD32YaqV4v2T6u2oCyGqegJenk8qHztZNQouNMG9jW+iGjaxpzVJRfEXHz+e5jQjU+1aYFUOqB0Sby6l2pTPZmGiWK/FqMAkBecHdGz8XCizwjnYC2uHMravPBa55bcVSmGPi+1HPwyABAce7uou0bUfGlxZR7m4V50Alib93BtnSD0eObm6oJCvJrNF5Dg+XBr+a93Nqv/H7YOelF9K7T9Ajroq4esagmcc89p+pDuxU/0Oq2a0RW0hN5gnZRUybhE1Y3t/fFFuq2E1CdbNenXPafC5n36vZ3Wb0aEHmn2qfBX8mj4QiAB5Zq7iWLj4htdrwhM0h+a+TYpN9jEMLgPmoHHLHHIj3L8DXh2UQxVwgJsE/8ECM2x9EOTzhVFZsgGv3xOIPqu6ipJUyrUh0d5xb2/U0OVkh7dVuybBiGUPDZNdxctCn2ucHeHJ9rBvokGW+OTd36Up7KI/FuOzKYKTGn/55ZuGmpra0y9XHKIg+WRgvrnIh87x8ZtCCjcWEYG2KAFkI/43MqSYzZ75b2sJ2f6ZlxC7UdUBAHtMjegY8ennLkfdUBal+NTKTtQ/05FgtPrGawcGNUwhBhkN5+n0XLieouL1W38swbqu8bkYYtM0jymTSx46DcTDNKTQhXoCc85AYk+QUA77JsyiDylX7mivkRG7dXZiqhL0YJT8kNQLVaQm8QMluc X-Forefront-Antispam-Report: CIP:195.60.68.100;CTRY:SE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.axis.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: axis.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Sep 2024 09:27:39.9695 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bf26b752-21d7-4d65-8f78-08dcce562736 X-MS-Exchange-CrossTenant-Id: 78703d3c-b907-432f-b066-88f7af9ca3af X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=78703d3c-b907-432f-b066-88f7af9ca3af;Ip=[195.60.68.100];Helo=[mail.axis.com] X-MS-Exchange-CrossTenant-AuthSource: AMS1EPF00000042.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR02MB5905 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 06 Sep 2024 09:27:54 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16544 From: Christian Lindeberg Add a go module fetcher for downloading module dependencies to the module cache directly from a git repository. The fetcher can be used with the go-mod class in OE-Core. A module dependency can be specified with: SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;srcrev=..." Signed-off-by: Christian Lindeberg --- Changes in V2: - Use separate gomodgit:// type instead of the direct=git parameter. - Use GO_MOD_CACHE_DIR variable or default instead of requiring the GOMODCACHE environment variable to be set. - Add more documentation to make it cleared what is downloaded and what is unpacked. lib/bb/fetch2/__init__.py | 1 + lib/bb/fetch2/gomod.py | 140 +++++++++++++++++++++++++++++++++++++- lib/bb/tests/fetch.py | 89 ++++++++++++++++++++++++ 3 files changed, 228 insertions(+), 2 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index f84ce5999..ddee4400b 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -2112,3 +2112,4 @@ methods.append(az.Az()) methods.append(crate.Crate()) methods.append(gcp.GCP()) methods.append(gomod.GoMod()) +methods.append(gomod.GoModGit()) diff --git a/lib/bb/fetch2/gomod.py b/lib/bb/fetch2/gomod.py index fe025e367..1b532d03f 100644 --- a/lib/bb/fetch2/gomod.py +++ b/lib/bb/fetch2/gomod.py @@ -1,12 +1,13 @@ """ BitBake 'Fetch' implementation for Go modules -The gomod fetcher is used to download Go modules to the module cache from a -module proxy. +The gomod/gomodgit fetchers are used to download Go modules to the module cache +from a module proxy or directly from a version control repository. Example SRC_URI: SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..." +SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;repo=go.googlesource.com/net;srcrev=..." Required SRC_URI parameters: @@ -27,6 +28,23 @@ Optional SRC_URI parameters: only the go.mod file. Alternatively, set the SRC_URI varible flag for "module@version.sha256sum". +- protocol + The method used when fetching directly from a version control repository. + The default is "https" for git. + +- repo + The URL when fetching directly from a version control repository. Required + when the URL is different from the module path. + +- srcrev + The revision identifier used when fetching directly from a version control + repository. Alternatively, set the SRCREV varible for "module@version". + +- subdir + The module subdirectory when fetching directly from a version control + repository. Required when the module is not located in the root of the + repository. + Related variables: - GO_MOD_PROXY @@ -41,14 +59,19 @@ See the Go modules reference, https://go.dev/ref/mod, for more information about the module cache, module proxies and version control systems. """ +import hashlib import os import re import shutil +import subprocess import zipfile import bb from bb.fetch2 import FetchError from bb.fetch2 import MissingParameterError +from bb.fetch2 import runfetchcmd +from bb.fetch2 import subprocess_setup +from bb.fetch2.git import Git from bb.fetch2.wget import Wget @@ -126,3 +149,116 @@ class GoMod(Wget): # If the module does not have a go.mod file, synthesize # one containing only a module statement. mf.write(f'module {module}\n'.encode()) + + +class GoModGit(Git): + """Class to fetch Go modules directly from a git repository""" + + def supports(self, ud, d): + """Check to see if a given URL is for this fetcher.""" + return ud.type == 'gomodgit' + + def urldata_init(self, ud, d): + """Set up to download the module from the git repository. + + Set up to download the git repository to the module cache directory and + unpack the module zip file and the go.mod file: + + cache/vcs/: The bare git repository. + cache/download//@v/.zip: The module zip file. + cache/download//@v/.mod: The go.mod file. + """ + + moddir = d.getVar('GO_MOD_CACHE_DIR') or 'pkg/mod' + + if 'version' not in ud.parm: + raise MissingParameterError('version', ud.url) + + module = ud.host + ud.path + ud.parm['module'] = module + + # Set host, path and srcrev for git download + if 'repo' in ud.parm: + repo = ud.parm['repo'] + idx = repo.find('/') + if idx != -1: + ud.host = repo[:idx] + ud.path = repo[idx:] + else: + ud.host = repo + ud.path = '' + if 'protocol' not in ud.parm: + ud.parm['protocol'] = 'https' + name = f"{module}@{ud.parm['version']}" + ud.names = [name] + srcrev = d.getVar('SRCREV_' + name) + if srcrev: + if 'srcrev' not in ud.parm: + ud.parm['srcrev'] = srcrev + else: + if 'srcrev' in ud.parm: + d.setVar('SRCREV_' + name, ud.parm['srcrev']) + if 'branch' not in ud.parm: + ud.parm['nobranch'] = '1' + + # Set subpath, subdir and bareclone for git unpack + if 'subdir' in ud.parm: + ud.parm['subpath'] = ud.parm['subdir'] + key = f"git3:{ud.parm['protocol']}://{ud.host}{ud.path}".encode() + ud.parm['key'] = key + ud.parm['subdir'] = os.path.join(moddir, 'cache/vcs', + hashlib.sha256(key).hexdigest()) + ud.parm['bareclone'] = '1' + + super().urldata_init(ud, d) + + def unpack(self, ud, rootdir, d): + """Unpack the module in the module cache.""" + + # Unpack the bare git repository + super().unpack(ud, rootdir, d) + + moddir = d.getVar('GO_MOD_CACHE_DIR') or 'pkg/mod' + + # Create the info file + module = ud.parm['module'] + repodir = os.path.join(rootdir, ud.parm['subdir']) + with open(repodir + '.info', 'wb') as f: + f.write(ud.parm['key']) + + # Unpack the go.mod file from the repository + unpackdir = os.path.join(rootdir, moddir, 'cache/download', + escape(module), '@v') + bb.utils.mkdirhier(unpackdir) + srcrev = ud.parm['srcrev'] + version = ud.parm['version'] + escaped_version = escape(version) + cmd = f"git ls-tree -r --name-only '{srcrev}'" + if 'subpath' in ud.parm: + cmd += f" '{ud.parm['subpath']}'" + files = runfetchcmd(cmd, d, workdir=repodir).split() + name = escaped_version + '.mod' + bb.note(f"Unpacking {name} to {unpackdir}/") + with open(os.path.join(unpackdir, name), mode='wb') as mf: + f = 'go.mod' + if 'subpath' in ud.parm: + f = os.path.join(ud.parm['subpath'], f) + if f in files: + cmd = ['git', 'cat-file', 'blob', srcrev + ':' + f] + subprocess.check_call(cmd, stdout=mf, cwd=repodir, + preexec_fn=subprocess_setup) + else: + # If the module does not have a go.mod file, synthesize one + # containing only a module statement. + mf.write(f'module {module}\n'.encode()) + + # Synthesize the module zip file from the repository + name = escaped_version + '.zip' + bb.note(f"Unpacking {name} to {unpackdir}/") + with zipfile.ZipFile(os.path.join(unpackdir, name), mode='w') as zf: + prefix = module + '@' + version + '/' + for f in files: + cmd = ['git', 'cat-file', 'blob', srcrev + ':' + f] + data = subprocess.check_output(cmd, cwd=repodir, + preexec_fn=subprocess_setup) + zf.writestr(prefix + f, data) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 2365a5096..832e0dd6a 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -3455,3 +3455,92 @@ class GoModTest(FetcherTest): downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip'))) self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod'))) + +class GoModGitTest(FetcherTest): + + @skipIfNoNetwork() + def test_gomodgit_url_repo(self): + urls = ['gomodgit://golang.org/x/net;version=v0.9.0;' + 'repo=go.googlesource.com/net;' + 'srcrev=694cff8668bac64e0864b552bffc280cd27f21b1'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.host, 'go.googlesource.com') + self.assertEqual(ud.path, '/net') + self.assertEqual(ud.names, ['golang.org/x/net@v0.9.0']) + self.assertEqual(self.d.getVar('SRCREV_golang.org/x/net@v0.9.0'), '694cff8668bac64e0864b552bffc280cd27f21b1') + + fetcher.download() + self.assertTrue(os.path.exists(ud.localpath)) + + fetcher.unpack(self.unpackdir) + vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs') + self.assertTrue(os.path.exists(os.path.join(vcsdir, 'ed42bd05533fd84ae290a5d33ebd3695a0a2b06131beebd5450825bee8603aca'))) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'golang.org/x/net/@v/v0.9.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'golang.org/x/net/@v/v0.9.0.mod'))) + + @skipIfNoNetwork() + def test_gomodgit_url_subdir(self): + urls = ['gomodgit://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;' + 'repo=github.com/Azure/azure-sdk-for-go;subdir=sdk/storage/azblob;' + 'srcrev=ec928e0ed34db682b3f783d3739d1c538142e0c3'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.host, 'github.com') + self.assertEqual(ud.path, '/Azure/azure-sdk-for-go') + self.assertEqual(ud.parm['subpath'], 'sdk/storage/azblob') + self.assertEqual(ud.names, ['github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0']) + self.assertEqual(self.d.getVar('SRCREV_github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0'), 'ec928e0ed34db682b3f783d3739d1c538142e0c3') + + fetcher.download() + self.assertTrue(os.path.exists(ud.localpath)) + + fetcher.unpack(self.unpackdir) + vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs') + self.assertTrue(os.path.exists(os.path.join(vcsdir, 'd31d6145676ed3066ce573a8198f326dea5be45a43b3d8f41ce7787fd71d66b3'))) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod'))) + + @skipIfNoNetwork() + def test_gomodgit_url_srcrev_var(self): + urls = ['gomodgit://gopkg.in/ini.v1;version=v1.67.0'] + self.d.setVar('SRCREV_gopkg.in/ini.v1@v1.67.0', 'b2f570e5b5b844226bbefe6fb521d891f529a951') + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.host, 'gopkg.in') + self.assertEqual(ud.path, '/ini.v1') + self.assertEqual(ud.names, ['gopkg.in/ini.v1@v1.67.0']) + self.assertEqual(ud.parm['srcrev'], 'b2f570e5b5b844226bbefe6fb521d891f529a951') + + fetcher.download() + fetcher.unpack(self.unpackdir) + vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs') + self.assertTrue(os.path.exists(os.path.join(vcsdir, 'b7879a4be9ba8598851b8278b14c4f71a8316be64913298d1639cce6bde59bc3'))) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod'))) + + @skipIfNoNetwork() + def test_gomodgit_url_no_go_mod_in_module(self): + urls = ['gomodgit://gopkg.in/ini.v1;version=v1.67.0;' + 'srcrev=b2f570e5b5b844226bbefe6fb521d891f529a951'] + + fetcher = bb.fetch2.Fetch(urls, self.d) + ud = fetcher.ud[urls[0]] + self.assertEqual(ud.host, 'gopkg.in') + self.assertEqual(ud.path, '/ini.v1') + self.assertEqual(ud.names, ['gopkg.in/ini.v1@v1.67.0']) + self.assertEqual(self.d.getVar('SRCREV_gopkg.in/ini.v1@v1.67.0'), 'b2f570e5b5b844226bbefe6fb521d891f529a951') + + fetcher.download() + fetcher.unpack(self.unpackdir) + vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs') + self.assertTrue(os.path.exists(os.path.join(vcsdir, 'b7879a4be9ba8598851b8278b14c4f71a8316be64913298d1639cce6bde59bc3'))) + downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download') + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip'))) + self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')))