From patchwork Fri Sep 5 06:20:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 69710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B162CA1015 for ; Fri, 5 Sep 2025 06:21:24 +0000 (UTC) Received: from AM0PR02CU008.outbound.protection.outlook.com (AM0PR02CU008.outbound.protection.outlook.com [52.101.72.84]) by mx.groups.io with SMTP id smtpd.web10.14988.1757053280733913428 for ; Thu, 04 Sep 2025 23:21:23 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@weidmueller.com header.s=selector2 header.b=1UvhUee4; spf=pass (domain: weidmueller.com, ip: 52.101.72.84, mailfrom: stefan.herbrechtsmeier-oss@weidmueller.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=g84yOtDmtl0xpjb/CHSepV4Ws2A4Fd94dYNaw6mGazorD3vcAffkysPYLmGekGrB3M7xek8MMVemmfuCTgakbcMvWo95h6SGinzrwpdo3VJFW0kuEpcI1bocQ/6i6ec5UYmS+bgz9P1bPO2UO95h7OKErJN9mvQP+AoU1tOqS7Tu57Eke73vOPMAH3Cf/E0XLVvzi89/sxWFIbb0YKnz4kvAVHrcTkRVG7hlN1rLW0VYZykDI/GikxBdV7tcuZ91wYSIHZN19ZpV1UiQWPWeOxtzZ6kVnhNu9+rGqTRcIbzup9Z3MhVMQd/Kov8tj/on1q/xxETZFX0HtPJ52rw5Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6BYOfzCZEnDeicVH5gylifvft4s0qoNVgkS+Ec8hDrs=; b=q1DSd1vZPWrkLQaeGPPPWlyAE5ketkEkgz0EL1yW1LjHUmiPYrof13DwP//HABPUCzrGJkGY0o0T6HyG65SQOVn1wJwxKGKna8k9WP6X9D2a7w208KB0X1igReCShq5FoHlSd96TKHa8PbqdXvjCOCoKGwnmBSY4aWmQhgVZSd9TnwnN5M33MONvVFMjy25lHbGgVJH8XDIn2Z3GanFg+rD6UCL9z61F7OpJaqxr2u/vkAN89J0lB1CzXHqURWrGuUCfXGC2g/3aI9njHnTH/iOB48W/IzamxWhxNn80Qnvrsgy/Vi7FddBOtgAdYdyQODsyl/Mii4DHAJLR52EkbA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6BYOfzCZEnDeicVH5gylifvft4s0qoNVgkS+Ec8hDrs=; b=1UvhUee4oBAC5XYimRROy4g6rO8nId/0DKaCTdZoz+reL3UdEYA2KWurxJHHLXgjtY6q2jyGSrM8jwgYWe3y/kbzBRcAFjiUqppzyPJkMG5MTLntHbVdbDOD/tJCmA9VmGDvcxpN+xVT814uZR16GHOqnKgBqLFHGS17b7zVe31MMnyvaq67ai9ZShlttNujVzqLgihfIK8IpIdV9jf6+kaBamS7qCpzR6HerZLgt0BBuvNKA6D3v72p4yOfO4BegTrVHB+xQR2sMKJCalVi2p0pfSy3RI4fD1qPmH3lDZj+WaiyCOid9mBrmP1WmpkjwkeuaR+5ISIfR9aiD8MzCA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from DU0PR08MB8422.eurprd08.prod.outlook.com (2603:10a6:10:406::6) by DU0PR08MB9632.eurprd08.prod.outlook.com (2603:10a6:10:447::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9094.17; Fri, 5 Sep 2025 06:21:16 +0000 Received: from DU0PR08MB8422.eurprd08.prod.outlook.com ([fe80::4ac7:a794:5e4b:fc25]) by DU0PR08MB8422.eurprd08.prod.outlook.com ([fe80::4ac7:a794:5e4b:fc25%4]) with mapi id 15.20.9094.017; Fri, 5 Sep 2025 06:21:16 +0000 From: Stefan Herbrechtsmeier To: bitbake-devel@lists.openembedded.org CC: Stefan Herbrechtsmeier Subject: [PATCH v2 15/15] fetch2: gitsm: remove indirection and nesting Date: Fri, 5 Sep 2025 08:20:58 +0200 Message-ID: <20250905062058.2563806-16-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250905062058.2563806-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20250905062058.2563806-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0130.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:97::17) To DU0PR08MB8422.eurprd08.prod.outlook.com (2603:10a6:10:406::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DU0PR08MB8422:EE_|DU0PR08MB9632:EE_ X-MS-Office365-Filtering-Correlation-Id: 4ea73753-869f-453c-a7b3-08ddec446b4c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|52116014|38350700014; X-Microsoft-Antispam-Message-Info: TovQo07o2MahrExpuKUWRkPIw47565b3OTo4jDOupsXmMoB5dC8TphPZ9qNLwXabCS8pSNWVVgAQ/HTGUseK3jN/TxqEooDm+GStesbziVHAxRtp4v8KuCekNS1TsI/ItS0ZR17/tNG2oxAkdct+9Ns4SCIPRYT0IA9D15g2wkNA/0bZVyOCl+hOtZElpNl+apEHjPPZeLiFJsfJfLjSEC69x1/bAqI+OARpDYb4d8Y63DEsvmMbzGvQZwQ6iTwYrQDND6FTz0prsHjxIBdldGRP880KgBuTBEOj0AJ0nBUsFDKqThtuq5QPRnLQMeRh0KAmuwW7kIBKVy1cxkwBmgflYBFnyjdlQUGFxvntNNisvZCyu0EGi/BL8aZt4+WSzMiTHGn5yqBanO+VUedQXMGOQi89stu6hjvBLMg0HIwR722LR96UDqTRZuqFi8hs7ztD7epHLfKsibJ3GxRIpLqQP3xOaNjR+4xmPwMOvKCCzjJtFiHGyxYKb7k10dM7/+LU4yzxyhgpnmxURnRS9ahO8LQ7if74bEH3YzGPXUDV+j5jSKF4OV9Zpdy9SSF7bsPlwReBTCTpJ0a94YiXHQ+NHvAfJUjNsQJDpxDONPlabH5ZsDtkX8LT44reTbmGrrLFu9bHyL4FGO/WxKK/eqN9Qje8UBr8D1qvDDoZZhMgW7XOZ1IfJ41sPayMJjCnswhDVjPQAI9+4LNY/Ghm1VpBOFCfBjZRyhWKopiSNOTt2fDJHsUf2LFmQbpWpNBV6lQXZZ53FjFYUvlpnUknYoVPqwHIBwdenQfMtoRzCVlXZsz4UZF7AOleiSuOCgfdqEMh5Z2R/YBK9PBmHVTMujpFODGYLGm2Ftys6WCBTbduCIUwIR8GFN/xolqvGKrJrrOq6oP0FMkLVQ44CdDEErHd8CDqd5ilhMAG6gSgkz6R5Hp6PObkS9UbotfC47khpMe90hX9qUhxYFjhjVnx7PaFsk78ocompendB0YCYAzlm3hF6dAPwOMAbSxqo9eRyEa3DIgyGV1Adh+ekKbh/3ngawBQ4cjY012Hu7V+27eeHw8EB+ajLvjy2q+UCL44TKVIxGIFjKVO6F7HQTNGzJQj8eJxfbDsCnHYEnPsV8vSBRHUgeXn8wEJkxm5F9X2imc9sTBaL3BqCzPYTWYgBCL4qSYsSRUh4kY81oa65G8G1wHS+Vo1aJba8Psi8PJt4wNySO96hxeMwkvj3rJCnPURU8qjzLN4AZad/G1uEDiQr5bgOLYI/uv/ANqiqK+4JCcSViXJuTbV9kHP5iNiTSaF8Wxf/oLUJ3/T48fSZ6wzOPzSkkNCFKm/MItrqyxuHBtZa48SO049EQQt1f4xvYkAisslgU5zwrmFFK2qtc0Igsw0PkxbLklbCJygQzYhp5eKm+DbyocGV2emypNBNtg39yw38N3mlBU09F4awM6AK+OXujo/qyclUaFOqsC9nyW7iGgKN9/puXby2fmYhA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DU0PR08MB8422.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(52116014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2d1tfC1XX7EEriN85elKWeUAy6tjc65qV95m2D09jeXSTm6wglnEV0uqYhSi80XhjflYgp23Htq+OyOnJSQzCWo5saD7zbdOSpRFFwe3Pb10aDxjibXiypOStB61marAe3TpaUd42qCJWK4/OAFshc027HIWFvUPMnaXKl+QBfYvsF1QfpO1xUh/0czpTWUDMrofHiQAfkyqohsyic1Ja4R4cjs1kyyscDBWGYEuWwIj+EthhBc4uWLzbibMetNGLx5eFamSPod2XX2LG3aUoVGdhG7G9LIXz0UL+mivROdLZGvuFd3GaCAJMWn1t7MR5zpA0kMddyFtXL/YImMkVjfQahTyn2LaQxNesUkT4FFCkPqAzO0BCotrL09HIouVC4FEd4J+AlxLN2NvnR8Ks5oKCIRVAPB0NvE6CzknVlVK6PRKrQ8mtXJwE6+r6RZkuXRgFRE3cgrAFmCfytz0GYqvZcY5hQh9hQVGrsyqX2drnsiHHRTniR09S2HSCYoZttMy1lquvjQNJjHknCUUMAib7kqUInbk3LUj2LlWxICCWuJqthAJ/hS/ltTkYUTtnbZBBlk0m4EaMG5nEoEgXZ+0f4Gg0ZkybWc5RPlpMFy70iWkyyZILak1OCKvH+vxWIJR59ZSXDU91f4KM6Pot+SUCUgPzJq7RQ6NfhHh0QQtLiQDQxEh6iQTrhrFA37jjgzwvd6gWou2s0XA+6GhEfsUG9PA1CA4K8XLMEjHv3ZO0d+oaAykpjwHdHfHT2H5l5IE1/GhtMTczJwhdLk09SMFAH2CZbJDCDoUboPO0t/GhnrPNjfaoczQgOhh4elQJBqi84uazL50gdgZI9maSD3sedA/Akg98TxWMaSG3c9fBf6GmN4/ZqfoRgQdamHyhg7EK06k1GBs2J3BlzyPeeVH29hjw7ZHnT7FJpnFvaAr4r/2G6lnIcmiSIvEBOs9WYSxiW+l39MapFesFf8m+PEBlZG0A2ge/zg5ICfAY/z8YBvDXjWILBCd1w5tDOGIt3PnEbfmJ4F768BCAnE5fbJEMdUuBgClOEvlu3uiZv8C9YvR21ETTyPVZ1TtcocqQAy358FODhU1OD2cSTBXBJz3iO5C7cq4234ljl1fB+dUqaro/gowuVD8n7Fxjx0r0qetNOL+0eV5USul4k69YU9CAh9WVy2iDOk/efB9gCMlsWYiYEfIy75R4RuOliVVQOgkYATYNcW19KjV/K63bPdm8y0cwn0ESWrdi/1+m7TT05WC2jgFT423OQ8PZK5dRAg7SL/jBUqY8J1JeG3oIqDDnfLa6GLsSltn+iXm+86BPu1c85F6XchqEeUo2hbHdkoWmc0LFlVaKjLC/9+Vvu231HkZY9Ye9lS0nMsiIjps0TYlQlAmvOD8aFAZ8tqZeMZ4LT7Mox4HOI4vRa3zOGQ1XWLQ5Jed4YCQ4YJz/KWbaH7ors0koIJxwxV/4zeqHF88YKtjXiANYm4nlPKbd9sU+HrBB2tsgMFHHXd+k4BCZjgHjluIA5rN+zcn/C0FQ12EpTv8bkAy6ViEI1/vxcDnKzNHGjaeaknXmuGmB4nkNL5cP8euXK0195Bb+AsPepV42qg/YCywGJgkaoptwA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4ea73753-869f-453c-a7b3-08ddec446b4c X-MS-Exchange-CrossTenant-AuthSource: DU0PR08MB8422.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2025 06:21:16.0272 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 67giG2g+EKjPmvARn+6TNF45LFyBCcOPVeAXXlN5UJsbhas3Aool0V2xerxQociqk0f5cbl4sC5vKwooH3u5xw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB9632 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 05 Sep 2025 06:21:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/17972 From: Stefan Herbrechtsmeier Use a single fetch object to reuse the fetch functionality. Remove indirection, nesting and duplicate code. Signed-off-by: Stefan Herbrechtsmeier --- (no changes since v1) lib/bb/fetch2/gitsm.py | 98 ++++++++++++++---------------------------- 1 file changed, 32 insertions(+), 66 deletions(-) diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py index 4b6c88baa..5ddc81f86 100644 --- a/lib/bb/fetch2/gitsm.py +++ b/lib/bb/fetch2/gitsm.py @@ -33,10 +33,10 @@ class GitSM(Git): """ return ud.type in ['gitsm'] - def process_submodules(self, ud, workdir, function, d): + def process_submodules(self, ud, workdir, d): """ - Iterate over all of the submodules in this repository and execute - the 'function' for each of them. + Iterate over all of the submodules in this repository and generate a + fetch object for all of them. """ submodules = [] @@ -91,6 +91,7 @@ class GitSM(Git): newud.path = os.path.normpath(os.path.join(newud.path, uris[m])) uris[m] = Git._get_repo_url(self, newud) + urls = [] for module in submodules: # Translate the module url into a SRC_URI @@ -138,21 +139,21 @@ class GitSM(Git): # the user (See comment in git.py). # So, we will not take patches adding "user=" support here. - function(ud, url, module, paths[module], workdir, d) + urls.append(url) - return submodules != [] + return Fetch(urls, d, cache=False) if urls else None - def call_process_submodules(self, ud, d, subfunc): + def call_process_submodules(self, ud, d): # If we're using a shallow mirror tarball it needs to be # unpacked temporarily so that we can examine the .gitmodules file # Unpack even when ud.clonedir is not available, # which may occur during a fast shallow clone if os.path.exists(ud.clonedir): - self.process_submodules(ud, ud.clonedir, subfunc, d) + return self.process_submodules(ud, ud.clonedir, d) elif ud.shallow and os.path.exists(ud.fullshallow): with tempfile.TemporaryDirectory(dir=d.getVar("DL_DIR")) as tmpdir: runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir) - self.process_submodules(ud, tmpdir, subfunc, d) + return self.process_submodules(ud, tmpdir, d) else: raise bb.fetch2.FetchError("Submodule source not available.") @@ -160,51 +161,26 @@ class GitSM(Git): if Git.need_update(self, ud, d): return True - need_update_list = [] - def need_update_submodule(ud, url, module, modpath, workdir, d): - try: - newfetch = Fetch([url], d, cache=False) - new_ud = newfetch.ud[url] - if new_ud.method.need_update(new_ud, d): - need_update_list.append(modpath) - except Exception as e: - logger.error('gitsm: submodule update check failed: %s %s' % (type(e).__name__, str(e))) - need_update_result = True - - self.call_process_submodules(ud, d, need_update_submodule) - - if need_update_list: - logger.debug('gitsm: Submodules requiring update: %s' % (' '.join(need_update_list))) - return True + need_update = False + fetch = self.call_process_submodules(ud, d) + if fetch: + for url in fetch.urls: + urldata = fetch.ud[url] + if urldata.method.need_update(urldata, d): + need_update = True - return False + return need_update def download(self, ud, d): - def download_submodule(ud, url, module, modpath, workdir, d): - # Is the following still needed? - #url += ";nocheckout=1" - - try: - newfetch = Fetch([url], d, cache=False) - newfetch.download() - except Exception as e: - logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e))) - raise - Git.download(self, ud, d) - self.call_process_submodules(ud, d, download_submodule) + + fetch = self.call_process_submodules(ud, d) + if fetch: + fetch.download() def unpack(self, ud, destdir, d): fulldestdir = self.destdir(ud, destdir, d) - def unpack_submodules(ud, url, module, modpath, workdir, d): - try: - newfetch = Fetch([url], d, cache=False) - newfetch.unpack(root=destdir) - except Exception as e: - logger.error('gitsm: submodule unpack failed: %s %s' % (type(e).__name__, str(e))) - raise - Git.unpack(self, ud, destdir, d) parentdir = ud.parm.get('parentdir') @@ -228,35 +204,25 @@ class GitSM(Git): logger.error("Unable to set git config core.bare to false for %s" % fulldestdir) raise - ret = self.process_submodules(ud, fulldestdir, unpack_submodules, d) + fetch = self.process_submodules(ud, fulldestdir, d) + if fetch: + fetch.unpack(destdir) - if not ud.bareclone and ret: + if not ud.bareclone and fetch: cmdprefix = "" # Avoid LFS smudging (replacing the LFS pointers with the actual content) when LFS shouldn't be used but git-lfs is installed. if not self._need_lfs(ud): cmdprefix = "GIT_LFS_SKIP_SMUDGE=1 " runfetchcmd("%s%s submodule update --recursive --no-fetch" % (cmdprefix, ud.basecmd), d, quiet=True, workdir=fulldestdir) def clean(self, ud, d): - def clean_submodule(ud, url, module, modpath, workdir, d): - try: - newfetch = Fetch([url], d, cache=False) - newfetch.clean() - except Exception as e: - logger.warning('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e))) - - self.call_process_submodules(ud, d, clean_submodule) - - # Clean top git dir + fetch = self.call_process_submodules(ud, d) + if fetch: + fetch.clean() Git.clean(self, ud, d) def implicit_urldata(self, ud, d): - import subprocess - - urldata = [] - def add_submodule(ud, url, module, modpath, workdir, d): - newfetch = Fetch([url], d, cache=False) - urldata.extend(newfetch.expanded_urldata()) - - self.call_process_submodules(ud, d, add_submodule) + fetch = self.call_process_submodules(ud, d) + if fetch: + return fetch.expanded_urldata() - return urldata + return []