From patchwork Fri Jul 31 09:26:37 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jamin Lin X-Patchwork-Id: 93998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33053C55172 for ; Fri, 31 Jul 2026 09:26:49 +0000 (UTC) Received: from TYPPR03CU001.outbound.protection.outlook.com (TYPPR03CU001.outbound.protection.outlook.com [52.101.126.106]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.2924.1785490002156443016 for ; Fri, 31 Jul 2026 02:26:44 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@aspeedtech.com header.s=selector1 header.b=YoN527rw; spf=pass (domain: aspeedtech.com, ip: 52.101.126.106, mailfrom: jamin_lin@aspeedtech.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Zi5VZoSmhcb2tdFOBU+YFj0ieawG8PD8DSd2y1tgDdUxAEIF8N9zaNinZIu4SWGGl/7n3xW73FFzR0ev56PzPhIfluz4KCQGvcU8kTIanMa8qV7LMi+zNklSq3ubb3tWjGcjJz3t1+kdWWYPmkwWpgD9ZmDxpBTITbR3CeLE6WbS4CSkCntTFq+YQc8fjnCQarHSh9yTbmhLLPo87Ezrpa5GSuF1JUMgvzCLTBTnDgVVEiXpn/ELifQHC5mXofYV1RfW2SQmBDZXBWsVV1FjujO8hmZKOJ8JMCJW+aQLQs7eLlL74Fcf7u0Xewha4pTzRbZ5JCuETMuGQNm0lvst9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rtRHyUsNC/9vReP8p3z7qb/vBHHiT/nQ6SjpkCKJvTE=; b=PkSQ2A9KCzVAHjl4B95ZXYz8R0DdE2iaTtntMXmfvc3OuSgg56bBpjPQioOHK7uar+GEjZ36YLKplxnbSWNJtAMXT1/gH70fuY9Cf52bklJi69JrgVVCC/cnq8isdl6qK3hTG68r2W8cbaULrb/MrD3BULdUcEye2na+Iqp2p4QZ1yjMw/vCgpPefGiX63hiy3pYceiEJolAHvA5zbm1RycdUWiK9Etn73VUSjt+GJG2D8YJhkUueHgbo5q+3SKUVgKE501AHXL1rCqyVGkhQAWQmXWBIrWnNGLvHQxfyVEdKA6hRK0f+5O1/NvDWbzGThaDxOVWepkGDg+j8VQwBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=aspeedtech.com; dmarc=pass action=none header.from=aspeedtech.com; dkim=pass header.d=aspeedtech.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aspeedtech.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rtRHyUsNC/9vReP8p3z7qb/vBHHiT/nQ6SjpkCKJvTE=; b=YoN527rwj06NUd9FZsYh/C3U2zO8WdGtO3RIpK2vV7AL0izvltL9+z2/I0M8niMW1CRpGyDY//4GqtEhj6zBSh7cDI4slVmxgBzcNG884nsDVbej8+iDR0FjtEEn8QDBxn0KXzX23h7BW1mEBlfIFsZl7PDhE85WRrM5neQ53Ix9VDCX7Z3yisfvITLRnjk0svLd2LUVXvxAF2AjaZ1qAtzcOCVC6ly1IhxkPd872x7arKBroRCqzZEJFx7KXN2eQiSNx2+T814AaeBy91JIgt/o0HIMTXCA1TiPGJyQ6pHDjdmxbGd8/XC87FKbFvj4VwShIYgMlAdhHl87HayX3w== Received: from TYZPR06MB4980.apcprd06.prod.outlook.com (2603:1096:400:1cc::10) by SEZPR06MB5295.apcprd06.prod.outlook.com (2603:1096:101:7b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.270.15; Fri, 31 Jul 2026 09:26:38 +0000 Received: from TYZPR06MB4980.apcprd06.prod.outlook.com ([fe80::ea8a:7cb7:4822:2fb3]) by TYZPR06MB4980.apcprd06.prod.outlook.com ([fe80::ea8a:7cb7:4822:2fb3%6]) with mapi id 15.21.0270.012; Fri, 31 Jul 2026 09:26:38 +0000 From: Jamin Lin To: "openembedded-core@lists.openembedded.org" , "alex.kanavin@gmail.com" , "paul@pbarker.dev" , "mathieu.dubois-briand@bootlin.com" CC: Troy Lee , Jamin Lin Subject: [PATCH v4 2/5] devtool: Register nested git repos before the initial commit Thread-Topic: [PATCH v4 2/5] devtool: Register nested git repos before the initial commit Thread-Index: AQHdIM6v+ernWD7tZ0ifQ5Gml3Ixmw== Date: Fri, 31 Jul 2026 09:26:37 +0000 Message-ID: <20260731092634.1127862-3-jamin_lin@aspeedtech.com> References: <20260731092634.1127862-1-jamin_lin@aspeedtech.com> In-Reply-To: <20260731092634.1127862-1-jamin_lin@aspeedtech.com> Accept-Language: zh-TW, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=aspeedtech.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: TYZPR06MB4980:EE_|SEZPR06MB5295:EE_ x-ms-office365-filtering-correlation-id: d955f3c9-90e6-4df6-9f79-08deeee5d2ff x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230040|366016|376014|23010399003|1800799024|38070700021|6133799003|56012099006|10067099003|22082099003|18002099003; x-microsoft-antispam-message-info: 7siHmWJiCgttpep65j8OQCPGZTQ9Ygq01e1XX3l3qZXpW5uNYUi88yarvtLcMq0sgKhJSKu6B4l68QsHuGLxh486HM5OshlgWvXxuavWlItOd9gJb2eetj8hMoUFi4BfD6tETVJJad7FJEenZrQy3lI9yVrpA94a2tUouqI8DK5ahoMZh5ArSNTIPna8v6TJXnVr4F7Ryi2Mdy9kFla0StoeCynu8725tGhzCeq8l4hOhDMOtBJ613TqbmhUAV/zLtPzC4wmjBEOa6AH28SDsP0oUTgdCbVaUWtezjRPWwi0elzbDqXTiw+vYT95jDXI/eOVNspDU+cvtAbbphhIaBXbKDpfZScQKHo8GvKQ5epGLBgLwrMjB/znxBfoN6edRMjS2yiD4rqevKvIj0LcMi6SlhjiEDpChD47P+qmKawn3RzhgpAt54B2fx2ZWQetJxRC+x8BD1DoIGfnFquu+7lh9wWH82hYjHlfMXvTWyv5396fWOIdMQgUopAFBneLGe8H+lYBGmqXrXjVT1JPaJzrD2cjLvywbw1zTuQtMAkfM+LA3Gmyp6wvAE7UOsKkafiIEF8wgxqxM/LGDeDtq1shEfmOWQxpQutAJdwZmnm6iPfZejDp6tNwv6hT0kdx5lHOM9GqQB5ntM5XfGrwDjhESn4HSK2TlbC39M9ZY+c1QCFdRa4HiD1WW/JGe8wWHERxAjUN4v4VFHfVAQBbOyXwxq2FX1+Q9sKPd2lRYt4= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:TYZPR06MB4980.apcprd06.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(23010399003)(1800799024)(38070700021)(6133799003)(56012099006)(10067099003)(22082099003)(18002099003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: 5uLgvdFHRtS6607Z7a0SW/Pmnnef8bncoOtBTY6QFQx1YHQ7T+gnhkanAQNhW9uO6DG+cTezbbG4l8/zrN0fQX9Sk5E6NoG394tWV+6bHSrH2OZ8bw/s+6VNzYCYwxt/h+Deda1sUczRX7IqYLqvyGKymQp7auuBk8UtmgJstO7x2J8pASpTH5/BqsQYr6S69Q4lYfO78cpdPbNIgd5gTZZLoj4j3abtJPo6gSiEX6hKIwQ70IrMCi7QdAf+0mrqCZijKRLzVffkCqdbueWjqINtUgJ9wzpLZsNrXNWUArTZPQrX1ukl9Rx/7THotdqjWKVAu2HTw1OQAIjWCEx8RfDshZavTjAt4gKoEaIu+r7UkQsyYIsMjmJJdRrdq8Gl3UWHs8HLJIXa7hGVZluUq3OIHiZmD1yolYxrjT5SlD2d0B6uY2+V2ydDgJgml9cflvPYsdgNKduqKVi0Feu+DvGHA3/dN6ufKqsqFzBcr14QCR9F7mWLCjo5RV+HWTzH8onra/3zYv2WWrDmnFFTX9Dy3BeJRl3GoB2NmjmLqMitVBc4YKGrR1iD4okxuDuAV+857XbmYLR1cT3CfDsnHEdPR1NYz8Ld9J9fiiRhTWRn6cbsWdm87pfHRUldVeR6SlVwbBSiubiMd6OY/awRuJU9FNOaPcrXmDRTbmaFv+H0ikPhBy04LP1k0yWcuvoGfYVuCBSoIVBiAl57q02kEeTkKvDSGA1o01DiBKEBF9Ylv0F9Q8G9G6B2FrqTpZ/Klg+sfke/QrnHjAfvwmzzzU25Poy5qrh6igV3s/tz8+uvt/XKkjYvwIuR3lmvu2zuMAQaGvxn/Nx9f2EzKfvuo+18XS+2QtQivI7i7k/xyxqcqxwtfu9gIhfWtqU9cUP6WCmDcb9so/PGpGW/ngaf585tJxYFl/oV9Nc42Ky/y63tGv6iMUZ166NYq9j20YoeeSzGBEm/De5bdTg2q4R2Eyul2sUEsCuEbiL4CzDiGZ4+g0MEwBFT32gbSWvaZU0gMCTkDnc2u/IMJchRb5sXX/dbiVok8HUUCOszdhadC+Z7lLxZLpqMm/R2/txAXIoPm8wHxLY+ojjl1sUac7sYrRZnIgYVN61EluGISRpAPci3iqiRZJu2Jphrf38qOPnmBX6wxqCute2KjU26hAkmVInZ5+88f4kxFq9onPMqxHGxHrCW0L79jn+1Tz/wd7jpG2f5exx8WcOM6p7JSXC2LczlZXalL56NpDrXo88f77omglcBqMrXuy2Y0KHr4QAS9Prr95M1x+CUgJP8uK8kGOyKLCzogS95+fg44t0HzvZz9ZUx1fI6koI1s9HURMGD2v9EIGeUc1fXOjW0X1qRGthDznOm8MnWtxuFR3G/vvq9bSGm5sbl/YxoYrBnPliUA7TGTJnGLGcOUYHywIXgWaH6DA7SH9FzNleXi9gN3g39lxCx/f7hHk2AUWSA0kzCJgGvu3+MhA79QDdsHEepKbmn5dJ5reeMMmBOQQs9REDlDa2xeSaKGyz5Kr76aACoI1xqArgJbm0d7J07E2gWa/w7Sh5TTkvbQK9BObui8SjFmRlhgFG93zv9MB7qfKAoKnsodBLVsxQ4PclYmrRTRwNSRkzmnTA94XAnKphlrTBaEswYtKfqjDJ+WlVF1gamDpj/Mxq98IiYTnTI0QhYmG+I3Lk70vW7KAjeW19D7ZtB5wGc+FF/wOCJk2lQyADr1FYaoSYABUPgx27rqiyMWg== MIME-Version: 1.0 X-Exchange-RoutingPolicyChecked: mUSETkIbhKqSLDgxAm676l3QeoWaXbi+mNNLsQGvHbO7GuPPqZ/VmBu3r0CZtbp8V/XqGnshq6UynM+ZhVjyl3DkPI3RzPrBrccZ2yc2k2dPB8mQo8zfaGh/WqNIoGyQ1MKllFujP/PcJ7sFjgNOCgodiu6BYrppcG1AObfrlVuMbhIqka9a/4lFL5erYczzhVjcYTT1AoIZF76y3xEo0pSlFPA3dPCGrUyQWG1sJGDMETZy2ezK/qKG6my8DJJH+KxLKOJxzF9rQEyG8m36P8MRGDWbBtLq8lgvmrjQ02kpUltfyI56UwNTbfcHSCM7147KVuj8bfVHF5GGJEGRxA== X-OriginatorOrg: aspeedtech.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: TYZPR06MB4980.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: d955f3c9-90e6-4df6-9f79-08deeee5d2ff X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Jul 2026 09:26:37.3823 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43d4aa98-e35b-4575-8939-080e90d5a249 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: kQV4iz4vPM5lY+yOKxfFmYxozbWon851Nnag8iVmoIxdfVL2UOhQayoDXk5ZVkh5v8SOzG/CW4t3a5tucdUtCjJocRqYNUeiYjF+6f+ISOw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEZPR06MB5295 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 31 Jul 2026 09:26:49 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/242414 setup_git_repo() is meant to convert a git repo that a recipe unpacks inside S (e.g. via multiple git SRC_URI entries with different destsuffix values) into a regular git submodule, so devtool can later tag branches on it and extract patches from it via finish/update. That detection never actually triggered, because of the order the function ran things in when it had to create the workspace repo itself: 1. 'git init' 2. 'git add -A .' + initial commit <- commits the nested repo as a bare, unregistered gitlink 3. checkout devbranch, tag basetag 4. scan 'git status --porcelain' for still-untracked directories ("?? /") and convert any that are git repos into submodules By the time step 4 ran, the nested repo had already been swept up by step 2's 'git add -A .': git treats a directory containing its own .git as an embedded repo and stages it as a gitlink pointing at its current HEAD, without registering it as a submodule. Once that gitlink is committed, 'git status --porcelain' reports it as e.g. " M " (already tracked) rather than "?? /" (untracked), so step 4's "line.endswith('/')" check could never match it and the conversion to a real submodule silently never happened. There is also a second entry path with the same root cause: when the recipe's top-level source is itself fetched via git://, repodir is already a git repo, so the 'if not .git' block above (init + initial commit) is skipped entirely - and so was the detection that lived inside it. In that case the nested repo instead gets committed as a bare gitlink later, by patch_task_postfunc's 'git add' after do_patch. Fix this by extracting the detection into a helper and calling it before anything can commit the nested repo as a bare gitlink, in both cases: - freshly-created workspace repo: right after 'git init', before 'git add -A .' and the initial commit; - repodir already a git repo: at function entry, before the later 'git add' in patch_task_postfunc. At those points the nested repo is still untracked and reported with a trailing "/", so it is correctly picked up and registered via 'git submodule add'. Nested repos are discovered top-down (so a repo that manages its own submodules via .gitmodules can be skipped rather than descended into), but registered bottom-up (deepest first): a parent's commit recording its child's HEAD must happen after that child is fully finalized, otherwise registering a still-deeper repo afterwards moves the child's HEAD forward again and leaves the parent pointing at a stale revision. Signed-off-by: Jamin Lin --- scripts/lib/devtool/__init__.py | 79 +++++++++++++++++++++++---------- 1 file changed, 56 insertions(+), 23 deletions(-) diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py index 58b02eb460..030ba7edd9 100644 --- a/scripts/lib/devtool/__init__.py +++ b/scripts/lib/devtool/__init__.py @@ -197,9 +197,59 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None): """ import bb.process import oe.patch + + def register_nested_git_submodules(): + # If the recipe unpacks another git repo inside S (e.g. multiple git + # SRC_URI entries with destsuffix), declare it as a regular git + # submodule now, so we will be able to tag branches on it and extract + # patches when doing finish/update on the recipe. This must happen + # before anything else commits that nested repo as a bare, + # unregistered gitlink: once that happens 'git status' no longer + # reports it as untracked ("?? /"), so this detection can never + # find it. That can happen either from 'git add -A .' below (for a + # freshly-initialized repo) or, when repodir is already its own git + # repo (e.g. a recipe fetched via plain git://), from a later 'git + # add' done elsewhere (patch_task_postfunc, after do_patch) - so this + # is called both from the fresh-repo branch below and from the + # already-a-repo branch, before either has a chance to do that. + # + # Discover nested repos top-down (so we can still skip descending into + # a repo that manages its own submodules via .gitmodules), but do the + # actual 'git submodule add' + commit bottom-up (deepest repo first): + # a parent's commit recording its child's current HEAD must happen + # after that child is fully finalized, otherwise a deeper repo added + # later on gets its own registration commit, moving the child's HEAD + # forward again and leaving the parent's already-made commit pointing + # at a stale, superseded revision of it. + stdout, _ = bb.process.run("git status --porcelain", cwd=repodir) + nested_repos = [] + for line in stdout.splitlines(): + if line.endswith("/"): + new_dir = line.split()[1] + for root, dirs, files in os.walk(os.path.join(repodir, new_dir)): + if ".git" in dirs + files: + nested_repos.append(root) + # Do not descend into nested git repos that have submodules themselves. + if ".gitmodules" in files: + logger.warning('Nested git repository with submodules %s; devtool will not recurse into it', root) + dirs[:] = [] + + for root in reversed(nested_repos): + parentdir = os.path.join(root, "..") + (stdout, _) = bb.process.run('git remote', cwd=root) + remote = stdout.splitlines()[0] + (stdout, _) = bb.process.run('git remote get-url %s' % remote, cwd=root) + remote_url = stdout.splitlines()[0] + logger.error(os.path.relpath(parentdir, root)) + bb.process.run('git submodule add %s %s' % (remote_url, os.path.relpath(root, parentdir)), cwd=parentdir) + oe.patch.GitApplyTree.commitIgnored("Add additional submodule from SRC_URI", dir=parentdir, d=d) + if not os.path.exists(os.path.join(repodir, '.git')): bb.process.run('git init', cwd=repodir) bb.process.run('git config --local gc.autodetach 0', cwd=repodir) + + register_nested_git_submodules() + bb.process.run('git add -f -A .', cwd=repodir) commit_cmd = ['git'] oe.patch.GitApplyTree.gitCommandUserOptions(commit_cmd, d=d) @@ -214,6 +264,12 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None): commitmsg = "Initial commit from upstream" commit_cmd += ['-m', commitmsg] bb.process.run(commit_cmd, cwd=repodir) + else: + # repodir is already a git repo in its own right (e.g. a recipe whose + # top-level source is fetched via plain git://), so there was no + # fresh init/initial commit above to interfere with detecting nested + # repos - do it here instead, at the earliest point available. + register_nested_git_submodules() # Ensure singletask.lock (as used by externalsrc.bbclass) is ignored by git gitinfodir = os.path.join(repodir, '.git', 'info') @@ -237,29 +293,6 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None): bb.process.run('git checkout -b %s' % devbranch, cwd=repodir) bb.process.run('git tag -f --no-sign %s' % basetag, cwd=repodir) - # if recipe unpacks another git repo inside S, we need to declare it as a regular git submodule now, - # so we will be able to tag branches on it and extract patches when doing finish/update on the recipe - stdout, _ = bb.process.run("git status --porcelain", cwd=repodir) - found = False - for line in stdout.splitlines(): - if line.endswith("/"): - new_dir = line.split()[1] - for root, dirs, files in os.walk(os.path.join(repodir, new_dir)): - if ".git" in dirs + files: - (stdout, _) = bb.process.run('git remote', cwd=root) - remote = stdout.splitlines()[0] - (stdout, _) = bb.process.run('git remote get-url %s' % remote, cwd=root) - remote_url = stdout.splitlines()[0] - logger.error(os.path.relpath(os.path.join(root, ".."), root)) - bb.process.run('git submodule add %s %s' % (remote_url, os.path.relpath(root, os.path.join(root, ".."))), cwd=os.path.join(root, "..")) - # Do not descend into nested git repos that have submodules themselves. - if ".gitmodules" in files: - logger.warning('Nested git repository with submodules %s; devtool will not recurse into it', root) - dirs[:] = [] - found = True - if found: - oe.patch.GitApplyTree.commitIgnored("Add additional submodule from SRC_URI", dir=os.path.join(root, ".."), d=d) - found = False if os.path.exists(os.path.join(repodir, '.gitmodules')): bb.process.run('git submodule foreach --recursive "git tag -f --no-sign %s"' % basetag, cwd=repodir)