From patchwork Fri Aug 8 08:49:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41728C87FDA for ; Fri, 8 Aug 2025 08:49:47 +0000 (UTC) Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by mx.groups.io with SMTP id smtpd.web10.17674.1754642981539729346 for ; Fri, 08 Aug 2025 01:49:41 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=UOEkf5HB; spf=pass (domain: gmail.com, ip: 209.85.215.178, mailfrom: akash.hadke27@gmail.com) Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-b4209a0d426so1931678a12.1 for ; Fri, 08 Aug 2025 01:49:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754642980; x=1755247780; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=jzk6GzJjKNvuZn2Lz3JKsjQTVz/JgU889oaPQwyPf84=; b=UOEkf5HBJ+LcHeHD9iPxi0NUFSNCy14VnzsEVpN5Q3cqdoCUqWhgzAUuiiVbRiK74U lu8sOV2TPdfS4Ob6Vx2AUkpMduAonQb//eFH3j+BoUvYvo3EkfbDKVHpnVpQc285+t9f 6wivmycxHbXW7OclyTXXtj4Pfynqh6cq1zYY+Z+/+w0vCpL165jTbzTdSmFEWXtA5XP+ NhCzOas0tXdja/6TDI5hK0No4scDt/8gGIXkf6BvvSsb53WHCCyp9iKqZdF2BXJRgG+k l6uAYlJLX6sa+XPMo5nyzSgnl0wsRErfPf01DxbrsQh8+e6DLHIe+uNqj78Y6AwDAuJr dWcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754642980; x=1755247780; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jzk6GzJjKNvuZn2Lz3JKsjQTVz/JgU889oaPQwyPf84=; b=m/8sMevyxg7G5TlUIhiiAjUZ7F2l4I8uQfUu2MZ6daeRTxZRaooHVhjATfGn3EqJdX MkBRXz4ViIfnhu3kwZCgZZErUn55CZwxkAIEK8NU/RyKSO0kmwPAxDoB06VmB97pvhit YAfORCnP9tx1zfGW/AfS1ePkpYnaMvsArJWb9thmDTxEhIADNSBDBry6NoMKM457xUj/ 6bUyT/UP8u7GNzQfTl/ZZMP53epwHFxS0Wpl3X/mWbiD2v2CvtbegDYNrNYW23uwylsu VC0q6bXAqa8d01iSSVtP+6X1ETqwqu6qle4BqlPW0AGdrFngmvNfYvFHPjq1RR6SE2dZ BQyg== X-Gm-Message-State: AOJu0YxGftYpuMIELS21bomd5DhVD4lGx7JLutJCj7VxEG06e1d1GrPF 9gDCzIdz1QWhAy2xGLBx9D2iJU5VyaUB96ppEp3c9we3ILnzYaXvgM+pLEzeJg== X-Gm-Gg: ASbGncuXoos30XWLJOLCId98nbSi5FFgtK0d+EmmPvvqTKRT58BNnSKMKHLFrFidNKL SN7MxVNvMaM55CEmyljfWSfi72H+UzYoFeXMHALDfTg4hi+xYCXhFKUQcrZgMFgWLivG0SSAtHD 2XBVhA44xHFjXUUPkBEasUfGyFV4UeitqWSDo+jmvCKP//H2Uq3il6tNaVz4KKq5/9ZIJ30OO3F dWkKz4JLn02CcUDPt3QoNLU2fg/TAhtiJwK5oDrlK6fP8NdWb/lBrwsvlow36NvFlpRkYzvL2TD 2dpsuI+V+JkDdJqIMhVMN5otvhU+RHjF8KORt+nhV/qYOi6Z8aN4qh9iZPG9LUtxv9sI0L3rsL1 Yt5qvb3ZyD+nLSzOvXfmoFO62MPjjEkyn X-Google-Smtp-Source: AGHT+IEhvtMoYw/SZxEpkfBAfHmikfrajZ4iPU/oXfXxtlZxQLCsAkNUJrRDr6LxNHSDbjR5wBWWhw== X-Received: by 2002:a17:902:c946:b0:237:f76f:ce34 with SMTP id d9443c01a7336-242c200831amr36000125ad.15.1754642980443; Fri, 08 Aug 2025 01:49:40 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.49.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:49:39 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Robert Yang , Richard Purdie Subject: [poky][scarthgap][PATCH 01/23] bitbake: fetch2/git: Use git shallow fetch to implement clone_shallow_local() Date: Fri, 8 Aug 2025 14:19:09 +0530 Message-Id: <20250808084931.2156763-1-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:49:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221598 From: Robert Yang This patch can make the following settings much more faster: BB_GIT_SHALLOW = "1" BB_GENERATE_MIRROR_TARBALLS = "1" * The previous implementation was: - Make a full clone for the repo from local ud.clonedir - Use git-make-shallow to remove unneeded revs It was very slow for recipes which have a lot of SRC_URIs, for example vulkan-samples and docker-compose, the docker-compose can't be done after 5 hours. $ bitbake vulkan-samples -cfetch Before: 12 minutes Now: 2 minutes $ bitbake docker-compose -cfetch Before: More than 300 minutes Now: 15 minutes * The patch uses git shallow fetch to fetch the repo from local ud.clonedir: - For BB_GIT_SHALLOW_DEPTH: git fetch --depth rev - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude= rev Then the git repo will be shallow, and git-make-shallow is not needed any more. And git shallow fetch will download less commits than before since it doesn't need "rev^" to parse the dependencies, the previous code always need 'rev^'. (Bitbake rev: a5a569c075224fe41707cfa9123c442d1fda2fbf) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit e2527cf58f4f390712641a99b276b9e0aeae01c6) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 78 ++++++++++++++++++++++++------------ 1 file changed, 52 insertions(+), 26 deletions(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 6029144601..2194c839dd 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -551,18 +551,31 @@ class Git(FetchMethod): runfetchcmd("touch %s.done" % ud.fullmirror, d) def clone_shallow_local(self, ud, dest, d): - """Clone the repo and make it shallow. + """ + Shallow fetch from ud.clonedir (${DL_DIR}/git2/ by default): + - For BB_GIT_SHALLOW_DEPTH: git fetch --depth rev + - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude= rev + """ + + bb.utils.mkdirhier(dest) + init_cmd = "%s init -q" % ud.basecmd + if ud.bareclone: + init_cmd += " --bare" + runfetchcmd(init_cmd, d, workdir=dest) + runfetchcmd("%s remote add origin %s" % (ud.basecmd, ud.clonedir), d, workdir=dest) - The upstream url of the new clone isn't set at this time, as it'll be - set correctly when unpacked.""" - runfetchcmd("%s clone %s %s %s" % (ud.basecmd, ud.cloneflags, ud.clonedir, dest), d) + # Check the histories which should be excluded + shallow_exclude = '' + for revision in ud.shallow_revs: + shallow_exclude += " --shallow-exclude=%s" % revision - to_parse, shallow_branches = [], [] for name in ud.names: revision = ud.revisions[name] depth = ud.shallow_depths[name] - if depth: - to_parse.append('%s~%d^{}' % (revision, depth - 1)) + + # The --depth and --shallow-exclude can't be used together + if depth and shallow_exclude: + raise bb.fetch2.FetchError("BB_GIT_SHALLOW_REVS is set, but BB_GIT_SHALLOW_DEPTH is not 0.") # For nobranch, we need a ref, otherwise the commits will be # removed, and for non-nobranch, we truncate the branch to our @@ -575,36 +588,49 @@ class Git(FetchMethod): else: ref = "refs/remotes/origin/%s" % branch - shallow_branches.append(ref) - runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) + fetch_cmd = "%s fetch origin %s" % (ud.basecmd, revision) + if depth: + fetch_cmd += " --depth %s" % depth + + if shallow_exclude: + fetch_cmd += shallow_exclude - # Map srcrev+depths to revisions - parsed_depths = runfetchcmd("%s rev-parse %s" % (ud.basecmd, " ".join(to_parse)), d, workdir=dest) + # Advertise the revision for lower version git such as 2.25.1: + # error: Server does not allow request for unadvertised object. + # The ud.clonedir is a local temporary dir, will be removed when + # fetch is done, so we can do anything on it. + adv_cmd = 'git branch -f advertise-%s %s' % (revision, revision) + runfetchcmd(adv_cmd, d, workdir=ud.clonedir) - # Resolve specified revisions - parsed_revs = runfetchcmd("%s rev-parse %s" % (ud.basecmd, " ".join('"%s^{}"' % r for r in ud.shallow_revs)), d, workdir=dest) - shallow_revisions = parsed_depths.splitlines() + parsed_revs.splitlines() + runfetchcmd(fetch_cmd, d, workdir=dest) + runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) # Apply extra ref wildcards - all_refs = runfetchcmd('%s for-each-ref "--format=%%(refname)"' % ud.basecmd, - d, workdir=dest).splitlines() + all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \ + d, workdir=dest).splitlines() + all_refs = [] + for line in all_refs_remote: + all_refs.append(line.split()[-1]) + extra_refs = [] for r in ud.shallow_extra_refs: if not ud.bareclone: r = r.replace('refs/heads/', 'refs/remotes/origin/') if '*' in r: matches = filter(lambda a: fnmatch.fnmatchcase(a, r), all_refs) - shallow_branches.extend(matches) + extra_refs.extend(matches) else: - shallow_branches.append(r) - - # Make the repository shallow - shallow_cmd = [self.make_shallow_path, '-s'] - for b in shallow_branches: - shallow_cmd.append('-r') - shallow_cmd.append(b) - shallow_cmd.extend(shallow_revisions) - runfetchcmd(subprocess.list2cmdline(shallow_cmd), d, workdir=dest) + extra_refs.append(r) + + for ref in extra_refs: + ref_fetch = os.path.basename(ref) + runfetchcmd("%s fetch origin --depth 1 %s" % (ud.basecmd, ref_fetch), d, workdir=dest) + revision = runfetchcmd("%s rev-parse FETCH_HEAD" % ud.basecmd, d, workdir=dest) + runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) + + # The url is local ud.clonedir, set it to upstream one + repourl = self._get_repo_url(ud) + runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=dest) def unpack(self, ud, destdir, d): """ unpack the downloaded src to destdir""" From patchwork Fri Aug 8 08:49:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4061CC87FD3 for ; Fri, 8 Aug 2025 08:49:47 +0000 (UTC) Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by mx.groups.io with SMTP id smtpd.web10.17675.1754642984340367776 for ; Fri, 08 Aug 2025 01:49:44 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=H3D1BZ1D; spf=pass (domain: gmail.com, ip: 209.85.215.178, mailfrom: akash.hadke27@gmail.com) Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-b271f3ae786so1534595a12.3 for ; Fri, 08 Aug 2025 01:49:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754642983; x=1755247783; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QY5MZdoMEig9wA2LT1bPpkytfyhCYTtHzjQuBh5StY0=; b=H3D1BZ1D9mCz4Q+G8ZwHbBNp/ZRUivPNUL5Wrt6e6Fnfi9HpmykCBswMRK0nlSD/m/ cWzjMer0Quy7XuMHdmkuIxaVHTFKqd7fVqwYDhTHIlGCQsr5DnHH2/Uw3HtPqjRQKCMW JxoDZFyTZzvYJF1Q0HAcuByZCZB702hRf+ao7NmgHFN1JbYJ8KCAu8iBnm47wS9qhc+v wTvYH63Xc9q+uyrTB+PRED7u+3t6peFLROZNLUhj1iz0CeLz2t774wzH1QY9XicWaOft B5dukBD+0HGvfu8kyFYjmzqwmZS82xrEGBDKLgHQ6TxYnIdBz4y3qD54Zfw+IJ1pJ398 jdKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754642983; x=1755247783; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QY5MZdoMEig9wA2LT1bPpkytfyhCYTtHzjQuBh5StY0=; b=N/8wGJmEZudyOtbf+9QFNtdwTddtx2dFmegdP0ubnJR66iGiVyv42UTHV9QEEkkQyf YzJ6zN4e4nIw6S8yQ9sYpzdWHi+YZ0zE/8eDDmmKe8LwHSjdpnTZzlaQVJxdkn/E6xL5 EifCNpccGAcVME/XCfo/4h6K/YMscuQP4+aJGBO/R7YcBPD8uo38qZii+GofsZhuUdSC dAlI4iOjmsHWS7lFyTDUKiLfJPw1PfxX2eBboUhwuUNHB9mmjcJ5OwLbeE7u3nVTBmSh q+jzxdR7x3THX3SgyWQ4vfdGzkt0vyznal0ubALimIK5lz0M5wKdMiicWRGxlYPZupw/ fyqA== X-Gm-Message-State: AOJu0Yx5T667a6ApHGhM1WQwqOLKXfLd1MGn8dUxWOSgrsBCswEcLrzO 0f+zc/IbPI79QyU4hluvW0egANrvs5GomPM+IwnUC7aoJHkzsoWiBKD3TCSzJw== X-Gm-Gg: ASbGnctpgJMU5Eiivf+eUXi+goZnYj1QfFvSin/BZhFlQ62YCqVD+v2ym/dy47bUeP3 coDWy7IA2BfgoQF2avPSND7x8A/xNvd+QhC4F2O/Mx2K1Qdnz2l3+OYxUxpSiw6yonh6fO4iQTI q77QxUTxSLDBv5PQS9/yMwF9fu97oVrg3ScsWtENbK52elKLYpXa4b8CKRb7AuwU6bof3N0g12z 9ajaxafbNV1ScykVUPlueRMGLPn+2CoGRRnuVKINFizGct1G+15PuqLEGft4nIjNxZiKSnq942d uOtLSkvP+wOExddQzzTk4xGN5qyHJc8ztVKXUmHh2SX6zG6bI5hN+Ccb7o0GWJq1Itt8Wb90XW7 wpx4q9xsqQhuoxm441YuoKK1xYRxDH4VK X-Google-Smtp-Source: AGHT+IH6x8k7Qdaowzg5QFPYwIQ1cJ8VoKKqEHrD7W5Ylu7wzB7qjdIlRAcArc500VoqwzNgxeltOw== X-Received: by 2002:a17:903:40cb:b0:242:b315:dda7 with SMTP id d9443c01a7336-242c1fdc02dmr31017785ad.3.1754642983298; Fri, 08 Aug 2025 01:49:43 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.49.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:49:42 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Robert Yang , Richard Purdie Subject: [poky][scarthgap][PATCH 02/23] bitbake: bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local() Date: Fri, 8 Aug 2025 14:19:10 +0530 Message-Id: <20250808084931.2156763-2-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:49:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221599 From: Robert Yang Update the test cases since the implementation is changed: * test_shallow_multi_one_uri() The a_branch and v0.0 had the same revision, and it required fetch a_branch and remove histories of v0.0 which were conflicted, and bitbake reported: fatal: no commits selected for shallow requests Make a_branch and v0.0 have different revs to fix the problem. And now the 'rev^' is not needed, so update self.assertRevCount() as well. * test_shallow_multi_one_uri_depths() Update self.assertRevCount(), now git only fetches the required revs. * test_shallow_fetch_missing_revs() The command is: $ git fetch --shallow-exclude=v0.0 master But master and v0.0 uses the same revision, so there is no commit to fetch. * test_shallow_fetch_missing_revs_fails() Two unneeded committs are not fetched now: - rev^ - One not specified or required tag. So update self.assertRevCount() (Bitbake rev: 48eff9d9a660ad6b9bd8b53a7dcec600ef42b1d1) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit 7bc521ed34656165fedaef1dd98cee865092291d) Signed-off-by: Akash Hadke --- bitbake/lib/bb/tests/fetch.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index b57cf511a4..eed92ad6ea 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -2012,9 +2012,9 @@ class GitShallowTest(FetcherTest): self.add_empty_file('b') self.git('checkout -b a_branch', cwd=self.srcdir) self.add_empty_file('c') + self.git('tag v0.0 HEAD', cwd=self.srcdir) self.add_empty_file('d') self.git('checkout master', cwd=self.srcdir) - self.git('tag v0.0 a_branch', cwd=self.srcdir) self.add_empty_file('e') self.git('merge --no-ff --no-edit a_branch', cwd=self.srcdir) self.add_empty_file('f') @@ -2030,7 +2030,7 @@ class GitShallowTest(FetcherTest): self.fetch_shallow(uri) - self.assertRevCount(5) + self.assertRevCount(4) self.assertRefs(['master', 'origin/master', 'origin/a_branch']) def test_shallow_multi_one_uri_depths(self): @@ -2177,7 +2177,7 @@ class GitShallowTest(FetcherTest): self.fetch_shallow() - self.assertRevCount(5) + self.assertRevCount(2) def test_shallow_invalid_revs(self): self.add_empty_file('a') @@ -2196,7 +2196,10 @@ class GitShallowTest(FetcherTest): self.git('tag v0.0 master', cwd=self.srcdir) self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0') self.d.setVar('BB_GIT_SHALLOW_REVS', 'v0.0') - self.fetch_shallow() + + with self.assertRaises(bb.fetch2.FetchError), self.assertLogs("BitBake.Fetcher", level="ERROR") as cm: + self.fetch_shallow() + self.assertIn("fatal: no commits selected for shallow requests", cm.output[0]) def test_shallow_fetch_missing_revs_fails(self): self.add_empty_file('a') @@ -2227,7 +2230,7 @@ class GitShallowTest(FetcherTest): revs = len(self.git('rev-list master').splitlines()) self.assertNotEqual(orig_revs, revs) self.assertRefs(['master', 'origin/master']) - self.assertRevCount(orig_revs - 1758) + self.assertRevCount(orig_revs - 1760) def test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist(self): self.add_empty_file('a') From patchwork Fri Aug 8 08:49:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68218 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29F4CC87FDA for ; Fri, 8 Aug 2025 08:49:57 +0000 (UTC) Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by mx.groups.io with SMTP id smtpd.web10.17676.1754642987184205044 for ; Fri, 08 Aug 2025 01:49:47 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=jehYeicH; spf=pass (domain: gmail.com, ip: 209.85.215.177, mailfrom: akash.hadke27@gmail.com) Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-b34a6d0c9a3so2200648a12.3 for ; Fri, 08 Aug 2025 01:49:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754642986; x=1755247786; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8p8vzDIdIjcqdEZQlgsyw2egD+F8dLWFbh55pam/sIA=; b=jehYeicHw5deq6NCnTfJayj2OdPRIIRl2zD2i02hK7Q6mYuktwzV+A8ZKKFc/fwVwD uXThNQxOigmR2C+rC0hNGT69yhQZRsunxsUrk98K7TB4f5QdTM1/eMt1+ZQKFmFR0DzK l2j7KMrR08QiWQeWMyCeYUxJrQ0rNDWoINX6xoBPsYvNXqgWPxZ/rbnenPcqDOLF7asl TrT6rMRYQHK8WLFgRg8FaFWoRhxuedO4bhslS1VBAmzCBRQTE/dGj9fdY+rNRsy910rr 4QG3e1blqwryJD87II/LrFUgDV04ZaB6oLIZTQpFCkHusVNcQHhXLuxzQKAbwAdygjx/ QUig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754642986; x=1755247786; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8p8vzDIdIjcqdEZQlgsyw2egD+F8dLWFbh55pam/sIA=; b=u4+e8Tn20g+qkVE1mSxBuek6tp/2nOvFXvfPKTKk0N5em6Uk1TXMqC3Y4EdZK9bfEM Tobg0HY2eD7I8BxVNmI/iDxUbRG+iGt6GWKJW+XFxQ6CuYYWLm2vklw6iT4OeilApTF4 v9WSzpclZbu8SlPv2tO64r6QZ9VkPaUVaE3+NIoktN2rJD5yZnWCe8+2kpCMjMBh6QtA ldnl5XGiyvk/qupgn1v9X/xz1nKI9wfa+LHea8ZIAGNDtN1usM2JVCRVJ7WTfXrmtS76 d3x0X9dHxhLB6rU90ASap1xQPFBzS1YtH8Nazo1GLhIpnpaCnH+I/X/rbm9oLORB/eJy 3Pnw== X-Gm-Message-State: AOJu0YwY2uuTV+EhSvYyVTFr5siZTFRLEJyDxs76fe7V01PSYAoApv8D wOTlTSEA/3BzMVkPVkmmogMv3fjwwIjMuUMm5dkelBHNMafUYJ3eTdKcJzjpKw== X-Gm-Gg: ASbGncveVl9iQMDx++IJnWKKHK2YVAzL1gXSKKrlBQZxOfnlnBZEUtQ9Z4e/oo4VgC/ axmiYDDY6D/bBOQwzl/G6dHXiK1curvVtd6ufTsh2JsIlwbH3IOxYCzglFTB7K8ykfwcN1g8zd/ jUk+m1uTbu7Y3HNrQ6JTXN1kxkBaynO4lYN9Dr2O1fFxi4/WleSLl5Stnr/LH1uNKXLKxJuLszd HKDrWik+IpkvtkbR6kAqsn2lLeTlBnWQ60ih9HBHI5smwOHMKCyGRmfPoBzX1R9vAiUsftNL9MS Lll5e2WgFwLsL9NLTy7c9oNBsyjE7lwGw1s4LGN059G4iqTAoBNSsfNioBZGK9S4B22lUUwr+Gq SpomqK1LIYH45VNMrOjzuoVT1ApQwG+aA X-Google-Smtp-Source: AGHT+IGV1aZIuxIWvJX1qSsFDwN/SyJPq5MmoW+Zib6I8gSNJLeDyDdMZWxGsEXK3CU4yD/tFCLemQ== X-Received: by 2002:a17:903:2391:b0:234:a139:11f0 with SMTP id d9443c01a7336-242c20703e0mr37580315ad.7.1754642986155; Fri, 08 Aug 2025 01:49:46 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.49.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:49:45 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Yuri D'Elia , Richard Purdie Subject: [poky][scarthgap][PATCH 03/23] bitbake: fetch2/git: Enforce default remote name to "origin" Date: Fri, 8 Aug 2025 14:19:11 +0530 Message-Id: <20250808084931.2156763-3-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:49:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221600 From: Yuri D'Elia Enforce the default remote name to "origin", as assumed in numerous places. This prevents build failures in case the system/user configuration sets this to a different value. (Bitbake rev: 1d7360031164f04887c792fb0b2dd86c6ccfcc23) Signed-off-by: Richard Purdie (cherry picked from commit b698371255799f94861af57c710450e4879af208) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 2194c839dd..5b678827ed 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -262,7 +262,7 @@ class Git(FetchMethod): for name in ud.names: ud.unresolvedrev[name] = 'HEAD' - ud.basecmd = d.getVar("FETCHCMD_git") or "git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all" + ud.basecmd = d.getVar("FETCHCMD_git") or "git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin" write_tarballs = d.getVar("BB_GENERATE_MIRROR_TARBALLS") or "0" ud.write_tarballs = write_tarballs != "0" or ud.rebaseable From patchwork Fri Aug 8 08:49:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29F12C87FD2 for ; Fri, 8 Aug 2025 08:49:57 +0000 (UTC) Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by mx.groups.io with SMTP id smtpd.web11.17936.1754642989923380453 for ; Fri, 08 Aug 2025 01:49:49 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Cc2xPS5r; spf=pass (domain: gmail.com, ip: 209.85.214.179, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-23636167afeso15961385ad.3 for ; Fri, 08 Aug 2025 01:49:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754642989; x=1755247789; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hXw8G6/lBiPMqvhA39b+1TweWZYLKsxhExx0liOHVz4=; b=Cc2xPS5rmBExOgWX9eAIWEejrrviQC7uMcu9+U6PiIEN7sALD3OL54TTEgh+l3MLjS HKOoRUdncGJMLJYwKzqIr1XqNMU8lWp4YkAtJLgmdeRd4IzCVk1SLlfE5OmcnfIVhNeW emWxnZCYRr8dYpGV8o+AaCO1xNvLEISelI6s/Dn6u84gcfbPNSMKRhtd5jiy0OXU2R4r WvprOJI62zziBipOAFlOqUfKx5sNjf0SS3RvcqtCO5k/SiTQ07+Ch4HFCCd7M2Tsc5no dB/1eaBvfE55z80bhSpiaMNfcwt8Dbo4jdD3MhJN7hhqxiknxwptxiJNftRSrlme3IMN H3Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754642989; x=1755247789; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hXw8G6/lBiPMqvhA39b+1TweWZYLKsxhExx0liOHVz4=; b=GOSbkzsWZvChcLj7TPUW/S4A0zA6HFR9KxwwRFKffH89KGvdLK835zWEnic2gXwBJb OcErMYAvamXN5vzx9RErDBo/G675Qkxzu6aTeKq2W5uHl2ikYwOayCo5o2Q2NuHu5co8 +vstbNTJOPc5VZnOHJ6jzG6Ds2yGoHWab1NNNAr1AQh+uTZ8WGBAw9rHtySp+BSc/aEt r3SAexRHC14WS7kIDbhQl5wmeK1yESdJejZet1zvvg4uD5UKCn/3H9ElRmZGkHfs3RXT dwn24VJJonVHN1994OBsf87cWzXsDkTLvgYOnlUEkKkqbmrROSKIiBa4JMRIsR9BF69F 29yg== X-Gm-Message-State: AOJu0YzxfVDqXX1aAXlq8oyFFI7ahEm6WXAgIVigkJ0UPoLpSIUp2O2B nRB88MOsor3aInimbsr5YkMBqbS3q/hqSRQ94MUBLdYTr10V+NqyD743XkBIBg== X-Gm-Gg: ASbGncurJD6urfMQD5yo68RShFm6fxvgFbWMkW3qaejNtNrZMOg5gIm/zXXayGOX3/6 Jr3dRyRzUvRdrg8TZEl9+TsmPmup1SjkGr4Iw9zdHpF7EJSiktMD/rquK3dLi2YD4HyxTIxqN02 h6CDTANf7r9Zo9od/as61cpYwMoc9nR7idvFJGVVmOjWVVgyf9OhrPy0ZF/IOYoNyKFnxxsBuXB WDkwP57x3MqO52gozugi4W2WSDnz046nRAYJ/kLrBkR6Y3DWaBEfePcrFs3r5FOSHg6JaeXD+CY pfc1Wsit+2dzrmJ+HpokQSAXUJ3MEmuA405D1dXOQyJ25PFhAMdeEilA5Fq/HXkGULx4rSKPG6s Spgj+znyBs88E+hBux4kY30DxRcldgNtF X-Google-Smtp-Source: AGHT+IFI+MPUCvd35hp+nuBr+K43YPwTPMOTeurusIYvVngvc2eNA9ax9+BNfZUHdyizX5UOXqkFpA== X-Received: by 2002:a17:902:ea11:b0:240:3239:21c7 with SMTP id d9443c01a7336-242c21dcd0bmr32476785ad.37.1754642988824; Fri, 08 Aug 2025 01:49:48 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.49.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:49:48 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Robert Yang , Richard Purdie Subject: [poky][scarthgap][PATCH 04/23] bitbake: gitsm: Add clean function Date: Fri, 8 Aug 2025 14:19:12 +0530 Message-Id: <20250808084931.2156763-4-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:49:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221601 From: Robert Yang Fixed: $ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall The downloads/git2/github.com.nemtrif.ftest won't be cleaned without this fix. (Bitbake rev: 79f25fc5c1b8d0e08540f4aa07875309f5325f47) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit 5bce38fbaea0a7d1228740e2cb313957c914cfdf) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/gitsm.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py index fab4b1164c..ba62517f08 100644 --- a/bitbake/lib/bb/fetch2/gitsm.py +++ b/bitbake/lib/bb/fetch2/gitsm.py @@ -249,6 +249,19 @@ class GitSM(Git): # should also be skipped as these files were already smudged in the fetch stage if lfs # was enabled. runfetchcmd("GIT_LFS_SKIP_SMUDGE=1 %s submodule update --recursive --no-fetch" % (ud.basecmd), d, quiet=True, workdir=ud.destdir) + def clean(self, ud, d): + def clean_submodule(ud, url, module, modpath, workdir, d): + url += ";bareclone=1;nobranch=1" + try: + newfetch = Fetch([url], d, cache=False) + newfetch.clean() + except Exception as e: + logger.warning('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e))) + + self.call_process_submodules(ud, d, True, clean_submodule) + + # Clean top git dir + Git.clean(self, ud, d) def implicit_urldata(self, ud, d): import shutil, subprocess, tempfile From patchwork Fri Aug 8 08:49:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36A17C87FD3 for ; Fri, 8 Aug 2025 08:49:57 +0000 (UTC) Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by mx.groups.io with SMTP id smtpd.web10.17677.1754642992612218504 for ; Fri, 08 Aug 2025 01:49:52 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=dwaQhijs; spf=pass (domain: gmail.com, ip: 209.85.214.178, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2405c0c431cso17673995ad.1 for ; Fri, 08 Aug 2025 01:49:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754642992; x=1755247792; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c3symzm59eEajvUi1Pn4TRyMO2CgS8eRrdaZ/eXp3VI=; b=dwaQhijsP/Id9wKO0EBOYntrR7u681c9Hlc+wKWaVe0yXkOFymoumjJyo8xKFJP7x8 GQ/Mz+t/vvTXDwfGagUCAlN3PnRyZTeU0pSfWjMc9zkfvQ2cua2wClQxXdIqVZPjQeTe wj5+9NhcjTzjniHvk49P2o9lfOsfOwQEK/jiwgNut6E7AWU+0kRlYM4QM2l22YTbk8a0 N5v3MTPEuHHJa2Smt1hMtnT7WMx6FRlOhH8o1fQM5K7oVTp+R0TYV6Oc+ffZhHlG16SL ieCGCox6fSrAzGpcE8GjdmkoDAkKRKWVSs9T5fMMvKuS/CCRFObDTBYnZ/x6dlXR3mD8 Srfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754642992; x=1755247792; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c3symzm59eEajvUi1Pn4TRyMO2CgS8eRrdaZ/eXp3VI=; b=CvL2yL1ee39+nLM1Zzfusm3/vUq1f4062FbByxBna5Nx+0SytNUQHt59SMygOfiPvD OQy54PkASUkPutnuLTzAELytubn08/EHZcXPUp/pBQD5cCPMqypUZ9F8xHBnmNJAL6DE TbKzvntDN/jkAfz2Cydk0lqH6wzMiKjyEnRF2pWxh2HkrJngbRs3Ng5X4Pct0fwRKKlo +HZCosj2E9uM8qw6ODq45fragoFHHax1HcZ1GF5kWfizRHiq+cEMfHUByfSCgLTl+h/Y lM7ETOhltF7g2XRxsXrlohtRwjNVnbmVAnte5/7e8pUuMPzeGj2rDKk9WL+RBR0+aNSZ MSlg== X-Gm-Message-State: AOJu0YwokUlE3xBQsymYTQic/mQfxmHMQpFbMUoxd6uU6NNCFV4uZ0Wo Q9KRStYaY94Ls1SynWYy5fTIRFLgD6/ya1O2q35arVoy/+OoqhdU/HI7ggRBbQ== X-Gm-Gg: ASbGncs0imhx46WEs/qOUj4It/fV3OeSYriFCfoIA281aR4M2ZBluwgR5HNeQo8SHQL 9g3g2P/eemM/6J4MHwQxfCTnNLqVUKboZAIsGh/iR3zjBW7XLDaN1O4gUxWWH3GOUUG9HeKBiR9 5DIMhXW9CfNYVSKcRp99PFG6W3vRYapfehfUui6mRzHMnH8cn8JZagpp1X2nAEGDUlN9b7QOIUf b+11IBqmeSli6q2tthqVr/dZnOuamAGyUlN8GnviIeb8fewkzVkq4yeaxOnqsTixO2Gs+CMYn+k IT13Js8Z+WIUDv9AbHUxSTApXbbG9SLKu6flrWBChHvhWO1ENlTu7pxxGDE6DNcGTJNLuBmeYWB wwTZ9vh3QSRt/bh76tTnsKzdpXOlXeaoB X-Google-Smtp-Source: AGHT+IF7UbLKTl1FgKuLI27R4LQTO2nof+njpDyXMfqAaawO/qFkO+8/s3/gkK7clvYXoP0q2R2ugA== X-Received: by 2002:a17:902:da83:b0:23f:f39b:eaf6 with SMTP id d9443c01a7336-242c226cac3mr34832595ad.46.1754642991363; Fri, 08 Aug 2025 01:49:51 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.49.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:49:51 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Robert Yang , Richard Purdie Subject: [poky][scarthgap][PATCH 05/23] bitbake: git: Clean shallow mirror tarball Date: Fri, 8 Aug 2025 14:19:13 +0530 Message-Id: <20250808084931.2156763-5-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:49:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221602 From: Robert Yang Fixed: BB_GIT_SHALLOW = "1" BB_GENERATE_SHALLOW_TARBALLS = "1" $ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall The downloads/gitsmshallow_github.com.nemtrif.*.tar.gz won't be cleaned without this fix. (Bitbake rev: bab7a8970a0237a9d24217685a595e76a1336c07) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit dd9d183621d583331cd17c7ea4c6dd7cc350e8aa) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 5b678827ed..0ea55e139d 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -727,6 +727,11 @@ class Git(FetchMethod): clonedir = os.path.realpath(ud.localpath) to_remove.append(clonedir) + # Remove shallow mirror tarball + if ud.shallow: + to_remove.append(ud.fullshallow) + to_remove.append(ud.fullshallow + ".done") + for r in to_remove: if os.path.exists(r): bb.note('Removing %s' % r) From patchwork Fri Aug 8 08:49:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68220 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4426ECA0EC0 for ; Fri, 8 Aug 2025 08:49:57 +0000 (UTC) Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by mx.groups.io with SMTP id smtpd.web10.17679.1754642995009838659 for ; Fri, 08 Aug 2025 01:49:55 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=J6Zpsbwf; spf=pass (domain: gmail.com, ip: 209.85.214.172, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2402bbb4bf3so22959505ad.2 for ; Fri, 08 Aug 2025 01:49:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754642994; x=1755247794; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7TZofw14SzG1UPmEVIWyW9VjhjZBe54YMK3FzEUSST4=; b=J6ZpsbwfGbDArczGEglyTlwJRnnNHZkffFgchzKhRrJJZPp6Ox4jx2GSxnUSG2Mc/L AGKRsQ7CIzZ4ic2ThIOXql3f9ZLygv3ZqIW3nlWyngVEIeXeR/4o4l7HUJacNtwvQBlR NFApCLRqOhvxCR7uSaCw6PoEfHWgb16+YebxvDBuYxVVMmADX3b0QOgS5uu26QYko7wA TCg/DFF1xu2DiZDKrOLyhNDLmgLvWqSkDuttipM2QfRN8kA7K8CFwxaap0VNrAB2P10r bfm6Gsy93ynQE+sSmp/7TC/JkZEOxtaJtFSYWHlOfkrMpsXqlPM+b5jDkVOTJgczNSf0 YK5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754642994; x=1755247794; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7TZofw14SzG1UPmEVIWyW9VjhjZBe54YMK3FzEUSST4=; b=MSZHpuoMZmXsT3rHFSeeyHoTyGIFyBWcTo+jdZ0V/eFUYC+pxSkKheC8Blo+ShajIE 66mHDG857phGE8qKjDdQWB8zptcRyggFBm3+KGz2ZJueurRuVIa3liYjgVQR+OpY3TaF wWVn05IIU6f6Z2imzj8eQ2GrM00c8gdVLl6jckrfvfdKo1uNuSLlUhkaTc3Td50zIV1t W3GSaqpo1L5VFvT18LBlpx2CMhu4V11952WqYHecap//bh3+qE+ciWqq3Pt3gj980puh OYJ2Qg1OLbjh8Gej7BUxp9KnynE5zzNHersamPwKiNcY/yuKJsLeU8XhQ36iiO4+wZc4 KfKg== X-Gm-Message-State: AOJu0Yy7xAujEO9TsGun49A3wDjsN80e5CqP/50G5pfkw4P0T/ORyaBF j8o02T9fXO7hnrrFwCUqpT8GUj1rL0pJrFNTuavP6V23vpHtSTFuZ0Unyl40Cg== X-Gm-Gg: ASbGnct61nAAJe03cRel+c/ShUKjt8cvcIgwpvVkBCAIaYGpo+HKYjC/A0Y8MTIkpm5 11VUEJIQ83AiR70Ffo9EtqsvS2pF3vePA8X75HnK2Lr6tFws2sQdlLf6+zhmwjjfWgjAJjwuz26 2oRu1Fhdl8yI+vEcVq4gHGA2fKndJ0S6avGR9vETZvVB2vgIFElQpEOiCR/XOjXdgKuxO1koPFe RRYPmFpsHsD+JTtKMw/IKmKQoRoz8hDEcBfHe7JdiS62rI+7zsGm35VXXuFD7l0yW4cGy3SxLhs eIu7jvCtaOKfHZ0NluvMysfegOOPBP6K5oHPhxVxT9LD2Rv0JniItTDF69A/ow3hi4pighZZXOV 01CXa6fOdQc1omSZ8IbyK51hNE6L4WLMA X-Google-Smtp-Source: AGHT+IH55Sa2kiIA7bfXMafeTFUniznALH7rU0FRmu9qx8s8K5CEJDbz8hHx+3Kx2IlxL1Ws/H9dAA== X-Received: by 2002:a17:903:2a8b:b0:240:2a0:c449 with SMTP id d9443c01a7336-242c1fdc1b2mr29844735ad.8.1754642993961; Fri, 08 Aug 2025 01:49:53 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.49.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:49:53 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Robert Yang , Richard Purdie Subject: [poky][scarthgap][PATCH 06/23] bitbake: git: Clean broken symlink Date: Fri, 8 Aug 2025 14:19:14 +0530 Message-Id: <20250808084931.2156763-6-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:49:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221603 From: Robert Yang The file may be a symlink when mirror from local disk, and the symlink will be broken when the linked file is removed, use os.path.islink() to check and remove the broken symlink. (Bitbake rev: d0deb5fe6e7a84b07b49ef685b209729d2fdc780) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit 0a45ba64adbc279928a4347c5073eb78da1bf00a) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 0ea55e139d..6badda5972 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -733,7 +733,7 @@ class Git(FetchMethod): to_remove.append(ud.fullshallow + ".done") for r in to_remove: - if os.path.exists(r): + if os.path.exists(r) or os.path.islink(r): bb.note('Removing %s' % r) bb.utils.remove(r, True) From patchwork Fri Aug 8 08:49:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68222 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42D3DC87FDA for ; Fri, 8 Aug 2025 08:50:07 +0000 (UTC) Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by mx.groups.io with SMTP id smtpd.web11.17937.1754642997708249122 for ; Fri, 08 Aug 2025 01:49:57 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=WZP/cDAS; spf=pass (domain: gmail.com, ip: 209.85.214.171, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2401b855980so13626235ad.1 for ; Fri, 08 Aug 2025 01:49:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754642997; x=1755247797; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Trb7PFGlellBbTNTlVhpdaNZn34qoReBUU2I/Htc0K4=; b=WZP/cDASnYZYE4dk6NUkWlKMmeMs6dkNykEHf2Pou28GgKfTw80RsPPV1gelv63Emr FtqCRH+O060iTiTcrcMD5tL7Sc0qxPazHj21LQO4k3cMohxN9/jXUDeBQjWHNfjWm+VR pPMRjvY9xtigxCaAD2a5faTS3uLJ68KShrgzgV4IV868oRZxBEj/OcFAOz2bP/kLi30o SL3vbQqBE9Z8St047+T1hUYRXVW90ZfNyqdk29KJT0RRUag9nEE18uzLJ2gbNJaJ791O WLi5Qhd1ALBZyiYahvUkWD3JS9Tgf+cqAoH/vgLYfTHTJAzhtArfYfY9eBS9mAhtUGMI ubdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754642997; x=1755247797; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Trb7PFGlellBbTNTlVhpdaNZn34qoReBUU2I/Htc0K4=; b=gtouZzpjwwMEiBtX+f7GuMThBpjwFZaccOdv8prfGUzBRD7rASMLPHVkSb37BPMYy6 JsLH05oW71EQ4agssQALQ29FKvGeuajPh/gpu3SP5IjKQXYt66ihumTf/XVtgFZmMMah zUz8BOjoJRIRHV4PimqQY/qtTbqp+Wavcu3J/HU5w+0wqnBQH+bZLV86lr6DBjiuZ/ME N0JVdf/DrDSDJHmTP6HuUzRgHc37C0dt51x/tqC4kq3yWfiJ29kXtPHoerk4Uvj8BazV a36ExZcMSUfmCxpW2avB1J1JXQp95UUkKdFKwwGaLx6lXk3tdTkoMe5VPMyjCMICiSuT wXeg== X-Gm-Message-State: AOJu0Yw4Ljriy2ojRc4pC+vIVOrQ5qMCazzkhM1ta6UHKY4jdl4JM6sV AsOoJkb41b5rmViNnolRKfI8OgDWT30/OgUXb8GRw8+n42Ez/nLaysTM/Hptlg== X-Gm-Gg: ASbGncsSX++I7UdsM5CHG8iupwbj45DFwHzVFnhJsMq/AlBcOKAGXt+YcXd/2pK8PF/ SD2VzuXmWCWGmcU2qnNZkN998fXtlYDF+CZcCZct++184wJri76KxCMbH3WV5Qv3qs2qucFUqOL vOt+qdx2CudDYKSwhvlO13YFLvUX3zj63doCc4hSL7CkjDKGyTl3kwPIztU/lPGf2IN4/GFv9KB Gs8sjWsiIPLbc7HYWmmhMjGMmRrVl02GBljmj9552+2y6eZAaScewlhjaQ8bPAnXnBG2h/8Ec/x xgldtjI8HrN+wrG3G8msvFzQQ5k/I+G5b8PWOHqMGT4wN6AbgDOdqRpF7W+qrUgxI9pvkkb/h1h sSTB4BGE1EDlVxoHhDuWUHJxJQW2mGDqB X-Google-Smtp-Source: AGHT+IHQbENPBBAZ1HKCB6TqPzJ2zt8xwoj0Sar1zA+iM8mBv5V+DivNi+kz1z4+9jQu+uj1MYvexQ== X-Received: by 2002:a17:902:d550:b0:234:1163:ff99 with SMTP id d9443c01a7336-242c222c806mr32410155ad.43.1754642996652; Fri, 08 Aug 2025 01:49:56 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.49.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:49:56 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Michael Estner , Richard Purdie Subject: [poky][scarthgap][PATCH 07/23] bitbake: lib: Remove double imports Date: Fri, 8 Aug 2025 14:19:15 +0530 Message-Id: <20250808084931.2156763-7-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221604 From: Michael Estner * Remove double imports mentioned by pylint (Bitbake rev: 741db6719efca5aa9ef2c15e60cdd624e4aa1a8d) Signed-off-by: Michael Estner Signed-off-by: Richard Purdie (cherry picked from commit e1836b205a2716e85f36346762805e2e505a5eb6) Signed-off-by: Akash Hadke --- bitbake/lib/bb/__init__.py | 1 - bitbake/lib/bb/acl.py | 2 -- bitbake/lib/bb/fetch2/git.py | 1 - bitbake/lib/bb/fetch2/gitsm.py | 2 +- bitbake/lib/bb/runqueue.py | 1 - bitbake/lib/bs4/__init__.py | 1 - bitbake/lib/bs4/diagnose.py | 1 - bitbake/lib/ply/yacc.py | 1 - bitbake/lib/toaster/orm/models.py | 3 +-- bitbake/lib/toaster/toastergui/templatetags/projecttags.py | 1 - bitbake/lib/toaster/toastergui/views.py | 2 -- bitbake/lib/toaster/toastermain/settings.py | 1 - 12 files changed, 2 insertions(+), 15 deletions(-) diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index cdec9e4d6c..a3c259479a 100644 --- a/bitbake/lib/bb/__init__.py +++ b/bitbake/lib/bb/__init__.py @@ -214,7 +214,6 @@ def deprecated(func, name=None, advice=""): # For compatibility def deprecate_import(current, modulename, fromlist, renames = None): """Import objects from one module into another, wrapping them with a DeprecationWarning""" - import sys module = __import__(modulename, fromlist = fromlist) for position, objname in enumerate(fromlist): diff --git a/bitbake/lib/bb/acl.py b/bitbake/lib/bb/acl.py index 0f41b275cf..e9dbdb617f 100755 --- a/bitbake/lib/bb/acl.py +++ b/bitbake/lib/bb/acl.py @@ -195,8 +195,6 @@ class ACL(object): def main(): import argparse - import pwd - import grp from pathlib import Path parser = argparse.ArgumentParser() diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 6badda5972..168f14d0c8 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -817,7 +817,6 @@ class Git(FetchMethod): """ Return True if git-lfs can be found, False otherwise. """ - import shutil return shutil.which("git-lfs", path=d.getVar('PATH')) is not None def _get_repo_url(self, ud): diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py index ba62517f08..bab1bf2580 100644 --- a/bitbake/lib/bb/fetch2/gitsm.py +++ b/bitbake/lib/bb/fetch2/gitsm.py @@ -264,7 +264,7 @@ class GitSM(Git): Git.clean(self, ud, d) def implicit_urldata(self, ud, d): - import shutil, subprocess, tempfile + import subprocess urldata = [] def add_submodule(ud, url, module, modpath, workdir, d): diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index db68f97e68..809324d3fd 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -476,7 +476,6 @@ class RunQueueData: self.runtaskentries = {} def runq_depends_names(self, ids): - import re ret = [] for id in ids: nam = os.path.basename(id) diff --git a/bitbake/lib/bs4/__init__.py b/bitbake/lib/bs4/__init__.py index e35725b86e..e226b9cf4a 100644 --- a/bitbake/lib/bs4/__init__.py +++ b/bitbake/lib/bs4/__init__.py @@ -463,6 +463,5 @@ class FeatureNotFound(ValueError): #By default, act as an HTML pretty-printer. if __name__ == '__main__': - import sys soup = BeautifulSoup(sys.stdin) print(soup.prettify()) diff --git a/bitbake/lib/bs4/diagnose.py b/bitbake/lib/bs4/diagnose.py index 083395fb46..8d6ce8f87e 100644 --- a/bitbake/lib/bs4/diagnose.py +++ b/bitbake/lib/bs4/diagnose.py @@ -16,7 +16,6 @@ import tempfile import time import traceback import sys -import cProfile def diagnose(data): """Diagnostic suite for isolating common problems.""" diff --git a/bitbake/lib/ply/yacc.py b/bitbake/lib/ply/yacc.py index 381b50cf0b..529f85b081 100644 --- a/bitbake/lib/ply/yacc.py +++ b/bitbake/lib/ply/yacc.py @@ -1122,7 +1122,6 @@ class LRParser: # manipulate the rules that make up a grammar. # ----------------------------------------------------------------------------- -import re # regex matching identifiers _is_identifier = re.compile(r'^[a-zA-Z0-9_-]+$') diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py index 19c9686206..e2f488ed89 100644 --- a/bitbake/lib/toaster/orm/models.py +++ b/bitbake/lib/toaster/orm/models.py @@ -79,7 +79,6 @@ if 'sqlite' in settings.DATABASES['default']['ENGINE']: # end of HACK class GitURLValidator(validators.URLValidator): - import re regex = re.compile( r'^(?:ssh|git|http|ftp)s?://' # http:// or https:// r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}\.?)|' # domain... @@ -1500,7 +1499,7 @@ class Layer_Version(models.Model): # code lifted, with adaptations, from the layerindex-web application # https://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/ def _handle_url_path(self, base_url, path): - import re, posixpath + import posixpath if base_url: if self.dirpath: if path: diff --git a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py index bd398f0012..aee9bbcd14 100644 --- a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py +++ b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py @@ -233,7 +233,6 @@ def filter_sizeovertotal(package_object, total_size): return '{:.1%}'.format(float(size)/float(total_size)) -from django.utils.safestring import mark_safe @register.filter def format_vpackage_rowclass(size): if size == -1: diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/toastergui/views.py index 40aed265dc..061e6436c8 100644 --- a/bitbake/lib/toaster/toastergui/views.py +++ b/bitbake/lib/toaster/toastergui/views.py @@ -372,7 +372,6 @@ def _get_parameters_values(request, default_count, default_order): # set cookies for parameters. this is usefull in case parameters are set # manually from the GET values of the link def _set_parameters_values(pagesize, orderby, request): - from django.urls import resolve current_url = resolve(request.path_info).url_name request.session['%s_count' % current_url] = pagesize request.session['%s_orderby' % current_url] =orderby @@ -699,7 +698,6 @@ class LazyEncoder(json.JSONEncoder): return super(LazyEncoder, self).default(obj) from toastergui.templatetags.projecttags import filtered_filesizeformat -import os def _get_dir_entries(build_id, target_id, start): node_str = { Target_File.ITYPE_REGULAR : '-', diff --git a/bitbake/lib/toaster/toastermain/settings.py b/bitbake/lib/toaster/toastermain/settings.py index e06adc5a93..d2a449627f 100644 --- a/bitbake/lib/toaster/toastermain/settings.py +++ b/bitbake/lib/toaster/toastermain/settings.py @@ -298,7 +298,6 @@ SOUTH_TESTS_MIGRATE = False # We automatically detect and install applications here if # they have a 'models.py' or 'views.py' file -import os currentdir = os.path.dirname(__file__) for t in os.walk(os.path.dirname(currentdir)): modulename = os.path.basename(t[0]) From patchwork Fri Aug 8 08:49:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B9D4C87FD2 for ; Fri, 8 Aug 2025 08:50:07 +0000 (UTC) Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by mx.groups.io with SMTP id smtpd.web10.17682.1754643000566462320 for ; Fri, 08 Aug 2025 01:50:00 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=SAGboEpp; spf=pass (domain: gmail.com, ip: 209.85.214.172, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-24031a3e05cso13504015ad.1 for ; Fri, 08 Aug 2025 01:50:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754642999; x=1755247799; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jrzoX/joTffAl1MPQX3ffgHU0uU261I4cyTWCxNEwXs=; b=SAGboEpp/xuA7JoCbudWWYTMoHehTIeSh3CKUTMR3anJAdisewpvvxQ4QuQ8hR/ldf dph12u6dHPv1rasKm2fJwhZvxXHk5bSb5GyD64FL8PobLRUMLKbPbcU5fWtkVBjv2/xh CwoHFwfBTTl0fGpnZFoUYtqdt0O0cWcaGWTPj6ZqRFzz2sTgJ/ZxB6R98r5/f02plYkH v2wjeDhYC6oQBu4raOljOfydVwATIt2SCtBK8BdTWqQJWTL0IS4AetzEHkvW/PRQ6mwU M9u+WyVXxxBZJ/UpMKLDhXTTHAPuja/aM4bs87w5nSeWrXKf0oJ6X1fA7mbuNqpx9eGn onjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754642999; x=1755247799; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jrzoX/joTffAl1MPQX3ffgHU0uU261I4cyTWCxNEwXs=; b=D6l+SqcBx0nQYUg8XWXAt9GIFfbSqW6JYokIm9Cq1Rqud+03Z6P2lWkAl6nyvKU7By FYlD3RbDObndFhXYo8IXQsm7ujHI70EXrBw1TRJxdqDko/1N1Tf5C6D2Gx4TyT1L2WFG tq8MWsxqcNLlC5cNqBXpMN68viKnb9DyxR+v06l5WAQAHHjWqyNVSurLCIz85+D8fNZS JIPQYtMANkq3hvlamhWiWVpdl/sKyZpfJSnJ9gqtcxMYOkXg5hCZ8kaGZFNQa0Kn7/6y ofhm6ZbZTEZqPPM5fBwdHR7rS0blDleXfSM0mRm1MsAJwBHv5voJbaatCbUG0VpR7PUp 0d+Q== X-Gm-Message-State: AOJu0YyaJO1yILO8yyl7joIuL/etb8/MwM2+QnLUUnj6Myi6n0WAo5CG wxpxupfzfYUeq/ATmPeFNFYNFFbpAumHCzldKSymhOBxHBcVaRmYs1LGlbqLgA== X-Gm-Gg: ASbGncusLXjuRPDJpPfYxB/rNKbZtHTSbH59tlsivowZWlVvzjkST+H+SEcMJNqsvRm G14NpzcRZIf/tEVEic3h0XqzU2GWlqjyUALxvRaM165dzyErJ2zeJXibpyRjRlHdnDXt/4IJ8Fg pDNAPSnYkeoGlQLbYqRsFI5AdDvvDeTwjgYgXf+6DkjEkWCukAlf+tv6L6jRPihlEbqvPzk4c1G Yq3rdC85w3y/V9lK2b63Kf9u5VVd5eUJ68ovfIu7aaQBVa8tM1cqVR1D2Ma9LYPiHsNyYNHjl27 ZiYax9uepQJ/PmfUCIhL5U2gyw6ctKDIfvWW3PEVyQC+57E5xkdOayMJfngaMihjyvtTEs/bdoe qwwmtlb8ugM3nq80pL/P9IAx0dwuEzfjr X-Google-Smtp-Source: AGHT+IHgpQbO9Dehk2Kcwx+Yhtn16RwqTCyDCC9JZFnJrqdswJ0M/3jS4qllO90OsPPukrMuaR/iXQ== X-Received: by 2002:a17:902:d505:b0:234:9fe1:8fc6 with SMTP id d9443c01a7336-242b072bbb4mr97586895ad.18.1754642999210; Fri, 08 Aug 2025 01:49:59 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.49.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:49:58 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Stefan Koch , Richard Purdie Subject: [poky][scarthgap][PATCH 08/23] bitbake: fetch2/git: Add support for fast initial shallow fetch Date: Fri, 8 Aug 2025 14:19:16 +0530 Message-Id: <20250808084931.2156763-8-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221605 From: Stefan Koch When `ud.shallow == 1`: - Prefer an initial shallow clone over an initial full bare clone, while still utilizing any already existing full bare clones. - If the Git error "Server does not allow request for unadvertised object" occurs, the initial full bare clone is fetched automatically. This may happen if the Git server does not allow the request or if the Git client has issues with this functionality, especially with the Git client from Ubuntu 20.04. This improves: - Resolve timeout issues during initial clones on slow internet connections by reducing the amount of data transferred. - Eliminate the need to use an HTTPS tarball `SRC_URI` to reduce data transfer. - Allow SSH-based authentication (e.g. cert and agent-based) when using non-public repos, so additional HTTPS tokens may not be required. (Bitbake rev: 457288b2fda86fd00cdcaefac616129b0029e1f9) Signed-off-by: Stefan Koch Signed-off-by: Richard Purdie (cherry picked from commit 65ae50cd16a3989699bea845566d38476f2ae9a7) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 114 ++++++++++++++++++++++++++--------- 1 file changed, 85 insertions(+), 29 deletions(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 168f14d0c8..9a15abaa79 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -207,6 +207,7 @@ class Git(FetchMethod): if ud.bareclone: ud.cloneflags += " --mirror" + ud.shallow_skip_fast = False ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1" ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or "").split() @@ -446,6 +447,24 @@ class Git(FetchMethod): if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url) progresshandler = GitProgressHandler(d) + + # Try creating a fast initial shallow clone + # Enabling ud.shallow_skip_fast will skip this + # If the Git error "Server does not allow request for unadvertised object" + # occurs, shallow_skip_fast is enabled automatically. + # This may happen if the Git server does not allow the request + # or if the Git client has issues with this functionality. + if ud.shallow and not ud.shallow_skip_fast: + try: + self.clone_shallow_with_tarball(ud, d) + # When the shallow clone has succeeded, use the shallow tarball + ud.localpath = ud.fullshallow + return + except: + logger.warning("Creating fast initial shallow clone failed, try initial regular clone now.") + + # When skipping fast initial shallow or the fast inital shallow clone failed: + # Try again with an initial regular clone runfetchcmd(clone_cmd, d, log=progresshandler) # Update the checkout if needed @@ -508,48 +527,74 @@ class Git(FetchMethod): if os.path.exists(os.path.join(ud.destdir, ".git", "lfs")): runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clonedir, d, workdir="%s/.git" % ud.destdir) - def build_mirror_data(self, ud, d): - - # Create as a temp file and move atomically into position to avoid races - @contextmanager - def create_atomic(filename): - fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) - try: - yield tfile - umask = os.umask(0o666) - os.umask(umask) - os.chmod(tfile, (0o666 & ~umask)) - os.rename(tfile, filename) - finally: - os.close(fd) + def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None): + """Helper method for fetching Git LFS data""" + try: + if self._need_lfs(ud) and self._contains_lfs(ud, d, clonedir) and self._find_git_lfs(d) and len(revision): + # Using worktree with the revision because .lfsconfig may exists + worktree_add_cmd = "%s worktree add wt %s" % (ud.basecmd, revision) + runfetchcmd(worktree_add_cmd, d, log=progresshandler, workdir=clonedir) + lfs_fetch_cmd = "%s lfs fetch %s" % (ud.basecmd, "--all" if fetchall else "") + runfetchcmd(lfs_fetch_cmd, d, log=progresshandler, workdir=(clonedir + "/wt")) + worktree_rem_cmd = "%s worktree remove -f wt" % ud.basecmd + runfetchcmd(worktree_rem_cmd, d, log=progresshandler, workdir=clonedir) + except: + logger.warning("Fetching LFS did not succeed.") + + @contextmanager + def create_atomic(self, filename): + """Create as a temp file and move atomically into position to avoid races""" + fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename)) + try: + yield tfile + umask = os.umask(0o666) + os.umask(umask) + os.chmod(tfile, (0o666 & ~umask)) + os.rename(tfile, filename) + finally: + os.close(fd) + def build_mirror_data(self, ud, d): if ud.shallow and ud.write_shallow_tarballs: if not os.path.exists(ud.fullshallow): if os.path.islink(ud.fullshallow): os.unlink(ud.fullshallow) - tempdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR')) - shallowclone = os.path.join(tempdir, 'git') - try: - self.clone_shallow_local(ud, shallowclone, d) - - logger.info("Creating tarball of git repository") - with create_atomic(ud.fullshallow) as tfile: - runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) - runfetchcmd("touch %s.done" % ud.fullshallow, d) - finally: - bb.utils.remove(tempdir, recurse=True) + self.clone_shallow_with_tarball(ud, d) elif ud.write_tarballs and not os.path.exists(ud.fullmirror): if os.path.islink(ud.fullmirror): os.unlink(ud.fullmirror) logger.info("Creating tarball of git repository") - with create_atomic(ud.fullmirror) as tfile: + with self.create_atomic(ud.fullmirror) as tfile: mtime = runfetchcmd("{} log --all -1 --format=%cD".format(ud.basecmd), d, quiet=True, workdir=ud.clonedir) runfetchcmd("tar -czf %s --owner oe:0 --group oe:0 --mtime \"%s\" ." % (tfile, mtime), d, workdir=ud.clonedir) runfetchcmd("touch %s.done" % ud.fullmirror, d) + def clone_shallow_with_tarball(self, ud, d): + ret = False + tempdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR')) + shallowclone = os.path.join(tempdir, 'git') + try: + try: + self.clone_shallow_local(ud, shallowclone, d) + except: + logger.warning("Fash shallow clone failed, try to skip fast mode now.") + bb.utils.remove(tempdir, recurse=True) + os.mkdir(tempdir) + ud.shallow_skip_fast = True + self.clone_shallow_local(ud, shallowclone, d) + logger.info("Creating tarball of git repository") + with self.create_atomic(ud.fullshallow) as tfile: + runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone) + runfetchcmd("touch %s.done" % ud.fullshallow, d) + ret = True + finally: + bb.utils.remove(tempdir, recurse=True) + + return ret + def clone_shallow_local(self, ud, dest, d): """ Shallow fetch from ud.clonedir (${DL_DIR}/git2/ by default): @@ -557,12 +602,20 @@ class Git(FetchMethod): - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude= rev """ + progresshandler = GitProgressHandler(d) + repourl = self._get_repo_url(ud) bb.utils.mkdirhier(dest) init_cmd = "%s init -q" % ud.basecmd if ud.bareclone: init_cmd += " --bare" runfetchcmd(init_cmd, d, workdir=dest) - runfetchcmd("%s remote add origin %s" % (ud.basecmd, ud.clonedir), d, workdir=dest) + # Use repourl when creating a fast initial shallow clone + # Prefer already existing full bare clones if available + if not ud.shallow_skip_fast and not os.path.exists(ud.clonedir): + remote = shlex.quote(repourl) + else: + remote = ud.clonedir + runfetchcmd("%s remote add origin %s" % (ud.basecmd, remote), d, workdir=dest) # Check the histories which should be excluded shallow_exclude = '' @@ -600,10 +653,14 @@ class Git(FetchMethod): # The ud.clonedir is a local temporary dir, will be removed when # fetch is done, so we can do anything on it. adv_cmd = 'git branch -f advertise-%s %s' % (revision, revision) - runfetchcmd(adv_cmd, d, workdir=ud.clonedir) + if ud.shallow_skip_fast: + runfetchcmd(adv_cmd, d, workdir=ud.clonedir) runfetchcmd(fetch_cmd, d, workdir=dest) runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) + # Fetch Git LFS data for fast shallow clones + if not ud.shallow_skip_fast: + self.lfs_fetch(ud, d, dest, ud.revisions[ud.names[0]]) # Apply extra ref wildcards all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \ @@ -629,7 +686,6 @@ class Git(FetchMethod): runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) # The url is local ud.clonedir, set it to upstream one - repourl = self._get_repo_url(ud) runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=dest) def unpack(self, ud, destdir, d): From patchwork Fri Aug 8 08:49:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68224 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42E0DCA0EC0 for ; Fri, 8 Aug 2025 08:50:07 +0000 (UTC) Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by mx.groups.io with SMTP id smtpd.web11.17940.1754643003101057433 for ; Fri, 08 Aug 2025 01:50:03 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=lq0EXhoz; spf=pass (domain: gmail.com, ip: 209.85.214.170, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-240763b322fso20982745ad.0 for ; Fri, 08 Aug 2025 01:50:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643002; x=1755247802; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XBk/jbk5ST5LKcMR3zmKPcmEr5eDfHb92Kbmt/x5pGM=; b=lq0EXhozxFTkGrzOhT0+zHFomc/3P/5GtdCL7IzXF5eZ46L7l4yBrEZEcqz5/92Ue6 9MF7WtFqSdDZYbiH8DtTkSznwvQuthD1SZ8flFUiaBPVZGbxAlyzV+kPSqYB2tCBv5ar EByjbfGX6bXfhr66bRfWXh/tfcwEgGe25kZnZdsek8futjxJASNGCQnbsRMHsingaVVy dSel/MGl6/C/cYPPdcmc6aVLR6x5JAO1dJZ+8JlOL+5bcCsXhBZcB6OeS3HH6A5XYhwT GTg5toU0IbdKhzrwglyPuzjoFKlEUFDOZ6RmxQjrBc81MliubMcu4Iw+o21/bZjiAhFW XGDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643002; x=1755247802; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XBk/jbk5ST5LKcMR3zmKPcmEr5eDfHb92Kbmt/x5pGM=; b=b7bOwNADtHUO3EQJTTqObAyHTqN76qmp82aAbFT/r1G/Cmq0jiOLpoXl3sxOSo4iBQ zc86QhRtEfyuJu4qCZQHXvywq/QfXltRAjsFXCxxsP4vTwA8Ho5Kxg8sGUO3kgMi6eml j4jR/SBhGNa70FSymvI1GwgiUBs1qZXWOOGie0vNza1AX/hIRhLxZfNiWU9gsRkbKWqi cv0cZLNbIIVbiLPrBUXhWYyeLY175wLG6czBaLiWnolHlIXvKf4Snh3F9yBiZ9JIXgAW EWibUIhuwdzzr8ZfDxXsy0+TpwO61KRv6Cw8h5u1igmJSMEOQKJTntQrXX+aogS+s/M7 AUTw== X-Gm-Message-State: AOJu0YwojSYaPMKVVzVYWeUNr1z7gAajKdhQjRvfziB6HlBAzOuXHjuw JBCnF6TJHMMlPHEcGkd2HJ1r/8CzGJrEOCM+hCefvBoRYBxIp/BP2nS0NB/M5g== X-Gm-Gg: ASbGncuv89Khzd3q1dCcmhxvs6exfjPFQQt0EzFIMDZUgVgKPleYqMx04FPfDdRk5ld s9oGytIZ8HaXhRYQ5ukRw7w0W4BLS3U956bWWWtw9mB29SNuI3pbdlFhnrQkYyjhUMQXyfwawo0 QluSoJ+lMgRJUTx/OEJpn1YCnnjzqgELr8359bp2XRrDbVGxb2kt7vCUxdsNTA1oxoAdiAzZhuM 54VvqmsTjv4jaWtWzGyrpvB3sW3NQ3SCSjzrkJmH3rl5U2c0gnv1P61KvLowdEKPw71CxM3CZip y3LpPy9aL4GvJ4P293cWClAteEnqujPuEfuTOprUNSaK5R4gLKJ4/EeoXvalxablZDp1fF5onk0 a46AmBXXMucCX4DQeWVjH9UKFrdGdXj9i X-Google-Smtp-Source: AGHT+IHxAZSTOJ+h3+AnX+deUsnuXlHL9GrWvcmsk7Ij2PpnDW8mE7v4SmpXB/ZseagmS7tS1gtneQ== X-Received: by 2002:a17:902:ecc4:b0:240:6fc0:342c with SMTP id d9443c01a7336-242c1fdb15emr33262025ad.11.1754643001982; Fri, 08 Aug 2025 01:50:01 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.49.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:01 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Stefan Koch , Richard Purdie Subject: [poky][scarthgap][PATCH 09/23] bitbake: fetch2/gitsm: Unpack even when `ud.clonedir` is not available Date: Fri, 8 Aug 2025 14:19:17 +0530 Message-Id: <20250808084931.2156763-9-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221606 From: Stefan Koch (Bitbake rev: 13d76361ec37faecd84e7b81da22ada7d4e0ba90) Signed-off-by: Stefan Koch Signed-off-by: Richard Purdie (cherry picked from commit 22728acd78d65e993215a7f94cbf692d3111a4c4) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/gitsm.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py index bab1bf2580..1c58230eac 100644 --- a/bitbake/lib/bb/fetch2/gitsm.py +++ b/bitbake/lib/bb/fetch2/gitsm.py @@ -150,7 +150,10 @@ class GitSM(Git): def call_process_submodules(self, ud, d, extra_check, subfunc): # If we're using a shallow mirror tarball it needs to be # unpacked temporarily so that we can examine the .gitmodules file - if ud.shallow and os.path.exists(ud.fullshallow) and extra_check: + # Unpack even when ud.clonedir is not available, + # which may occur during a fast shallow clone + unpack = extra_check or not os.path.exists(ud.clonedir) + if ud.shallow and os.path.exists(ud.fullshallow) and unpack: tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR")) try: runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir) From patchwork Fri Aug 8 08:49:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D0E8CA0EC1 for ; Fri, 8 Aug 2025 08:50:07 +0000 (UTC) Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by mx.groups.io with SMTP id smtpd.web10.17686.1754643005805613109 for ; Fri, 08 Aug 2025 01:50:05 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=DhQLZyEo; spf=pass (domain: gmail.com, ip: 209.85.210.173, mailfrom: akash.hadke27@gmail.com) Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-76bf3dafaa5so2969814b3a.0 for ; Fri, 08 Aug 2025 01:50:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643005; x=1755247805; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3S4/nMDbpKsHyrer8rYv5NE5Ste4bP4b8delBGYI+o4=; b=DhQLZyEo0XDaBQE3kaDhQvn3O5/IOF5ZWTin6fGeGWfgWEMvzsns9JHmdojacACvsZ vM+jbzxCljhj44Y/hkLYyT+bu4ywFbv/6qcrJHG55uQfw55i/YOg8MNbERJwL7SVACFL 8OS4ZxgK1oS7WYjc0Pu5UoZCQDtfvAUQE25yoq6ibwYsPvErdodx0cj48U4HWFUSw07M vbjfQCKQRDu/89iaYUNjW5HdC3kp9huw8q3x6+3NulcG1JDvK0CQ34K7AWr95cHFqXo2 IhyFOz1habEj2ymdNqU5IavVGd88IP+zjQz67tiWCs7o50kC8NdJkePGqvnvCV/x7s9t PzAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643005; x=1755247805; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3S4/nMDbpKsHyrer8rYv5NE5Ste4bP4b8delBGYI+o4=; b=DaC4cTacoBDBBGHPdsud88ExsL7HkiByFAQyQ9sEV0QdIEg0OqThCtSbrZP9SYkJoP +VGEor9TEohD5vNKZmPZrWoMM/ZYWQjvTa7J0FQLDrkFesrzQVU7TLxMbEBMsePDW4EL nw1Lajcknn1eKW9ZQqGMrgF6O+R/4fuUXaVwThCIjJUsH4viBrx5IxVCp0JQuJhzsSJx IHMQNTOKnIqbytKQz7Rv0pS4Waqm2FOEwBbrKhtmDctnQRcYkDso1zaxFax/qfZkC/R4 E3iOFVkiV5uvkvCRPl2WsoTrBBD/LZjNNWG86VKpL83E79B4ORgOnyR2Lq3Z0rwbGlB4 08Cg== X-Gm-Message-State: AOJu0YyvPxTSUlU8zEZCHTmyT/v14Vr1DsSRIoJsb8jI4mB9tqseD0ox cxu9qhInQ7+++yod4ZTET5ytml2hqsCCG1sxhltWSD3WWhaVHiynqRty4C+yig== X-Gm-Gg: ASbGncvxwvFbQ1rbdh694DqSjkXZObk0pc/7gN0YxxJJ8Lk3cfrQLdEMhr4aXW64aY6 T0lYafTes+Cy70AD5v1qIXJKdVUjG7m2HCAeVb6Zk+nyEqq1DcIExw6kTcjwyBINFd/5rcqQzYs PF3Fn7iheKwluhrsbcJIPruWhM7tYpSnTFaAV/mVMdVixCuW/e/y6dIqMuVz8cTP4XI2ZgpJZ4d TAjdLBLl8g0aYpK5gtii0lcE4gi0ihGTrQ+rAlwsobIaGhGaRyPuhGQG/JXdX9TmcuQRybjBnYS GHIfZS5ECQdi79n4eCGae7lRZW4cwc7H69UqRjeNYbfEmZKkLqciw1J5nvz748DFsH0U/N+gy+u I6u/DArkT8sEDdJef/08s0UwWew8m9bUY X-Google-Smtp-Source: AGHT+IGi+eN0taqq0mPgmEXA+O36f1Pdr/HDRjBJTvCBgVwgx2dRuEANYNEUf9X8Ys4w0q3vT4yq6Q== X-Received: by 2002:a17:903:187:b0:240:8381:45b9 with SMTP id d9443c01a7336-242c2afb941mr37249075ad.8.1754643004605; Fri, 08 Aug 2025 01:50:04 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:04 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Stefan Koch , Richard Purdie Subject: [poky][scarthgap][PATCH 10/23] bitbake: tests/fetch: Adapt test cases for fast shallow fetches Date: Fri, 8 Aug 2025 14:19:18 +0530 Message-Id: <20250808084931.2156763-10-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:07 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221607 From: Stefan Koch - Address the absence of an initial full bare clone - Utilize the initial shallow clone - Modify existing test cases for this behavior - Remove incompatible test cases (Bitbake rev: 599fedacd7782dcb52825c22200f35344c102548) Signed-off-by: Stefan Koch Signed-off-by: Richard Purdie (cherry picked from commit db267f2c94ec86e4a7c046c49d3a2260760ccbfa) Signed-off-by: Akash Hadke --- bitbake/lib/bb/tests/fetch.py | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index eed92ad6ea..ea91d20d1f 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -1763,7 +1763,6 @@ class GitShallowTest(FetcherTest): def fetch_shallow(self, uri=None, disabled=False, keepclone=False): """Fetch a uri, generating a shallow tarball, then unpack using it""" fetcher, ud = self.fetch_and_unpack(uri) - assert os.path.exists(ud.clonedir), 'Git clone in DLDIR (%s) does not exist for uri %s' % (ud.clonedir, uri) # Confirm that the unpacked repo is unshallow if not disabled: @@ -1771,9 +1770,10 @@ class GitShallowTest(FetcherTest): # fetch and unpack, from the shallow tarball bb.utils.remove(self.gitdir, recurse=True) - bb.process.run('chmod u+w -R "%s"' % ud.clonedir) - bb.utils.remove(ud.clonedir, recurse=True) - bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True) + if os.path.exists(ud.clonedir): + bb.process.run('chmod u+w -R "%s"' % ud.clonedir) + bb.utils.remove(ud.clonedir, recurse=True) + bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True) # confirm that the unpacked repo is used when no git clone or git # mirror tarball is available @@ -1856,7 +1856,12 @@ class GitShallowTest(FetcherTest): self.add_empty_file('c') self.assertRevCount(3, cwd=self.srcdir) + # Clone without tarball + self.d.setVar('BB_GIT_SHALLOW', '0') + fetcher, ud = self.fetch() + # Clone and generate mirror tarball + self.d.setVar('BB_GIT_SHALLOW', '1') fetcher, ud = self.fetch() # Ensure we have a current mirror tarball, but an out of date clone @@ -1868,6 +1873,7 @@ class GitShallowTest(FetcherTest): fetcher, ud = self.fetch() fetcher.unpack(self.d.getVar('WORKDIR')) self.assertRevCount(1) + assert os.path.exists(os.path.join(self.d.getVar('WORKDIR'), 'git', 'c')) def test_shallow_single_branch_no_merge(self): self.add_empty_file('a') @@ -2065,11 +2071,12 @@ class GitShallowTest(FetcherTest): self.add_empty_file('b') # Fetch once to generate the shallow tarball + self.d.setVar('BB_GIT_SHALLOW', '0') fetcher, ud = self.fetch() - assert os.path.exists(os.path.join(self.dldir, ud.mirrortarballs[0])) # Fetch and unpack with both the clonedir and shallow tarball available bb.utils.remove(self.gitdir, recurse=True) + self.d.setVar('BB_GIT_SHALLOW', '1') fetcher, ud = self.fetch_and_unpack() # The unpacked tree should *not* be shallow @@ -2244,20 +2251,6 @@ class GitShallowTest(FetcherTest): self.assertIn("No up to date source found", context.exception.msg) self.assertIn("clone directory not available or not up to date", context.exception.msg) - @skipIfNoNetwork() - def test_that_unpack_does_work_when_using_git_shallow_tarball_but_tarball_is_not_available(self): - self.d.setVar('SRCREV', 'e5939ff608b95cdd4d0ab0e1935781ab9a276ac0') - self.d.setVar('BB_GIT_SHALLOW', '1') - self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') - fetcher = bb.fetch.Fetch(["git://git.yoctoproject.org/fstests;branch=master;protocol=https"], self.d) - fetcher.download() - - bb.utils.remove(self.dldir + "/*.tar.gz") - fetcher.unpack(self.unpackdir) - - dir = os.listdir(self.unpackdir + "/git/") - self.assertIn("fstests.doap", dir) - class GitLfsTest(FetcherTest): def skipIfNoGitLFS(): import shutil From patchwork Fri Aug 8 08:49:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 487D2C87FDA for ; Fri, 8 Aug 2025 08:50:17 +0000 (UTC) Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by mx.groups.io with SMTP id smtpd.web10.17687.1754643008647558516 for ; Fri, 08 Aug 2025 01:50:08 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=WzHTYnUh; spf=pass (domain: gmail.com, ip: 209.85.214.170, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-23c8f179e1bso26767545ad.1 for ; Fri, 08 Aug 2025 01:50:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643008; x=1755247808; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vpF1IxqAU9/PEGB+MoIlW6eZdhSPZKkKBw3mRaTrLvY=; b=WzHTYnUh0rLixeR8fkE1Jl+Dp9DGvfvxuKUutcxpmY2D8qROEiixZEm0oKPvO3dCq6 zunor/zsuqgE31EliPX5xgg08oLqaySdTbwc1MyDgGM2+lQB/EOgChPadw1NPW1zeLEi jE0Mv77v3meIRJDZp0Rc1KX/o2GkrB6YuJAahmllEytkDNeE7HAIgLV8EqcEsqIZWJjH UTspymMp1rgYj0qQlVPW7dox4BBn3boJClGJNQQmRPU7S7W0nSq+5/2FeYbsBaW+07wN 9qlzEaDSteTlO0//qgTYU/cFGqJhvl9uXgCifHRFGHkcXIhflTPrFEwKABly5uH0O2og u0kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643008; x=1755247808; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vpF1IxqAU9/PEGB+MoIlW6eZdhSPZKkKBw3mRaTrLvY=; b=cDMedlGFDqF7kw3gi64RuqKlfn0+XOCRCfvCuFRl4J0Njaf9M4ekXX18pXM9UORCe/ 0pH0XXHK54KcSvQjSi9DNMDONIowGlVTuUG8JghYuoCq1k2xxdQxls55NaQJmKnudclL 69+n+RPs4lpAt2DStf39oFdPTKTe52VUqABJcRnWDf74tg4Gq6JiBabg6JTOUMYX7rvl jmhE6y5qInAZIRpsFUOcpkd0Nu0+id1vALvD0z09HC0XoIFn+/+1xT7ip7x7aJb5y3TO W9lY2jGQR+ZR5mY+KJXU9N5grFxkinOivQzTd1WnnEvrdFf72x7DL3zYi9tA/2DgPCqZ n/5Q== X-Gm-Message-State: AOJu0YwSv+Le9cT9tNMPNQbcEL/bOuvQn3XAsydBIfFs6S5lyQRk5Ne6 1/LrTC9JDd0M5mjtS9u1+xUtXWhhrs0zRVFVNazmVAlcS9cduW2Zwj3G02+sjw== X-Gm-Gg: ASbGncsxFdQLxzCejYEf3ThJgFhpBvJtfS7VWU0iMh1+uZ3hDkBtjU9Smu9h6eOyow9 Zy+6bHn+FIqP/llj6VfeGYoExKm8bxVvGUnf7U1h7+bswuLhoeSi6I0qkpsWDopxU2H19UZnjdC +AN/uY9r1KbS8k9/9jv2fIr8lEp2vjxgHqmDO1+FdqLfy26fjhVTQ7Uq5OCJJxZ3s22e29c5V6R rp9b1f+9TgzRffPSHHDNU4T9Mb1SwNTf3xb86EU5j1tVGmtcS9sZxNMpOjuDDY0+lVk3x62OLHs CwqCmL+kx9yIe3lybaJqFE+GiKDwRciOd19jtlq+K9InqcrTC2ln4r5dwa88IPDy+aZAZePyzo9 zMewqTefrhfhmlpiIqz2nEoFtb7Q4WYYU X-Google-Smtp-Source: AGHT+IEjxaMh+rDfF3rObomgg0IwoxRusmYHfChjzb0L4Peh0JTKJYClpgkwPgmENw8PLGCIVf/G2w== X-Received: by 2002:a17:903:2ace:b0:234:b123:b4ff with SMTP id d9443c01a7336-242c2011167mr35311685ad.21.1754643007672; Fri, 08 Aug 2025 01:50:07 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:07 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Patrik Nordvall , Richard Purdie Subject: [poky][scarthgap][PATCH 11/23] bitbake: fetch2/git: Restore escape quoting for the git url when used Date: Fri, 8 Aug 2025 14:19:19 +0530 Message-Id: <20250808084931.2156763-11-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221608 From: Patrik Nordvall This fixes a bug where escapes in the url path would not be properly restored for the git commands in the git fetcher. For example, a space which is encoded as '%20' was not properly encoded before the clone command. e.g. SRC_URI="git://git.openembedded.org/bitbake%20example/bitbake;protocol=https" resulted in git clone 'https://git.openembedded.org/bitbake example/bitbake' instead of git clone 'https://git.openembedded.org/bitbake%20example/bitbake' (Bitbake rev: be48024253b93215bb110cd1d05925e789ec9680) Signed-off-by: Patrik Nordvall Signed-off-by: Richard Purdie (cherry picked from commit 8180865c22000adc5114a062dbe67f4b02ea06d5) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 3 ++- bitbake/lib/bb/tests/fetch.py | 3 +++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 9a15abaa79..a73fb79ac8 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -81,6 +81,7 @@ import shlex import shutil import subprocess import tempfile +import urllib import bb import bb.progress from contextlib import contextmanager @@ -888,7 +889,7 @@ class Git(FetchMethod): username = ud.user + '@' else: username = "" - return "%s://%s%s%s" % (ud.proto, username, ud.host, ud.path) + return "%s://%s%s%s" % (ud.proto, username, ud.host, urllib.parse.quote(ud.path)) def _revision_key(self, ud, d, name): """ diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index ea91d20d1f..2d0bd29799 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -2442,11 +2442,13 @@ class GitURLWithSpacesTest(FetcherTest): test_git_urls = { "git://tfs-example.org:22/tfs/example%20path/example.git;branch=master" : { 'url': 'git://tfs-example.org:22/tfs/example%20path/example.git;branch=master', + 'repo_url': 'git://tfs-example.org:22/tfs/example%20path/example.git', 'gitsrcname': 'tfs-example.org.22.tfs.example_path.example.git', 'path': '/tfs/example path/example.git' }, "git://tfs-example.org:22/tfs/example%20path/example%20repo.git;branch=master" : { 'url': 'git://tfs-example.org:22/tfs/example%20path/example%20repo.git;branch=master', + 'repo_url': 'git://tfs-example.org:22/tfs/example%20path/example%20repo.git', 'gitsrcname': 'tfs-example.org.22.tfs.example_path.example_repo.git', 'path': '/tfs/example path/example repo.git' } @@ -2469,6 +2471,7 @@ class GitURLWithSpacesTest(FetcherTest): self.assertEqual(ud.lockfile, os.path.join(self.dldir, "git2", ref['gitsrcname'] + '.lock')) self.assertEqual(ud.clonedir, os.path.join(self.dldir, "git2", ref['gitsrcname'])) self.assertEqual(ud.fullmirror, os.path.join(self.dldir, "git2_" + ref['gitsrcname'] + '.tar.gz')) + self.assertEqual(ud.method._get_repo_url(ud), ref['repo_url']) class CrateTest(FetcherTest): @skipIfNoNetwork() From patchwork Fri Aug 8 08:49:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68226 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48794C87FD2 for ; Fri, 8 Aug 2025 08:50:17 +0000 (UTC) Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by mx.groups.io with SMTP id smtpd.web11.17943.1754643011353013658 for ; Fri, 08 Aug 2025 01:50:11 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=OkUpfEDN; spf=pass (domain: gmail.com, ip: 209.85.215.178, mailfrom: akash.hadke27@gmail.com) Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-b26f7d2c1f1so2017530a12.0 for ; Fri, 08 Aug 2025 01:50:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643010; x=1755247810; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gYO7nTzdW7nDfJdfr7C9/s1jRijsiBXL8Ore6KODz9A=; b=OkUpfEDNEu+0EIUPUhnxCLtSsm3SYWxaJuQdClu1Lm873QCpAe6muK3bweag904KuX kbtnH8+QpZZXBm96D2uqDluQZI5K+y1GTg1JwEucgU61WBHiutCTtdHR4rGWzjk1Kz2H ZxWMpQtP2nCMt4WLtTk1XHqtq8uCH/czjEzpqMPDpLVUi3JXdnKAeRx8JcEkb62W7mDE tx4Tkf+nPnioY676kSoP18IgYekOKeztzf+UgArI3sJhUaDyUo7ABTbHTwyYVCreQt5p YXUNhLtzC+W8k92Xx2lKKGOGyzZJyRf1YLc0edzg8PnA3ueIkRPTE/mcOjfP+R0MMNf+ zMHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643010; x=1755247810; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gYO7nTzdW7nDfJdfr7C9/s1jRijsiBXL8Ore6KODz9A=; b=DEBUN2ugb9tA+mq74y/Oc5cGam15IoL7TRVh5xmbnYyCaCebrF2iHwcvvuUFanbS3t ONsZFGe4+mLGsTm8qfj9Gef6JKsS/N3QLvskdsEtFRY7tP/VGkGg4lvL3+8pBksu8+CS 8g6SO/wiRFq+4aMZCSpmvWESq6lAZMZS5m+/SULD3iLVpPNIMfTD7xpMVYYlqIvGmh+z /i4As9Sdy/gl/CrVOokDZaclDBLmnqdF3ZYPzGaselcv4F3hE3TxXO5e9mCddH3uimpa nmSNQaUtcBe7mIdnPhoX2axQZDr3TKDPZDqJppW1dOMxMn/aL9yZtjCv5zfpc84AvePN HklQ== X-Gm-Message-State: AOJu0YzFT2ZMQpc6Nmty4AXjD9merMmR9q4lc9alIyDHtyMT9JkU46oW 4djPwQHoB/wrAg2tZ3mMEnOPtZwCEtjR/WTbaUQffcTyOs/WLDLdgOtAak600g== X-Gm-Gg: ASbGncumYwsqBGku+oiPsgFS6jp9f7xmuuPG4SnFRY9z2ph5P70moaPDbddhaW5jbOn 0Q5CaAM7sLeUV6eqcZhk7lXyhxNY+f7HD/yQFkq5qBvGm/n5m5okLkX2G/VKPGlrXuyos82wJSb eSdc+FZXcCQzq+Hr43hw1RPFL3RbhW3zFIM/PwBOdFtkH/sPzQCDbLJEsbaU5Bu13YaZ7dNBwif U96jz6+XPZJFYTQkTRbhFIax1PBw4R7EGtQQ672KK4Ae8qi7YjVDinUPSA/rQ+x+DOezRRDqATl TiY0SF5SZZ/jsikSsQj4Hzp8ecCAtHQCJrrNTKRTAf1RoGUdu5W21a9AnPvZ7eTx8uLLefv5eIB Waj77b6oWN7XQ0svjpnST74wOjMRc/MhEfegmFw== X-Google-Smtp-Source: AGHT+IHqU067G3gWaCeYyNyH5SfRmAOhRdOEMUmJW6v2fpwDn9a/585DAazk1c1JHaCiuA4MQcGJPw== X-Received: by 2002:a17:902:ec89:b0:235:15f3:ef16 with SMTP id d9443c01a7336-242c20a1558mr31658315ad.13.1754643010396; Fri, 08 Aug 2025 01:50:10 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:10 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Peter Marko , Mathieu Dubois-Briand , Richard Purdie Subject: [poky][scarthgap][PATCH 12/23] bitbake: fetch/git: always fetch lfs when creating shallow tarball Date: Fri, 8 Aug 2025 14:19:20 +0530 Message-Id: <20250808084931.2156763-12-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221609 From: Peter Marko Before fast shallow tarballs the shallow tarball with lfs worked only if the original .../downloads/git2/... directory existed. Once it was not there, do_unpack on shallow tarball without lfs files failed due to disabled network. It was fixed for fast shallow tarballs and this commit fixes the other scenarios, too. This can happen when such shallow tarball is put to mirror or if someone does cleanup to reclaim disk space. (Bitbake rev: 4e6fb31a88448cdacf4e9f84cf0a8a8035f84d63) Signed-off-by: Peter Marko Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie (cherry picked from commit d1dae30b7932c2e6c6829a04b19a7f9e440cdd5c) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index a73fb79ac8..9e0950006e 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -659,9 +659,8 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=dest) runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) - # Fetch Git LFS data for fast shallow clones - if not ud.shallow_skip_fast: - self.lfs_fetch(ud, d, dest, ud.revisions[ud.names[0]]) + # Fetch Git LFS data + self.lfs_fetch(ud, d, dest, ud.revisions[ud.names[0]]) # Apply extra ref wildcards all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \ From patchwork Fri Aug 8 08:49:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68228 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 552F8CA0EC0 for ; Fri, 8 Aug 2025 08:50:17 +0000 (UTC) Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by mx.groups.io with SMTP id smtpd.web10.17689.1754643014055437676 for ; Fri, 08 Aug 2025 01:50:14 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=I1JKO1nN; spf=pass (domain: gmail.com, ip: 209.85.214.171, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-23fc5aedaf0so13100635ad.2 for ; Fri, 08 Aug 2025 01:50:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643013; x=1755247813; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JiTrGtb8GFkMb6sdBHFmR4RyTh6P85DiX0RqQYPUKbY=; b=I1JKO1nNtF5y2pQgdbflBgs2EcKMVQ09H8KgIpLcd8XKaLXHRYocD0S6AJW8W+8qGk JKND97bSKhjzKJU8fep+wrzz+4NTecUQEF6K1+rWfdqxct50Sg3uk/NGaLE9FiiEiY4c UAxW9yaJNPcnOPmWGRSP9xJxSJiOgezDq5+8P0s7sRfIzQl+SEC9FyRuJdugK3YIEYw4 nwMHjhp4s0jryglzqKzbGNf9tcAym+t94tlTohUvVwi6V0XKhQUaMMT7tsJXjpcmt3Cx GJtAmzZbuvBUK8pp2tE5VLe0+/th04x1q/3irIZjE/YheBarCyN2TZ0grWm0u4OyvlBh pyVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643013; x=1755247813; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JiTrGtb8GFkMb6sdBHFmR4RyTh6P85DiX0RqQYPUKbY=; b=JlxJrx1Gtoc5HBdZ9SEE98w8N8lYGepRHlbHd9UICZ2IfdB3H7CvFsMF6N8mB5Fneu Nw9gfrWNPxG4yiNTn1VhhpI3QRxdPPV21fZdYf6zYz0tQYDU8Xz2Un0925ZX8m+JMGro MD5hbQbw47fqu5V4E3ZA5qsAkaXxXQ1r7eLJYdgeIRo/B62jXW8U3elyjjKaF3Zz5yJh r0IRl3iA2lx47vp1y3176C9J131FfOEete/GbMCES8WkWyHcyPRKqMMrazwmgaMMMe7d 4yZPsBpl9cH7tv4fhyVYP++wICrnwqybh0O9K82HSEHkrTdmEyr23+l9mVcQtr4sJ/vF qFHg== X-Gm-Message-State: AOJu0YwxL41z0TExqWikh/geXKN5lrpmUgtw22gYHW3Ic5D4lyazQGms C7of4Cna1NyPO3Ad3yxQlmlxAgo8EkKyMZSgv2StUERKfhIEycPXF+r8gFz0Eg== X-Gm-Gg: ASbGncvzaRmEOtYz4LJau/smx+ccwCDRDkS5YF09r7Q6uWWHg76i06DS4ML2oIRCVCk YgJnuX3RzKZzes9935oNyilNcVe2wzLbuk5NaF003Zqy1JQ2qZ5T7Z0g8os53CX9hgU382mudnW PmF9/tBlNM7k+SrCwm/mjPZwLRr2GC2qLak2U8B4ewrZYRdK5OW+lyyolnzLtU0lu3wB88H6Vsq XPm8JHuXju0qSUGi9ObZJqY7MXwPska9Ios2eFWpBS4OnEVfL+UX8pp5E6MIkzH9QWNlYoWTFbi p7oWNo5PMln+Rkwd9hHsd61nqQROM2/AZqROvkLMuixLvU2HjMsjQlFPcwnrub+RGGC1r4qXo36 vJfNGvFO8KcCsQ09j7LnlWi6ufrFpbrClelmbaQ== X-Google-Smtp-Source: AGHT+IFaFBifrUo8T3Uv3v+ONvco1IGwhGuO+qAy+qRHuc0Cl/dUSF67LUijiMhjnHJHHYEt95N9tw== X-Received: by 2002:a17:903:1b30:b0:240:79ef:ae80 with SMTP id d9443c01a7336-242c1ff40d5mr33404445ad.8.1754643013090; Fri, 08 Aug 2025 01:50:13 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:12 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Philip Lorenz , Richard Purdie Subject: [poky][scarthgap][PATCH 13/23] bitbake: tests/fetch: Move commonly used imports to top Date: Fri, 8 Aug 2025 14:19:21 +0530 Message-Id: <20250808084931.2156763-13-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221610 From: Philip Lorenz Avoid multiple import statements for anything that is used more than once. Additionally, drop no longer used imports. (Bitbake rev: 7c74310440f4d6ec47cf5bacf597e18308b3bb20) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie (cherry picked from commit 62c4daafcef2c920607868a0629b9bda59b61951) Signed-off-by: Akash Hadke --- bitbake/lib/bb/tests/fetch.py | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-) diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index 2d0bd29799..14480d2d65 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -7,7 +7,9 @@ # import contextlib +import shutil import unittest +import urllib.parse import hashlib import tempfile import collections @@ -1250,7 +1252,6 @@ class FetcherNetworkTest(FetcherTest): class SVNTest(FetcherTest): def skipIfNoSvn(): - import shutil if not shutil.which("svn"): return unittest.skip("svn not installed, tests being skipped") @@ -1373,7 +1374,6 @@ class TrustedNetworksTest(FetcherTest): self.assertFalse(bb.fetch.trusted_network(self.d, url)) class URLHandle(unittest.TestCase): - datatable = { "http://www.google.com/index.html" : ('http', 'www.google.com', '/index.html', '', '', {}), "cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg" : ('cvs', 'cvs.handhelds.org', '/cvs', 'anoncvs', '', {'module': 'familiar/dist/ipkg'}), @@ -2253,7 +2253,6 @@ class GitShallowTest(FetcherTest): class GitLfsTest(FetcherTest): def skipIfNoGitLFS(): - import shutil if not shutil.which('git-lfs'): return unittest.skip('git-lfs not installed') return lambda f: f @@ -2373,8 +2372,6 @@ class GitLfsTest(FetcherTest): @skipIfNoGitLFS() def test_lfs_enabled(self): - import shutil - uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir self.d.setVar('SRC_URI', uri) @@ -2385,8 +2382,6 @@ class GitLfsTest(FetcherTest): @skipIfNoGitLFS() def test_lfs_disabled(self): - import shutil - uri = 'git://%s;protocol=file;lfs=0;branch=master' % self.srcdir self.d.setVar('SRC_URI', uri) @@ -2396,8 +2391,6 @@ class GitLfsTest(FetcherTest): fetcher.unpack(self.d.getVar('WORKDIR')) def test_lfs_enabled_not_installed(self): - import shutil - uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir self.d.setVar('SRC_URI', uri) @@ -2418,8 +2411,6 @@ class GitLfsTest(FetcherTest): ud.method._find_git_lfs = old_find_git_lfs def test_lfs_disabled_not_installed(self): - import shutil - uri = 'git://%s;protocol=file;lfs=0;branch=master' % self.srcdir self.d.setVar('SRC_URI', uri) @@ -2593,7 +2584,6 @@ class CrateTest(FetcherTest): class NPMTest(FetcherTest): def skipIfNoNpm(): - import shutil if not shutil.which('npm'): return unittest.skip('npm not installed') return lambda f: f @@ -3272,7 +3262,6 @@ class FetchPremirroronlyNetworkTest(FetcherTest): self.d.setVar("PREMIRRORS", self.recipe_url + " " + "file://{}".format(self.mirrordir) + " \n") def make_git_repo(self): - import shutil self.mirrorname = "git2_git.yoctoproject.org.fstests.tar.gz" os.makedirs(self.clonedir) self.git("clone --bare --shallow-since=\"01.01.2013\" {}".format(self.recipe_url), self.clonedir) @@ -3301,7 +3290,6 @@ class FetchPremirroronlyMercurialTest(FetcherTest): the test covers also basic hg:// clone (see fetch_and_create_tarball """ def skipIfNoHg(): - import shutil if not shutil.which('hg'): return unittest.skip('Mercurial not installed') return lambda f: f @@ -3357,7 +3345,6 @@ class FetchPremirroronlyBrokenTarball(FetcherTest): targz.write("This is not tar.gz file!") def test_mirror_broken_download(self): - import sys self.d.setVar("SRCREV", "0"*40) fetcher = bb.fetch.Fetch([self.recipe_url], self.d) with self.assertRaises(bb.fetch2.FetchError), self.assertLogs() as logs: From patchwork Fri Aug 8 08:49:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5532ECA0EC1 for ; Fri, 8 Aug 2025 08:50:17 +0000 (UTC) Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by mx.groups.io with SMTP id smtpd.web11.17945.1754643016860276486 for ; Fri, 08 Aug 2025 01:50:16 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=BnRaO8nQ; spf=pass (domain: gmail.com, ip: 209.85.215.173, mailfrom: akash.hadke27@gmail.com) Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-b3507b63c6fso1908326a12.2 for ; Fri, 08 Aug 2025 01:50:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643016; x=1755247816; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=icROQGy6UiGt5iJvHhaVy3ZR0M7Vr2Q4r0JodrrkCIA=; b=BnRaO8nQQlSuIqAPVHWxzt8Z4rFV8hkoxXoe5tJBevSv0XrOdW0An1Xx8HnodvnT9D tubiTlpg0nEvwx25dD7knd1JJhrN1pv3dSvMu3iBzaO0W2DCDHvCsRg905bqpOwWdaPK rBBmasuKKjubYkEX1P0ev0BDAwmebRJ/TAJ3ElR9OqTe5sYGcnZsj18d/f2/R7ljCxNZ 6q68lZHHM3NjCGSsaKCEqUXU1ny+Eg3QFCP8zyrrLCBzH7Y9qmntCXb7yF1foWIaJ1K3 luwM8fbJWv1mJOpuAp02Gd7ptkeylFng5znJxpzudQEENbkkXAJSvUENWoS4PbwYb7zb NvBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643016; x=1755247816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=icROQGy6UiGt5iJvHhaVy3ZR0M7Vr2Q4r0JodrrkCIA=; b=vqnBwaDM2navbgkqstwTU3cfIv1Ih9r4mNHNIyodBepMCHWLO0reoNVxzrh9hCF1B9 bX5rN5PkvLGxo1FjE0K4RVN/k6QBalbMI074I5AeDIzaO5AHOwENaoklo2CKS4cBvrjT NdL39s84y5Us8HBK233gFssVL6gX1iLdDXBCogdc3ICAV1H/s1eyKBKpvNKzzET1MJjW 5GiebSmIJ6MAi1XR8QQsf+6nafurDAzonYRKs2EQE2E/YOCvBG/I0bo0lTGqlyAy+MMP CjTu4BxBu3cUpyReKvIoA/FP06pPL8Oj7kpyEziuQeHchlYaW/MKSoltVqnyzVYKm29w FzLg== X-Gm-Message-State: AOJu0Yy65ie+VH0BNpGJjFD7SlcxVWVzlrRgVieJYG177XqjWrxDGQ2n CYwQ7KGU1sluQBHKGmlJjvN2yxHcka6kv+LwAi+QL6S+Ei1fjsT4SDlp6ZxF9A== X-Gm-Gg: ASbGncvJ/dDmzLwplbn6dfq34zQknRaJfoKJjG7DwectvLj4dvE459o9VJYBqr/ZKzI gkEIuZdkq8Zw+rAZSd0adNfUY/Hlhw2g1HIuPtldcBLLOzQWkYm1R9AT5iM65l1VImSv9P6ha3s 53nS8M/ByclPaYfisLm52viaASjMsFFuxIPzoFy/q+j7WVCYH/mrekAwQKshnElomVI5UUu6/rL gJkwAno+ETjtuOQ6sNF5BjK37jnXuEhR2/70Eqla6tyuO1yrGWxUGtsSjlD79EJ9T4EhFU0bXkN bAjfzRnYgHRL//iHQuhfdXDAp+33S9/FmrYQ7CzhKqASY98N6vymM6qiKWGHBUvJbSdGjh4m9DC sMs/n9vICROA7C8Dqo9194XOKSihWg2wUaISTYw== X-Google-Smtp-Source: AGHT+IE1scSUExoFf1swbMHNbe5cGlbY2w3JXqPrb8x5KwsiChhu24gW/lEiwNyvip1yDFmD7o9ctw== X-Received: by 2002:a17:902:e94e:b0:240:79d5:8dc7 with SMTP id d9443c01a7336-242c22500b0mr36863255ad.46.1754643015854; Fri, 08 Aug 2025 01:50:15 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:15 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Philip Lorenz , Richard Purdie Subject: [poky][scarthgap][PATCH 14/23] bitbake: fetch2: Check for git-lfs existence before using it Date: Fri, 8 Aug 2025 14:19:22 +0530 Message-Id: <20250808084931.2156763-14-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:17 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221611 From: Philip Lorenz So far, existence of `git-lfs` was only checked during unpacking. As the binary is also used in earlier steps also check for its existence there. Additionally, factor out the LFS existence check into a dedicated function and call it wherever git-lfs is used for the first time. (Bitbake rev: 5818367db9b261b7e07c347d38044e6cba8f9727) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie (cherry picked from commit 7642477dc8550eeff75d34bd3bbdf574e2f5687c) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 26 +++++++++---- bitbake/lib/bb/tests/fetch.py | 73 ++++++++++++++++++++++++----------- 2 files changed, 68 insertions(+), 31 deletions(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 9e0950006e..10ee248473 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -346,6 +346,9 @@ class Git(FetchMethod): return False def lfs_need_update(self, ud, d): + if not self._need_lfs(ud): + return False + if self.clonedir_need_update(ud, d): return True @@ -531,7 +534,9 @@ class Git(FetchMethod): def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None): """Helper method for fetching Git LFS data""" try: - if self._need_lfs(ud) and self._contains_lfs(ud, d, clonedir) and self._find_git_lfs(d) and len(revision): + if self._need_lfs(ud) and self._contains_lfs(ud, d, clonedir) and len(revision): + self._ensure_git_lfs(d, ud) + # Using worktree with the revision because .lfsconfig may exists worktree_add_cmd = "%s worktree add wt %s" % (ud.basecmd, revision) runfetchcmd(worktree_add_cmd, d, log=progresshandler, workdir=clonedir) @@ -750,11 +755,11 @@ class Git(FetchMethod): runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=destdir) if self._contains_lfs(ud, d, destdir): - if need_lfs and not self._find_git_lfs(d): - raise bb.fetch2.FetchError("Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 to ignore it)" % (repourl)) - elif not need_lfs: + if not need_lfs: bb.note("Repository %s has LFS content but it is not being fetched" % (repourl)) else: + self._ensure_git_lfs(d, ud) + runfetchcmd("%s lfs install --local" % ud.basecmd, d, workdir=destdir) if not ud.nocheckout: @@ -817,9 +822,11 @@ class Git(FetchMethod): Verifies whether the LFS objects for requested revisions have already been downloaded """ # Bail out early if this repository doesn't use LFS - if not self._need_lfs(ud) or not self._contains_lfs(ud, d, wd): + if not self._contains_lfs(ud, d, wd): return True + self._ensure_git_lfs(d, ud) + # The Git LFS specification specifies ([1]) the LFS folder layout so it should be safe to check for file # existence. # [1] https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md#intercepting-git @@ -869,11 +876,14 @@ class Git(FetchMethod): pass return False - def _find_git_lfs(self, d): + def _ensure_git_lfs(self, d, ud): """ - Return True if git-lfs can be found, False otherwise. + Ensures that git-lfs is available, raising a FetchError if it isn't. """ - return shutil.which("git-lfs", path=d.getVar('PATH')) is not None + if shutil.which("git-lfs", path=d.getVar('PATH')) is None: + raise bb.fetch2.FetchError( + "Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 " + "to ignore it)" % self._get_repo_url(ud)) def _get_repo_url(self, ud): """ diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index 14480d2d65..5e76988570 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -9,6 +9,7 @@ import contextlib import shutil import unittest +import unittest.mock import urllib.parse import hashlib import tempfile @@ -2276,12 +2277,18 @@ class GitLfsTest(FetcherTest): self.git_init(cwd=self.srcdir) self.commit_file('.gitattributes', '*.mp3 filter=lfs -text') - def commit_file(self, filename, content): - with open(os.path.join(self.srcdir, filename), "w") as f: + def commit(self, *, cwd=None): + cwd = cwd or self.srcdir + self.git(["commit", "-m", "Change"], cwd=cwd) + return self.git(["rev-parse", "HEAD"], cwd=cwd).strip() + + def commit_file(self, filename, content, *, cwd=None): + cwd = cwd or self.srcdir + + with open(os.path.join(cwd, filename), "w") as f: f.write(content) - self.git(["add", filename], cwd=self.srcdir) - self.git(["commit", "-m", "Change"], cwd=self.srcdir) - return self.git(["rev-parse", "HEAD"], cwd=self.srcdir).strip() + self.git(["add", filename], cwd=cwd) + return self.commit(cwd=cwd) def fetch(self, uri=None, download=True): uris = self.d.getVar('SRC_URI').split() @@ -2390,6 +2397,21 @@ class GitLfsTest(FetcherTest): fetcher, ud = self.fetch() fetcher.unpack(self.d.getVar('WORKDIR')) + @skipIfNoGitLFS() + def test_lfs_enabled_not_installed_during_unpack(self): + uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir + self.d.setVar('SRC_URI', uri) + + # Careful: suppress initial attempt at downloading + fetcher, ud = self.fetch(uri=None, download=False) + + fetcher.download() + # If git-lfs cannot be found, the unpack should throw an error + with self.assertRaises(bb.fetch2.FetchError): + with unittest.mock.patch("shutil.which", return_value=None): + shutil.rmtree(self.gitdir, ignore_errors=True) + fetcher.unpack(self.d.getVar('WORKDIR')) + def test_lfs_enabled_not_installed(self): uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir self.d.setVar('SRC_URI', uri) @@ -2397,18 +2419,10 @@ class GitLfsTest(FetcherTest): # Careful: suppress initial attempt at downloading fetcher, ud = self.fetch(uri=None, download=False) - # Artificially assert that git-lfs is not installed, so - # we can verify a failure to unpack in it's absence. - old_find_git_lfs = ud.method._find_git_lfs - try: - # If git-lfs cannot be found, the unpack should throw an error + # If git-lfs cannot be found, the download should throw an error + with unittest.mock.patch("shutil.which", return_value=None): with self.assertRaises(bb.fetch2.FetchError): fetcher.download() - ud.method._find_git_lfs = lambda d: False - shutil.rmtree(self.gitdir, ignore_errors=True) - fetcher.unpack(self.d.getVar('WORKDIR')) - finally: - ud.method._find_git_lfs = old_find_git_lfs def test_lfs_disabled_not_installed(self): uri = 'git://%s;protocol=file;lfs=0;branch=master' % self.srcdir @@ -2417,17 +2431,30 @@ class GitLfsTest(FetcherTest): # Careful: suppress initial attempt at downloading fetcher, ud = self.fetch(uri=None, download=False) - # Artificially assert that git-lfs is not installed, so - # we can verify a failure to unpack in it's absence. - old_find_git_lfs = ud.method._find_git_lfs - try: - # Even if git-lfs cannot be found, the unpack should be successful + # Even if git-lfs cannot be found, the download / unpack should be successful + with unittest.mock.patch("shutil.which", return_value=None): + fetcher.download() + shutil.rmtree(self.gitdir, ignore_errors=True) + fetcher.unpack(self.d.getVar('WORKDIR')) + + def test_lfs_enabled_not_installed_but_not_needed(self): + srcdir = os.path.join(self.tempdir, "emptygit") + bb.utils.mkdirhier(srcdir) + self.git_init(srcdir) + self.commit_file("test", "test content", cwd=srcdir) + + uri = 'git://%s;protocol=file;lfs=1;branch=master' % srcdir + self.d.setVar('SRC_URI', uri) + + # Careful: suppress initial attempt at downloading + fetcher, ud = self.fetch(uri=None, download=False) + + # It shouldnt't matter that git-lfs cannot be found as the repository configuration does not + # specify any LFS filters. + with unittest.mock.patch("shutil.which", return_value=None): fetcher.download() - ud.method._find_git_lfs = lambda d: False shutil.rmtree(self.gitdir, ignore_errors=True) fetcher.unpack(self.d.getVar('WORKDIR')) - finally: - ud.method._find_git_lfs = old_find_git_lfs class GitURLWithSpacesTest(FetcherTest): test_git_urls = { From patchwork Fri Aug 8 08:49:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C0F8C87FD3 for ; Fri, 8 Aug 2025 08:50:27 +0000 (UTC) Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by mx.groups.io with SMTP id smtpd.web10.17691.1754643019522755395 for ; Fri, 08 Aug 2025 01:50:19 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=fNUKmyCc; spf=pass (domain: gmail.com, ip: 209.85.210.170, mailfrom: akash.hadke27@gmail.com) Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-76bfd457607so2095471b3a.0 for ; Fri, 08 Aug 2025 01:50:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643018; x=1755247818; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WvzK5n3PwGDDQmqcYH0JdNvOVPe/CWZZMAURAxGoslo=; b=fNUKmyCcbDpK/YsZJtUG8Dye/hILU6ZW2+J3d7HW3SjYdfNNC3/3TpqcgTC9hRwK0N 6OYcodQuSs+ve/vFFIaXxSW4EVfyzAikNiuYlkKMdtrJHbaJyOPfcoBlbOsmkIyxisYU ld2XvvE6TdQMgcqChrJ5PffXcA46Jy9Iqv0WAAlUOsb2o2lLOSF3L+pzLSJfsRyFb1zN V1wS13QH4aqskJKyen8uI0Uk04L4M2/bGcFZ+lw7u2B/9xFLDG63+3x4KOvWEJK4UuQI OA7X0oRRSl76Bn4XjB15g3f5eUDKNyblNox2pvIGjJrw2u6UBn9OrwB3kuZKTXaHPFKZ +XVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643018; x=1755247818; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WvzK5n3PwGDDQmqcYH0JdNvOVPe/CWZZMAURAxGoslo=; b=gS1XQ1n8hR52zSLaPQPGz9GVRn0/J8TSvYo3iP2DzhVRUfvHtcFlbnTAONprdbEV2W FZ4dNzar5IM3UUYQt5iCg3WkzmcOCh06eOjLZh0lc0RcNBicC43Is1iB1EijSLUGAS68 ztALChe+R3XtaYnwFUIbmrDUclWngNG6EIc1/ykp6++8oC9wCyS8HjPkMwIqKqhE9Zil PqILwsdMmGPlROKCvC5/lIpG4HO36LJlwJ3HNT6Cy/F/5xhkT0cKufYH3l/C/iCDPdxx LbErwKCiMEZ9XK9Gp7F3tW6Aab5tdbBHECpOxenVspFcLvdEN5X6kG/qjmBK+6aZQ3Jh Un+Q== X-Gm-Message-State: AOJu0Yz5A/hHSGrj2mqNfn1EmVcayIDcHKe+yDj5diZAPLOm+ZSYOc7o 2WVhWKiwBrTnaAY1gBp/nljIyQqO/87lAHSm/BYubgccHSC6LlW/9PKdEP+huQ== X-Gm-Gg: ASbGncubqCQVEAXvdC+JsMCco3yCd6bcyT7NBhrNTgz35JcZ6/qIqgflyvkVku2G5ro Bblj5wBguwnRuVtzxd2UNm+TyRg7TcqHJrApq1b5rFXS1+I6Yt9EE3bu9+M09AJJ+y6uxkUjpmY f3eekV0kkXLaYCrGFCRdjqFu4o1PdVahvqp98ycYQo2NWghDSi+/pRS43bUdXyPXiu7/9lpYenz itKuYWTygiMX6obSn6HvdFlfmcZp1KR7EOMGhrtFE5EoM4UNqjAoJ3k6nY6bB83/lghFyV6Ay8L Q3yoH4OkB20m7zznmk98mM5rYS9DBaIRMZfRcZPKqQAXt9j7QkfwPvKfTV2SUJGjmGsuk56sRzy SJBmX1zWRPKlHmGSZajJkhJjXhxsS4JsA X-Google-Smtp-Source: AGHT+IFy1UgIub0ppfoE3N6cLUUHDvJ5ZfNvyBfJUmQBdWqfHhPUVDplLEUVOGL5x+lmb9JEqzKnGQ== X-Received: by 2002:a17:902:ebc7:b0:242:4b7e:fad1 with SMTP id d9443c01a7336-242c2206d7cmr24535685ad.28.1754643018573; Fri, 08 Aug 2025 01:50:18 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:18 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Philip Lorenz , Richard Purdie Subject: [poky][scarthgap][PATCH 15/23] bitbake: fetch2: Simplify git LFS detection Date: Fri, 8 Aug 2025 14:19:23 +0530 Message-Id: <20250808084931.2156763-15-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221612 From: Philip Lorenz Its unclear why this function does not operate on the desired source revision to begin with (which really should be the decider on whether a particular source revision uses LFS or not). Simplify the decision logic by always checking the `.gitattributes` file of the target revision. (Bitbake rev: b3faa0ce5c0a6945f26b5b303a7f38c00d132397) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie (cherry picked from commit 74945e467895d6dd444f7ccf8d2a41eac63604ae) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 10ee248473..e211f7230e 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -855,18 +855,8 @@ class Git(FetchMethod): """ Check if the repository has 'lfs' (large file) content """ - - if ud.nobranch: - # If no branch is specified, use the current git commit - refname = self._build_revision(ud, d, ud.names[0]) - elif wd == ud.clonedir: - # The bare clonedir doesn't use the remote names; it has the branch immediately. - refname = ud.branches[ud.names[0]] - else: - refname = "origin/%s" % ud.branches[ud.names[0]] - cmd = "%s grep lfs %s:.gitattributes | wc -l" % ( - ud.basecmd, refname) + ud.basecmd, ud.revision) try: output = runfetchcmd(cmd, d, quiet=True, workdir=wd) From patchwork Fri Aug 8 08:49:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C134C87FDA for ; Fri, 8 Aug 2025 08:50:27 +0000 (UTC) Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by mx.groups.io with SMTP id smtpd.web11.17947.1754643022199600425 for ; Fri, 08 Aug 2025 01:50:22 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=VLJ/A0mt; spf=pass (domain: gmail.com, ip: 209.85.214.181, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2401b855980so13627935ad.1 for ; Fri, 08 Aug 2025 01:50:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643021; x=1755247821; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PFEDeMyDY8E4dQQzQ1pHRBCuQeiD2XocXYO9YNyQhuI=; b=VLJ/A0mtUyNmdGSld8lT0YOHVONM5BeaXw/1S8K1aumyk8XCzq+EBYdzOhbI3TQACx azefHv1TWBX5misrXnUXN+9Q1knO0xtwyAjqMyTIlJJeY8PhIgx0Utue5vH/CCyBhFqG YqfGxmQfcuHEr0Suil2i1Q1AHx4KnbzgKSBEit0jC9zevp0uKaVsSZux3ueKkMiLsx2V c8zVip8KTmzWjeS67GenX9rBcrZ3sUECNIyb4bMQTrgUlp8MfyAMG8/39HkMzhyLgWzM 5khOEKNiB4CArq3ARbg1wa7pb46sik6DgDh9ha9X5t54h+6aAoCpy/6IcGFw04vwxgWd nj4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643021; x=1755247821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PFEDeMyDY8E4dQQzQ1pHRBCuQeiD2XocXYO9YNyQhuI=; b=IQSt+VOynrrPdz8HlWPu4UHncSfEbKpNNf4z/2kRvLs3FAEvkOJqTX2XjhAaVOyk5D HODQiWi0pvK03dk9BRFmpQqdN65eeHKZ2DXdQQnTAYpMngJyTwn8vtL55y/20D7I6lAj qvt/o66ve5vKzLhKiYRbcRDEXLs1li1DaOT71jGKnl/W6pLBdlsn7s7ioC044v8dpzkB 8e+TgjRtPgVIPFJyd60ZzcTpLr4hKdBx0fUHeA8FYPaFxWnBN7/7SVAJ14Ih83yPaEsJ ZUU5sZHDYvQ8NcbAk0bjO29Z04cS4jBpm8Qf/u3C+omVTd2OtM0ymYg1DzOTugq7h9AH XMqQ== X-Gm-Message-State: AOJu0YwjjgJIXihbnn26cR9pBywQ0XtdkNzhHRdY98f3pIXuxRSALzzg 9psqIxZg5Mj2hcdjpAEY4FjvpOo3Ak4OmWvkRsaUN5hSpM/feAHpiaBE2bjx2w== X-Gm-Gg: ASbGncsCDiabX2eTqMEsE2kmxh2fq2C5TwKUz6gUCQ6FVW/vKoouYGVNrzm/uZqPlRg CoIh5+z9/1Ps2UoCKMd0C/PV8dTtUEp5025Dl0iIFoUeQ6OB8/ZVYirDS0stchBdbfESt3Ozrhy DIai20dfkmg/4QPVr62yXh2Ri2GeldIqoAZ0YMPrl9cMfkPwTtiDiHvrHxZ7QejznVkBE+xdhc9 d4EXznbrPyqC6+Wk7PgAA+xGP9iih4TeaHkrv3z8ukj9oDgVmu9PFFSsYRPTQkxfHXOzqHsLUHz r9qXu9CL/KGK1ocG/4cZl2kwX9xorwRWs7gOnERy9UfSrXxA+crbjk0MTHahQat8BapVGdkRS4N qRw9fnOaf50BKy8TZBt3o4fgJu4bi+Gbv X-Google-Smtp-Source: AGHT+IGc8uYyX/dKY1/sJDv1ii8huD6+9FnitbCS/uM5tkyj5RHmhNsJ/NtLZqWr+qNdkl/ENu6hgg== X-Received: by 2002:a17:902:db12:b0:242:1ce4:e4f9 with SMTP id d9443c01a7336-242c22641cemr29742215ad.53.1754643021133; Fri, 08 Aug 2025 01:50:21 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:20 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Philip Lorenz , Richard Purdie Subject: [poky][scarthgap][PATCH 16/23] bitbake: fetch2: Use git-lfs fetch to download objects Date: Fri, 8 Aug 2025 14:19:24 +0530 Message-Id: <20250808084931.2156763-16-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221613 From: Philip Lorenz Its not clear which exact git-lfs versions failed to work with bare repositories, however git-lfs 2.13.2 which is shipped by Debian 10 (i.e. the oldest supported distribution by scarthgap) shows no issue when fetching into a bare repository. Switch to git-lfs fetch which in turn eliminates issues seen when using the gitsm fetcher with submodules utilizing lfs. In these scenarios, fetching of LFS objects did not actually happen as the gitsm fetcher parametrizes the to be fetched repositories with `bareclone=1` which in turn means that the target revision was never checked out (and therefore no LFS objects were fetched). (Bitbake rev: 501d3317df5780a5b0e3c2562b26ed276c1dbe90) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie (cherry picked from commit 9962d198c82d64a1562844f66ba67aba590fa7e4) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 25 +------------------------ 1 file changed, 1 insertion(+), 24 deletions(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index e211f7230e..2b6e49362f 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -506,30 +506,7 @@ class Git(FetchMethod): raise bb.fetch2.FetchError("Unable to find revision %s even from upstream" % missing_rev) if self.lfs_need_update(ud, d): - # Unpack temporary working copy, use it to run 'git checkout' to force pre-fetching - # of all LFS blobs needed at the srcrev. - # - # It would be nice to just do this inline here by running 'git-lfs fetch' - # on the bare clonedir, but that operation requires a working copy on some - # releases of Git LFS. - with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir: - # Do the checkout. This implicitly involves a Git LFS fetch. - Git.unpack(self, ud, tmpdir, d) - - # Scoop up a copy of any stuff that Git LFS downloaded. Merge them into - # the bare clonedir. - # - # As this procedure is invoked repeatedly on incremental fetches as - # a recipe's SRCREV is bumped throughout its lifetime, this will - # result in a gradual accumulation of LFS blobs in /lfs - # corresponding to all the blobs reachable from the different revs - # fetched across time. - # - # Only do this if the unpack resulted in a .git/lfs directory being - # created; this only happens if at least one blob needed to be - # downloaded. - if os.path.exists(os.path.join(ud.destdir, ".git", "lfs")): - runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clonedir, d, workdir="%s/.git" % ud.destdir) + self.lfs_fetch(ud, d, ud.clonedir, ud.revision) def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None): """Helper method for fetching Git LFS data""" From patchwork Fri Aug 8 08:49:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6887CCA0EC0 for ; Fri, 8 Aug 2025 08:50:27 +0000 (UTC) Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by mx.groups.io with SMTP id smtpd.web10.17692.1754643024862239007 for ; Fri, 08 Aug 2025 01:50:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=F/L+VIrN; spf=pass (domain: gmail.com, ip: 209.85.215.173, mailfrom: akash.hadke27@gmail.com) Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-b427f4a9652so1769533a12.2 for ; Fri, 08 Aug 2025 01:50:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643024; x=1755247824; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qZs6TGjYaZ081FSWxT14xbCbpWIPZvh6J4LSGL+/NOQ=; b=F/L+VIrNJbjbyDteLn0YHD3tpju4+GrQcHH9dHDUR0Kj6JdLc52dAMKywyN0KrN9gz tcjOyzQIYdEHFa39mkmj+04jK2DNgIjrWYzgywEiCHWyMapCiN31UGIp2biXeccDAhDJ LQB08te66Bvhye9lb9Fu/GnSOLfH67W2CCOX6QCvl5KQUE9V3fPtYYRdDaACmqq0rxp7 tr17Hvf1upoiiML8jBbVbD6fsX8rYqCudRaHDLG6UQ2pe1P7hoTJmvxP7U3Nb0iVElo+ ca4JtOeFpQY41rC8vE8Fxh5otUmbphGHpWw7xGtWcwaHkZBgojmtoJbT7+fVZ3hA2oxC XXcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643024; x=1755247824; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qZs6TGjYaZ081FSWxT14xbCbpWIPZvh6J4LSGL+/NOQ=; b=XbUqqT4pD7NvmFAyJOPntsUgpnkNX19Vxsuqq0LXeQJntcn/nrQm7X0tI3vIISDpfV ROjbz6RgeLgCCfDM5qp0US43f1qBejANbh+dasNtwgbD/xYeoG/Cn5VKCO1h2SQJkETi IZMrL1B8TL/OmROgVlXFSuAAxDyQAor+LNlIPns+dZYKKbn5b5oQdN+3K/VOVYY/UtEt O0yByl0BZOqOnqJ6kBIuvLZHXXy8vS94eVbaYa3Ab1EBiKtoEP/Dgf8fpsiCVXi+XTg8 Z2a3QfJaJDtu38hCwZ8TjvJlGO9MzVBmBrAunA07Ww5QTuEbsWgfmq2K5AEzrbwtNClM kQ0A== X-Gm-Message-State: AOJu0YwSlYneTLt3PAA/rOztQT0Xjf3OMqgPeXy4KOXkfQgKLDaVDUzb x6a8dwEHaY+t+7Ad05CUI0MqLTKblUN7MYNFLUptoaAnhO6t50JACtfZEuNo5A== X-Gm-Gg: ASbGncuSoetT71R4Rk6l3DFbsRWWNbgU1Il4VnmMYM3SJqMgP5QSJ1BooZ2F67ua6NQ DXeUH6PE1Fc95OPc8UW7qbQG+ChyV5GcTTRkt7BG32Tvf82QmPlWtHad//NyF4PcnPRJy7/NFur H3R35tuZdOufa9syRQSjCAs/DG3K4ytXylPTdd8AOp32m26B/mqWA8Dnkk3SRneJgkK2XMcFHmi jICwgNZfqELrHTO4bNSS+Qo8zoPu9RuLXx8YHf1YXbZP8+EtSkrR6aqMQ8QyIfgCjrtb95YO1go HRT5zIPIXwe98VkTOM4ndpB8SJUD8IfpGZAUFQG3iazzxDqQl4pKkbckvh/Q+2rBbQ6S1ED2KQZ WxVEAZN6hui3GE4eOATiQfRXe/YObYczvp2NAKIYAChg= X-Google-Smtp-Source: AGHT+IHa/VPsVK0w/atHqxIQ2BsqJ081LdMzGHw7tV8BMlciJNyKXwblNwubK/LC1AMwwVdDpMQ9LA== X-Received: by 2002:a17:903:1b64:b0:240:7f7d:2b57 with SMTP id d9443c01a7336-242c22a0552mr27951975ad.28.1754643023881; Fri, 08 Aug 2025 01:50:23 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:23 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Philip Lorenz , Richard Purdie Subject: [poky][scarthgap][PATCH 17/23] bitbake: fetch2: Fix incorrect lfs parametrization for submodules Date: Fri, 8 Aug 2025 14:19:25 +0530 Message-Id: <20250808084931.2156763-17-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221614 From: Philip Lorenz The existing code would pass `True` or `False` to the git fetcher. As the fetcher expects `lfs` to be set to `1` this always lead to LFS fetching being disabled. (Bitbake rev: 5e487a5a096400271ed1e29b0df72903f2304e49) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie (cherry picked from commit eb6d89e9e6f27b683da6f2ba2227707a965a0094) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/gitsm.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py index 1c58230eac..992de8276a 100644 --- a/bitbake/lib/bb/fetch2/gitsm.py +++ b/bitbake/lib/bb/fetch2/gitsm.py @@ -123,7 +123,7 @@ class GitSM(Git): url += ";name=%s" % module url += ";subpath=%s" % module url += ";nobranch=1" - url += ";lfs=%s" % self._need_lfs(ud) + url += ";lfs=%s" % ("1" if self._need_lfs(ud) else "0") # Note that adding "user=" here to give credentials to the # submodule is not supported. Since using SRC_URI to give git:// # URL a password is not supported, one have to use one of the From patchwork Fri Aug 8 08:49:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A750C87FD2 for ; Fri, 8 Aug 2025 08:50:37 +0000 (UTC) Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by mx.groups.io with SMTP id smtpd.web11.17949.1754643027773034910 for ; Fri, 08 Aug 2025 01:50:27 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=D+fNqztr; spf=pass (domain: gmail.com, ip: 209.85.210.181, mailfrom: akash.hadke27@gmail.com) Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-76bd2b11f80so1868194b3a.3 for ; Fri, 08 Aug 2025 01:50:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643027; x=1755247827; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=00z6XZq54Cby1WUq1VS6PdUzIlxPUvCNNz2PR+XFg4g=; b=D+fNqztrw+c5NQ9s3jBCd3VZWSmxgPbGUCbXoqZIpa7z6DeyAKVwLfLWYeXHn6Ka8K NzG9W4wgUC2rRbp+XTts1axtETr2VEA3jSCGJWJgwLnRIZ8cHzWxgGnK+SdAawOddFVK yv23Db18tVaiDX5jkqz3BCkdTplMCpDrbTKYxiLXvs4WmvNAvnWuz+kMmzwMuYZSxV7P fk/zsjA91rVrzuKvRhvfoeeTBVXjQrYkI7o6xP8n522yBmHYp2JDFGqfnxcmLZkVoNh+ NkK5ryDlGuiSz+LWqrxPkuwK+zsnsYFTpNGu2PI8oPFgz3KaHfXku3a0hjSadO61o2st 60Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643027; x=1755247827; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=00z6XZq54Cby1WUq1VS6PdUzIlxPUvCNNz2PR+XFg4g=; b=It3lG2VTP15nKfLUkllGyVIuDfr0FWtaIYyaGssGsqKHHZslGvuqOJ4GOzUgT+Bkag tJr2yv/U08QsQaGEDBYd8DHZplzhSPUj+oO6peouBKpOqavuwIf5/74hE33cy150pGAb 9reHgDwmLkyjYUKcByUQpHLmbyQ6PRXas4+1PoXnpUjJ8+YZr9zUjAF/6VWrNyAyIAGA qo36Utlv+twYZrW5CtkHThT0KBCrqieO0TCytLlA5CGZbew6niRStuRKJ10Z5cA25h16 D4bmlKfN0HkWTxlTr9bSYBejLihA8Qk9dNrcwiEi/SAzkoVGhQ2RLYgmTXkd8okM3Pkr jNQg== X-Gm-Message-State: AOJu0YzFM8CJD5e780aO4B6Rm8NN3abEFVv8jiXMq086M9Qdk4rU6fkJ K79S3ay1ML1RwKWNECob4jCHcuddr80r8lXMnDOirXi2TPRDl3qjW+j04N8LXA== X-Gm-Gg: ASbGncsTYaBngQIH7joSK03o5kuX3Nhlmuf5666qkJdz7QOd247WJlCuFofIKQ66Dd6 x4bqM5Jf16IZaq5YiUhTUQyMNVB3l5TNASbjPN5gg1zcpWPFx8oynhGJ3rZX9hsKi+vdZVzX+1Z jCyII8gzrmOF05rWFSUgk+1n+UjzTHhx56u2y8XKbeXC4AnyrdbSeqgD9uM6kZ9gq2CXQjhbXNl VP8S1immCJk9wcD/AFYHdAh8bO/uW18J8tYkOJgiAtCDvr9U/JiMgAWh+dFPlfHeZ7D5Yvg6Izw jyy3dnoDmA0WrVncHJ6jTdlEl9MA8ulnntPZEkwW04+xWeAVDQc0f39qU4lzZKX0x2fXckbz09t Yqkdf9eMbObX6VjW7CrGppihI5ZXTtIsz X-Google-Smtp-Source: AGHT+IGAGIsTTE1vkivP66cYjIp2S8tP1SGpZSnsaU+k8gE5X8xs27mxvCnL5HUGtPUZ/XczbF2QHg== X-Received: by 2002:a17:903:1b66:b0:240:7753:3bec with SMTP id d9443c01a7336-242c228e4e1mr32120345ad.51.1754643026794; Fri, 08 Aug 2025 01:50:26 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:26 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Philip Lorenz , Richard Purdie Subject: [poky][scarthgap][PATCH 18/23] bitbake: fetch2: Fix LFS object checkout in submodules Date: Fri, 8 Aug 2025 14:19:26 +0530 Message-Id: <20250808084931.2156763-18-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:37 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221615 From: Philip Lorenz Skipping smudging prevents the LFS objects from replacing their placeholder files when `git submodule update` actually checks out the target revision in the submodule. Smudging cannot happen earlier as the clone stored in `.git/modules` is bare. This should be fine as long as all LFS objects are available in the download cache (which they are after the other fixes are applied). (Bitbake rev: d270e33a07c50bb9c08861cf9a6dc51e1fd2d874) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie (cherry picked from commit 3eeac69385e8f29a08d022a17b28b5d504deed66) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/gitsm.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py index 992de8276a..9f397699a5 100644 --- a/bitbake/lib/bb/fetch2/gitsm.py +++ b/bitbake/lib/bb/fetch2/gitsm.py @@ -246,12 +246,11 @@ class GitSM(Git): ret = self.process_submodules(ud, ud.destdir, unpack_submodules, d) if not ud.bareclone and ret: - # All submodules should already be downloaded and configured in the tree. This simply - # sets up the configuration and checks out the files. The main project config should - # remain unmodified, and no download from the internet should occur. As such, lfs smudge - # should also be skipped as these files were already smudged in the fetch stage if lfs - # was enabled. - runfetchcmd("GIT_LFS_SKIP_SMUDGE=1 %s submodule update --recursive --no-fetch" % (ud.basecmd), d, quiet=True, workdir=ud.destdir) + cmdprefix = "" + # Avoid LFS smudging (replacing the LFS pointers with the actual content) when LFS shouldn't be used but git-lfs is installed. + if not self._need_lfs(ud): + cmdprefix = "GIT_LFS_SKIP_SMUDGE=1 " + runfetchcmd("%s%s submodule update --recursive --no-fetch" % (cmdprefix, ud.basecmd), d, quiet=True, workdir=ud.destdir) def clean(self, ud, d): def clean_submodule(ud, url, module, modpath, workdir, d): url += ";bareclone=1;nobranch=1" From patchwork Fri Aug 8 08:49:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A78DC87FDA for ; Fri, 8 Aug 2025 08:50:37 +0000 (UTC) Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by mx.groups.io with SMTP id smtpd.web11.17951.1754643030723984579 for ; Fri, 08 Aug 2025 01:50:30 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=RD1lW44z; spf=pass (domain: gmail.com, ip: 209.85.214.179, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2402774851fso17240375ad.1 for ; Fri, 08 Aug 2025 01:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643030; x=1755247830; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Mju10L1DzfvHYtVqUH/MpwVBjl+k6reslyfZ5tqzU5A=; b=RD1lW44zQVTwwvzBrfnw8Xety6T9YktqcuAUFH8Hfnmb48SiNfQ5HqW3CSnlebqsIF QLKbhRBmii1svehDg8OMrGQ95AOrmCB/ZiUpSTG6KSckGXyfNu1vKoYx1azLKONSTGjI bMxkwk/T5V7VK5KTmBQe/yn/6upbVbzuTZ99fiyEmr4Kf7FpbrWc77CVXKFALcXFfrAq aFPYfpCFEc24CnsW+xrrQd0K8Twi/rlM1p6lMnHP5G3a+Gbw5o4SsmjlNB91kMexORd3 JHM9S2pVxMyEfxGXnDG6J9k7Kix9Dlf1RD8GZJEUg2CXfoaDUjBdHoKQBpu+70SjbJdn SkvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643030; x=1755247830; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Mju10L1DzfvHYtVqUH/MpwVBjl+k6reslyfZ5tqzU5A=; b=jxoGRPbJgTpgW2KYHnG4/qHtWSS2XQM0fFvEu+fJzB2JKHLnHD3dILK7S0VTBONBN0 n4A7LzPpsBYXcUGgjiuR84hzP//k4DGRbtiQQqmfTCTnCaJ+slSW/Hl3BQwcBfHaTVz5 /ny8QQiGeulhHcygSlDd+ymegymgrvD782Ozp3CxVxyw7xhbHdlquF6UGmhFfGgy0EIa kdGM+qgxOSbCqr7/pP5IOUcbi8GqxZiuM7FTVssgzOKTUK+KEbpQU7osTfWWGLswWMzS Wnd6X/aEDCppKrGZzrUm/MQWcu292xI4MrQvVuxMxbHmt8Pv9HkqZCTHsEm8ljQDxhF+ XFNg== X-Gm-Message-State: AOJu0YzTMIjVS/S3PqlwSrGrlPOfNMRdxJVgbfm/cDmXs/y5bM9eGM3t kccNHoAr/eH4sIbzKdt+gn5/v673yJ2j8HT9yk8mCL8Gj+6NiDnarXA8GbRupQ== X-Gm-Gg: ASbGncvNON71E+Q8v3EJgrOE6EXQ5kpZlXjzJPfhupsfWzKogEhcZKGF88DBG0LkJhx 2xfBwOcUuLR/3NMZHxG7l/9qvYvtIg+diEza/gMMd/OpxmHbhvw7cq0AJy/N/8vH7gOxBZXgTDb dAWmHJ4e3p2C04jLssLwzE9kJ9gzQNp4yXR5TbULTg/TH+P/Gsq/oPbAmeWDRtBWAAbRmCPBlrB m/Hatonhd39D3m4Tvr5AtLZPKo0Ty68HKSF3q2xENeIyowUBHgi4UpWItIyr8gqzMm/bXcOf1zW x2DK3xZQkLBBJTfzY9JqHbc/5d4eAt7Q2WE0MdXbvdskpfOVLTvF70BFCrmopmFwL9k7veYHBUH qk7zESzxr6qv/IrY0tVdj3ymkpZPxiV3m X-Google-Smtp-Source: AGHT+IHQMnkpEIqgXusRVHL43DimxdCB8yevc/uLlz5nuQYX13rveBlpB/642UThxRltdvd7V8gZwQ== X-Received: by 2002:a17:903:98f:b0:23f:f96f:9de7 with SMTP id d9443c01a7336-242c22fa800mr35051855ad.51.1754643029672; Fri, 08 Aug 2025 01:50:29 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:29 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Philip Lorenz , Richard Purdie Subject: [poky][scarthgap][PATCH 19/23] bitbake: tests/fetch: Test gitsm with LFS Date: Fri, 8 Aug 2025 14:19:27 +0530 Message-Id: <20250808084931.2156763-19-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:37 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221616 From: Philip Lorenz Add a test case to verify that the gitsm fetcher properly handles repositories storing objects with LFS. The test case verifies that LFS objects are fetched on the initial clone but also ensures that consecutive updates extend the original clone with any newly referenced LFS objects. (Bitbake rev: 2a8722ddd155596862029f6ea34e1e92c77e0b7f) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie (cherry picked from commit a7331c399252c8b365e51290c8eba3d2f6aa6fa1) Signed-off-by: Akash Hadke --- bitbake/lib/bb/tests/fetch.py | 122 +++++++++++++++++++++++++++++++--- 1 file changed, 111 insertions(+), 11 deletions(-) diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index 5e76988570..6ca4745f8f 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -20,6 +20,7 @@ import tarfile from bb.fetch2 import URI from bb.fetch2 import FetchMethod import bb +import bb.utils from bb.tests.support.httpserver import HTTPService def skipIfNoNetwork(): @@ -27,6 +28,18 @@ def skipIfNoNetwork(): return unittest.skip("network test") return lambda f: f + +@contextlib.contextmanager +def hide_directory(directory): + """Hide the given directory and restore it after the context is left""" + temp_name = directory + ".bak" + os.rename(directory, temp_name) + try: + yield + finally: + os.rename(temp_name, directory) + + class TestTimeout(Exception): # Indicate to pytest that this is not a test suite __test__ = False @@ -2308,26 +2321,113 @@ class GitLfsTest(FetcherTest): unpacked_lfs_file = os.path.join(self.d.getVar('WORKDIR'), 'git', "Cat_poster_1.jpg") return unpacked_lfs_file + @skipIfNoGitLFS() + def test_gitsm_lfs(self): + """Test that the gitsm fetcher caches objects stored via LFS""" + self.git(["lfs", "install", "--local"], cwd=self.srcdir) + + def fetch_and_verify(revision, filename, content): + self.d.setVar('SRCREV', revision) + fetcher, ud = self.fetch() + + with hide_directory(submoduledir), hide_directory(self.srcdir): + workdir = self.d.getVar('WORKDIR') + fetcher.unpack(workdir) + + with open(os.path.join(workdir, "git", filename)) as f: + self.assertEqual(f.read(), content) + + # Create the git repository that will later be used as a submodule + submoduledir = self.tempdir + "/submodule" + bb.utils.mkdirhier(submoduledir) + self.git_init(submoduledir) + self.git(["lfs", "install", "--local"], cwd=submoduledir) + self.commit_file('.gitattributes', '*.mp3 filter=lfs -text', cwd=submoduledir) + + submodule_commit_1 = self.commit_file("a.mp3", "submodule version 1", cwd=submoduledir) + _ = self.commit_file("a.mp3", "submodule version 2", cwd=submoduledir) + + # Add the submodule to the repository at its current HEAD revision + self.git(["-c", "protocol.file.allow=always", "submodule", "add", submoduledir, "submodule"], + cwd=self.srcdir) + base_commit_1 = self.commit() + + # Let the submodule point at a different revision + self.git(["checkout", submodule_commit_1], self.srcdir + "/submodule") + self.git(["add", "submodule"], cwd=self.srcdir) + base_commit_2 = self.commit() + + # Add a LFS file to the repository + base_commit_3 = self.commit_file("a.mp3", "version 1") + # Update the added LFS file + base_commit_4 = self.commit_file("a.mp3", "version 2") + + self.d.setVar('SRC_URI', "gitsm://%s;protocol=file;lfs=1;branch=master" % self.srcdir) + + # Verify that LFS objects referenced from submodules are fetched and checked out + fetch_and_verify(base_commit_1, "submodule/a.mp3", "submodule version 2") + # Verify that the repository inside the download cache of a submodile is extended with any + # additional LFS objects needed when checking out a different revision. + fetch_and_verify(base_commit_2, "submodule/a.mp3", "submodule version 1") + # Verify that LFS objects referenced from the base repository are fetched and checked out + fetch_and_verify(base_commit_3, "a.mp3", "version 1") + # Verify that the cached repository is extended with any additional LFS objects required + # when checking out a different revision. + fetch_and_verify(base_commit_4, "a.mp3", "version 2") + + @skipIfNoGitLFS() + def test_gitsm_lfs_disabled(self): + """Test that the gitsm fetcher does not use LFS when explicitly disabled""" + self.git(["lfs", "install", "--local"], cwd=self.srcdir) + + def fetch_and_verify(revision, filename, content): + self.d.setVar('SRCREV', revision) + fetcher, ud = self.fetch() + + with hide_directory(submoduledir), hide_directory(self.srcdir): + workdir = self.d.getVar('WORKDIR') + fetcher.unpack(workdir) + + with open(os.path.join(workdir, "git", filename)) as f: + # Assume that LFS did not perform smudging when the expected content is + # missing. + self.assertNotEqual(f.read(), content) + + # Create the git repository that will later be used as a submodule + submoduledir = self.tempdir + "/submodule" + bb.utils.mkdirhier(submoduledir) + self.git_init(submoduledir) + self.git(["lfs", "install", "--local"], cwd=submoduledir) + self.commit_file('.gitattributes', '*.mp3 filter=lfs -text', cwd=submoduledir) + + submodule_commit_1 = self.commit_file("a.mp3", "submodule version 1", cwd=submoduledir) + + # Add the submodule to the repository at its current HEAD revision + self.git(["-c", "protocol.file.allow=always", "submodule", "add", submoduledir, "submodule"], + cwd=self.srcdir) + base_commit_1 = self.commit() + + # Add a LFS file to the repository + base_commit_2 = self.commit_file("a.mp3", "version 1") + + self.d.setVar('SRC_URI', "gitsm://%s;protocol=file;lfs=1;branch=master;lfs=0" % self.srcdir) + + # Verify that LFS objects referenced from submodules are not fetched nor checked out + fetch_and_verify(base_commit_1, "submodule/a.mp3", "submodule version 1") + # Verify that the LFS objects referenced from the base repository are not fetched nor + # checked out + fetch_and_verify(base_commit_2, "a.mp3", "version 1") + @skipIfNoGitLFS() def test_fetch_lfs_on_srcrev_change(self): """Test if fetch downloads missing LFS objects when a different revision within an existing repository is requested""" self.git(["lfs", "install", "--local"], cwd=self.srcdir) - @contextlib.contextmanager - def hide_upstream_repository(): - """Hide the upstream repository to make sure that git lfs cannot pull from it""" - temp_name = self.srcdir + ".bak" - os.rename(self.srcdir, temp_name) - try: - yield - finally: - os.rename(temp_name, self.srcdir) - def fetch_and_verify(revision, filename, content): self.d.setVar('SRCREV', revision) fetcher, ud = self.fetch() - with hide_upstream_repository(): + with hide_directory(self.srcdir): workdir = self.d.getVar('WORKDIR') fetcher.unpack(workdir) From patchwork Fri Aug 8 08:49:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61993C87FD3 for ; Fri, 8 Aug 2025 08:50:37 +0000 (UTC) Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by mx.groups.io with SMTP id smtpd.web10.17693.1754643033209480045 for ; Fri, 08 Aug 2025 01:50:33 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=W8jI3nlQ; spf=pass (domain: gmail.com, ip: 209.85.215.181, mailfrom: akash.hadke27@gmail.com) Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-b4209a0d426so1932398a12.1 for ; Fri, 08 Aug 2025 01:50:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643032; x=1755247832; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AL2WkQaHXSacRiHRpk5kDaah6EYWRRp3/8aLrZ8f3Xg=; b=W8jI3nlQUVpjjmufCotC/UQmg7jrOKFbihvnJRnbTsH0u+kGaqnV0Qz0I647V9vXhJ 4Hahq9sXN5c6LwzTPsp7FKYaQWRstIHUxaBd1wq3EYRw4ersWuM3MVCWNnQE8mvm6OHs /om5kGr/jXgGzvSUxYlTRVEFVTRzlki+8Ii1vFsXoIbG7O2wNSB6WRiD3XE2gfdYBqgc ubrP9BhtZ6b7LKNtZ8N4dFmQUdB4q2IYjl8V/DUpzSeCWoZuMURjT4yXoIOa6TwWxuKZ HrLJ6d16QBAAzPtIPJRpqgdYqLaJaSh7ydWP9wKFrSW70PMob+ulIXHkyZ3iY5WtztWH W21w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643032; x=1755247832; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AL2WkQaHXSacRiHRpk5kDaah6EYWRRp3/8aLrZ8f3Xg=; b=keHKon6Oi6APgfTcOkqbpdDwhUUszuRUc8mk3WjW1s759sGCKHYndE3LW4rSc5yZwU HOQtlN6pw1Eaa++dJvuy0GphRXxe5kb2rr6SSyAiQ/H0pxAqSIjcoos8IcvVF81+5XBN 8T02el9ElVEM27O2aYp9h9RglaeKY8YsDjP7qmH7AfJIQ0fQPCO00XHL54ttHCHQ7c/P KPjUAJPTamb7L5wCfKtjLG4Y4Exj7PX5A1TyLnu1dfp883azZk1AFGQXpQXIgrnj4WQJ akg1cy9DXO/ugwf0pVS+hm6Y1hF0SpSDcC8CoAAcOdZkpnlANykL8mrtYciPabGCoNGN nRlw== X-Gm-Message-State: AOJu0YzJdVjlBkDiw744jZPM65pmZudqdhXXmKzwTv+yOaDJy57MP5IK 6PYA0hH5jd+NoKfZKqliWcPmdGuSuFd6TkXO13CSJG4B9llBfYblM6Byklx7cw== X-Gm-Gg: ASbGncv3fqKf1mIE5KkpokebecS81i/J2Iw+T11X05GFkqUJeipXldD+b1zjMRjJiRV kbBV+B/g6DpDMRanOrJSR72By+SjnSyY2zsphLZ5t+h3sdCpIQomjjnH3c7PUjM3rBgS6x7Lm/b gno/fEny3Fm72WGauupGEjWX5n9Jr7PW1xohyxn2oTUynJAPxWZV222Jh5yVm0BweZKSmGnygHw xy073pfY58mlcbgm+KR08nFN2Lmd82w9bRJ5Ue+yavq4cK9rr0uiHGXngRmACd2ckbwieZc8Iu2 ajbJmNzgQBL+yROU2yb9KqjbV76ITDIxJuspLRlaxCml5TT1hyww6hllLWQq/yFpIJUeSOGXFk1 wsKrKAI8qYgIDSrhoLJMizFuDU8lT9l+W X-Google-Smtp-Source: AGHT+IGOqWy4JS3H+LdBVia2o5xTPTN5p4air7Egt52BjxpGDc7M4SX3r9AvYO8Q1vvP3jofCFRFFg== X-Received: by 2002:a17:903:110f:b0:23f:df36:5f0c with SMTP id d9443c01a7336-242c21dda3emr32084925ad.29.1754643032317; Fri, 08 Aug 2025 01:50:32 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:31 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Chen Qi , Richard Purdie Subject: [poky][scarthgap][PATCH 20/23] bitbake: fetch2/git: fix shallow clone for tag containing slash Date: Fri, 8 Aug 2025 14:19:28 +0530 Message-Id: <20250808084931.2156763-20-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:37 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221617 From: Chen Qi If a tag contains slash, e.g., debian/5.22, then shallow clone fails because it's using a wrong ref. To reproduce the issue, add the following lines in local.conf: BB_GIT_SHALLOW = "1" BB_GENERATE_SHALLOW_TARBALLS = "1" And then run 'bitbake debianutils -c fetch'. What the original os.path.basename(ref) wanted to do is to remove the strings such as refs/heads/. So we do it explitly to fix this issue. Fixes: [YOCTO #15862] (Bitbake rev: c6d6999f1ed01e7445b8f177a888038edacf555c) Signed-off-by: Chen Qi Signed-off-by: Richard Purdie (cherry picked from commit 07f08063c1d6eff52fe95d0b6c1239cb1336c7a7) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 2b6e49362f..764e34ccc2 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -662,7 +662,7 @@ class Git(FetchMethod): extra_refs.append(r) for ref in extra_refs: - ref_fetch = os.path.basename(ref) + ref_fetch = ref.replace('refs/heads/', '').replace('refs/remotes/origin/', '').replace('refs/tags/', '') runfetchcmd("%s fetch origin --depth 1 %s" % (ud.basecmd, ref_fetch), d, workdir=dest) revision = runfetchcmd("%s rev-parse FETCH_HEAD" % ud.basecmd, d, workdir=dest) runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) From patchwork Fri Aug 8 08:49:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66DFCCA0EC0 for ; Fri, 8 Aug 2025 08:50:37 +0000 (UTC) Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by mx.groups.io with SMTP id smtpd.web11.17952.1754643035670431370 for ; Fri, 08 Aug 2025 01:50:35 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=VDIwFJx8; spf=pass (domain: gmail.com, ip: 209.85.210.178, mailfrom: akash.hadke27@gmail.com) Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-76bd041c431so1837037b3a.2 for ; Fri, 08 Aug 2025 01:50:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643035; x=1755247835; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Cw+E5rg6VQSrLCUOHoi5NkEzNglUKjgkX55lA8tYBHU=; b=VDIwFJx8EFH9JPflyPTFLIjXsWo2aN5XxQ+TZlqvazqncJ6gygprcyrdfhoSRuUVLH BfflEs86gUuA9FiHrlQqU0jBndmW3TiLKk042dTMa7APvI82htCNVRePhrkGuPGs6OB0 FhULdFEmA/Z60uNSnxeMoy3juaqBD+Kb8/ZujxPOEahMxp72OYS88ChU6yOS5NKQYXWi 6nEjhTviVooRWcWJ8S8gsAx7UCpVyovAnkJ7ck9GhkZ/WXaFbRFjja/F1zxBsA98T1CH EkLAwMxcO2LerMeD1wFvGM3VLpCo9+AI4nCuifVD6qdrS4scSv5eXTDoGaiZymVyja+m t1EA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643035; x=1755247835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cw+E5rg6VQSrLCUOHoi5NkEzNglUKjgkX55lA8tYBHU=; b=HBnFe052E/TuN3ouWx4BVqB00IVXiVBiyDpDiqSvT7hS5MtZghwJRNlybSqFCy/cVR MAjhUXtOt/HuOV/+EpTJBq/HLOoFRhII+QCGCY/wOEJPNyOz+GtjoPLejSB1srVDgMwQ k1UHpwqk+15CrVdbBTM3/ac3nSqz5ZzzE5WW1ouo9RiwwEQtBXH3pWdglSzSWdE2oqnz +SUz5C2Uq48i1I6Id/1y5ePH8ih+35K967rJaF1JDx0lFIexMPYeNSeOwGh89eCI4M91 uEtCZh0MyJ8oV6XOPhiX/ekvtpflut9kES+9v+lgxgIPEe5OLyO7/HX61z3JCSIzu3wZ Nj8Q== X-Gm-Message-State: AOJu0YyDzZrcNgISNsifB46njY9zaAfGpfl+QSvNZ6NjAtsMihuDLHoG iV9fB2Mh2LqUAP6/li2UiEQqvNBQbfjOXKKRX3QOSIaftanzERrHZCbyAQBcWg== X-Gm-Gg: ASbGncthAyqZO4En9WsPT44HbngEHQaWbFRYFPUGK2kFUkZNBaiYYBtEnV9ToOm9rNX 8sWqzFZK1a0AYoenxMF1VZBjEAD1G0WfOkEWNmJnm2wyWgLNUrhj0w8lI5kKkL0H0YmVLMvEsPX ycjo4b+YKNXEpPhY8n65aaZ5IBtj4DLlRusKCySlWWgyFICC21vOR4uJlBk4s7Yiq0xf1vZ+NHp aBXvqZR+giS3gNOoFxJ9KPwmrPv1dJoonG+P3RAnf/K90Q5oTISjwYrgbvNvVRFwXkMtcIONVbg JbAZpsnjlcC0+v110bp8d+7ZdRFQGFXuQ4yBuIlBl1jC1RlXT1s/g6pi8FrSplcP8dYlUUaARpk X3ALZNwhrNRw2EU0WsUiOezez/pA2Q3/4E2oCFwqfHx8Lbqrb X-Google-Smtp-Source: AGHT+IG4oCkz3XTtdWgQwyQf+H5un9WGTxRvXmLGOfoxbwI3zfc6HBqYLGNbly2/ilJgS/Gg+Ie8Hg== X-Received: by 2002:a17:902:f544:b0:240:c9b6:f8ee with SMTP id d9443c01a7336-242c225f966mr31040815ad.50.1754643034754; Fri, 08 Aug 2025 01:50:34 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:34 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Stefan Koch , Richard Purdie Subject: [poky][scarthgap][PATCH 21/23] bitbake: fetch2: Move the `ensure_symlink()` function into the `FetchMethod` class Date: Fri, 8 Aug 2025 14:19:29 +0530 Message-Id: <20250808084931.2156763-21-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:37 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221618 From: Stefan Koch (Bitbake rev: d934763f91b9bda85f273a44f12a04a3ca19fc28) Signed-off-by: Stefan Koch Signed-off-by: Richard Purdie (cherry picked from commit 0e9ce41a1cc0e1158f84d5c5c08cf23b5c161155) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/__init__.py | 36 +++++++++++++++---------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index 1a6ff25d4d..49dca75025 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -1127,25 +1127,6 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): if ud.lockfile and ud.lockfile != origud.lockfile: bb.utils.unlockfile(lf) - -def ensure_symlink(target, link_name): - if not os.path.exists(link_name): - dirname = os.path.dirname(link_name) - bb.utils.mkdirhier(dirname) - if os.path.islink(link_name): - # Broken symbolic link - os.unlink(link_name) - - # In case this is executing without any file locks held (as is - # the case for file:// URLs), two tasks may end up here at the - # same time, in which case we do not want the second task to - # fail when the link has already been created by the first task. - try: - os.symlink(target, link_name) - except FileExistsError: - pass - - def try_mirrors(fetch, d, origud, mirrors, check = False): """ Try to use a mirrored version of the sources. @@ -1635,6 +1616,23 @@ class FetchMethod(object): """ bb.utils.remove(urldata.localpath) + def ensure_symlink(self, target, link_name): + if not os.path.exists(link_name): + dirname = os.path.dirname(link_name) + bb.utils.mkdirhier(dirname) + if os.path.islink(link_name): + # Broken symbolic link + os.unlink(link_name) + + # In case this is executing without any file locks held (as is + # the case for file:// URLs), two tasks may end up here at the + # same time, in which case we do not want the second task to + # fail when the link has already been created by the first task. + try: + os.symlink(target, link_name) + except FileExistsError: + pass + def try_premirror(self, urldata, d): """ Should premirrors be used? From patchwork Fri Aug 8 08:49:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66D90C87FDA for ; Fri, 8 Aug 2025 08:50:47 +0000 (UTC) Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by mx.groups.io with SMTP id smtpd.web10.17694.1754643038311462121 for ; Fri, 08 Aug 2025 01:50:38 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=NCWUCq7X; spf=pass (domain: gmail.com, ip: 209.85.214.178, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-24014cd385bso20642535ad.0 for ; Fri, 08 Aug 2025 01:50:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643037; x=1755247837; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4mK3/9Ohtbj4n0nl7V8n0G82kH9pv7Al6B/pJQG37lI=; b=NCWUCq7Xaw/V9mn2xID52LGLF6spUiyWTxJYYcHYdklC9Ofqexol7lToh1a/GXwnxr MysYuJIFM/hk35r8X44a3IV+fsM3DQXDuZvTy+xU/mvce8DALd8oL0ixMzwjKgGPq3y+ g4RJhpaILSYCqIljVtZTkO91xsSOeEUjlxNZLpzeUa7qSG+KH0gqu6JSAJ3DuYAxtfTd qu3DY8LN39ufUuecRh1mhCWPFTiLTDk89O1ardL0q7t9/CO3AqP7kI6fUEE3ynnXdi1M sbczGJHcKOAa65DjnePdVFFeHM6fDKcBVIfHyLRRgtnu560ZcRkSGjP9K/991llCU8Nh 5xVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643037; x=1755247837; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4mK3/9Ohtbj4n0nl7V8n0G82kH9pv7Al6B/pJQG37lI=; b=A5X9nr3CT70R6dmLZuY1HRaJzb/7ZjgHIq2sZ33Siz8RicW8ZJF1AY/WDzij71AoA6 IFOZj6axuG7ZOLitpVZozsmeT84a97nvuqRfDZO99zijf+3mRz+w3rLotxJIcpLQl+wa LzmwV93xzTNjEfsI/h3VW+cLV7Ak9lzOFdpsu91SA9giXeiI/tTJyM4+yXcjS90WXewj Yo5HxJPPEzH1uOemytnHrxMhJ6MRJZYjxrYroeLpTsCHuv9AO4DjdGgimGx5uSPNrjT6 ua+2e89ep9kGnOpQ3yia+0msnxor/4TTGxkWqjWAZ6Q3gb8+1Hoc86LuQcErqGZ4sMmy RPMA== X-Gm-Message-State: AOJu0Yws6ycdK1npre+se9ZiBakyrH89b3i02kvLTHP04b1qiG7ZK7FJ IUPSN0K5XcmKOc/FfPjUQUj9bjm/oiwdIX7cJZkuEjNg9TilAh6zwoIq+EiWEw== X-Gm-Gg: ASbGnctZwyU78JRj3QugX+Dqy7cgLyeQf+XUsU0oaaJZDnkXYXaMVd4gM8uaE3M+hCJ KEtn9BG9IAUgER3gmeXyczmsXcAb8dqCP/E4VV/ZOwaBnRdEopuzuo0M76AKk98wUXaazUwkBtN BPQfz4O1Opb2cSpNRKFwCjThnHSkrZxDMr1gVdk1jaoHpZ6J/5xULQbUMb79W4UM8Mso4Z9RCyj U5GJvPQFRp2k9JVI5JDQUsZnF5Js9EaMh+aABhjBhGpJIZ+m+EVfbDqkN4at7Sb/k8caoD0EaIp dFLd7mMWXNUPGTtE70q3YQMdH6uYj7fXfht0SykynOd89mO3haOFVbdys/8Wekvcsbvl2cscTCf T7OoWyOnzoIZpJFtaFEw/t7R9zLrySUjS2TfvnTx6YcU= X-Google-Smtp-Source: AGHT+IHenumyEIVo1yN1gJ6PGhvo8ZesrMIONdPgsH6LDHsELDVEW4G9AfGG+iHtWPz/BSO33Q7zZQ== X-Received: by 2002:a17:902:f683:b0:235:f078:4746 with SMTP id d9443c01a7336-242c224b697mr37714075ad.42.1754643037354; Fri, 08 Aug 2025 01:50:37 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:37 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Cc: Stefan Koch , Richard Purdie Subject: [poky][scarthgap][PATCH 22/23] bitbake: fetch2: Ensure a valid symlink in `PREMIRRORS` case when using shallow cloning Date: Fri, 8 Aug 2025 14:19:30 +0530 Message-Id: <20250808084931.2156763-22-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221619 From: Stefan Koch - Since `ud.path` contains in that case the `PREMIRRORS` prefix path, this change ensures that a correct symlink is set up. (Bitbake rev: 37ed18e45aa17406162efc5ee3ddb2d6b33d07b9) Signed-off-by: Stefan Koch Signed-off-by: Richard Purdie (cherry picked from commit f5ae37fbd86c1328ebd81166573fbad3028abe5b) Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/__init__.py | 10 +++++++++- bitbake/lib/bb/fetch2/git.py | 7 +++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index 49dca75025..9b1aa8858b 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -1093,7 +1093,10 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): origud.method.build_mirror_data(origud, ld) return origud.localpath # Otherwise the result is a local file:// and we symlink to it - ensure_symlink(ud.localpath, origud.localpath) + # This may also be a link to a shallow archive + # When using shallow mode, add a symlink to the original fullshallow + # path to ensure a valid symlink even in the `PREMIRRORS` case + origud.method.update_mirror_links(ud, origud) update_stamp(origud, ld) return ud.localpath @@ -1633,6 +1636,11 @@ class FetchMethod(object): except FileExistsError: pass + def update_mirror_links(self, ud, origud): + # For local file:// results, create a symlink to them + # This may also be a link to a shallow archive + self.ensure_symlink(ud.localpath, origud.localpath) + def try_premirror(self, urldata, d): """ Should premirrors be used? diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 764e34ccc2..390d9ddafc 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -371,6 +371,13 @@ class Git(FetchMethod): def tarball_need_update(self, ud): return ud.write_tarballs and not os.path.exists(ud.fullmirror) + def update_mirror_links(self, ud, origud): + super().update_mirror_links(ud, origud) + # When using shallow mode, add a symlink to the original fullshallow + # path to ensure a valid symlink even in the `PREMIRRORS` case + if ud.shallow and not os.path.exists(origud.fullshallow): + self.ensure_symlink(ud.localpath, origud.fullshallow) + def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout with only premirrors # is not possible From patchwork Fri Aug 8 08:49:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akash Hadke X-Patchwork-Id: 68237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66D55C87FD3 for ; Fri, 8 Aug 2025 08:50:47 +0000 (UTC) Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by mx.groups.io with SMTP id smtpd.web11.17953.1754643040395765932 for ; Fri, 08 Aug 2025 01:50:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=FnTF+BN/; spf=pass (domain: gmail.com, ip: 209.85.214.177, mailfrom: akash.hadke27@gmail.com) Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2400f746440so15071455ad.2 for ; Fri, 08 Aug 2025 01:50:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1754643039; x=1755247839; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=SL7Nq93SL1rXHkLsyteZmXDVJ6IrYtk6+gjnqxApHdM=; b=FnTF+BN/3D2DZ67J6qABl/QPnW6Yu+kogVJ/mSh//+M8uheIb3244nqX0+YCc9O9Aw e9D1sZiCFR6CKkE9N7IWSjg0vl5e/hSbLld7douK5DzGoj0kkspOTGsW6oYpeHst2bAK +LtPTv5Q8G21Pc7dsP3lwH/lvjlX5Ltc3cgvF5tBVInMJU6FaZ3vPU9jwdOIJaG1AXOe wusRlxCfQPhdgYlGPDUQZdN9LgMD/Styw+Fh9HpApwlxx4tkviPX5YRqu8+MZAb+RFzr EAnaL7D34XuWMRWfCz3kpcqXpMtuvuoJkP9u8I3+ktetHePsVAmGpwSHprRWSixXu7tO 4SOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754643039; x=1755247839; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SL7Nq93SL1rXHkLsyteZmXDVJ6IrYtk6+gjnqxApHdM=; b=Cf6yi76487soFu6uIOOV3CXakOb1mdsklmai1BaSKAfW6q4qVwyRkUeite/B+L3YEr dIkcQrm4TS5/9pgFivAyNlER7438h3AlwNugQmGPIWWfCR6Dur+mufQ/S+m5s0mKHUyv ceqvIZsjI8RRdNgC2adCxkN31gIDIv129mkrPMXQU/v6j8rGrQNub6zxTNhDGWEnGdG6 UXv7zmOhIqJic3X6w+z7Jas82CfgTGtUiadPZAlqi3I80J4oiynXx6pYugtWnVo+ZxAK rfl8Iq0l2d4/F2njzLOggEh3cM+dTTJYT9VChuBiQtuF7bLCgbwie695m7c/URT0sAmr Op+Q== X-Gm-Message-State: AOJu0YxYZsEcw+pFQEwcsimQoXu0F29PUoXdTgV+nqao77ES/m0fQ52T nKIqvxkBZ1V0xZ8e7ZKDS1bgR7m8oCYPBw2MThuL1lSUgG39oPKdLrgTXjAz7Q== X-Gm-Gg: ASbGnctUKZ3MCV3lHmlDBHi3WO65DTPm9w6FJd17Et/yCUXtFO+smKSDQaa3WqEyBx4 j8yitGgs0JrXOaOLMxAnSK3mnIXY8dOdXlpfgG4MtCUSlyfRhnz8eCfjRZydqvmLrJ63hEjn0zi NNTpqiRN8740E5ld5RnyVA5DtiV9lBuS4Rjkj7fDuMc9OFKrl52I1toF/qyGtzYLTaGhSqddmSA iGQM/MWGSx2siZ8xdPfPvnvj18SYICzrVGeRYtLnDzFMHbBYMYGlE52s+N239ETP3N8uwq2fjvo sW6eG57WJiu1fuuK/asw2nz5/1PYHehXdVdQVS0JdmpIBl7A75B62Xb6FB9QdGuKJihbeDPWyCn 4DD5hQIsGTqkw+K9UikEOZfqeUIi1gr5PKbB3hw== X-Google-Smtp-Source: AGHT+IFJRDAdBgMyjveUSqYqoc4vwPIsEGsejePeir5ZZ3+vqUKKTENO1Rtl3nn3/ZIPEWBv4ThkJw== X-Received: by 2002:a17:903:4b47:b0:240:8704:fb9 with SMTP id d9443c01a7336-242c22fb175mr28304075ad.53.1754643039371; Fri, 08 Aug 2025 01:50:39 -0700 (PDT) Received: from L-18010L.kpit.com ([49.36.49.248]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-24218d8413asm188782455ad.63.2025.08.08.01.50.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 01:50:38 -0700 (PDT) From: Akash Hadke To: openembedded-core@lists.openembedded.org Subject: [poky][scarthgap][PATCH 23/23] bitbake: fetch2/git: Add multiple revision support Date: Fri, 8 Aug 2025 14:19:31 +0530 Message-Id: <20250808084931.2156763-23-akash.hadke27@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com> References: <20250808084931.2156763-1-akash.hadke27@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 08 Aug 2025 08:50:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/221620 In scarthgap single SRC_URI can still have multiple revisions update the code in order to iterate over these revisions and do the fetch. Signed-off-by: Akash Hadke --- bitbake/lib/bb/fetch2/git.py | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 390d9ddafc..88fa119e1d 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -513,12 +513,13 @@ class Git(FetchMethod): raise bb.fetch2.FetchError("Unable to find revision %s even from upstream" % missing_rev) if self.lfs_need_update(ud, d): - self.lfs_fetch(ud, d, ud.clonedir, ud.revision) + for name in ud.names: + self.lfs_fetch(ud, d, name, ud.clonedir, ud.revisions[name]) - def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None): + def lfs_fetch(self, ud, d, name, clonedir, revision, fetchall=False, progresshandler=None): """Helper method for fetching Git LFS data""" try: - if self._need_lfs(ud) and self._contains_lfs(ud, d, clonedir) and len(revision): + if self._need_lfs(ud) and self._contains_lfs(ud, d, name, clonedir) and len(revision): self._ensure_git_lfs(d, ud) # Using worktree with the revision because .lfsconfig may exists @@ -649,7 +650,7 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=dest) runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest) # Fetch Git LFS data - self.lfs_fetch(ud, d, dest, ud.revisions[ud.names[0]]) + self.lfs_fetch(ud, d, name, dest, ud.revisions[ud.names[0]]) # Apply extra ref wildcards all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \ @@ -738,7 +739,7 @@ class Git(FetchMethod): repourl = self._get_repo_url(ud) runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=destdir) - if self._contains_lfs(ud, d, destdir): + if any(self._contains_lfs(ud, d, name, destdir) for name in ud.names): if not need_lfs: bb.note("Repository %s has LFS content but it is not being fetched" % (repourl)) else: @@ -806,8 +807,9 @@ class Git(FetchMethod): Verifies whether the LFS objects for requested revisions have already been downloaded """ # Bail out early if this repository doesn't use LFS - if not self._contains_lfs(ud, d, wd): - return True + for name in ud.names: + if not self._contains_lfs(ud, d, name, wd): + return True self._ensure_git_lfs(d, ud) @@ -835,12 +837,12 @@ class Git(FetchMethod): def _need_lfs(self, ud): return ud.parm.get("lfs", "1") == "1" - def _contains_lfs(self, ud, d, wd): + def _contains_lfs(self, ud, d, name, wd): """ Check if the repository has 'lfs' (large file) content """ cmd = "%s grep lfs %s:.gitattributes | wc -l" % ( - ud.basecmd, ud.revision) + ud.basecmd, ud.revisions[name]) try: output = runfetchcmd(cmd, d, quiet=True, workdir=wd)