From patchwork Sun Mar 29 18:32:48 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFFB6FC97E4 for ; Sun, 29 Mar 2026 18:34:30 +0000 (UTC) Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net [185.136.64.228]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.34484.1774809263057920385 for ; Sun, 29 Mar 2026 11:34:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=ST2gykNZ; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.64.228, mailfrom: fm-1329275-20260329183419e6f874b93e00020791-2a0nrt@rts-flowmailer.siemens.com) Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 20260329183419e6f874b93e00020791 for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=Nnmo35oV07aTzoAP8J5D6m73Apiz4salVBQcM/EJGEg=; b=ST2gykNZgV7Nu/Ztb2chLqWy/GmL0HT4oF4y0AhuQLxKGxLD9B1Cj8nAZCgmQyYZAdUKvR PBvMJIi1u/tZa4QiaQi0HX7si6AY3rCbUYEHYhAryg3HTKJF7pHGgDc8+WOGbghcIC9aUF0t 2N4XfJ0PQGN6gv9LvcoRc8gu7AlSkO5GES4Igy+MjJcoJGMCszqEzHmhtuFYAJYFmzR1KREw F16ASHp8ukB8yXkL3UJxlqtRsjmQU0ikTE2khzzyxnlKdJ4JUJDln+krGwemxtGd8gCUiEK+ iRFUEcL0coNDyQVgqaaQbFeTgkE0rirH2/mgUO5GxjQPL/pMpRHP/p2g==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 01/14] bitbake-setup: Fix type annotation for descriptions parameter in print_configs Date: Sun, 29 Mar 2026 20:32:48 +0200 Message-ID: <20260329183332.1996183-2-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:30 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19271 From: Adrian Freihofer Fixup for "bitbake-setup: Set function default to None": when the default was changed from [] to None to avoid the mutable default argument pitfall (B006), the type annotation was not updated accordingly. The function body already explicitly handles None via: if not descriptions: descriptions = ["" for _ in choices] Update the annotation to use the union type list[str] | None (Python 3.10+) to accurately reflect that None is a valid value for this parameter. Signed-off-by: Adrian Freihofer --- bin/bitbake-setup | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index ce712e517..f0240cae5 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -46,7 +46,7 @@ def color_enabled() -> bool: def get_diff_color_param() -> str: return "--color=always" if color_enabled() else "--color=never" -def print_configs(prompt: str, choices: list[str], descriptions: list[str] = None): +def print_configs(prompt: str, choices: list[str], descriptions: list[str] | None = None): """ Helper function to print a list of choices and align the output. Each option name is made bold to stand out, unless color is not enabled in From patchwork Sun Mar 29 18:32:49 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E999BFC97E6 for ; Sun, 29 Mar 2026 18:34:30 +0000 (UTC) Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net [185.136.65.227]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.34711.1774809263055890021 for ; Sun, 29 Mar 2026 11:34:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=W/akWgDe; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.65.227, mailfrom: fm-1329275-20260329183419d7365fead000020754-0sjrd0@rts-flowmailer.siemens.com) Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20260329183419d7365fead000020754 for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=NPm4s7iKzEDeuqxpl9IVYBGfOkpj6hREhHtW5vkvOt8=; b=W/akWgDeylNu55Zx0OwcukWwOv68PeQdhPaBC2oZKB1MDiA2Lzy7dbFWc9OA51gVkHyJTA GC6rcj3jjVgR6AxgP9WPkiuEDZOXoHxUpcnBfaJCQgT86xGa5enLYHyD1ibVO14PZQVGuOhW MK/evhXDZjV96djx1/69WGYdkAVo2CiNAqLfK2n0UCRF6RswW9sUIjq8f9aDJhtt+CKTqoeK wc7eYFMS/2N4308kd0vfW0zAz0Es7UMTL6KRtoERSuzhFT4Bf33U1hr5ZXqlqJoC8BULbb7G DKqw7fkB1wnDpnCB8uZThE9BuQ60OCsyFGdjVemmooNh3Jcdmf/e5f5A==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 02/14] fetch2: add LocalModificationsError and RebaseError exceptions Date: Sun, 29 Mar 2026 20:32:49 +0200 Message-ID: <20260329183332.1996183-3-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:30 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19270 From: Adrian Freihofer When unpack_update() detects either of two blocking conditions, it now raises a dedicated exception subclass of UnpackError instead of a generic UnpackError: - LocalModificationsError: for uncommitted changes in the working tree that prevent the update. Includes git status output and instructs the user to commit, stash or discard changes. - RebaseError: for local commits that could not be rebased onto the new upstream revision. Includes the git rebase output and a hint that the 'dldir' remote points to the local download cache and may be used to resolve conflicts manually. Both exceptions take (repodir, url, git_output) as arguments, build the full user-facing message in __init__, and preserve the is-a relationship with UnpackError so existing callers are not broken. The fetch and rebase operations are also split into separate try/except blocks so that a failure to fetch from dldir raises a plain UnpackError while a failed rebase raises RebaseError. Signed-off-by: Adrian Freihofer --- lib/bb/fetch2/__init__.py | 17 +++++++++++++++++ lib/bb/fetch2/git.py | 7 +++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index e7213dea4..d104dfaf3 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -96,6 +96,23 @@ class UnpackError(BBFetchException): BBFetchException.__init__(self, msg) self.args = (message, url) +class LocalModificationsError(UnpackError): + """Exception raised when a checkout cannot be updated due to local modifications""" + def __init__(self, repodir, url, git_output): + message = ("Repository at %s has uncommitted changes, unable to update:\n%s\n" + "Commit, stash or discard your changes and re-run the update." % (repodir, git_output)) + UnpackError.__init__(self, message, url) + self.args = (repodir, url, git_output) + +class RebaseError(UnpackError): + """Exception raised when a checkout has local commits that could not be rebased onto the new upstream revision""" + def __init__(self, repodir, url, git_output): + message = ("Repository at %s has local commits that could not be rebased onto the new upstream revision:\n%s\n" + "Note: the 'dldir' remote points to the local download cache and may be used to resolve the conflict manually.\n" + "Once resolved, re-run the update." % (repodir, git_output)) + UnpackError.__init__(self, message, url) + self.args = (repodir, url, git_output) + class NoMethodError(BBFetchException): """Exception raised when there is no method to obtain a supplied url or set of urls""" def __init__(self, url): diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 645746340..6ed14005b 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -730,7 +730,7 @@ class Git(FetchMethod): output = runfetchcmd("%s status --untracked-files=no --porcelain" % (ud.basecmd), d, workdir=destdir) if output: - raise bb.fetch2.UnpackError("Repository at %s has uncommitted changes, unable to update:\n%s" % (destdir, output), ud.url) + raise bb.fetch2.LocalModificationsError(destdir, ud.url, output) # Set up remote for the download location if it doesn't exist try: @@ -740,6 +740,9 @@ class Git(FetchMethod): runfetchcmd("%s remote add dldir file://%s" % (ud.basecmd, ud.clonedir), d, workdir=destdir) try: runfetchcmd("%s fetch dldir" % (ud.basecmd), d, workdir=destdir) + except bb.fetch2.FetchError as e: + raise bb.fetch2.UnpackError("Failed to fetch from dldir remote: %s" % str(e), ud.url) + try: runfetchcmd("%s rebase --no-autosquash --no-autostash %s" % (ud.basecmd, ud.revision), d, workdir=destdir) except bb.fetch2.FetchError as e: # If rebase failed, abort it @@ -747,7 +750,7 @@ class Git(FetchMethod): runfetchcmd("%s rebase --abort" % (ud.basecmd), d, workdir=destdir) except Exception: pass - raise bb.fetch2.UnpackError("Failed to update checkout in place: %s" % str(e), ud.url) + raise bb.fetch2.RebaseError(destdir, ud.url, str(e)) return True # If there is a tag parameter in the url and we also have a fixed srcrev, check the tag From patchwork Sun Mar 29 18:32:50 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30533FC97EA for ; Sun, 29 Mar 2026 18:34:31 +0000 (UTC) Received: from mta-65-228.siemens.flowmailer.net (mta-65-228.siemens.flowmailer.net [185.136.65.228]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.34708.1774809263055289428 for ; Sun, 29 Mar 2026 11:34:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=aw5n6bjB; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.65.228, mailfrom: fm-1329275-202603291834206f3f2f54ba00020776-bwshzk@rts-flowmailer.siemens.com) Received: by mta-65-228.siemens.flowmailer.net with ESMTPSA id 202603291834206f3f2f54ba00020776 for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=127zOZlTVAH4o2b6Tcyx/iixb6dflLDR8X0TSCHCUL4=; b=aw5n6bjBIsuEreRKJDwOsusKESP0RFpNedKGc8InvmuE8v5qLP/5pQ6Elh31aR17oiQKIi 2simpzhSy6b1NHOtjQvI5JQrR38o2ss79jkQbHhEyIKsKmde0Ja4n/FfYHmVQpY5benhHjU+ q1V5NYHZcOr4IN+U8uIF/jMQXRzjb5DvxV0gjFDEyCqtSLA3cS/ww3h6aMBQVD28wI/Hu5NT WJ+1hw3XGOwjwlesZ+luTYV2aA822kH6OAwYy4BEgwbdgf4CnzHIl6zQR6xW26uFuNDaWI+2 c1LnqkrfK41TTNAjAFEdOnSeH2hYVV2GLlOuOv94jKHFha7H9DarqS6g==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 03/14] bitbake-selftest: add GitUnpackUpdateTest Date: Sun, 29 Mar 2026 20:32:50 +0200 Message-ID: <20260329183332.1996183-4-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19268 From: Adrian Freihofer Add a test class that exercises the new unpack_update() code path in the Git fetcher, ordered from basic building blocks to advanced workflow and error cases: test_unpack_update_full_clone Basic update to a newer upstream revision succeeds and the working tree reflects the new content. test_unpack_update_dldir_remote_setup The "dldir" remote pointing to ud.clonedir is created during the initial unpack and is present for subsequent update calls. test_unpack_update_ff_with_local_changes Full workflow: after a normal unpack the "dldir" remote is verified, a local commit is added, download() brings updated_rev into the clonedir, and unpack_update() fetches from dldir and rebases the local commit fast forward on top. The commit graph (HEAD^ == updated_rev) and both file contents are asserted. test_unpack_update_already_at_target_revision Calling unpack_update() when the checkout is already at SRCREV is a no-op: it succeeds and the working tree is left unchanged. test_unpack_update_with_untracked_file The status check uses --untracked-files=no so untracked files are invisible to it; the update succeeds and the untracked file survives the rebase unchanged. test_unpack_update_with_staged_changes Staged (but not committed) changes are detected by "git status --untracked-files=no --porcelain" and block the update; LocalModificationsError is raised so the caller can fall back to backup + re-fetch. test_unpack_update_with_modified_tracked_file An unstaged modification to a tracked file is detected by "git status --untracked-files=no --porcelain" and blocks the update; LocalModificationsError is raised. test_unpack_update_conflict_raises_rebase_error A local commit that conflicts with the incoming upstream change raises RebaseError; the repository is left in a clean state with no pending rebase (rebase --abort was called). test_unpack_update_untracked_file_overwritten_by_upstream An untracked file that would be overwritten by an incoming upstream commit causes git to refuse the rebase; RebaseError is raised and the repository is not left in a mid-rebase state. Two sub-cases are covered: a top-level file clash and a clash inside a subdirectory (xxx/somefile). test_unpack_update_shallow_clone_fails Shallow clones do not carry enough history; UnpackError is raised. test_unpack_update_stale_dldir_remote When the clonedir has been removed after the initial unpack the dldir remote no longer resolves; UnpackError is raised so the caller can fall back to a full re-fetch. test_fetch_unpack_update_toplevel_api The public Fetch.unpack_update(root) API (used by callers such as bitbake-setup) dispatches correctly end-to-end through to the Git fetcher. Signed-off-by: Adrian Freihofer --- lib/bb/tests/fetch.py | 559 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 559 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 7b8297a78..1d018671a 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -3793,3 +3793,562 @@ class GoModGitTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod'))) self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod')), '0dc9ccc660ad21cebaffd548f2cc6efa27891c68b4fbc1f8a3893b00f1acec96') + + +class GitUnpackUpdateTest(FetcherTest): + """Test the unpack_update functionality for git fetcher. + + Intended workflow + 1. First-time setup: + 1. download() - clones the upstream repo into DL_DIR/git2/... (clonedir). + 2. unpack() - clones from clonedir into the workspace (S/workdir) and + registers a 'dldir' git remote pointing at + file://DL_DIR/git2/... for later offline use. + + 2. Subsequent updates (what unpack_update is designed for): + 1. The user works in the unpacked source tree. + 2. Upstream advances - SRCREV changes in the recipe. + 3. download() - fetches the new revision into the local clonedir. + 4. unpack_update() - instead of wiping the workspace and re-cloning: + * fetches the new revision from the local 'dldir' remote + * rebases the user's local commits on top of the new SRCREV + * raises LocalModificationsError if uncommitted changes block the + update, RebaseError if local commits cannot be rebased, or a + plain UnpackError for other failures (shallow clone, stale dldir); + in all cases the caller (e.g. bitbake-setup) can fall back to + backup + re-clone. + + Key design constraints: + * unpack_update() never deletes existing data (unlike unpack()). + * Only staged/modified tracked files block the update; untracked files and + committed local work are handled gracefully. + * The 'dldir' remote is intentionally visible to users outside the + fetcher (e.g. for manual 'git log dldir/master'). + * Currently only git is supported. + """ + + def setUp(self): + """Set up a local bare git source repository with two commits on 'master'. + + self.initial_rev - the first commit (testfile.txt: 'initial content') + self.updated_rev - the second commit (testfile.txt: 'updated content') + + SRCREV is initialised to self.initial_rev so individual tests can + advance it to self.updated_rev (or create further commits) as needed. + """ + FetcherTest.setUp(self) + + self.gitdir = os.path.join(self.tempdir, 'gitrepo') + self.srcdir = os.path.join(self.tempdir, 'gitsource') + + self.d.setVar('WORKDIR', self.tempdir) + self.d.setVar('S', self.gitdir) + self.d.delVar('PREMIRRORS') + self.d.delVar('MIRRORS') + + # Create a source git repository + bb.utils.mkdirhier(self.srcdir) + self.git_init(cwd=self.srcdir) + + # Create initial commit + with open(os.path.join(self.srcdir, 'testfile.txt'), 'w') as f: + f.write('initial content\n') + self.git(['add', 'testfile.txt'], cwd=self.srcdir) + self.git(['commit', '-m', 'Initial commit'], cwd=self.srcdir) + self.initial_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip() + + # Create a second commit + with open(os.path.join(self.srcdir, 'testfile.txt'), 'w') as f: + f.write('updated content\n') + self.git(['add', 'testfile.txt'], cwd=self.srcdir) + self.git(['commit', '-m', 'Update commit'], cwd=self.srcdir) + self.updated_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip() + + self.d.setVar('SRCREV', self.initial_rev) + self.d.setVar('SRC_URI', 'git://%s;branch=master;protocol=file' % self.srcdir) + + def test_unpack_update_full_clone(self): + """Test that unpack_update updates an existing checkout in place for a full clone. + + Steps: + 1. Fetch and unpack at self.initial_rev - verify 'initial content'. + 2. Advance SRCREV to self.updated_rev and re-download. + 3. Call unpack_update() instead of unpack() - the existing checkout + must be updated via 'git fetch dldir' + 'git rebase' without + re-cloning the directory. + 4. Verify testfile.txt now contains 'updated content'. + """ + # First fetch at initial revision + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + # Verify initial state + unpack_path = os.path.join(self.unpackdir, 'git') + self.assertTrue(os.path.exists(os.path.join(unpack_path, 'testfile.txt'))) + with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'initial content\n') + + # Update to new revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + # Use unpack_update + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + # Verify updated state + with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'updated content\n') + + def test_unpack_update_dldir_remote_setup(self): + """Test that unpack() adds a 'dldir' git remote pointing at ud.clonedir. + + The 'dldir' remote is used by subsequent unpack_update() calls to fetch + new commits from the local download cache (${DL_DIR}/git2/…) without + requiring network access. After a normal unpack the remote must exist + and its URL must be 'file://'. + """ + # First fetch + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Check that dldir remote exists + remotes = self.git(['remote'], cwd=unpack_path).strip().split('\n') + self.assertIn('dldir', remotes) + + # Verify it points to the clonedir + dldir_url = self.git(['remote', 'get-url', 'dldir'], cwd=unpack_path).strip() + self.assertEqual(dldir_url, 'file://{}'.format(ud.clonedir)) + + def test_unpack_update_ff_with_local_changes(self): + """Test that unpack_update rebases local commits fast forward. + + Full workflow: + 1. Fetch + unpack at initial_rev - verify 'dldir' remote is created + pointing at ud.clonedir. + 2. Add a local commit touching localfile.txt. + 3. Advance SRCREV to updated_rev and call download() - verify that + ud.clonedir (the dldir bare clone) now contains updated_rev. + 4. Call unpack_update() - it fetches updated_rev from dldir into the + working tree and rebases the local commit on top. + 5. Verify the final commit graph: HEAD's parent is updated_rev, and + both testfile.txt ('updated content') and localfile.txt ('local + change') are present. + + Note: git rebase operates the same way regardless of whether HEAD is + detached or on a named branch (e.g. 'master' or a local feature branch), + so this test covers those scenarios implicitly. + """ + # Step 1 - fetch + unpack at initial_rev + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + unpack_path = os.path.join(self.unpackdir, 'git') + + # The normal unpack must have set up the 'dldir' remote pointing at + # ud.clonedir so that subsequent unpack_update() calls work offline. + dldir_url = self.git(['remote', 'get-url', 'dldir'], cwd=unpack_path).strip() + self.assertEqual(dldir_url, 'file://{}'.format(ud.clonedir)) + + # Step 2 - add a local commit that touches a new file + with open(os.path.join(unpack_path, 'localfile.txt'), 'w') as f: + f.write('local change\n') + self.git(['add', 'localfile.txt'], cwd=unpack_path) + self.git(['commit', '-m', 'Local commit'], cwd=unpack_path) + local_commit = self.git(['rev-parse', 'HEAD'], cwd=unpack_path).strip() + + # Step 3 - advance SRCREV and download; clonedir must now contain + # updated_rev so that unpack_update can fetch it without network access. + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + ud = fetcher.ud[uri] + clonedir_refs = self.git(['rev-parse', self.updated_rev], cwd=ud.clonedir).strip() + self.assertEqual(clonedir_refs, self.updated_rev, + "clonedir must contain updated_rev after download()") + + # Step 4 - unpack_update fetches from dldir and rebases + git_fetcher = ud.method + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + # Step 5 - verify the commit graph and working tree content + # HEAD is the rebased local commit; its parent must be updated_rev + head_rev = self.git(['rev-parse', 'HEAD'], cwd=unpack_path).strip() + parent_rev = self.git(['rev-parse', 'HEAD^'], cwd=unpack_path).strip() + self.assertNotEqual(head_rev, local_commit, + "local commit should have a new SHA after rebase") + self.assertEqual(parent_rev, self.updated_rev, + "HEAD's parent must be updated_rev after fast-forward rebase") + + with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'updated content\n') + with open(os.path.join(unpack_path, 'localfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'local change\n') + + def test_unpack_update_already_at_target_revision(self): + """Test that unpack_update is a no-op when the checkout is already at SRCREV. + + Calling unpack_update() without advancing SRCREV must succeed and leave + the working tree unchanged. No rebase should be attempted because the + checkout already points at ud.revision. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + with open(os.path.join(unpack_path, 'testfile.txt')) as f: + self.assertEqual(f.read(), 'initial content\n') + + # Call unpack_update with SRCREV still at initial_rev - no upstream change + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + result = git_fetcher.unpack_update(ud, self.unpackdir, self.d) + self.assertTrue(result) + + # Content must be unchanged + with open(os.path.join(unpack_path, 'testfile.txt')) as f: + self.assertEqual(f.read(), 'initial content\n') + + def test_unpack_update_with_untracked_file(self): + """Test that unpack_update succeeds when the checkout has an untracked file. + + The status check uses '--untracked-files=no', so untracked files are not + detected and do not trigger the fallback path. git rebase also leaves + untracked files untouched, so both the upstream update and the untracked + file must be present after the call. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Create an untracked file (not staged, not committed) + untracked = os.path.join(unpack_path, 'untracked.txt') + with open(untracked, 'w') as f: + f.write('untracked content\n') + + # Update to new upstream revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # --untracked-files=no means the status check passes; rebase preserves the file + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f: + self.assertEqual(f.read(), 'updated content\n') + + # Untracked file must survive the rebase + self.assertTrue(os.path.exists(untracked)) + with open(untracked, 'r') as f: + self.assertEqual(f.read(), 'untracked content\n') + + def test_unpack_update_with_staged_changes(self): + """Test that unpack_update fails when the checkout has staged (but not committed) changes. + + The rebase is run with --no-autostash so git refuses to rebase over a + dirty index. The caller (bitbake-setup) is expected to catch the + resulting LocalModificationsError and fall back to backup + re-fetch. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Stage a new file without committing it + staged = os.path.join(unpack_path, 'staged.txt') + with open(staged, 'w') as f: + f.write('staged content\n') + self.git(['add', 'staged.txt'], cwd=unpack_path) + + # Update to new upstream revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # Should fail - git rebase refuses to run with a dirty index + with self.assertRaises(bb.fetch2.LocalModificationsError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + def test_unpack_update_with_modified_tracked_file(self): + """Test that unpack_update fails when a tracked file has unstaged modifications. + + 'git status --untracked-files=no --porcelain' reports unstaged modifications + to tracked files (output line ' M filename'), which must block the update so + the caller can fall back to backup + re-fetch rather than silently discarding + work in progress. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Modify a tracked file without staging or committing + with open(os.path.join(unpack_path, 'testfile.txt'), 'w') as f: + f.write('locally modified content\n') + + # Update to new upstream revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # Should fail - unstaged modification to tracked file is detected by + # 'git status --untracked-files=no --porcelain' + with self.assertRaises(bb.fetch2.LocalModificationsError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + def test_unpack_update_conflict_raises_rebase_error(self): + """Test that unpack_update raises RebaseError on a rebase conflict. + + When a local commit modifies the same lines as an incoming upstream commit, + git rebase cannot resolve the conflict automatically. unpack_update must + abort the failed rebase and raise RebaseError so the caller can fall back + to a backup + re-fetch. + """ + # Fetch and unpack at the initial revision + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Make a local commit that edits the same lines as the upcoming upstream commit + with open(os.path.join(unpack_path, 'testfile.txt'), 'w') as f: + f.write('conflicting local content\n') + self.git(['add', 'testfile.txt'], cwd=unpack_path) + self.git(['commit', '-m', 'Local conflicting commit'], cwd=unpack_path) + + # Add a third upstream commit that also edits testfile.txt differently + with open(os.path.join(self.srcdir, 'testfile.txt'), 'w') as f: + f.write('conflicting upstream content\n') + self.git(['add', 'testfile.txt'], cwd=self.srcdir) + self.git(['commit', '-m', 'Upstream conflicting commit'], cwd=self.srcdir) + conflict_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip() + + # Update SRCREV to the new upstream commit + self.d.setVar('SRCREV', conflict_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # unpack_update must fail and clean up (rebase --abort) rather than + # leaving the repo in a mid-rebase state + with self.assertRaises(bb.fetch2.RebaseError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + # Verify the repo is not left in a conflicted / mid-rebase state + rebase_merge = os.path.join(unpack_path, '.git', 'rebase-merge') + rebase_apply = os.path.join(unpack_path, '.git', 'rebase-apply') + self.assertFalse(os.path.exists(rebase_merge), + "rebase-merge dir should not exist after failed unpack_update") + self.assertFalse(os.path.exists(rebase_apply), + "rebase-apply dir should not exist after failed unpack_update") + + def test_unpack_update_untracked_file_overwritten_by_upstream(self): + """Test that unpack_update raises RebaseError when an untracked file would be + overwritten by an incoming upstream commit. + + We skip untracked files in the pre-check (git rebase doesn't touch harmless + untracked files), but git itself refuses to rebase when an untracked file would + be overwritten by the incoming changes. The resulting FetchError must be caught + and re-raised as RebaseError without leaving the repo in a mid-rebase state. + + Two sub-cases are covered: + - top-level untracked file clashing with an incoming upstream file + - untracked file inside a subdirectory (xxx/somefile) clashing with an + upstream commit that adds the same path + """ + def _run_case(upstream_path, local_rel_path, commit_msg): + """ + Add upstream_path to self.srcdir, create local_rel_path as an + untracked file in the checkout, then assert that unpack_update + raises RebaseError and leaves no mid-rebase state, and that the + local file is untouched. + """ + # Fresh fetch + unpack at the current SRCREV + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Upstream adds the file (potentially inside a subdirectory) + full_upstream = os.path.join(self.srcdir, upstream_path) + os.makedirs(os.path.dirname(full_upstream), exist_ok=True) + with open(full_upstream, 'w') as f: + f.write('upstream content\n') + self.git(['add', upstream_path], cwd=self.srcdir) + self.git(['commit', '-m', commit_msg], cwd=self.srcdir) + new_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip() + + # Create the clashing untracked file in the checkout + full_local = os.path.join(unpack_path, local_rel_path) + os.makedirs(os.path.dirname(full_local), exist_ok=True) + with open(full_local, 'w') as f: + f.write('local untracked content\n') + + self.d.setVar('SRCREV', new_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + # git rebase refuses because the untracked file would be overwritten + with self.assertRaises(bb.fetch2.RebaseError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + # Repo must not be left in a mid-rebase state + self.assertFalse(os.path.exists(os.path.join(unpack_path, '.git', 'rebase-merge'))) + self.assertFalse(os.path.exists(os.path.join(unpack_path, '.git', 'rebase-apply'))) + + # The local untracked file must be untouched + self.assertTrue(os.path.exists(full_local)) + with open(full_local) as f: + self.assertEqual(f.read(), 'local untracked content\n') + + # Reset unpackdir for the next sub-case + import shutil as _shutil + _shutil.rmtree(self.unpackdir) + os.makedirs(self.unpackdir) + + # Sub-case 1: top-level file clash + _run_case('newfile.txt', 'newfile.txt', + 'Upstream adds newfile.txt') + + # Sub-case 2: file inside a subdirectory (xxx/somefile) + _run_case('xxx/somefile.txt', 'xxx/somefile.txt', + 'Upstream adds xxx/somefile.txt') + + def test_unpack_update_shallow_clone_fails(self): + """Test that unpack_update raises UnpackError for shallow-tarball checkouts. + + Shallow clones lack full history, which makes an in-place rebase impossible + without network access. After fetching with BB_GIT_SHALLOW=1 the clonedir + is deleted so that unpack() is forced to use the shallow tarball. + A subsequent call to unpack_update() must raise UnpackError and the message + must mention 'shallow clone' so callers can distinguish this case. + """ + self.d.setVar('BB_GIT_SHALLOW', '1') + self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') + + # First fetch at initial revision + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + # Remove clonedir to force use of shallow tarball + clonedir = os.path.join(self.dldir, 'git2') + if os.path.exists(clonedir): + shutil.rmtree(clonedir) + + fetcher.unpack(self.unpackdir) + + # Update to new revision + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + # unpack_update should fail for shallow clones + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + git_fetcher = ud.method + + with self.assertRaises(bb.fetch2.UnpackError) as context: + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + self.assertIn("shallow clone", str(context.exception).lower()) + + def test_unpack_update_stale_dldir_remote(self): + """Test that unpack_update raises UnpackError when the dldir remote URL is stale. + + If the clonedir has been removed after the initial unpack (e.g. DL_DIR was + cleaned) the 'dldir' remote URL no longer resolves. The fetch inside + update_mode will fail with a FetchError which must be re-raised as + UnpackError so the caller can fall back to a full re-fetch. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + + # Advance SRCREV to trigger update_mode + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + uri = self.d.getVar('SRC_URI') + ud = fetcher.ud[uri] + + # Delete the clonedir and corrupt the dldir remote URL so that + # 'git fetch dldir' fails, simulating a missing or relocated DL_DIR. + shutil.rmtree(ud.clonedir) + self.git(['remote', 'set-url', 'dldir', 'file://' + ud.clonedir], + cwd=unpack_path) + + git_fetcher = ud.method + with self.assertRaises(bb.fetch2.UnpackError): + git_fetcher.unpack_update(ud, self.unpackdir, self.d) + + def test_fetch_unpack_update_toplevel_api(self): + """Test that the top-level Fetch.unpack_update() dispatches to Git.unpack_update(). + + Callers such as bitbake-setup use fetcher.unpack_update(root) rather than + calling the method on the Git fetcher directly. Verify that the public API + works end-to-end: fetch at initial_rev, unpack, advance to updated_rev, + fetch again, then call fetcher.unpack_update(root) and confirm the content + is updated. + """ + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + + unpack_path = os.path.join(self.unpackdir, 'git') + with open(os.path.join(unpack_path, 'testfile.txt')) as f: + self.assertEqual(f.read(), 'initial content\n') + + self.d.setVar('SRCREV', self.updated_rev) + fetcher = bb.fetch2.Fetch([self.d.getVar('SRC_URI')], self.d) + fetcher.download() + + # Use the public Fetch.unpack_update() rather than the method directly + fetcher.unpack_update(self.unpackdir) + + with open(os.path.join(unpack_path, 'testfile.txt')) as f: + self.assertEqual(f.read(), 'updated content\n') From patchwork Sun Mar 29 18:32:51 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59DAAFD0052 for ; Sun, 29 Mar 2026 18:34:30 +0000 (UTC) Received: from mta-65-225.siemens.flowmailer.net (mta-65-225.siemens.flowmailer.net [185.136.65.225]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.34483.1774809263057676862 for ; Sun, 29 Mar 2026 11:34:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=E08vHLEV; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.65.225, mailfrom: fm-1329275-20260329183420369d5ed52200020719-mfrge6@rts-flowmailer.siemens.com) Received: by mta-65-225.siemens.flowmailer.net with ESMTPSA id 20260329183420369d5ed52200020719 for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=gabGuLbbibLPYOMUoZ3amckwHUWnrOh/pQU8zTZjIXc=; b=E08vHLEVmm/SqgvWAeWt6nWkYk403qg1F2Fe9BuxMmcmfsxmLqs7TZH3f1g0OChk0j8k+y OQx3RdDhz+XI/dZ60AV2XJyLLQ/urP4vnxVdxUCNfUwuYcopsHebQfsuMywUVrfQQASxicmV ypHBAcytnSfGj/nqLrrf3P5O4c1SI41YQJ/WVLI/6ip5VRerNkT117+yNRUk+rND15PpHxfW ftch4whRNdN2uRGBLXzgMBVxKQOooKmsKfTFOpdZ+W/CRy2F0JJRW2u0wzJ7HdDwTILV3DSK Xf3Rysi5+1jh0cO3aV+taEn27Danj/+Bn8+FzhLLuNFPvbdUshBzIlqA==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 04/14] tests/fetch: remove unused import, fix trailing whitespace Date: Sun, 29 Mar 2026 20:32:51 +0200 Message-ID: <20260329183332.1996183-5-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:30 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19274 From: Adrian Freihofer Just cleanup, no functional change. Signed-off-by: Adrian Freihofer --- lib/bb/tests/fetch.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 1d018671a..5b3fc8a41 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -18,7 +18,6 @@ import os import signal import tarfile from bb.fetch2 import URI -from bb.fetch2 import FetchMethod import bb import bb.utils from bb.tests.support.httpserver import HTTPService @@ -551,8 +550,8 @@ class MirrorUriTest(FetcherTest): fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d) mirrors = bb.fetch2.mirror_from_string(mirrorvar) uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) - self.assertEqual(uris, ['file:///somepath/downloads/bitbake-1.0.tar.gz', - 'file:///someotherpath/downloads/bitbake-1.0.tar.gz', + self.assertEqual(uris, ['file:///somepath/downloads/bitbake-1.0.tar.gz', + 'file:///someotherpath/downloads/bitbake-1.0.tar.gz', 'http://otherdownloads.yoctoproject.org/downloads/bitbake-1.0.tar.gz', 'http://downloads2.yoctoproject.org/downloads/bitbake-1.0.tar.gz']) @@ -1390,7 +1389,7 @@ class URLHandle(unittest.TestCase): "https://somesite.com/somerepo.git;user=anyUser:idtoken=1234" : ('https', 'somesite.com', '/somerepo.git', '', '', {'user': 'anyUser:idtoken=1234'}), 'git://s.o-me_ONE:%s@git.openembedded.org/bitbake;branch=main;protocol=https' % password: ('git', 'git.openembedded.org', '/bitbake', 's.o-me_ONE', password, {'branch': 'main', 'protocol' : 'https'}), } - # we require a pathname to encodeurl but users can still pass such urls to + # we require a pathname to encodeurl but users can still pass such urls to # decodeurl and we need to handle them decodedata = datatable.copy() decodedata.update({ From patchwork Sun Mar 29 18:32:52 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4698FFC97EE for ; Sun, 29 Mar 2026 18:34:31 +0000 (UTC) Received: from mta-65-225.siemens.flowmailer.net (mta-65-225.siemens.flowmailer.net [185.136.65.225]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.34479.1774809263056646482 for ; Sun, 29 Mar 2026 11:34:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=Ko4ujvs+; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.65.225, mailfrom: fm-1329275-20260329183420abf5bff7c90002075e-o_ph0s@rts-flowmailer.siemens.com) Received: by mta-65-225.siemens.flowmailer.net with ESMTPSA id 20260329183420abf5bff7c90002075e for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=u9P8iHjY758z1s97xk1S5g+9p+0GWZEJkZE1QI9MrqA=; b=Ko4ujvs+w7E6fePlIQpD4ogz2FcScRNBNUqY54Pzw5m0MgwougFVXKE8LR6r9O+UyUtX5e wDSG6Gh/btXMcNjgiC7RtEkyhaT7k9DZuLC7yHgHTxCAUYkeT3LYysDey34aA76yZ5cFZQVB uEjWzQyiqhogjO57BnDhfpl3LLKv7DelfYAzEUBZwIHJg0qgvD6jR418VOyXp91FJClOKLBe z3++PKjYcQq2XlCklUqR7clBFCmyBraonNWhUko5zWgKw4xudQPtV7XJ33nfPyqRQl+oy1Yp IzxpdQE9951HqBktoI771nECXGtYkcj7q+Mseh0wJeOe3o5b/L1w12Qw==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 05/14] doc: document the unpack_update() non-destructive git update method Date: Sun, 29 Mar 2026 20:32:52 +0200 Message-ID: <20260329183332.1996183-6-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19273 From: Adrian Freihofer Add a new "The Non-Destructive Update (unpack_update)" section to the fetching chapter, describing the in-place update mode introduced in "fetch/git: Add an 'update' unpack mode to the fetchers (git only for now)" e7d5e156275782948d3346f299780389ab263ab6. The section covers: - what unpack_update() does and when to use it vs. unpack() - the three-step internal process: add/update the dldir remote, fetch the new upstream revision, rebase local commits on top - the error conditions: LocalModificationsError (uncommitted changes), RebaseError (conflicting local commits), and the shallow/stale-dldir case - the current limitation that only the Git fetcher supports it Signed-off-by: Adrian Freihofer --- .../bitbake-user-manual-fetching.rst | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index 6af803591..c2747c401 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -186,6 +186,51 @@ cloning the tree into the final directory. The process is completed using references so that there is only one central copy of the Git metadata needed. +.. _bb-the-unpack-update: + +The Non-Destructive Update (``unpack_update``) +============================================== + +.. note:: + + This is a specialised method intended for tools that manage persistent + layer checkouts on behalf of the user, such as + :ref:`bitbake-setup update `. It is not part of the + normal recipe fetch/unpack flow. + +For Git URLs, an alternative ``unpack_update()`` method is available +that updates an existing checkout *in place* rather than removing and +re-cloning it. This is useful when the target directory may contain +local commits that should be preserved across updates. + +The code to call the non-destructive update looks like the following:: + + rootdir = l.getVar('UNPACKDIR') + fetcher.unpack_update(rootdir) + +``unpack_update()`` performs the following steps: + +1. A ``dldir`` git remote is added (or updated) in the existing + checkout, pointing at the local download cache. This remote may + also be useful for manually resolving conflicts outside the fetcher. +2. The new upstream revision is fetched from the ``dldir`` remote into + the existing repository. +3. Any local commits are rebased on top of the new upstream revision, + preserving local work. + +The method raises an error if: + +- The working tree contains staged or unstaged changes to tracked + files (``LocalModificationsError``). +- Local commits cannot be cleanly rebased onto the new upstream + revision (``RebaseError``). A failed rebase is automatically aborted + before the exception is raised. +- The download cache does not contain a sufficiently recent clone + of the repository, or the checkout is a shallow clone. + +Currently only the Git fetcher supports ``unpack_update()``. All other +fetcher types raise ``RuntimeError`` if it is called. + .. _bb-fetchers: Fetchers From patchwork Sun Mar 29 18:32:53 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B160FC97E2 for ; Sun, 29 Mar 2026 18:34:30 +0000 (UTC) Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net [185.136.65.226]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.34480.1774809263056843324 for ; Sun, 29 Mar 2026 11:34:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=bTVKkeWn; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.65.226, mailfrom: fm-1329275-20260329183420db6e8cd13600020707-wqrymk@rts-flowmailer.siemens.com) Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 20260329183420db6e8cd13600020707 for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=HAq7eXLFjZheEakXRqCPGzR8luram90pCOe2hsMqG9I=; b=bTVKkeWn7jwgY3rtP6ktpiepEj9InmiR8vDotEZXEfDZ5I0uk+pZhp41Au2ONH7UxtE95L PoCRcC3TY3hu74654wE8C1Zq7zg+YPfkDzzv5ens9IBJ/ukt7aDJvqlENGbeL3RBuaPhrW64 JmZBgSpJPYXNPdLmh4n+FrVi7gJdjIpWTkke03d8be105x8pOK9/nZkENxiOXPNUzhqCF9fZ Llw6qxRES1XpZJp7QOHusR6mjo5yF8UqoBW9ee8LpquItlbbqdHI+M+vN1YtGpWRgIGw2lYD 12LEtpqp3Yma1UFn++00hH9qOoFfvwm+VVAJRlZ1pZ8k5ygxm9xpAeIQ==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 06/14] bitbake-setup: always restore sys.stdout Date: Sun, 29 Mar 2026 20:32:53 +0200 Message-ID: <20260329183332.1996183-7-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:30 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19272 From: Adrian Freihofer While working on the bitbake-setup update with a non destructive fetcher, I noticed that if the fetcher raises an exception, sys.stdout is not restored, which can lead to issues in the rest of the code. This change ensures that sys.stdout is always restored, even if an exception occurs during the fetch and unpack process. Showing the full Traceback of the error would be a bit confusing because it happened in code which is not yet ready for review, but the final error was: ValueError: I/O operation on closed file. and this little change seems to be a reasonable improvement to avoid such issues. Signed-off-by: Adrian Freihofer --- bin/bitbake-setup | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index f0240cae5..335927dae 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -794,9 +794,11 @@ def do_fetch(fetcher, dir): with open(fetchlog, 'a') as f: oldstdout = sys.stdout sys.stdout = f - fetcher.download() - fetcher.unpack_update(dir) - sys.stdout = oldstdout + try: + fetcher.download() + fetcher.unpack_update(dir) + finally: + sys.stdout = oldstdout def update_registry(registry, cachedir, d): registrydir = 'configurations' From patchwork Sun Mar 29 18:32:54 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 476C1FC97EF for ; Sun, 29 Mar 2026 18:34:31 +0000 (UTC) Received: from mta-64-227.siemens.flowmailer.net (mta-64-227.siemens.flowmailer.net [185.136.64.227]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.34713.1774809263056144717 for ; Sun, 29 Mar 2026 11:34:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=PWeUFVIj; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.64.227, mailfrom: fm-1329275-202603291834201dc8a9c48a00020736-gdxpdc@rts-flowmailer.siemens.com) Received: by mta-64-227.siemens.flowmailer.net with ESMTPSA id 202603291834201dc8a9c48a00020736 for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=ZvsrtByuHhRGhYvFKB/jsVqT3GPrbDYzgxb3raFvR1M=; b=PWeUFVIj0emmJyMKBcHN6bzB8DyFo1zdGmaDp/Vj2TV/Hby2164vfMghKI+lcAlvPRELOe s14dwp2BGwOWmtWdQI08b6QlJUIopRfPdajM69vXPaREmtmi+7m6G4zYB6TJhh2FmpVZW46a dif2ipSEgyU4SDVSJXJsjU5XQEzZXBo4Q4NPbDO9eD4/F/apdTrLYTDL9YjCaT1Ae3ccTdbV omtSbAISQh1L9Aaq2Fs3hr/vFEaemtgd92nYVXwoX10715wYlukeH1M2uLHITIsaskWiM6+m wrlAf6XZGy6SWDRkk57eNlarFJwMtGOubZ6UI9pEIKUmIbdM7MNoBinw==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 07/14] tests/setup: cleanup imports Date: Sun, 29 Mar 2026 20:32:54 +0200 Message-ID: <20260329183332.1996183-8-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19267 From: Adrian Freihofer Add missing top-level imports for os, stat, bb and bb.process and remove the redundant inline 'import os' and 'import stat' that were inside individual methods. Signed-off-by: Adrian Freihofer --- lib/bb/tests/setup.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index a66f05b36..d52a81395 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -5,9 +5,13 @@ # from bb.tests.fetch import FetcherTest -import json -import hashlib +import bb +import bb.process import glob +import hashlib +import json +import os +import stat from bb.tests.support.httpserver import HTTPService class BitbakeSetupTest(FetcherTest): @@ -208,7 +212,6 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) with open(fullname, 'w') as f: f.write(content) if script: - import stat st = os.stat(fullname) os.chmod(fullname, st.st_mode | stat.S_IEXEC) self.git('add {}'.format(name), cwd=self.testrepopath) @@ -279,7 +282,6 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) def test_setup(self): # unset BBPATH to ensure tests run in isolation from the existing bitbake environment - import os if 'BBPATH' in os.environ: del os.environ['BBPATH'] From patchwork Sun Mar 29 18:32:55 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F0CFFC97E8 for ; Sun, 29 Mar 2026 18:34:31 +0000 (UTC) Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net [185.136.64.228]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.34481.1774809263057265018 for ; Sun, 29 Mar 2026 11:34:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=O71HaIF7; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.64.228, mailfrom: fm-1329275-20260329183420b82c7a7855000207f5-6byvbg@rts-flowmailer.siemens.com) Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 20260329183420b82c7a7855000207f5 for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=tHatzJkj8v41S00Pvw0rYn2RiqmtE0gHkGSd5U3RR+E=; b=O71HaIF7NZ145WHsgshFw09pivT/NAMECMZ5598U1grj5MOwXlbD7qWgog9cGdo7c8Y9Vn VN2LGVweV3KdstWKHrbuEUGzFzwfdwhj2gOuUkDITeZlYXSRvoX1mbxKM/kxQCKTRW2RvcqV iVs67i3CY0WDJO7rJKBiFbqRyGBk6jMAv4xUwLWBeDztaM70agXbE7Ec/R2X5MXvrrMtQf9g EZFAcr9LVotvxpX4Bh/Y7XBM5HEbTHCfIe7pdJI+WHq7SBZBduUj6TlE77WIZUxxPszQ24cu agJ9HJvHFUgX/VpeC0+7OPjcEf02sP4Mkra8t7n2HY7P71z77vI8/eZQ==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 08/14] tests/setup: fix dead check_setupdir_files guards Date: Sun, 29 Mar 2026 20:32:55 +0200 Message-ID: <20260329183332.1996183-9-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19277 From: Adrian Freihofer Two conditions in check_setupdir_files used wrong key names and so never triggered: - 'oe-fragment' -> 'oe-fragments' (plural): fragment-existence assertions were never reached for any variant that has fragments. - 'bb-environment-passthrough' -> 'bb-env-passthrough-additions': BB_ENV_PASSTHROUGH_ADDITIONS assertions were never reached for the gizmo-env-passthrough variant. Also drop the redundant .keys() call on both guards. Signed-off-by: Adrian Freihofer --- lib/bb/tests/setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index d52a81395..1af3d8b50 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -260,11 +260,11 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) ) self.assertIn(filerelative_layer, bblayers) - if 'oe-fragment' in bitbake_config.keys(): + if 'oe-fragments' in bitbake_config: for f in bitbake_config["oe-fragments"]: self.assertTrue(os.path.exists(os.path.join(bb_conf_path, f))) - if 'bb-environment-passthrough' in bitbake_config.keys(): + if 'bb-env-passthrough-additions' in bitbake_config: with open(os.path.join(bb_build_path, 'init-build-env'), 'r') as f: init_build_env = f.read() self.assertTrue('BB_ENV_PASSTHROUGH_ADDITIONS' in init_build_env) From patchwork Sun Mar 29 18:32:56 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A371F3D61F for ; Sun, 29 Mar 2026 18:34:30 +0000 (UTC) Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net [185.136.64.226]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.34482.1774809263057518081 for ; Sun, 29 Mar 2026 11:34:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=aoEn/zwg; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.64.226, mailfrom: fm-1329275-20260329183420113e28acb000020767-oikszb@rts-flowmailer.siemens.com) Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 20260329183420113e28acb000020767 for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=bVJEMd+YZNNfSmAr28g+BXoH8DgRJqlIR/7JLIIume8=; b=aoEn/zwgMO4bPb6QpNNV3kQeXNl1FpxRaF7HWXEr/VPh5DDVsXtH5wWZsQAtJGH6m50nBD Uklii18zzGsQWSgM8QvXeqGXtzVoUm+FdEB50rzqelcG2m2Zu9qM7FB6coUiWTetIKNBo1c2 u/PAsAikIf2/mik46WPz6A6Pc72oD0Y1BpnHQSV+TZcg/S4OBNImw6oT5VaXhFjNtL7UVJYz utFuIT3JddjyDKux2+xnrdqh/jRyPmrR9ufoNBFsiz8wK08ML6ZEqut+tSG1ITJeEuZpM9lW BxbV4EEpfdbuuQaZIXoxOxsqltU7RwFyp9wAytMJJ6ETqw3LSivCfM0w==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 09/14] bitbake-setup: generate config files for VSCode Date: Sun, 29 Mar 2026 20:32:56 +0200 Message-ID: <20260329183332.1996183-10-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:30 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19276 From: Adrian Freihofer This change introduces a function to generate a VSCode workspace file (bitbake.code-workspace). The --init-vscode flag added to bitbake-setup init defaults to True when the code binary is found on PATH, and can be passed explicitly to exercise the feature on machines without code (e.g. when running tests on an autobuilder). Once the workspace file exists, it is updated automatically on every subsequent run. This workspace file is preferred over a project-specific .vscode/settings.json for several reasons: - It allows for a multi-root workspace, which is ideal for a bitbake project structure setup with bitbake-setup. This enables including all layer repositories and the build configuration directory as top-level folders in the explorer. - The workspace file can be located at the top level of the setup, outside of any version-controlled source directory. This avoids cluttering the git repositories with editor-specific configuration. - It provides a centralized place for all VSCode settings related to the project, including those for the bitbake extension, Python language server, and file associations, ensuring a consistent development environment for all users of the project. The Python analysis paths (python.analysis.extraPaths) are configured with absolute paths. This is a workaround for a limitation in the Pylance extension, which does not correctly resolve ${workspaceFolder:...} variables in a multi-root workspace context for import resolution. Using absolute paths ensures that Pylance can find all necessary modules from the various layers. There is room for improvement. Starting a terminal (bitbake or any other) is cumbersome, as VSCode wants to start it for one of the layers rather than the build directory. Signed-off-by: Adrian Freihofer --- bin/bitbake-setup | 175 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 168 insertions(+), 7 deletions(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index 335927dae..ce32c3f13 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -240,7 +240,7 @@ bitbake-setup init -L {} /path/to/repo/checkout""".format( return layers_fixed_revisions -def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf): +def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf, init_vscode=False): def _setup_build_conf(layers, filerelative_layers, build_conf_dir): os.makedirs(build_conf_dir) layers_s = [] @@ -359,6 +359,7 @@ def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_c init_script = os.path.join(bitbake_builddir, "init-build-env") + workspace_file = os.path.join(setupdir, "bitbake.code-workspace") shell = "bash" fragments = bitbake_config.get("oe-fragments", []) + sorted(bitbake_config.get("oe-fragment-choices",{}).values()) if fragments: @@ -370,6 +371,8 @@ def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_c logger.plain('New bitbake configuration from upstream is the same as the current one, no need to update it.') shutil.rmtree(bitbake_confdir) os.rename(backup_bitbake_confdir, bitbake_confdir) + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script) return logger.plain('Upstream bitbake configuration changes were found:') @@ -385,6 +388,8 @@ def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_c logger.plain(f'Leaving the upstream configuration in {upstream_bitbake_confdir}') os.rename(bitbake_confdir, upstream_bitbake_confdir) os.rename(backup_bitbake_confdir, bitbake_confdir) + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script) return logger.plain('Applying upstream bitbake configuration changes') @@ -392,17 +397,21 @@ def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_c fragment_note = "Run 'bitbake-config-build enable-fragment ' to enable additional fragments or replace built-in ones (e.g. machine/ or distro/ to change MACHINE or DISTRO)." + readme_extra = "" + if init_vscode: + readme_extra = "\n\nTo edit the code in VSCode, open the workspace: code {}\n".format(workspace_file) + readme = """{}\n\nAdditional information is in {} and {}\n Source the environment using '. {}' to run builds from the command line.\n {}\n -The bitbake configuration files (local.conf, bblayers.conf and more) can be found in {}/conf -""".format( +The bitbake configuration files (local.conf, bblayers.conf and more) can be found in {}/conf{}""".format( bitbake_config["description"], os.path.join(bitbake_builddir,'conf/conf-summary.txt'), os.path.join(bitbake_builddir,'conf/conf-notes.txt'), init_script, fragment_note, - bitbake_builddir + bitbake_builddir, + readme_extra ) readme_file = os.path.join(bitbake_builddir, "README") with open(readme_file, 'w') as f: @@ -413,6 +422,11 @@ The bitbake configuration files (local.conf, bblayers.conf and more) can be foun logger.plain("To run builds, source the environment using\n . {}\n".format(init_script)) logger.plain("{}\n".format(fragment_note)) logger.plain("The bitbake configuration files (local.conf, bblayers.conf and more) can be found in\n {}/conf\n".format(bitbake_builddir)) + if init_vscode: + logger.plain("To edit the code in VSCode, open the workspace:\n code {}\n".format(workspace_file)) + + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script) def get_registry_config(registry_path, id): for root, _dirs, files in os.walk(registry_path): @@ -428,12 +442,12 @@ def merge_overrides_into_sources(sources, overrides): layers[k] = v return layers -def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt"): +def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt", init_vscode=False): layer_config = merge_overrides_into_sources(config["data"]["sources"], config["source-overrides"]["sources"]) sources_fixed_revisions = checkout_layers(layer_config, confdir, layerdir, d) bitbake_config = config["bitbake-config"] thisdir = os.path.dirname(config["path"]) if config["type"] == 'local' else None - setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf) + setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf, init_vscode) write_sources_fixed_revisions(confdir, layerdir, sources_fixed_revisions) commit_config(confdir) @@ -622,6 +636,151 @@ def obtain_overrides(args): return overrides +def configure_vscode(setupdir, layerdir, builddir, init_script): + """ + Configure the VSCode environment by creating or updating a workspace file. + + Create or update a bitbake.code-workspace file with folders for the layers and build/conf. + Managed folders are regenerated; user-added folders are kept. Settings are merged, with + managed keys (bitbake.*, python extra paths) always overwritten. + """ + logger.debug("configure_vscode: setupdir={}, layerdir={}, builddir={}, init_script={}".format( + setupdir, layerdir, builddir, init_script)) + + # Get git repository directories + git_repos = [] + if os.path.exists(layerdir): + for entry in os.listdir(layerdir): + entry_path = os.path.join(layerdir, entry) + if os.path.isdir(entry_path) and not os.path.islink(entry_path): + # Check if it's a git repository + if os.path.exists(os.path.join(entry_path, '.git')): + git_repos.append(entry) + logger.debug("configure_vscode: found {} git repos: {}".format(len(git_repos), git_repos)) + + conf_path = os.path.relpath(os.path.join(builddir, "conf"), setupdir) + repo_paths = [os.path.relpath(os.path.join(layerdir, repo), setupdir) for repo in git_repos] + logger.debug("configure_vscode: conf_path={}, repo_paths={}".format(conf_path, repo_paths)) + + # Load existing workspace + workspace_file = os.path.join(setupdir, "bitbake.code-workspace") + workspace = { + "extensions": { + "recommendations": [ + "yocto-project.yocto-bitbake" + ] + } + } + if os.path.exists(workspace_file): + logger.debug("configure_vscode: loading existing workspace file: {}".format(workspace_file)) + try: + with open(workspace_file, 'r') as f: + workspace = json.load(f) + logger.debug("configure_vscode: loaded workspace with {} folders, {} settings".format( + len(workspace.get("folders", [])), len(workspace.get("settings", {})))) + except (json.JSONDecodeError, OSError) as e: + logger.error( + "Unable to read existing workspace file {}: {}. Skipping update.".format( + workspace_file, str(e) + ) + ) + return + else: + logger.debug("configure_vscode: creating new workspace file: {}".format(workspace_file)) + + # Update folders + existing_folders = workspace.get("folders", []) + new_folders = [{"name": "conf", "path": conf_path}] + for rp in repo_paths: + repo_name = os.path.basename(rp) + new_folders.append({"name": repo_name, "path": rp}) + # Keep any user-added folders that are not managed + managed_paths = {f["path"] for f in new_folders} + for f in existing_folders: + if f["path"] not in managed_paths: + new_folders.append(f) + logger.debug("configure_vscode: keeping user-added folder: {}".format(f["path"])) + workspace["folders"] = new_folders + logger.debug("configure_vscode: updated workspace with {} folders".format(len(new_folders))) + + # Build Python extra paths for each layer - only check top level of each repo + extra_paths = [] + subdirs_to_check = ['lib', 'scripts'] + for repo in git_repos: + repo_path_abs = os.path.join(layerdir, repo) + for subdir in subdirs_to_check: + sub_path = os.path.join(repo_path_abs, subdir) + if os.path.isdir(sub_path): + extra_paths.append(sub_path) + + # Update settings + existing_settings = workspace.get("settings", {}) + new_settings = { + "bitbake.disableConfigModification": True, + "bitbake.pathToBitbakeFolder": os.path.join(layerdir, "bitbake"), + "bitbake.pathToBuildFolder": builddir, + "bitbake.pathToEnvScript": init_script, + "bitbake.workingDirectory": builddir, + "files.associations": { + "*.conf": "bitbake", + "*.inc": "bitbake" + }, + "files.exclude": { + "**/.git/**": True + }, + "search.exclude": { + "**/.git/**": True, + "**/logs/**": True + }, + "files.watcherExclude": { + "**/.git/**": True, + "**/logs/**": True + }, + "python.analysis.exclude": [ + "**/.git/**", + "**/logs/**" + ], + "python.autoComplete.extraPaths": extra_paths, + "python.analysis.extraPaths": extra_paths + } + + # Merge settings: add missing, always update bitbake paths and python extra paths + for key, value in new_settings.items(): + if key not in existing_settings: + existing_settings[key] = value + elif key.startswith("bitbake.") or key in [ + "python.autoComplete.extraPaths", + "python.analysis.extraPaths", + ]: + # Always replace - these are managed/machine-generated settings + existing_settings[key] = value + elif key in [ + "files.associations", + "files.exclude", + "search.exclude", + "files.watcherExclude", + "python.analysis.exclude", + ]: + # For dicts and lists, merge new values in without removing user additions + if isinstance(value, dict): + if not isinstance(existing_settings[key], dict): + existing_settings[key] = {} + for k, v in value.items(): + if k not in existing_settings[key]: + existing_settings[key][k] = v + elif isinstance(value, list): + if not isinstance(existing_settings[key], list): + existing_settings[key] = [] + for item in value: + if item not in existing_settings[key]: + existing_settings[key].append(item) + + workspace["settings"] = existing_settings + logger.debug("configure_vscode: merged settings, total {} keys".format(len(existing_settings))) + + with open(workspace_file, 'w') as f: + json.dump(workspace, f, indent=4) + logger.debug("configure_vscode: wrote workspace file: {}".format(workspace_file)) def init_config(top_dir, settings, args): create_siteconf(top_dir, args.non_interactive, settings) @@ -691,7 +850,7 @@ def init_config(top_dir, settings, args): bb.event.register("bb.build.TaskProgress", handle_task_progress, data=d) write_upstream_config(confdir, upstream_config) - update_build(upstream_config, confdir, setupdir, layerdir, d, update_bb_conf="yes") + update_build(upstream_config, confdir, setupdir, layerdir, d, update_bb_conf="yes", init_vscode=args.init_vscode) bb.event.remove("bb.build.TaskProgress", None) @@ -1087,6 +1246,8 @@ def main(): parser_init.add_argument('--skip-selection', action='append', help='Do not select and set an option/fragment from available choices; the resulting bitbake configuration may be incomplete.') parser_init.add_argument('-L', '--use-local-source', default=[], action='append', nargs=2, metavar=('SOURCE_NAME', 'PATH'), help='Symlink local source into a build, instead of getting it as prescribed by a configuration (useful for local development).') + parser_init.add_argument('--init-vscode', action=argparse.BooleanOptionalAction, default=bool(shutil.which('code')), + help='Generate VSCode workspace configuration (default: %(default)s)') parser_init.set_defaults(func=init_config) parser_status = subparsers.add_parser('status', help='Check if the setup needs to be synchronized with configuration') From patchwork Sun Mar 29 18:32:57 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B3A3FC72A1 for ; Sun, 29 Mar 2026 18:34:30 +0000 (UTC) Received: from mta-65-225.siemens.flowmailer.net (mta-65-225.siemens.flowmailer.net [185.136.65.225]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.34485.1774809263058022126 for ; Sun, 29 Mar 2026 11:34:25 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=LlcPhtzO; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.65.225, mailfrom: fm-1329275-20260329183420f7371f9b24000207ac-3gp3tg@rts-flowmailer.siemens.com) Received: by mta-65-225.siemens.flowmailer.net with ESMTPSA id 20260329183420f7371f9b24000207ac for ; Sun, 29 Mar 2026 20:34:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=5slPa932EOIuwdVa2QjpcXBG0NCAeGQZyZ5bVhsu8sk=; b=LlcPhtzOUHUcpDQ5Qqf3JU9AbhrahdHfdZZVMkFVftbHrNxWaRQmU4H5mVczlGwNV4GOrC I2zJu3XLtotqatXeHzB9wJRc6IUAjtyfKgbctVm9OKOtmd9QAr9wGp/v1Y/aP9F9GJFNVyMS 4CfEHQiUNm67t4HXtbxK9FnfytIbdSqBy+Bb1R+T6reAb7DKYYE3KkFCrnp6LGsmcVDCP9E7 3xSmmhhO5o8RlxIYgUSBEjR6NRClrYKBo+MdYEnXismuDT4A5D1lNP2oo39GGpxzQ2DDrjwu bV8Ca80/xDegU0r0vJI4vRTIfaiIJQoilePGIMUO8VPrYEq6NW2XN5iw==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 10/14] tests/setup: add test_vscode for VSCode workspace generation Date: Sun, 29 Mar 2026 20:32:57 +0200 Message-ID: <20260329183332.1996183-11-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:30 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19275 From: Adrian Freihofer Add test_vscode to BitbakeSetupTest covering the --init-vscode option introduced in 'bitbake-setup: generate config files for VSCode': - init --init-vscode creates bitbake.code-workspace with the expected top-level structure (folders, settings, extensions). - Folders list conf and each non-symlink git repo in layers/; all paths are relative. - Bitbake extension settings (pathToBuildFolder, pathToEnvScript, disableConfigModification) are set correctly. - File associations (*.conf, *.inc) and python.analysis.extraPaths / python.autoComplete.extraPaths are populated. - init --no-init-vscode does not create a workspace file. - update after a layer change preserves user-added folders and settings while updating managed ones. - update with a corrupt workspace file logs an error and leaves the file unchanged. Signed-off-by: Adrian Freihofer --- lib/bb/tests/setup.py | 93 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index 1af3d8b50..52120546c 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -538,3 +538,96 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) custom_setup_dir = 'special-setup-dir-with-cmdline-overrides' out = self.runbbsetup("init --non-interactive -L test-repo {} --setup-dir-name {} test-config-1 gadget".format(self.testrepopath, custom_setup_dir)) _check_local_sources(custom_setup_dir) + + def test_vscode(self): + if 'BBPATH' in os.environ: + del os.environ['BBPATH'] + os.chdir(self.tempdir) + + self.runbbsetup("settings set default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)) + self.add_file_to_testrepo('test-file', 'initial\n') + self.add_json_config_to_registry('test-config-1.conf.json', 'master', 'master') + + # --init-vscode should create bitbake.code-workspace + self.runbbsetup("init --non-interactive --init-vscode test-config-1 gadget") + setuppath = self.get_setup_path('test-config-1', 'gadget') + workspace_file = os.path.join(setuppath, 'bitbake.code-workspace') + self.assertTrue(os.path.exists(workspace_file), + "bitbake.code-workspace should be created with --init-vscode") + + with open(workspace_file) as f: + workspace = json.load(f) + + # top-level structure + self.assertIn('folders', workspace) + self.assertIn('settings', workspace) + self.assertIn('extensions', workspace) + self.assertIn('yocto-project.yocto-bitbake', + workspace['extensions']['recommendations']) + + # folders: conf dir + test-repo (symlinks like oe-init-build-env-dir are skipped) + folder_names = {f['name'] for f in workspace['folders']} + self.assertIn('conf', folder_names) + self.assertIn('test-repo', folder_names) + + # folder paths must be relative so the workspace is portable + for f in workspace['folders']: + self.assertFalse(os.path.isabs(f['path']), + "Folder path should be relative, got: {}".format(f['path'])) + + # bitbake extension settings + settings = workspace['settings'] + self.assertTrue(settings.get('bitbake.disableConfigModification')) + self.assertEqual(settings['bitbake.pathToBuildFolder'], + os.path.join(setuppath, 'build')) + self.assertEqual(settings['bitbake.pathToEnvScript'], + os.path.join(setuppath, 'build', 'init-build-env')) + + # file associations + self.assertIn('*.conf', settings.get('files.associations', {})) + self.assertIn('*.inc', settings.get('files.associations', {})) + + # python extra paths: test-repo/scripts/ exists and should be listed + extra_paths = settings.get('python.analysis.extraPaths', []) + self.assertTrue(any('scripts' in p for p in extra_paths), + "python.analysis.extraPaths should include the scripts dir") + self.assertEqual(settings.get('python.analysis.extraPaths'), + settings.get('python.autoComplete.extraPaths')) + + # --no-init-vscode should NOT create a workspace file + self.runbbsetup("init --non-interactive --no-init-vscode test-config-1 gadget-notemplate") + notemplate_path = self.get_setup_path('test-config-1', 'gadget-notemplate') + self.assertFalse( + os.path.exists(os.path.join(notemplate_path, 'bitbake.code-workspace')), + "bitbake.code-workspace should not be created with --no-init-vscode") + + # update with --init-vscode after a layer change should preserve + # user-added folders and settings while still rewriting managed ones + workspace['folders'].append({"name": "user-folder", "path": "user/custom"}) + workspace['settings']['my.user.setting'] = 'preserved' + with open(workspace_file, 'w') as f: + json.dump(workspace, f, indent=4) + + self.add_file_to_testrepo('test-file', 'updated\n') + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + self.runbbsetup("update --update-bb-conf='no'") + del os.environ['BBPATH'] + + with open(workspace_file) as f: + updated = json.load(f) + self.assertIn('user/custom', {f['path'] for f in updated['folders']}, + "User-added folder was removed during update") + self.assertIn('my.user.setting', updated['settings'], + "User-added setting was removed during update") + + # update with a corrupt workspace file should log an error and leave it unchanged + self.add_file_to_testrepo('test-file', 'updated-again\n') + with open(workspace_file, 'w') as f: + f.write('{invalid json') + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + self.runbbsetup("update --update-bb-conf='no'") + del os.environ['BBPATH'] + with open(workspace_file) as f: + content = f.read() + self.assertEqual(content, '{invalid json', + "Corrupt workspace file should not be modified") From patchwork Sun Mar 29 18:32:58 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DA80FC97F3 for ; Sun, 29 Mar 2026 18:34:31 +0000 (UTC) Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net [185.136.64.228]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.34710.1774809263055482082 for ; Sun, 29 Mar 2026 11:34:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=IHFo4ABG; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.64.228, mailfrom: fm-1329275-20260329183420ad2ddcdc2000020729-edxgwn@rts-flowmailer.siemens.com) Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 20260329183420ad2ddcdc2000020729 for ; Sun, 29 Mar 2026 20:34:21 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=oSI7b6jTNl6qYNAHRL/07Gn5qIcP6gP/yQHbmtlIROw=; b=IHFo4ABGSGlkICRyntD0A7kLZxGk3gThck0EegpyPNz3UW7PmcisdBYpyKTPDF0aiNjI7k IGzVjUoPCmXu6fU3NjrBOsbniWml0mKbLOA3pwbLuHEnDubRk2Xb6OUfWmtaJLNtqTeyN1v1 bruiOJYDo7rUKFEiTUzT+S1WJX5X1fS8d1327Y6hnRfyrxdIcUtk4Fu8RF4PD5Bh1tTwNf22 RaQptz9K1s4lX/WsJ/M8F4J4BanPFZsEk2sOJsMVtdv2qwEEAMpmwAK+ur73rxAFQfu984Gp ZZ74kEzNw2IOjthGf30jxf8uAyBst+7WvYNZ/YDF8qaW/T/TCmi/EIow==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 11/14] bitbake-setup: add --rebase-conflicts-strategy to the update command Date: Sun, 29 Mar 2026 20:32:58 +0200 Message-ID: <20260329183332.1996183-12-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19265 From: Adrian Freihofer When unpack_update() raises LocalModificationsError (uncommitted changes) or RebaseError (local commits that could not be rebased), the caller now has a choice of how to handle it, controlled by the new --rebase-conflicts-strategy option on the 'update' subcommand: - abort (default): re-raise the error so the update stops with a clear message that names the affected source and its path. The fetcher has already aborted the rebase and restored the checkout to its previous state. - backup: rename the directory to a timestamped -backup path to preserve local work, then re-clone from upstream via fetcher.unpack(). Both exception types share a single except clause; the exception message already describes the specific failure (uncommitted changes vs. failed rebase with git output), so it is forwarded directly to the warning log. The strategy is threaded from the CLI argument through build_status() and update_build() down to checkout_layers(). Signed-off-by: Adrian Freihofer --- bin/bitbake-setup | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index ce32c3f13..596722e48 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -167,7 +167,7 @@ def _get_remotes(r_remote): return remotes -def checkout_layers(layers, confdir, layerdir, d): +def checkout_layers(layers, confdir, layerdir, d, rebase_conflicts_strategy='abort'): def _checkout_git_remote(r_remote, repodir, layers_fixed_revisions): rev = r_remote['rev'] branch = r_remote.get('branch', None) @@ -183,7 +183,23 @@ def checkout_layers(layers, confdir, layerdir, d): else: src_uri = f"{fetchuri};protocol={prot};rev={rev};nobranch=1;destsuffix={repodir}" fetcher = bb.fetch.Fetch([src_uri], d) - do_fetch(fetcher, layerdir) + repodir_path = os.path.join(layerdir, repodir) + try: + do_fetch(fetcher, layerdir) + except (bb.fetch2.LocalModificationsError, bb.fetch2.RebaseError) as e: + if rebase_conflicts_strategy != 'backup': + e.msg += ("\nUse 'bitbake-setup update --rebase-conflicts-strategy=backup'" + " to automatically back up the directory and re-clone from upstream," + " or use 'bitbake-setup init -L %s /path/to/local/checkout'" + " to work with a local checkout instead." % r_name) + raise + backup_path = add_unique_timestamp_to_path(repodir_path + '-backup') + logger.warning( + "%s\n" + "Renaming %s to %s to preserve your work, then re-cloning from upstream.", + e, repodir_path, backup_path) + os.rename(repodir_path, backup_path) + fetcher.unpack(layerdir) urldata = fetcher.ud[src_uri] revision = urldata.revision layers_fixed_revisions[r_name]['git-remote']['rev'] = revision @@ -442,9 +458,9 @@ def merge_overrides_into_sources(sources, overrides): layers[k] = v return layers -def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt", init_vscode=False): +def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt", init_vscode=False, rebase_conflicts_strategy='abort'): layer_config = merge_overrides_into_sources(config["data"]["sources"], config["source-overrides"]["sources"]) - sources_fixed_revisions = checkout_layers(layer_config, confdir, layerdir, d) + sources_fixed_revisions = checkout_layers(layer_config, confdir, layerdir, d, rebase_conflicts_strategy=rebase_conflicts_strategy) bitbake_config = config["bitbake-config"] thisdir = os.path.dirname(config["path"]) if config["type"] == 'local' else None setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf, init_vscode) @@ -927,7 +943,7 @@ def build_status(top_dir, settings, args, d, update=False): logger.plain('\nConfiguration in {} has changed:\n{}'.format(setupdir, config_diff)) if update: update_build(new_upstream_config, confdir, setupdir, layerdir, d, - update_bb_conf=args.update_bb_conf) + update_bb_conf=args.update_bb_conf, rebase_conflicts_strategy=args.rebase_conflicts_strategy) else: bb.process.run('git -C {} restore config-upstream.json'.format(confdir)) return @@ -936,7 +952,7 @@ def build_status(top_dir, settings, args, d, update=False): if are_layers_changed(layer_config, layerdir, d): if update: update_build(current_upstream_config, confdir, setupdir, layerdir, - d, update_bb_conf=args.update_bb_conf) + d, update_bb_conf=args.update_bb_conf, rebase_conflicts_strategy=args.rebase_conflicts_strategy) return logger.plain("\nConfiguration in {} has not changed.".format(setupdir)) @@ -1257,6 +1273,10 @@ def main(): parser_update = subparsers.add_parser('update', help='Update a setup to be in sync with configuration') add_setup_dir_arg(parser_update) parser_update.add_argument('--update-bb-conf', choices=['prompt', 'yes', 'no'], default='prompt', help='Update bitbake configuration files (bblayers.conf, local.conf) (default: prompt)') + parser_update.add_argument('--rebase-conflicts-strategy', choices=['abort', 'backup'], default='abort', + help="What to do when a layer repository has local modifications that prevent " + "an in-place update: 'abort' (default) aborts with an error message; " + "'backup' renames the directory to a timestamped backup and re-clones from upstream.") parser_update.set_defaults(func=build_update) parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine') From patchwork Sun Mar 29 18:32:59 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C2CEFC97F2 for ; Sun, 29 Mar 2026 18:34:31 +0000 (UTC) Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net [185.136.64.226]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.34707.1774809263055080653 for ; Sun, 29 Mar 2026 11:34:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=XWR9zeIA; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.64.226, mailfrom: fm-1329275-20260329183420de34f92b4200020707-pmg0pb@rts-flowmailer.siemens.com) Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 20260329183420de34f92b4200020707 for ; Sun, 29 Mar 2026 20:34:21 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=bm4TVf7D3XJqxfBx165Dn9+QyV7c0mIQxIpap0W+Emc=; b=XWR9zeIATGBBISXUTeqINlsR/yJhdH+aAW01LL2zu/6Gxc1C6zLS0Q31yYGQ64Lhlo0IeI 3Rv4iW4dwICp7RjiwnbmbyPeRNzg0UL/D859C60FbKxvqJfmuOk8D2JcaIgkZHuGaQlfq2YN fvLdcquWVY5qySbqy2Rac4hTROce4cPIVVtpFpFUAaBvfOxEyf38Rf9M8JgF/oi4wrmQr83X E3BJEL2I60maJRive07wcSE2Z2DlLHitBKtqtpEzuZ69a6GaMa+rXYd1EpjgGOEP69clZq5k HdbbNJRqIGMKKj6G2vxprXDxJkCYtNXZxFS9bSvs/BlHreIq7cEpskMA==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 12/14] tests/setup: add test_update_rebase_conflicts_strategy Date: Sun, 29 Mar 2026 20:32:59 +0200 Message-ID: <20260329183332.1996183-13-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19269 From: Adrian Freihofer Add a dedicated test for the --rebase-conflicts-strategy option introduced in 'bitbake-setup: add --rebase-conflicts-strategy to the update command'. Three scenarios are covered that are not exercised by the existing test_setup: 1. Uncommitted tracked-file change (LocalModificationsError), default 'abort' strategy: update must fail with an error that contains 'has uncommitted changes' and the '--rebase-conflicts-strategy=backup' hint; no backup directory is created. 2. Same uncommitted change, 'backup' strategy: the layer directory is renamed to a timestamped backup, the layer is re-cloned from upstream, and the result is clean (upstream content, no local modifications). 3. Committed local change that conflicts with the next upstream commit (RebaseError): a. Default 'abort' strategy: update must fail with an error that contains 'Merge conflict' and the '--rebase-conflicts-strategy=backup' hint; no backup directory is created. b. 'backup' strategy: instead of the hard rebase-conflict failure, the conflicted directory is backed up and re-cloned successfully. A small _count_layer_backups() helper is added to the class and reused across scenarios. Signed-off-by: Adrian Freihofer --- lib/bb/tests/setup.py | 95 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py index 52120546c..638d56d3b 100644 --- a/lib/bb/tests/setup.py +++ b/lib/bb/tests/setup.py @@ -631,3 +631,98 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) content = f.read() self.assertEqual(content, '{invalid json', "Corrupt workspace file should not be modified") + + def _count_layer_backups(self, layers_path): + return len([f for f in os.listdir(layers_path) if 'backup' in f]) + + def test_update_rebase_conflicts_strategy(self): + """Test the --rebase-conflicts-strategy option for the update command. + + Covers three scenarios not exercised by test_setup: + 1. Uncommitted tracked-file change (LocalModificationsError) + default 'abort' + strategy → clean error message containing 'has uncommitted changes' and a + hint at --rebase-conflicts-strategy=backup; no backup directory is created. + 2. Same uncommitted change + 'backup' strategy → directory is renamed to a + timestamped backup and the layer is re-cloned cleanly. + 3. Committed local change that conflicts with an incoming upstream commit + (RebaseError): + a. Default 'abort' strategy → error containing 'Merge conflict' and the + --rebase-conflicts-strategy=backup hint; no backup directory is created. + b. 'backup' strategy → backup + re-clone instead of a hard failure. + """ + if 'BBPATH' in os.environ: + del os.environ['BBPATH'] + os.chdir(self.tempdir) + + self.runbbsetup("settings set default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)) + self.add_file_to_testrepo('test-file', 'initial\n') + self.add_json_config_to_registry('test-config-1.conf.json', 'master', 'master') + self.runbbsetup("init --non-interactive test-config-1 gadget") + + setuppath = self.get_setup_path('test-config-1', 'gadget') + layer_path = os.path.join(setuppath, 'layers', 'test-repo') + layers_path = os.path.join(setuppath, 'layers') + + # Scenario 1: uncommitted tracked change, default 'abort' strategy + # Advance upstream so an update is required. + self.add_file_to_testrepo('test-file', 'upstream-v2\n') + # Modify the same tracked file in the layer without committing. + with open(os.path.join(layer_path, 'test-file'), 'w') as f: + f.write('locally-modified\n') + + os.environ['BBPATH'] = os.path.join(setuppath, 'build') + with self.assertRaises(bb.process.ExecutionError) as ctx: + self.runbbsetup("update --update-bb-conf='no'") + self.assertIn('has uncommitted changes', str(ctx.exception)) + self.assertIn('--rebase-conflicts-strategy=backup', str(ctx.exception)) + # No backup directory must have been created. + self.assertEqual(self._count_layer_backups(layers_path), 0, + "abort strategy must not create any backup") + + # Scenario 2: same uncommitted change, 'backup' strategy + out = self.runbbsetup("update --update-bb-conf='no' --rebase-conflicts-strategy=backup") + # One backup directory must now exist. + self.assertEqual(self._count_layer_backups(layers_path), 1, + "backup strategy must create exactly one backup") + # The re-cloned layer must be clean and at the upstream revision. + with open(os.path.join(layer_path, 'test-file')) as f: + self.assertEqual(f.read(), 'upstream-v2\n', + "re-cloned layer must contain the upstream content") + status = self.git('status --porcelain', cwd=layer_path).strip() + self.assertEqual(status, '', + "re-cloned layer must have no local modifications") + del os.environ['BBPATH'] + + # Scenario 3: committed conflicting change, 'backup' strategy + # Re-initialise a fresh setup so we start from a clean state. + self.runbbsetup("init --non-interactive --setup-dir-name rebase-conflict-setup test-config-1 gadget") + conflict_setup = os.path.join(self.tempdir, 'bitbake-builds', 'rebase-conflict-setup') + conflict_layer = os.path.join(conflict_setup, 'layers', 'test-repo') + conflict_layers = os.path.join(conflict_setup, 'layers') + + # Commit a local change that touches the same file as the next upstream commit. + with open(os.path.join(conflict_layer, 'test-file'), 'w') as f: + f.write('conflicting-local\n') + self.git('add test-file', cwd=conflict_layer) + self.git('commit -m "Local conflicting change"', cwd=conflict_layer) + + # Advance upstream with a conflicting edit. + self.add_file_to_testrepo('test-file', 'conflicting-upstream\n') + + os.environ['BBPATH'] = os.path.join(conflict_setup, 'build') + # Default stop strategy must still fail with a conflict error and include + # the --rebase-conflicts-strategy=backup hint (same handler as LocalModificationsError). + with self.assertRaises(bb.process.ExecutionError) as ctx: + self.runbbsetup("update --update-bb-conf='no'") + self.assertIn('Merge conflict in test-file', str(ctx.exception)) + self.assertIn('--rebase-conflicts-strategy=backup', str(ctx.exception)) + self.assertEqual(self._count_layer_backups(conflict_layers), 0) + + # Backup strategy must succeed: backup the conflicted dir and re-clone. + self.runbbsetup("update --update-bb-conf='no' --rebase-conflicts-strategy=backup") + self.assertEqual(self._count_layer_backups(conflict_layers), 1, + "backup strategy must create exactly one backup after a conflict") + with open(os.path.join(conflict_layer, 'test-file')) as f: + self.assertEqual(f.read(), 'conflicting-upstream\n', + "re-cloned layer must contain the upstream content after conflict backup") + del os.environ['BBPATH'] From patchwork Sun Mar 29 18:33:00 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CDEFFC72A0 for ; Sun, 29 Mar 2026 18:34:30 +0000 (UTC) Received: from mta-65-228.siemens.flowmailer.net (mta-65-228.siemens.flowmailer.net [185.136.65.228]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.34712.1774809263055963453 for ; Sun, 29 Mar 2026 11:34:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=EaMUvPIS; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.65.228, mailfrom: fm-1329275-2026032918342118c5e0ffeb000207f9-kmjacq@rts-flowmailer.siemens.com) Received: by mta-65-228.siemens.flowmailer.net with ESMTPSA id 2026032918342118c5e0ffeb000207f9 for ; Sun, 29 Mar 2026 20:34:21 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=L/kndJXkAdGDHDSqcJ9pO5OZMr8c2Rqq2Atnz6V3m/c=; b=EaMUvPISkf0W3ITEwwI1Y6uzlKjVCZkb/dZQKpty7FdGj/JS6TztfAHx0q7JuPBFP9Gt7N lFUr3O716x1h2kZD21u01AR6C/4w/Oi8uxBel9MsAbS3neM+m5qhE30EHULFK/Ihmtw3cQ/q ONENCBWS0TPK1SVGwfSy6SQr3W6LqUCS2eHbnCvPxwBQXPXN3ruxzPrRFMU3OWrvvemhCN6u 01IC1KDezEGPpVF9cGNCzBH41D0qdxkBWOGe0Z4j/W8WiOIKCkCAIra7XM5V9rkP3c9ve+AR 5UoPANwYumoUjsEJjw+2YE8ROy6rPSo3sF8WdOFWYyXOFb+SOwIPWDKA==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 13/14] bitbake-setup: catch unexpected exceptions and show a clean error message Date: Sun, 29 Mar 2026 20:33:00 +0200 Message-ID: <20260329183332.1996183-14-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:30 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19264 From: Adrian Freihofer Without a top-level exception handler, any unexpected Python exception reaches the user as a raw traceback, which is confusing and exposes implementation details. Add a try/except block around the main dispatch in main() that: - catches all unexpected exceptions and logs them as a single ERROR line, consistent with how other errors are already presented - re-raises when --debug is active, so the full traceback is still available for diagnosis - explicitly re-raises SystemExit and KeyboardInterrupt so sys.exit() calls and Ctrl-C continue to work normally Signed-off-by: Adrian Freihofer --- bin/bitbake-setup | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/bin/bitbake-setup b/bin/bitbake-setup index 596722e48..ddb6d2af1 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -1361,13 +1361,21 @@ def main(): logger.info('Bitbake-setup is using {} as top directory.'.format(top_dir)) - if args.func == init_config: - init_config(top_dir, all_settings, args) - else: - d = init_bb_cache(top_dir, all_settings, args) - args.func(top_dir, all_settings, args, d) + try: + if args.func == init_config: + init_config(top_dir, all_settings, args) + else: + d = init_bb_cache(top_dir, all_settings, args) + args.func(top_dir, all_settings, args, d) - save_bb_cache() + save_bb_cache() + except (SystemExit, KeyboardInterrupt): + raise + except Exception as e: + if args.debug: + raise + logger.error(str(e)) + sys.exit(1) else: parser.print_help() From patchwork Sun Mar 29 18:33:01 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Freihofer, Adrian" X-Patchwork-Id: 84728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EE0EFC97EB for ; Sun, 29 Mar 2026 18:34:31 +0000 (UTC) Received: from mta-64-227.siemens.flowmailer.net (mta-64-227.siemens.flowmailer.net [185.136.64.227]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.34714.1774809263056260880 for ; Sun, 29 Mar 2026 11:34:24 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=adrian.freihofer@siemens.com header.s=fm1 header.b=ECrtDx19; spf=pass (domain: rts-flowmailer.siemens.com, ip: 185.136.64.227, mailfrom: fm-1329275-20260329183421f305c8e3a30002072b-mtiucb@rts-flowmailer.siemens.com) Received: by mta-64-227.siemens.flowmailer.net with ESMTPSA id 20260329183421f305c8e3a30002072b for ; Sun, 29 Mar 2026 20:34:21 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=adrian.freihofer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=CnK2Cmu+v2s96HGHUKyPbB8QDzt/La9E/MSU1CFpLjI=; b=ECrtDx195unNsphkmsMVbXberhiQh39OsgG6YKZ4qq5LLFi7wwyAueuct9jTF4wIf8a1xo 7OrUFPZaQdmURdF9x0IiIR/bfYw7cYgbafWr4a77/X3k81gn3XMyIrzg1eNeBNw6oC5A2fTr yLP297f5zXFihSs07oK344S+fJ1YCqY0wgrMFTQDN/Ds1SoH0POftCBMxBISoyjyjYx7foiw lg9FkwSKf3cWA6eEBh2bj6W4IOsnkMoXYRcZzhUcClQW4GU1fFCbfUVpJbad4Da/Fq7XLx05 OBJx+sp0wlHg2W5r7xEalMkL+N+qtSkWgKf77nhgcGNV2nRyw0fjGelQ==; From: AdrianF To: bitbake-devel@lists.openembedded.org Cc: Adrian Freihofer Subject: [PATCH v3 14/14] doc: document new bitbake-setup init and update features Date: Sun, 29 Mar 2026 20:33:01 +0200 Message-ID: <20260329183332.1996183-15-adrian.freihofer@siemens.com> In-Reply-To: <20260329183332.1996183-1-adrian.freihofer@siemens.com> References: <20260329183332.1996183-1-adrian.freihofer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1329275:519-21489:flowmailer List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sun, 29 Mar 2026 18:34:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19266 From: Adrian Freihofer Document the two new options introduced in the setup command: - --init-vscode / --no-init-vscode (66c26e5f0): generates a bitbake.code-workspace file for VS Code in the Setup directory, with layer folders, BitBake extension settings, and file associations. Default is enabled when the 'code' binary is found on PATH. - --rebase-conflicts-strategy (60ce61cca): controls the fallback behaviour when a layer repository cannot be updated in place due to local modifications or conflicting local commits. Documents both the 'abort' (default) and 'backup' values. Also documents the in-place update behaviour of the update command (fetch via dldir remote + rebase), with a cross-reference to the unpack_update() section in the fetching chapter. Also document that the update command is intended to be run from a shell with the BitBake environment sourced, and why --autostash is not used (transparency: uncommitted changes are surfaced rather than silently stashed and re-applied). Add examples for the four distinct update scenarios, using real output from an actual update session: - normal update with local commits rebased and preserved - update blocked by uncommitted changes, with stash + retry steps - rebase conflict with manual resolution via the dldir git remote, including the rebase --continue + re-run sequence - rebase conflict resolved via --rebase-conflicts-strategy=backup, showing the real WARNING output and timestamped backup directory name, with a step-by-step git workflow (cd into the fresh clone, fetch the backup branch by local path, cherry-pick oldest-first) to recover commits from the backup, and cleanup instructions (rm -rf the backup, or remove the VS Code workspace folder via the UI or by re-running bitbake-setup update after deletion) Signed-off-by: Adrian Freihofer --- .../bitbake-user-manual-environment-setup.rst | 205 ++++++++++++++++++ 1 file changed, 205 insertions(+) diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst index 37518fceb..c22e19bd6 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst +++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst @@ -327,6 +327,16 @@ In addition, the command can take the following arguments: with a ``local`` source in it. See the :ref:`ref-bbsetup-source-overrides` section for more information on source overrides. +- ``--init-vscode`` / ``--no-init-vscode``: generate (or skip generating) a + ``bitbake.code-workspace`` file in the :term:`Setup` directory. The workspace + file configures the `Yocto Project BitBake + `_ + VS Code extension with paths to the build directory and the init script, and + lists the layer directories as workspace folders. Any user-added folders or + settings in an existing workspace file are preserved across updates. + The default is ``true`` when ``code`` (the VS Code binary) is found on + ``PATH``, and ``false`` otherwise. + ``bitbake-setup init`` Examples ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -410,6 +420,30 @@ the latest changes from the :term:`Configuration Template` it was constructed fr The :ref:`ref-bbsetup-command-status` command can be used to show the current status of the :term:`Setup` before updating it. +This command is intended to be run from a shell where the BitBake environment +has been sourced (e.g. after ``source build/init-build-env``), so that +``bitbake-setup`` can automatically identify the current :term:`Setup` without +requiring the ``--setup-dir`` argument. + +When a layer repository already exists in the :term:`Setup` (i.e. it has been +previously checked out by ``bitbake-setup init`` or a prior ``update``), the +update is performed *in place* using the fetcher's +:ref:`unpack_update ` method: the new upstream revision +is fetched into the local download cache and then rebased on top of the +checkout's current HEAD. This means any local commits in the layer directory +are preserved and rebased onto the new upstream revision. If the working tree +contains staged or unstaged changes to tracked files, the update is blocked +until those changes are committed, stashed or discarded. + +.. note:: + + ``bitbake-setup`` performs the rebase to fast-forward local commits onto + the new upstream revision, but intentionally does not go further by using + ``--autostash`` (which would silently stash uncommitted changes before the + rebase and pop them afterwards). Any uncommitted modifications are surfaced + to the user before the update proceeds, so there are no surprises from an + automatic stash/pop cycle. + In addition, the command can take the following arguments: - ``--update-bb-conf``: whether to update the :term:`BitBake Build` @@ -420,10 +454,181 @@ In addition, the command can take the following arguments: - ``yes``: update the configuration files. - ``no``: don't update the configuration files. +- ``--rebase-conflicts-strategy``: what to do when a layer repository has + local modifications or commits that prevent an in-place update. Accepted + values are: + + - ``abort`` (default): stop with an error message describing the problem. + The repository is left in its previous state (the failed rebase is + automatically aborted). The error message includes a hint to re-run with + ``--rebase-conflicts-strategy=backup``. + - ``backup``: rename the conflicting layer directory to a timestamped + ``-backup-`` path (preserving local work), then + re-clone the layer from upstream into a fresh directory. + - ``--setup-dir``: path to the :term:`Setup` to update. Not required if the command is invoked from an initialized BitBake environment that contains :term:`BBPATH`. +``bitbake-setup update`` Examples +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- This example performs a standard update from an initialized BitBake + environment. Local commits in any layer directory are rebased on top of + the new upstream revision and preserved: + + .. code-block:: shell + + $ bitbake-setup update + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory. + NOTE: Looking up config poky-master in configuration registry + NOTE: Layer repository https://git.openembedded.org/openembedded-core + checked out into /path/to/bitbake-builds/poky-master/layers/openembedded-core + updated revision master from d383ea3... to b50d6de... + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers + bitbake + meta-yocto + openembedded-core + +- This example shows what happens when a layer directory contains staged or + unstaged changes to tracked files. The update is blocked with an error: + + .. code-block:: shell + + $ bitbake-setup update + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory. + NOTE: Looking up config poky-master in configuration registry + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers + bitbake + meta-yocto + openembedded-core + ERROR: Unpack failure for URL: + 'git://git.openembedded.org/openembedded-core;protocol=https;rev=master;branch=master;destsuffix=openembedded-core'. + Repository at /path/to/bitbake-builds/poky-master/layers/openembedded-core has uncommitted changes, unable to update: + M meta/recipes-devtools/ccache/ccache_4.13.1.bb + + Commit, stash or discard your changes and re-run the update. + Use 'bitbake-setup update --rebase-conflicts-strategy=backup' + to automatically back up the directory and re-clone from upstream, + or use 'bitbake-setup init -L openembedded-core /path/to/local/checkout' + to work with a local checkout instead. + + Stashing the changes and re-running resolves the issue: + + .. code-block:: shell + + $ git -C layers/openembedded-core stash + $ bitbake-setup update + $ git -C layers/openembedded-core stash pop + +- This example shows what happens when a layer directory contains local + commits that conflict with the incoming upstream changes. The failed rebase + is automatically aborted, and the ``dldir`` remote is left in the repository + for manual resolution: + + .. code-block:: shell + + $ bitbake-setup update + ERROR: Repository at layers/openembedded-core has local commits that could + not be rebased onto the new upstream revision: + ... + Note: the 'dldir' remote points to the local download cache and may be + used to resolve the conflict manually. + Once resolved, re-run the update. + + The conflict can be resolved manually using the ``dldir`` remote that + ``bitbake-setup`` adds to the repository: + + .. code-block:: shell + + $ git -C layers/openembedded-core rebase dldir/master + # fix conflicts in an editor, then stage the resolved files: + $ git -C layers/openembedded-core add meta/recipes-core/base-files/base-files.bb + $ git -C layers/openembedded-core rebase --continue + $ bitbake-setup update + +- When manual conflict resolution is not desired, the + ``--rebase-conflicts-strategy=backup`` option can be used instead. It + preserves the conflicting directory under a timestamped backup path and + re-clones the layer cleanly from upstream: + + .. code-block:: shell + + $ bitbake-setup update --rebase-conflicts-strategy=backup + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory. + NOTE: Looking up config poky-master in configuration registry + NOTE: Layer repository https://git.openembedded.org/openembedded-core checked + out into /path/to/bitbake-builds/poky-master/layers/openembedded-core + updated revision master from 2ec283e... to b50d6de... + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers + bitbake + meta-yocto + openembedded-core + WARNING: Unpack failure for URL: + 'git://git.openembedded.org/openembedded-core;protocol=https;rev=master;branch=master;destsuffix=openembedded-core'. + Repository at /path/to/bitbake-builds/poky-master/layers/openembedded-core + has local commits that could not be rebased onto the new upstream revision: + ... + Note: the 'dldir' remote points to the local download cache and may be used to resolve the conflict manually. + Once resolved, re-run the update. + Renaming /path/to/bitbake-builds/poky-master/layers/openembedded-core to + /path/to/bitbake-builds/poky-master/layers/openembedded-core-backup.20260329160426 + to preserve your work, then re-cloning from upstream. + + The backup directory is a complete git repository. Local commits can be + recovered from it after the update by fetching a branch from the backup + into the fresh clone (git accepts local paths as remote URLs) and then + cherry-picking the desired commits. For example, given a ``my-wip`` branch + with two commits existing in the backup repository and not in the fresh clone, + the following commands can be used to apply these commits on top of the new + upstream revision in the fresh clone: + + .. code-block:: shell + + $ git -C layers/openembedded-core-backup.20260329160426 log --oneline my-wip + a1b2c3d u-boot: fix compilation with newer GCC + 2ec283e base-files: update version + ... + + $ cd layers/openembedded-core + $ git checkout -b my-wip + Switched to a new branch 'my-wip' + $ git fetch ../openembedded-core-backup.20260329160426 my-wip + + $ git cherry-pick 2ec283e + # resolve any conflicts, then stage the resolved files: + $ git add meta/recipes-core/base-files/base-files.bb + $ git cherry-pick --continue + + $ git cherry-pick a1b2c3d + # resolve any conflicts, then stage the resolved files: + $ git add meta/recipes-devtools/u-boot/u-boot_2026.04.bb + $ git cherry-pick --continue + + $ cd ../.. + + The sequence above is: + + #. Inspect the backup's branch history to identify the commits to recover. + #. Change into the fresh clone and create a matching branch. + #. Fetch the backup branch so its objects become available locally. Git + accepts filesystem paths as remote URLs, and from inside + ``layers/openembedded-core/``, ``../`` points to ``layers/``, where the + backup directory sits. + #. Cherry-pick the commits in oldest-first order. + + Once all desired commits have been recovered and verified, the backup + directory can be removed: + + .. code-block:: shell + + $ rm -rf layers/openembedded-core-backup.20260329160426 + + If a VSCode workspace is in use, the backup directory will appear as an + additional workspace folder until it is cleaned up. It can be removed from + the workspace via the VS Code UI by right-clicking the folder and selecting + *Remove Folder from Workspace*. + .. _ref-bbsetup-command-install-buildtools: ``bitbake-setup install-buildtools``