[10/10] bitbake-setup: correct (and test) several scenarios in layer checkout

From: Alexander Kanavin <alex@linutronix.de>

There were untested scenarios in layer updates which did not work properly.

The most serious of them is updating a remote layer which had user-made local
modifications (e.g. newly made local commits or simply edited files
without a commit). Bitbake-setup relies on git fetcher's unpack operation
which simply wipes it all out. Yes, the user should've used -L option
to symlink a local clone from elsewhere; this doesn't help when their
work is gone.

The commit adds a function that checks if such modifications have been
made, and a code that moves the layer into a backup location if the function
returns true. I considered asking the user what to do (similar to bitbake
config handling), but this would've resulted in a half-updated, possibly
broken layer set which ultimately just would be more confusing.

The above check is a part of a broader logic block that handles already
existing sources, and in particular addresses another issue: when a local
source replaces a git remote or another local source, this scenario failed
with 'file exists' errors (thanks to Anibal Limon for spotting that). In
this case the original symlink or source tree is removed before the new
one is unpacked from git or symlinked as a local source.

Adjust commit_config() to allow updates with no changes to configs: this can
happen when local modifications to a layer are undone and the original layer
commit is unpacked.

Add tests for all of this: both transitions between remote and local sources,
and ensuring that layer backups do happen when that is expected.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 bin/bitbake-setup     | 30 ++++++++++++++++++--
 lib/bb/tests/setup.py | 65 ++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 92 insertions(+), 3 deletions(-)

I tested the whole series + last patch to fix revisions path and it is
working as expected.

Cheers!,
Anibal

On Fri, Jan 2, 2026 at 1:34 PM Alexander Kanavin via lists.openembedded.org
<alex.kanavin=gmail.com@lists.openembedded.org> wrote:

> From: Alexander Kanavin <alex@linutronix.de>
>
> There were untested scenarios in layer updates which did not work properly.
>
> The most serious of them is updating a remote layer which had user-made
> local
> modifications (e.g. newly made local commits or simply edited files
> without a commit). Bitbake-setup relies on git fetcher's unpack operation
> which simply wipes it all out. Yes, the user should've used -L option
> to symlink a local clone from elsewhere; this doesn't help when their
> work is gone.
>
> The commit adds a function that checks if such modifications have been
> made, and a code that moves the layer into a backup location if the
> function
> returns true. I considered asking the user what to do (similar to bitbake
> config handling), but this would've resulted in a half-updated, possibly
> broken layer set which ultimately just would be more confusing.
>
> The above check is a part of a broader logic block that handles already
> existing sources, and in particular addresses another issue: when a local
> source replaces a git remote or another local source, this scenario failed
> with 'file exists' errors (thanks to Anibal Limon for spotting that). In
> this case the original symlink or source tree is removed before the new
> one is unpacked from git or symlinked as a local source.
>
> Adjust commit_config() to allow updates with no changes to configs: this
> can
> happen when local modifications to a layer are undone and the original
> layer
> commit is unpacked.
>
> Add tests for all of this: both transitions between remote and local
> sources,
> and ensuring that layer backups do happen when that is expected.
>
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  bin/bitbake-setup     | 30 ++++++++++++++++++--
>  lib/bb/tests/setup.py | 65 ++++++++++++++++++++++++++++++++++++++++++-
>  2 files changed, 92 insertions(+), 3 deletions(-)
>
> diff --git a/bin/bitbake-setup b/bin/bitbake-setup
> index 06255b112..58273e4dd 100755
> --- a/bin/bitbake-setup
> +++ b/bin/bitbake-setup
> @@ -81,7 +81,7 @@ def write_sources_fixed_revisions(config_dir, layer_dir,
> config_data):
>
>  def commit_config(config_dir):
>      bb.process.run("git -C {} add .".format(config_dir))
> -    bb.process.run("git -C {} commit --no-verify -a -m 'Configuration at
> {}'".format(config_dir, time.asctime()))
> +    bb.process.run("git -C {} commit --allow-empty --no-verify -a -m
> 'Configuration at {}'".format(config_dir, time.asctime()))
>
>  def _write_layer_list(dest, repodirs):
>      layers = []
> @@ -117,6 +117,17 @@ def checkout_layers(layers, layerdir, d):
>          logger.plain("Making a symbolic link {} pointing to
> {}".format(dst, src))
>          os.symlink(src, dst)
>
> +    def _has_local_modifications(r_name, r_path):
> +        fixed_revisions = json.load(open(os.path.join(layerdir,
> "sources-fixed-revisions.json")))
> +        rev = fixed_revisions['sources'][r_name]['git-remote']['rev']
> +        status = bb.process.run('git -C {} status
> --porcelain'.format(r_path))[0]
> +        if status:
> +            return True
> +        diff = bb.process.run('git -C {} diff {}'.format(r_path, rev))[0]
> +        if diff:
> +            return True
> +        return False
> +
>      layers_fixed_revisions = copy.deepcopy(layers)
>      repodirs = []
>      oesetupbuild = None
> @@ -130,10 +141,25 @@ def checkout_layers(layers, layerdir, d):
>          r_local = r_data.get('local')
>          if r_remote and r_local:
>              raise Exception("Source {} contains both git-remote and local
> properties.".format(r_name))
> +
> +        repodir_path = os.path.join(layerdir, repodir)
> +        if os.path.lexists(repodir_path):
> +            if os.path.islink(repodir_path):
> +                os.remove(repodir_path)
> +            elif _has_local_modifications(r_name, repodir_path):
> +                timestamp = time.strftime("%Y%m%d%H%M%S")
> +                backup_path = repodir_path +
> '-backup.{}'.format(timestamp)
> +                logger.warning("""Source {} in {} contains local
> modifications. Renaming to {} to preserve them.
> +For local development work it is recommended to clone the needed layers
> separately and use 'bitbake-setup init' with -L option to use
> them.""".format(
> +                    r_name, repodir_path, backup_path))
> +                os.rename(repodir_path, backup_path)
> +            else:
> +                shutil.rmtree(repodir_path)
> +
>          if r_remote:
>              _checkout_git_remote(r_remote, repodir,
> layers_fixed_revisions)
>          if r_local:
> -            _symlink_local(os.path.expanduser(r_local["path"]),
> os.path.join(layerdir,repodir))
> +            _symlink_local(os.path.expanduser(r_local["path"]),
> repodir_path)
>
>          if os.path.exists(os.path.join(layerdir, repodir,
> 'scripts/oe-setup-build')):
>              oesetupbuild = os.path.join(layerdir, repodir,
> 'scripts/oe-setup-build')
> diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
> index 438dc0cd8..5ff489cbe 100644
> --- a/lib/bb/tests/setup.py
> +++ b/lib/bb/tests/setup.py
> @@ -175,6 +175,16 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
>  """ % (self.testrepopath, branch, rev)
>          return self._add_json_config_to_registry_helper(name, sources)
>
> +    def add_local_json_config_to_registry(self, name, path):
> +        sources = """
> +        "test-repo": {
> +            "local": {
> +                "path": "%s"
> +            }
> +        }
> +""" % (path)
> +        return self._add_json_config_to_registry_helper(name, sources)
> +
>      def add_file_to_testrepo(self, name, content, script=False):
>          fullname = os.path.join(self.testrepopath, name)
>          os.makedirs(os.path.join(self.testrepopath,
> os.path.dirname(name)), exist_ok=True)
> @@ -413,7 +423,6 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
>              sums_after = _conf_chksum(f"{setuppath}/build/conf")
>              self.assertEqual(sums_before, sums_after)
>
> -        # check source overrides, local sources provided with symlinks,
> and custom setup dir name
>          def _check_local_sources(custom_setup_dir):
>              custom_setup_path = os.path.join(self.tempdir,
> 'bitbake-builds', custom_setup_dir)
>              custom_layer_path = os.path.join(custom_setup_path, 'layers',
> 'test-repo')
> @@ -421,6 +430,60 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
>              self.assertEqual(self.testrepopath,
> os.path.realpath(custom_layer_path))
>              self.config_is_unchanged(custom_setup_path)
>
> +        # Change the configuration to refer to a local source, then to
> another local source, then back to a git remote
> +        # Run status/update after each change and verify that nothing
> breaks
> +        c = 'gadget'
> +        setuppath = self.get_setup_path('test-config-1', c)
> +        self.config_is_unchanged(setuppath)
> +
> +        json_1 =
> self.add_local_json_config_to_registry('test-config-1.conf.json',
> self.testrepopath)
> +        os.environ['BBPATH'] = os.path.join(setuppath, 'build')
> +        out = self.runbbsetup("update --update-bb-conf='yes'")
> +        _check_local_sources(setuppath)
> +
> +        prev_path = self.testrepopath
> +        self.testrepopath = prev_path + "-2"
> +        self.git("clone {} {}".format(prev_path, self.testrepopath),
> cwd=self.tempdir)
> +        json_1 =
> self.add_local_json_config_to_registry('test-config-1.conf.json',
> self.testrepopath)
> +        os.environ['BBPATH'] = os.path.join(setuppath, 'build')
> +        out = self.runbbsetup("update --update-bb-conf='yes'")
> +        _check_local_sources(setuppath)
> +
> +        self.testrepopath = prev_path
> +        json_1 =
> self.add_json_config_to_registry('test-config-1.conf.json', branch, branch)
> +        os.environ['BBPATH'] = os.path.join(setuppath, 'build')
> +        out = self.runbbsetup("update --update-bb-conf='yes'")
> +        self.check_setupdir_files(setuppath, test_file_content)
> +
> +        # Also check that there are no layer backups up to this point,
> then make a change that should
> +        # result in a layer backup, and check that it does happen.
> +        def _check_layer_backups(layer_path, expected_backups):
> +            files = os.listdir(layer_path)
> +            backups = len([f for f in files if 'backup' in f])
> +            self.assertEqual(backups, expected_backups, msg = "Expected
> {} layer backups, got {}, directory listing: {}".format(expected_backups,
> backups, files))
> +
> +        layers_path = os.path.join(setuppath, 'layers')
> +        layer_path = os.path.join(layers_path, 'test-repo')
> +        _check_layer_backups(layers_path, 0)
> +
> +        ## edit a file without making a commit
> +        with open(os.path.join(layer_path, 'local-modification'), 'w') as
> f:
> +            f.write('locally-modified\n')
> +        test_file_content = "modified-again\n"
> +        self.add_file_to_testrepo('test-file', test_file_content)
> +        os.environ['BBPATH'] = os.path.join(setuppath, 'build')
> +        out = self.runbbsetup("update --update-bb-conf='yes'")
> +        _check_layer_backups(layers_path, 1)
> +
> +        ## edit a file and make a commit
> +        with open(os.path.join(layer_path, 'local-modification'), 'w') as
> f:
> +            f.write('locally-modified-again\n')
> +        self.git('add .', cwd=layer_path)
> +        self.git('commit -m "Adding a local modification"',
> cwd=layer_path)
> +        out = self.runbbsetup("update --update-bb-conf='yes'")
> +        _check_layer_backups(layers_path, 2)
> +
> +        # check source overrides, local sources provided with symlinks,
> and custom setup dir name
>          source_override_content = """
>  {
>      "sources": {
> --
> 2.47.3
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#18680):
> https://lists.openembedded.org/g/bitbake-devel/message/18680
> Mute This Topic: https://lists.openembedded.org/mt/117046550/8181911
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> anibal@limonsoftware.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>