| Message ID | 20251215103256.902711-1-corentin.guillevic@smile.fr |
|---|---|
| State | Accepted, archived |
| Commit | 7941a5dc4dba81ab2141531b8af94371a923b32b |
| Headers | show |
| Series | [v2,1/3] bitbake-setup: add inline URI | expand |
On Mon, 15 Dec 2025 at 11:33, Corentin Guillevic via lists.openembedded.org <corentin.guillevic=smile.fr@lists.openembedded.org> wrote: > + if 'remotes' in r_remote: > + if not 'bitbake-setup-uri' in r_remote['remotes']: > + r_name = 'bitbake-setup-uri' > + else: > + idx = 1 > + while 'bitbake-setup-uri_{}'.format(idx) in r_remote['remotes']: > + idx += 1 > + r_name = 'bitbake-setup-uri_{}'.format(idx) > + else: > + r_name = 'bitbake-setup-uri' > + > + remotes.update({r_name: {'uri': r_remote['uri']}}) Thanks for continuing to work on this. I think we can avoid adding the above code, see below. > - remotes = r_remote['remotes'] > + > + remotes = _get_remotes(r_remote) > > for remote in remotes: > prot,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) The names of the remotes are not actually used anywhere for anything. So the _get_remotes() can return a flat list of URIs instead, and then it doesn't have to do complicated name calculations like implemented above. Then this loop can iterate directly on URIs in the list. Alex
Thank you for your review, I have taken it into account.
However, what is it expected of the registered remote(s) once the source
has been cloned? When using several remotes for one source, I discovered
that they are all cloned but only one is registered in the final layer (in
the layers/ directory). For example, we can add another remote to the
bitbake layer in the 'oe-nodistro-master' configuration, but this results
in only one remote in bitbake-builds/oe-nodistro-master/layers/bitbake/.
How to reproduce:
# In-place edition, add a new remote called "origin-2"
$ jq --indent 4 '.sources.bitbake."git-remote".remotes += {"origin-2":
{"uri": "https://github.com/distro-core-curated-mirrors/bitbake/"}}' \
default-registry/configurations/oe-nodistro-master.conf.json | sponge
default-registry/configurations/oe-nodistro-master.conf.json
# Clear previous work
$ rm -rf bitbake-builds/oe-nodistro-master/
# Choose 'oe-nodistro-master'
$ ./bin/bitbake-setup init
$ git -C bitbake-builds/oe-nodistro-master/layers/bitbake/ remote -v
origin https://github.com/distro-core-curated-mirrors/bitbake/ (fetch)
origin https://github.com/distro-core-curated-mirrors/bitbake/ (push)
What should print the command "git remote -v"?
Corentin
Le lun. 15 déc. 2025 à 12:01, Alexander Kanavin <alex.kanavin@gmail.com> a
écrit :
> On Mon, 15 Dec 2025 at 11:33, Corentin Guillevic via
> lists.openembedded.org
> <corentin.guillevic=smile.fr@lists.openembedded.org> wrote:
> > + if 'remotes' in r_remote:
> > + if not 'bitbake-setup-uri' in r_remote['remotes']:
> > + r_name = 'bitbake-setup-uri'
> > + else:
> > + idx = 1
> > + while 'bitbake-setup-uri_{}'.format(idx) in
> r_remote['remotes']:
> > + idx += 1
> > + r_name = 'bitbake-setup-uri_{}'.format(idx)
> > + else:
> > + r_name = 'bitbake-setup-uri'
> > +
> > + remotes.update({r_name: {'uri': r_remote['uri']}})
>
> Thanks for continuing to work on this.
>
> I think we can avoid adding the above code, see below.
>
> > - remotes = r_remote['remotes']
> > +
> > + remotes = _get_remotes(r_remote)
> >
> > for remote in remotes:
> > prot,host,path,user,pswd,params =
> bb.fetch.decodeurl(remotes[remote]["uri"])
>
> The names of the remotes are not actually used anywhere for anything.
> So the _get_remotes() can return a flat list of URIs instead, and then
> it doesn't have to do complicated name calculations like implemented
> above. Then this loop can iterate directly on URIs in the list.
>
> Alex
>
On Tue, 16 Dec 2025 at 17:11, Corentin GUILLEVIC
<corentin.guillevic@smile.fr> wrote:
> However, what is it expected of the registered remote(s) once the source has been cloned? When using several remotes for one source, I discovered that they are all cloned but only one is registered in the final layer (in the layers/ directory). For example, we can add another remote to the bitbake layer in the 'oe-nodistro-master' configuration, but this results in only one remote in bitbake-builds/oe-nodistro-master/layers/bitbake/.
Hello Corentin,
bitbake-setup is using bitbake's git fetcher to perform fetch/unpack:
fetcher = bb.fetch.Fetch([src_uri], d)
fetcher.download()
fetcher.unpack(dir)
and I think that doesn't have support for multiple remotes that
unpack into the same directory.
It isn't entirely clear how to map multiple remotes (when specified in
json) into bb-setup's behavior, but I think the current behavior at
least makes sure they're all cloned into DL_DIR, so DL_DIR has a local
copy for all of them. We could probably add some after-the fact 'git
remote add <some-remote> ...' operation to the directory that gets
unpacked by the fetcher, but that's a separate issue, and probably a
fairly marginal use case. I have a hunch almost everyone will put just
one remote per source into the json config.
Alex
> It isn't entirely clear how to map multiple remotes (when specified in > json) into bb-setup's behavior, but I think the current behavior at > least makes sure they're all cloned into DL_DIR, so DL_DIR has a local > copy for all of them. Yes I confirm that all URIs are cloned into DL_DIR. Currently, this is the last treated URI that will always be used in the final layer, as every git unpack prunes any previous destination directory. Corentin Le mar. 16 déc. 2025 à 20:18, Alexander Kanavin <alex.kanavin@gmail.com> a écrit : > On Tue, 16 Dec 2025 at 17:11, Corentin GUILLEVIC > <corentin.guillevic@smile.fr> wrote: > > However, what is it expected of the registered remote(s) once the source > has been cloned? When using several remotes for one source, I discovered > that they are all cloned but only one is registered in the final layer (in > the layers/ directory). For example, we can add another remote to the > bitbake layer in the 'oe-nodistro-master' configuration, but this results > in only one remote in bitbake-builds/oe-nodistro-master/layers/bitbake/. > > Hello Corentin, > > bitbake-setup is using bitbake's git fetcher to perform fetch/unpack: > > fetcher = bb.fetch.Fetch([src_uri], d) > fetcher.download() > fetcher.unpack(dir) > > and I think that doesn't have support for multiple remotes that > unpack into the same directory. > > It isn't entirely clear how to map multiple remotes (when specified in > json) into bb-setup's behavior, but I think the current behavior at > least makes sure they're all cloned into DL_DIR, so DL_DIR has a local > copy for all of them. We could probably add some after-the fact 'git > remote add <some-remote> ...' operation to the directory that gets > unpacked by the fetcher, but that's a separate issue, and probably a > fairly marginal use case. I have a hunch almost everyone will put just > one remote per source into the json config. > > Alex >
diff --git a/bin/bitbake-setup b/bin/bitbake-setup index b1d751899..c97e63479 100755 --- a/bin/bitbake-setup +++ b/bin/bitbake-setup @@ -89,6 +89,33 @@ def _write_layer_list(dest, repodirs): with open(layers_f, 'w') as f: json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4) +def _get_remotes(r_remote): + remotes = {} + + if not 'remotes' in r_remote and not 'uri' in r_remote: + raise Exception("Expected key(s): 'remotes', 'uri'") + + if 'remotes' in r_remote: + remotes = r_remote['remotes'].copy() + + if 'uri' in r_remote: + r_name = '' + + if 'remotes' in r_remote: + if not 'bitbake-setup-uri' in r_remote['remotes']: + r_name = 'bitbake-setup-uri' + else: + idx = 1 + while 'bitbake-setup-uri_{}'.format(idx) in r_remote['remotes']: + idx += 1 + r_name = 'bitbake-setup-uri_{}'.format(idx) + else: + r_name = 'bitbake-setup-uri' + + remotes.update({r_name: {'uri': r_remote['uri']}}) + + return remotes + def checkout_layers(layers, layerdir, d): layers_fixed_revisions = copy.deepcopy(layers) repodirs = [] @@ -102,7 +129,8 @@ def checkout_layers(layers, layerdir, d): r_remote = r_data['git-remote'] rev = r_remote['rev'] branch = r_remote.get('branch', None) - remotes = r_remote['remotes'] + + remotes = _get_remotes(r_remote) for remote in remotes: prot,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"]) @@ -571,7 +599,8 @@ def are_layers_changed(layers, layerdir, d): r_remote = r_data['git-remote'] rev = r_remote['rev'] branch = r_remote.get('branch', None) - remotes = r_remote['remotes'] + + remotes = _get_remotes(r_remote) for remote in remotes: type,host,path,user,pswd,params = bb.fetch.decodeurl(remotes[remote]["uri"])
Most of the time, when we describe a remote, a bitbake-setup source looks like this: "bitbake": { "git-remote": { "remotes": { "origin": { "uri": "https://git.openembedded.org/bitbake" } }, ... } } i.e. an URI with the common name 'origin'. Alternatively, we could simplify this, by using a shorter structure with the property 'uri' only: "bitbake": { "git-remote": { "uri": "https://git.openembedded.org/bitbake", ... } } These properties can be used together. Signed-off-by: Corentin Guillevic <corentin.guillevic@smile.fr> --- Changes in v2: - The keys are complementary, rather than mutually exclusive - Updated configurations in default-registry/configurations/ bin/bitbake-setup | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-)