[v3,1/1] fetch2: try_mirror_url(): Skip invalid local url

From: Robert Yang <liezhi.yang@windriver.com>

There can be multiple PREMIRRORs each PREMIRROR contains specifics sources for
each layer, each recipe will try the PREMIRRORs one by one until succeed, but
the trying would be failed if the PREMIRROR doesn't contain the required
sources, so return it immediately to make log.do_fetch clean, and tt also can
fix a warning when BB_GIT_SHALLOW and is enabled and failed to fetch the source
from the PREMIRROR:

WARNING: Fast shallow clone failed, try to skip fast mode now.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/fetch2/__init__.py | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

On Tue, 2025-08-12 at 01:04 -0700, Robert Yang via
lists.openembedded.org wrote:
> From: Robert Yang <liezhi.yang@windriver.com>
> 
> There can be multiple PREMIRRORs each PREMIRROR contains specifics sources for
> each layer, each recipe will try the PREMIRRORs one by one until succeed, but
> the trying would be failed if the PREMIRROR doesn't contain the required
> sources, so return it immediately to make log.do_fetch clean, and tt also can
> fix a warning when BB_GIT_SHALLOW and is enabled and failed to fetch the source
> from the PREMIRROR:
> 
> WARNING: Fast shallow clone failed, try to skip fast mode now.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/fetch2/__init__.py | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/bitbake/lib/bb/fetch2/__init__.py
> b/bitbake/lib/bb/fetch2/__init__.py
> index 0ad987c596..6149b1726a 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -1067,6 +1067,22 @@ def try_mirror_url(fetch, origud, ud, ld,
> check = False):
>      # Return of None or a value means we're finished
>      # False means try another url
>  
> +    # Skip fetching it when the local url's path doesn't exist
> +    if ud.parm.get('protocol', '') == 'file':
> +        found = False
> +        check_paths = [ud.path]
> +        # Git works with or without '.git' suffix
> +        if ud.path.endswith('.git'):
> +            check_paths.append(ud.path[:-4])
> +        else:
> +            check_paths.append('%s.git' % ud.path)
> +        for check_path in check_paths:
> +            if os.path.exists(check_path):
> +                found = True
> +                break
> +        if not found:
> +            return False
> +
>      if ud.lockfile and ud.lockfile != origud.lockfile:
>          lf = bb.utils.lockfile(ud.lockfile)

Absolutely not.

This is adding git specific code into the main fetcher functions and
the whole idea is this kind of thing is meant to be abstracted.

The fetcher logging is a mess and we should probably be improving that
up rather than trying to add special cases to the code.

Cheers,

Richard

Hi RP,

On 8/12/25 17:12, Richard Purdie wrote:
> On Tue, 2025-08-12 at 01:04 -0700, Robert Yang via
> lists.openembedded.org wrote:
>> From: Robert Yang <liezhi.yang@windriver.com>
>>
>> There can be multiple PREMIRRORs each PREMIRROR contains specifics sources for
>> each layer, each recipe will try the PREMIRRORs one by one until succeed, but
>> the trying would be failed if the PREMIRROR doesn't contain the required
>> sources, so return it immediately to make log.do_fetch clean, and tt also can
>> fix a warning when BB_GIT_SHALLOW and is enabled and failed to fetch the source
>> from the PREMIRROR:
>>
>> WARNING: Fast shallow clone failed, try to skip fast mode now.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/fetch2/__init__.py | 16 ++++++++++++++++
>>   1 file changed, 16 insertions(+)
>>
>> diff --git a/bitbake/lib/bb/fetch2/__init__.py
>> b/bitbake/lib/bb/fetch2/__init__.py
>> index 0ad987c596..6149b1726a 100644
>> --- a/bitbake/lib/bb/fetch2/__init__.py
>> +++ b/bitbake/lib/bb/fetch2/__init__.py
>> @@ -1067,6 +1067,22 @@ def try_mirror_url(fetch, origud, ud, ld,
>> check = False):
>>       # Return of None or a value means we're finished
>>       # False means try another url
>>   
>> +    # Skip fetching it when the local url's path doesn't exist
>> +    if ud.parm.get('protocol', '') == 'file':
>> +        found = False
>> +        check_paths = [ud.path]
>> +        # Git works with or without '.git' suffix
>> +        if ud.path.endswith('.git'):
>> +            check_paths.append(ud.path[:-4])
>> +        else:
>> +            check_paths.append('%s.git' % ud.path)
>> +        for check_path in check_paths:
>> +            if os.path.exists(check_path):
>> +                found = True
>> +                break
>> +        if not found:
>> +            return False
>> +
>>       if ud.lockfile and ud.lockfile != origud.lockfile:
>>           lf = bb.utils.lockfile(ud.lockfile)
> 
> Absolutely not.
> 
> This is adding git specific code into the main fetcher functions and
> the whole idea is this kind of thing is meant to be abstracted.
> 
> The fetcher logging is a mess and we should probably be improving that

The problem is the following warning:

WARNING: Fast shallow clone failed, try to skip fast mode now.

So how about move the code into git.py?

Add the code to __init__.py can also reduce the logs in log.do_fetch, so I added 
it here.

// Robert

> up rather than trying to add special cases to the code.
> 
> Cheers,
> 
> Richard