| Message ID | 20260305-add_alt_fetch_method_curl-v1-0-0d0220e5fa59@se.com |
|---|---|
| Headers | show |
| Series | fetch2: add alternative fetch method based on curl | expand |
On Thu, 2026-03-05 at 16:32 +0100, Pascal Eberhard via B4 Relay via lists.openembedded.org wrote: > wget fetch method is used for web downloads with http, https, ftp, ftps > protocols. wget cmdline tool is missing some features such as hostname > resolution by the proxy when using SOCKS5 proxy. SSH tunnel provides > this feature for example. > > This curl fetch method is based on curl cmdline tool and provides the > same protocols as wget and supports SOCKS5 hostname resolution. > This class inherits wget method in order to avoid code duplication as > much as possible. > > wget remains the default download method. curl fetch method can be > enabled by setting a new bitbake variable: > BB_FETCH_METHOD_HTTP = "curl" > > The hostname resolution by SOCKS5 proxy is activated by setting > environment variable: > all_proxy="socks5h://..."" > > a patch on ce-core will be sent if this proposal is accepted to > conditionnaly add curl to HOSTTOOLS variable in meta/conf/bitbake.conf: > HOSTTOOLS += "${@bb.utils.contains('BB_FETCH_METHOD_HTTP', 'curl', 'curl','', d)}" > > Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com> Hi Pascal, Adding a second HTTP fetcher to bitbake increases complexity and maintenance cost. The behaviour of wget and curl is unlikely to match exactly and so issues will be caused when people switch from one to the other (for whatever reason). Is it just socks5h:// support that you need? Do you know if there is a bug/feature request open with wget upstream regarding socks5h support? Also, do you know the minimum curl version required to support socks5h? This may be important as we support multiple distros which may have different curl versions. Best regards,
On Thu Mar 05, 2026 at 19:09, Paul Barker wrote: > On Thu, 2026-03-05 at 16:32 +0100, Pascal Eberhard via B4 Relay via > lists.openembedded.org wrote: > > > wget fetch method is used for web downloads with http, https, ftp, ftps > > protocols. wget cmdline tool is missing some features such as hostname > > resolution by the proxy when using SOCKS5 proxy. SSH tunnel provides > > this feature for example. > > > > This curl fetch method is based on curl cmdline tool and provides the > > same protocols as wget and supports SOCKS5 hostname resolution. > > This class inherits wget method in order to avoid code duplication as > > much as possible. > > > > wget remains the default download method. curl fetch method can be > > enabled by setting a new bitbake variable: > > BB_FETCH_METHOD_HTTP = "curl" > > > > The hostname resolution by SOCKS5 proxy is activated by setting > > environment variable: > > all_proxy="socks5h://..."" > > > > a patch on ce-core will be sent if this proposal is accepted to > > conditionnaly add curl to HOSTTOOLS variable in meta/conf/bitbake.conf: > > HOSTTOOLS += "${@bb.utils.contains('BB_FETCH_METHOD_HTTP', 'curl', 'curl','', d)}" > > > > Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com> > > > Hi Pascal, Hi Paul, > Adding a second HTTP fetcher to bitbake increases complexity and > maintenance cost. The behaviour of wget and curl is unlikely to match > exactly and so issues will be caused when people switch from one to the > other (for whatever reason). Indeed, I understand having to ponder the value given the increase in the maintenance burden. > Is it just socks5h:// support that you need? Do you know if there is a > bug/feature request open with wget upstream regarding socks5h support? Actually, wget has no support at all for SOCKS proxy :(. A workaround with tsocks works for SOCKS4 and SOCKS5, but hostname resolution isn’t possible that way. So implementing socks5h in wget would require implementing SOCKS proxy support from scratch, which explain the choice of curl. > Also, do you know the minimum curl version required to support socks5h? > This may be important as we support multiple distros which may have > different curl versions. The 'socks5h' scheme has been supported since 2011. The motivation for this proposal was SOCKS proxy support, but curl may have other benefits than I am not aware of. Best regards, Pascal
wget fetch method is used for web downloads with http, https, ftp, ftps protocols. wget cmdline tool is missing some features such as hostname resolution by the proxy when using SOCKS5 proxy. SSH tunnel provides this feature for example. This curl fetch method is based on curl cmdline tool and provides the same protocols as wget and supports SOCKS5 hostname resolution. This class inherits wget method in order to avoid code duplication as much as possible. wget remains the default download method. curl fetch method can be enabled by setting a new bitbake variable: BB_FETCH_METHOD_HTTP = "curl" The hostname resolution by SOCKS5 proxy is activated by setting environment variable: all_proxy="socks5h://..."" a patch on ce-core will be sent if this proposal is accepted to conditionnaly add curl to HOSTTOOLS variable in meta/conf/bitbake.conf: HOSTTOOLS += "${@bb.utils.contains('BB_FETCH_METHOD_HTTP', 'curl', 'curl','', d)}" Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com> --- Pascal Eberhard (4): fetch2: add curl method to fetch web content fetch2: make curl method activable with BB_FETCH_METHOD_HTTP lib/tests/fetch2: add tests for curl method doc: bitbake-user-manual-ref-variables: describe BB_FETCH_METHOD_HTTP variable .../bitbake-user-manual-ref-variables.rst | 5 + lib/bb/fetch2/__init__.py | 2 + lib/bb/fetch2/curl.py | 162 +++++++++++++++++++++ lib/bb/fetch2/wget.py | 10 ++ lib/bb/tests/fetch.py | 68 +++++++++ 5 files changed, 247 insertions(+) --- base-commit: 48efc36b4e03f736e7521d269ced3417522784e9 change-id: 20260305-add_alt_fetch_method_curl-7a0bf96ad223 Best regards,