| Message ID | 20260310-add_alt_fetch_method_curl-v2-0-4789639b28f1@se.com |
|---|---|
| Headers | show |
| Series | fetch2: add alternative fetch method based on curl | expand |
On Tue, 10 Mar 2026 at 00:31, Pascal Eberhard via B4 Relay via lists.openembedded.org <pascal.eberhard=gmail.com@lists.openembedded.org> wrote: > wget fetch method is used for web downloads with http, https, ftp, ftps > protocols. wget cmdline tool is missing some features such as hostname > resolution by the proxy when using SOCKS5 proxy. SSH tunnel provides > this feature for example. > > This curl fetch method is based on curl cmdline tool and provides the > same protocols as wget and supports SOCKS5 hostname resolution. > This class inherits wget method in order to avoid code duplication as > much as possible. There has not been much demand for adding curl support (meaning, none :) Would you be okay with just setting FETCHCMD_wget to a translation wrapper to curl? Alex
wget fetch method is used for web downloads with http, https, ftp, ftps protocols. wget cmdline tool is missing some features such as hostname resolution by the proxy when using SOCKS5 proxy. SSH tunnel provides this feature for example. This curl fetch method is based on curl cmdline tool and provides the same protocols as wget and supports SOCKS5 hostname resolution. This class inherits wget method in order to avoid code duplication as much as possible. wget remains the default download method. curl fetch method can be enabled by setting a new bitbake variable: BB_FETCH_METHOD_HTTP = "curl" The hostname resolution by SOCKS5 proxy is activated by setting environment variable: all_proxy="socks5h://..."" a patch on ce-core will be sent if this proposal is accepted to conditionnaly add curl to HOSTTOOLS variable in meta/conf/bitbake.conf: HOSTTOOLS += "${@bb.utils.contains('BB_FETCH_METHOD_HTTP', 'curl', 'curl','', d)}" Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com> --- Changes in v2: - removed type hint unions and tested with Python 3.9, thanks Mathieu Dubois-Briand for reporting this. - curl tests are now skipped when curl cmdline tool is not found. - Link to v1: https://lore.kernel.org/r/20260305-add_alt_fetch_method_curl-v1-0-0d0220e5fa59@se.com --- Pascal Eberhard (4): fetch2: add curl method to fetch web content fetch2: make curl method activable with BB_FETCH_METHOD_HTTP lib/tests/fetch2: add tests for curl method doc: bitbake-user-manual-ref-variables: describe BB_FETCH_METHOD_HTTP variable .../bitbake-user-manual-ref-variables.rst | 5 + lib/bb/fetch2/__init__.py | 2 + lib/bb/fetch2/curl.py | 162 +++++++++++++++++++++ lib/bb/fetch2/wget.py | 10 ++ lib/bb/tests/fetch.py | 75 ++++++++++ 5 files changed, 254 insertions(+) --- base-commit: 48efc36b4e03f736e7521d269ced3417522784e9 change-id: 20260305-add_alt_fetch_method_curl-7a0bf96ad223 Best regards,