From patchwork Mon Mar 9 23:29:38 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pascal Eberhard via B4 Relay X-Patchwork-Id: 82954 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59051F4180D for ; Mon, 9 Mar 2026 23:31:50 +0000 (UTC) Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.28508.1773099105364651423 for ; Mon, 09 Mar 2026 16:31:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=cBx7vTrZ; spf=pass (domain: kernel.org, ip: 172.234.252.31, mailfrom: devnull+pascal.eberhard.se.com@kernel.org) Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 8EE0F43B3C; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPS id 6C39EC2BC9E; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773099104; bh=Gedtx1s8QCFLRBtxBzgtGmQWx3BTCNPq/TpibUmkpGI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=cBx7vTrZ7D3K94naV+Ofb2tfwUHnqcgoOXJMl687PMtoIxrEIrsAx0WCDcyPF7Yok 7WYwER1qgJRg/98iNNxNy7+m53yjj1JulPAOrHeBecf/xyXGmfug26Au0MKL+nBV6L SvvATIj3a2/l5LXVW3Dmlhed0s50HZNBz78mP0LJjs/+K4lKE0ypWie8lbbz4yxIZ5 JvU7vqKntcE7md6bZgbZYnUddfaOthzseiChLPPqBQ1fl5Tzhi/21REO8uSkBM6dYV ubLrPFJ4g2PCSyMYZPVj5FyX1Y/+WI1V+3EkG0Tg5IUbL48GxhLJmw3RzmyfdP09Um 0Q1WWiTqqlkhQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58BA3F3C277; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) From: Pascal Eberhard via B4 Relay Date: Tue, 10 Mar 2026 00:29:38 +0100 Subject: [PATCH v2 1/4] fetch2: add curl method to fetch web content MIME-Version: 1.0 Message-Id: <20260310-add_alt_fetch_method_curl-v2-1-4789639b28f1@se.com> References: <20260310-add_alt_fetch_method_curl-v2-0-4789639b28f1@se.com> In-Reply-To: <20260310-add_alt_fetch_method_curl-v2-0-4789639b28f1@se.com> To: bitbake-devel@lists.openembedded.org Cc: Pascal Eberhard X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1773099103; l=7321; i=pascal.eberhard@se.com; s=20260304; h=from:subject:message-id; bh=yb76WcTlje9CTcQKALnQhfM8wIKHvARpJCjPybI9J1k=; b=8hwOnC08Amg914qFvfUp0uABVXCyZZ9RQthdPL3g2ANF5VWvUgB9rnU/oiPtJFLlh2GSPVneH 0Z49DkLRfL7BL02IOQbBg5gpJ/bQK+madBbUxeh5WipxELGSil7UzdT X-Developer-Key: i=pascal.eberhard@se.com; a=ed25519; pk=J7TbFctjt1RVuN5K0juhf/w1E9bAfSeoQ/JG1vV9mWg= X-Endpoint-Received: by B4 Relay for pascal.eberhard@se.com/20260304 with auth_id=661 X-Original-From: Pascal Eberhard Reply-To: pascal.eberhard@se.com List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 09 Mar 2026 23:31:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19136 From: Pascal Eberhard curl fetch method is an alternative fetch method for web downloads. It is based on curl cmdline tool and provides the same http, https, ftp and ftps protocols as wget. It supports some new features as well such as hostname resolution by the proxy when using SOCKS5 proxy. Signed-off-by: Pascal Eberhard --- lib/bb/fetch2/curl.py | 162 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 162 insertions(+) diff --git a/lib/bb/fetch2/curl.py b/lib/bb/fetch2/curl.py new file mode 100644 index 000000000..ad32372fb --- /dev/null +++ b/lib/bb/fetch2/curl.py @@ -0,0 +1,162 @@ +""" +BitBake 'Fetch' implementations for web downloads based on curl. + +curl fetch method is an alternative to existing wget method and can be enabled +by setting bitbake variable: + BB_FETCH_METHOD_HTTP = "curl" + +curl fetch method provides new features such as hostname resolution by the +proxy itself when using SOCKS5 proxy. It can be set with environment variable: + all_proxy="socks5h://..."" +""" + +# Copyright (C) 2026, Schneider Electric +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Based on the wget fetcher method, Copyright 2003 Holger Schurig + +import os +import re +import shlex +import tempfile + +import bb +import bb.fetch2 +import bb.progress +import bb.utils +from bb.data_smart import DataSmart +from bb.fetch2 import FetchData, FetchError, logger, runfetchcmd +from bb.fetch2.wget import Wget + + +class CurlProgressHandler(bb.progress.LineFilterProgressHandler): + """ + Extract progress information from curl commandline output. + Note: relies on --progress-bar being specified on the curl command line. + """ + + def __init__(self, d: DataSmart): + super(CurlProgressHandler, self).__init__(d) + # Send an initial progress event so the bar gets shown + self._fire_progress(0) + + def writeline(self, line: str): + matches = re.findall(r' ([\d]+)\.\d%', line) + if matches: + progress = int(matches[0]) + self.update(progress) + return False + return True + + +class Curl(Wget): + """ + Class to fetch urls via curl cmdline tool. + The code not related to the cmdline is the same between wget and curl. + Curl class inherits Wget class to avoid code duplication. + """ + + def is_enabled(self, d) -> bool: + """ + curl method is enabled when BB_FETCH_METHOD_HTTP = "curl" only. + """ + method_http: str = d.getVar("BB_FETCH_METHOD_HTTP") + return method_http == "curl" + + def supports(self, ud: FetchData, d: DataSmart) -> bool: + """ + Check if a given url can be fetched with curl. + """ + if not self.is_enabled(d): + return False + if ud.type not in ['http', 'https', 'ftp', 'ftps']: + return False + logger.debug2("Fetch method 'curl' enabled") + return True + + def urldata_init(self, ud: FetchData, d: DataSmart): + if 'protocol' in ud.parm: + if ud.parm['protocol'] == 'git': + raise bb.fetch2.ParameterError("Invalid protocol - if you wish to fetch from a git repository using http, you need to instead use the git:// prefix with protocol=http", ud.url) + + if 'downloadfilename' in ud.parm: + ud.basename: str = ud.parm['downloadfilename'] + else: + ud.basename: str = os.path.basename(ud.path) + + ud.localfile = ud.basename + if not ud.localfile: + ud.localfile = ud.host + ud.path.replace("/", ".") + + # --retry 1: equivalent to --tries=2 of wget. + # --speed-limit 1 --speed-time 100 --connect-timeout 100: equivalent to --timeout=100 option of wget. + # --location: redo request on new location when a page as moved, indicated with 3xx response code. + # --fail: fails with exit code when server generates HTML error rather than writing HTML error to output. + self.basecmd: str = d.getVar("FETCHCMD_curl") or "/usr/bin/env curl --retry 1 --speed-limit 1 --speed-time 100 --connect-timeout 100 --location --fail" + + if ud.type == 'ftp' or ud.type == 'ftps': + self.basecmd += " --ftp-pasv" + + if not self.check_certs(d): + self.basecmd += " --insecure" + + def _runcurl(self, ud: FetchData, d: DataSmart, command: str, quiet: bool, workdir: str = None): + progresshandler = CurlProgressHandler(d) + + logger.debug2("Fetching %s using command '%s'" % (ud.url, command)) + bb.fetch2.check_network_access(d, command, ud.url) + runfetchcmd(command + " --progress-bar", d, quiet, log=progresshandler, workdir=workdir) + + def download(self, ud: FetchData, d: DataSmart): + """Fetch urls""" + fetchcmd: str = self.basecmd + dldir: str = os.path.realpath(d.getVar("DL_DIR")) + localpath: str = os.path.join(dldir, ud.localfile) + ".tmp" + bb.utils.mkdirhier(os.path.dirname(localpath)) + fetchcmd += " --output %s" % shlex.quote(localpath) + + if ud.user and ud.pswd: + fetchcmd += " --anyauth" + if ud.parm.get("redirectauth", "1") == "1": + fetchcmd += f" --user={ud.user}:{ud.pswd}" + + uri: str = ud.url.split(";")[0] + fetchcmd += f" --continue-at - '{uri}'" + + self._runcurl(ud, d, fetchcmd, False) + + # Sanity check since curl can pretend it succeed when it didn't + # Also, this used to happen if sourceforge sent us to the mirror page + if not os.path.exists(localpath): + raise FetchError(f"The fetch command returned success for url {uri} but {localpath} doesn't exist?!", uri) + + if os.path.getsize(localpath) == 0: + os.remove(localpath) + raise FetchError(f"The fetch of {uri} resulted in a zero size file?! Deleting and failing since this isn't right.", uri) + + # Try and verify any checksum now, meaning if it isn't correct, we don't remove the + # original file, which might be a race (imagine two recipes referencing the same + # source, one with an incorrect checksum) + bb.fetch2.verify_checksum(ud, d, localpath=localpath, fatal_nochecksum=False) + + # Remove the ".tmp" and move the file into position atomically + # Our lock prevents multiple writers but mirroring code may grab incomplete files + os.rename(localpath, localpath[:-4]) + + return True + + def _fetch_index(self, uri: str, ud: FetchData, d: DataSmart): + """ + Run fetch checkstatus to get directory information + """ + with tempfile.TemporaryDirectory(prefix="curl-index-") as workdir, tempfile.NamedTemporaryFile(dir=workdir, prefix="curl-listing-") as f: + fetchcmd: str = self.basecmd + fetchcmd += f" --output {f.name} '{uri}'" + try: + self._runcurl(ud, d, fetchcmd, True, workdir=workdir) + fetchresult = f.read() + except bb.fetch2.BBFetchException: + fetchresult = "" + + return fetchresult From patchwork Mon Mar 9 23:29:39 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pascal Eberhard via B4 Relay X-Patchwork-Id: 82953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9455FFCC9A4 for ; Mon, 9 Mar 2026 23:31:50 +0000 (UTC) Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.28532.1773099105369204505 for ; Mon, 09 Mar 2026 16:31:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=l12AxWVN; spf=pass (domain: kernel.org, ip: 172.234.252.31, mailfrom: devnull+pascal.eberhard.se.com@kernel.org) Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 8EE4844101; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPS id 6F1FEC2BCAF; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773099104; bh=kJPkb8kjs+qAATwbnBMiapdAUb7uPdzbqBSdLN+Oq60=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=l12AxWVN6tAuomYfY6uCwDSP08VEBRW/2EVXHwc1Yst/oZ5kIxHSTM9Hiy1TquqOb kaQb5/2YJ0JhNGfmjP+dqboFacZBsR2ur1nsim3tym0f5QtYajEndMmVkDBnEk9vXu CB0lnBcxXZjclu7pD+ABrDL3XpjgIiyy4EAOUtT8aG9FJ4J3vwp6v7aP1AZ06Jqhvk 5TlFm+vvT9KWPGK88ORsaLNLwYZmg0vTKk3h7DuwZGb1w5ZbyGg+M35RbYoRjZ0Q5v Iw3hjkbe21D9xS10snRXIMSxsD/Ar1vJASK+LTjIIAdXYOZB24mHhcL6mNIvwMc7QH QLfXPIBzMHINg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64C64F3C24D; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) From: Pascal Eberhard via B4 Relay Date: Tue, 10 Mar 2026 00:29:39 +0100 Subject: [PATCH v2 2/4] fetch2: make curl method activable with BB_FETCH_METHOD_HTTP MIME-Version: 1.0 Message-Id: <20260310-add_alt_fetch_method_curl-v2-2-4789639b28f1@se.com> References: <20260310-add_alt_fetch_method_curl-v2-0-4789639b28f1@se.com> In-Reply-To: <20260310-add_alt_fetch_method_curl-v2-0-4789639b28f1@se.com> To: bitbake-devel@lists.openembedded.org Cc: Pascal Eberhard X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1773099103; l=1806; i=pascal.eberhard@se.com; s=20260304; h=from:subject:message-id; bh=64aFNHzuzQvTD39bZT3zcb0N4XDTAFGjqhXZfbW5M2o=; b=7BS9N/jdWn7VYoBfgirMIBO0hUudTtFTbYbqL7+8eMIlpyjVjQGF4rVnQ4UUve6MlugLX0uA2 fa5O7HM89FJBdV51AZQb1jCwMbcgcPRPwNG1pbQEYw3he+QcW6e0Dzb X-Developer-Key: i=pascal.eberhard@se.com; a=ed25519; pk=J7TbFctjt1RVuN5K0juhf/w1E9bAfSeoQ/JG1vV9mWg= X-Endpoint-Received: by B4 Relay for pascal.eberhard@se.com/20260304 with auth_id=661 X-Original-From: Pascal Eberhard Reply-To: pascal.eberhard@se.com List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 09 Mar 2026 23:31:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19137 From: Pascal Eberhard wget is the default fetch method. curl fetch method can be activated by setting bitbake variable: BB_FETCH_METHOD_HTTP = "curl" Signed-off-by: Pascal Eberhard --- lib/bb/fetch2/__init__.py | 2 ++ lib/bb/fetch2/wget.py | 10 ++++++++++ 2 files changed, 12 insertions(+) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index aaefd8602..dc1158b38 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -2107,6 +2107,7 @@ from . import gitannex from . import local from . import svn from . import wget +from . import curl from . import ssh from . import sftp from . import s3 @@ -2123,6 +2124,7 @@ from . import gomod methods.append(local.Local()) methods.append(wget.Wget()) +methods.append(curl.Curl()) methods.append(svn.Svn()) methods.append(git.Git()) methods.append(gitsm.GitSM()) diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index 4e3505599..3b1993f29 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -60,10 +60,20 @@ class Wget(FetchMethod): """ return (d.getVar("BB_CHECK_SSL_CERTS") or "1") != "0" + def is_enabled(self, d): + """ + wget method is enabled when BB_FETCH_METHOD_HTTP = "wget" or by default + when BB_FETCH_METHOD_HTTP variable is not set. + """ + method_http: str = d.getVar("BB_FETCH_METHOD_HTTP") or "wget" + return method_http == "wget" + def supports(self, ud, d): """ Check to see if a given url can be fetched with wget. """ + if not self.is_enabled(d): + return False return ud.type in ['http', 'https', 'ftp', 'ftps'] def recommends_checksum(self, urldata): From patchwork Mon Mar 9 23:29:40 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pascal Eberhard via B4 Relay X-Patchwork-Id: 82951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32F79F41808 for ; Mon, 9 Mar 2026 23:31:50 +0000 (UTC) Received: from tor.source.kernel.org (tor.source.kernel.org [172.105.4.254]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.28533.1773099105763126604 for ; Mon, 09 Mar 2026 16:31:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oGn/7Awb; spf=pass (domain: kernel.org, ip: 172.105.4.254, mailfrom: devnull+pascal.eberhard.se.com@kernel.org) Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id C722D60054; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPS id 77ABBC2BC86; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773099104; bh=+A3FdQDUkn0AZTnosOSfEZzGoBXWcnnu+5djFl2CwYk=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=oGn/7AwbUGamlR3JGzMjaXbANnGIcSbEPc3P+qzW9Jv12DN0QsDBjhSgWBIxaAdbM nlioJc9htrMpv2rLl2/ZPym1hIDOTjN5X38JFD7doXCkTgH6uwJznu7WTyDkRcd0rX g4vl6J+Nu67A5YKOHzQT3JbCyBlYcBsTBY2ruinBE3gAFSslHMrkaVSl7LIqGDGoBw QlXo8rWYdpsQH4dT/BwqI8mMKJ8ItmQu1Z/aykomvy1YH6Q2bdulDln/K1a1GlSyEE LLfN2gixQYBF3xkolz6GOtSSdgXcDiSXWKceO42g6yRoaxTHH/syxIj2l4SOm792HP AWcCOVyCycxng== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70E0CF3C27D; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) From: Pascal Eberhard via B4 Relay Date: Tue, 10 Mar 2026 00:29:40 +0100 Subject: [PATCH v2 3/4] lib/tests/fetch2: add tests for curl method MIME-Version: 1.0 Message-Id: <20260310-add_alt_fetch_method_curl-v2-3-4789639b28f1@se.com> References: <20260310-add_alt_fetch_method_curl-v2-0-4789639b28f1@se.com> In-Reply-To: <20260310-add_alt_fetch_method_curl-v2-0-4789639b28f1@se.com> To: bitbake-devel@lists.openembedded.org Cc: Pascal Eberhard X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1773099103; l=5399; i=pascal.eberhard@se.com; s=20260304; h=from:subject:message-id; bh=IFaMueW2762ELu2lI2L6fxbF1Qag0z/rfsvviir0rz4=; b=mAXuBqPboTozSC7lnXNsTUZIv3gEZXE3VRXB86cOtqb6sJVRqr3xlTrbJI8J9LLeAgoUMwE4/ nqvyNm2RLeQDQzJnGCIEF1mOvUA+Ruv4xIk8iYJdfSd7cySe/L6d5qP X-Developer-Key: i=pascal.eberhard@se.com; a=ed25519; pk=J7TbFctjt1RVuN5K0juhf/w1E9bAfSeoQ/JG1vV9mWg= X-Endpoint-Received: by B4 Relay for pascal.eberhard@se.com/20260304 with auth_id=661 X-Original-From: Pascal Eberhard Reply-To: pascal.eberhard@se.com List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 09 Mar 2026 23:31:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19138 From: Pascal Eberhard Duplicate wget fetch method test cases to test curl fetch method. Signed-off-by: Pascal Eberhard --- lib/bb/tests/fetch.py | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 74eb73472..72257a130 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -28,6 +28,10 @@ def skipIfNoNetwork(): return unittest.skip("network test") return lambda f: f +def skipIfNoCurl(): + if not shutil.which("curl"): + return unittest.skip("'curl' not found") + return lambda f: f @contextlib.contextmanager def hide_directory(directory): @@ -1550,6 +1554,7 @@ class FetchLatestVersionTest(FetcherTest): url = "http://127.0.0.1:%s/%s" % (port, data.path) ud = bb.fetch2.FetchData(url, self.d) + self.assertEqual(ud.method.__class__.__name__, "Wget") pupver = ud.method.latest_versionstring(ud, self.d) verstring = pupver[0] self.assertTrue(verstring, msg="Could not find upstream version for %s" % data.pn) @@ -1558,6 +1563,36 @@ class FetchLatestVersionTest(FetcherTest): finally: server.stop() + @skipIfNoCurl() + def test_curl_latest_versionstring(self): + self.d.setVar("BB_FETCH_METHOD_HTTP", "curl") + testdata = os.path.dirname(os.path.abspath(__file__)) + "/fetch-testdata" + server = HTTPService(testdata, host="127.0.0.1") + server.start() + port = server.port + try: + for data, v in self.test_wget_uris.items(): + with self.subTest(pn=data.pn): + self.d.setVar("PN", data.pn) + self.d.setVar("PV", data.pv) + if data.check_uri: + checkuri = "http://127.0.0.1:%s/%s" % (port, data.check_uri) + self.d.setVar("UPSTREAM_CHECK_URI", checkuri) + if data.check_regex: + self.d.setVar("UPSTREAM_CHECK_REGEX", data.check_regex) + + url = "http://127.0.0.1:%s/%s" % (port, data.path) + ud = bb.fetch2.FetchData(url, self.d) + self.assertEqual(ud.method.__class__.__name__, "Curl") + pupver = ud.method.latest_versionstring(ud, self.d) + verstring = pupver[0] + self.assertTrue(verstring, msg="Could not find upstream version for %s" % data.pn) + r = bb.utils.vercmp_string(v, verstring) + self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (data.pn, v, verstring)) + finally: + server.stop() + self.d.delVar("BB_FETCH_METHOD_HTTP") + @skipIfNoNetwork() def test_crate_latest_versionstring(self): for k, v in self.test_crate_uris.items(): @@ -1590,6 +1625,7 @@ class FetchCheckStatusTest(FetcherTest): with self.subTest(url=u): ud = fetch.ud[u] m = ud.method + self.assertEqual(ud.method.__class__.__name__, "Wget") ret = m.checkstatus(fetch, ud, self.d) self.assertTrue(ret, msg="URI %s, can't check status" % (u)) @@ -1605,10 +1641,49 @@ class FetchCheckStatusTest(FetcherTest): with self.subTest(url=u): ud = fetch.ud[u] m = ud.method + self.assertEqual(ud.method.__class__.__name__, "Wget") + ret = m.checkstatus(fetch, ud, self.d) + self.assertTrue(ret, msg="URI %s, can't check status" % (u)) + + connection_cache.close_connections() + + @skipIfNoCurl() + @skipIfNoNetwork() + def test_curl_checkstatus(self): + self.d.setVar("BB_FETCH_METHOD_HTTP", "curl") + + fetch = bb.fetch2.Fetch(self.test_wget_uris, self.d) + for u in self.test_wget_uris: + with self.subTest(url=u): + ud = fetch.ud[u] + m = ud.method + self.assertEqual(ud.method.__class__.__name__, "Curl") + ret = m.checkstatus(fetch, ud, self.d) + self.assertTrue(ret, msg="URI %s, can't check status" % (u)) + + self.d.delVar("BB_FETCH_METHOD_HTTP") + + @skipIfNoCurl() + @skipIfNoNetwork() + def test_curl_checkstatus_connection_cache(self): + from bb.fetch2 import FetchConnectionCache + + self.d.setVar("BB_FETCH_METHOD_HTTP", "curl") + + connection_cache = FetchConnectionCache() + fetch = bb.fetch2.Fetch(self.test_wget_uris, self.d, + connection_cache = connection_cache) + + for u in self.test_wget_uris: + with self.subTest(url=u): + ud = fetch.ud[u] + m = ud.method + self.assertEqual(ud.method.__class__.__name__, "Curl") ret = m.checkstatus(fetch, ud, self.d) self.assertTrue(ret, msg="URI %s, can't check status" % (u)) connection_cache.close_connections() + self.d.delVar("BB_FETCH_METHOD_HTTP") class GitMakeShallowTest(FetcherTest): From patchwork Mon Mar 9 23:29:41 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pascal Eberhard via B4 Relay X-Patchwork-Id: 82952 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C4DDFCC9A2 for ; Mon, 9 Mar 2026 23:31:50 +0000 (UTC) Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.28531.1773099105366453658 for ; Mon, 09 Mar 2026 16:31:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=f18DzLkj; spf=pass (domain: kernel.org, ip: 172.234.252.31, mailfrom: devnull+pascal.eberhard.se.com@kernel.org) Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id A5ABB434A4; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPS id 8616DC2BCB0; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773099104; bh=cvbiLheeJnhWQf/i0mAc32p4j272BkKF0rHKq4OAZSo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=f18DzLkjMZ6pPgoCMWPGUYv23geTEiey6h6oAK3HrZ8cLHOaInbYEXe4OqBjRxeck +P350/2TrSuH8RKusyo/ZhoNaEmhbGZ3tOvdAzS51Dpc8nCdFOTQFSHkFOuyaX9kRC 5Ompf+IVA7VaCEtShe7GW7YzALdHaqrA5b9NVTTqDvklRYVNRYifa1/GEBOF7RJ1bY m9qakJpj/tgIb01ABrPBbh0BwAeVsN9Wa06G4y0vGf/9GSmPh2XwYfRfBXwIGf2pAf jOj65cRGe/HpajMRCQLqtgVSyLlK7URlB/iep7Ju8/yJtdZXlSVOH1e1bd/vthqj+I FvmcHZ3wmuMqw== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B99DFCA16F; Mon, 9 Mar 2026 23:31:44 +0000 (UTC) From: Pascal Eberhard via B4 Relay Date: Tue, 10 Mar 2026 00:29:41 +0100 Subject: [PATCH v2 4/4] doc: bitbake-user-manual-ref-variables: describe BB_FETCH_METHOD_HTTP variable MIME-Version: 1.0 Message-Id: <20260310-add_alt_fetch_method_curl-v2-4-4789639b28f1@se.com> References: <20260310-add_alt_fetch_method_curl-v2-0-4789639b28f1@se.com> In-Reply-To: <20260310-add_alt_fetch_method_curl-v2-0-4789639b28f1@se.com> To: bitbake-devel@lists.openembedded.org Cc: Pascal Eberhard X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1773099103; l=1168; i=pascal.eberhard@se.com; s=20260304; h=from:subject:message-id; bh=mMN0eDTcP33G0a41mnOnUsgTrlLpjDXr5F9CTc4gENE=; b=tCoOqYT9U8lc+3v45Zk2wD/AotwoTq55GzTJzxnDpMj+cD/Fa10N+AFP1hQHU/3LZqRfYhWgc zlKbv2tgStqButDIcDaFdwSiDSvTK62jjEljlfkHtuSKCd+3hDqFhJv X-Developer-Key: i=pascal.eberhard@se.com; a=ed25519; pk=J7TbFctjt1RVuN5K0juhf/w1E9bAfSeoQ/JG1vV9mWg= X-Endpoint-Received: by B4 Relay for pascal.eberhard@se.com/20260304 with auth_id=661 X-Original-From: Pascal Eberhard Reply-To: pascal.eberhard@se.com List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 09 Mar 2026 23:31:50 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19135 From: Pascal Eberhard add BB_FETCH_METHOD_HTTP variable to select between wget and curl method for http, https, ftp, ftps protocols. Signed-off-by: Pascal Eberhard --- doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst index 06bd53619..c459ed3af 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -311,6 +311,11 @@ overview of their function and contents. search the main :term:`SRC_URI` or :term:`MIRRORS`. + :term:`BB_FETCH_METHOD_HTTP` + When set to "curl", cause BitBake's fetcher to use curl method for + downloading web artifacts via http, https, ftp or ftps. + When not set (default) or set to "wget", wget method is used. + :term:`BB_FILENAME` Contains the filename of the recipe that owns the currently running task. For example, if the ``do_fetch`` task that resides in the