From patchwork Sat Oct 5 13:44:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 49971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F2E2CF34C4 for ; Sat, 5 Oct 2024 13:45:08 +0000 (UTC) Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by mx.groups.io with SMTP id smtpd.web11.11807.1728135901114852173 for ; Sat, 05 Oct 2024 06:45:01 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=IejxgbNQ; spf=softfail (domain: sakoman.com, ip: 209.85.215.178, mailfrom: steve@sakoman.com) Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-7db90a28cf6so2744963a12.0 for ; Sat, 05 Oct 2024 06:45:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1728135900; x=1728740700; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=FcXh5BpwvASJQJ4wEn0Jo/vEfqE2uuxFa7SwAZZK89g=; b=IejxgbNQJRvPH2A0Gkkw6lbe/EBRhH/CfTbZ8ytXhEy/F2zjbWEP+SMAYCUGY2vSsp j0f8t+rXY9VwjDLV6Glmh/cajZ/StwaPBSxGEUQSdh/SXLaO7EjutzDFsCFfGE9+mTBL 2EcxLfUIzqDploRMStXvhOtbAOsbn285naWROTS0xqvl9d10Oa0/XVd+UApl0A6X6fU3 TCjoV2nOITdhxRe2rPy+1Tpxow6I+UJdOPu5iyGqs2L8N24B3zmlpNxfjjgnW81nEjM+ Jk5/yOL2aeaA7A5ppm4D/jfFSf1X0WiF0RO/+GJktFqD7EFWEgAHbeGbteKRuHbqyKaM iYEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728135900; x=1728740700; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FcXh5BpwvASJQJ4wEn0Jo/vEfqE2uuxFa7SwAZZK89g=; b=ldMSt6jTGN2EF0iRqdOslawzqf5mkn6GsR8VDimm152ldTro+nf7UAWKeW7ap3znMU 11QCSO/VapFS53dmfcIAg+Hwi6bKF8+y3+CY4ifmM48Rl8ayYVmSXz6lkY4Rgal8FO4+ 8vTogFxm1oflo5+QAjDUpIBF+KliA2pnV38/JqkkcSsSJ1O0GjB4tWObFZu3QasrCa2r 1lU5sR/MV2onfy1D7EkswcAsmQoiyNm4dqadtzDBCazvgXzeEcbn5Ewz4GgonZxKzKfb 5UoPG/S+5V6j3852W2aZfnzNyPuDaaBEzTUU9F86XhynrMG4Ixn9kNLYNAjiI9Eyo/Wh 2CCw== X-Gm-Message-State: AOJu0Yxtpj5RWuouQHLsBnim5Oqgr3fuRn4IzYDPKsDfuIzuEVziie1P N+Nasaz2VKXs4si9CJ6NK2lZqZYjWZgg7Nbh67CjBFMLGp9GDknErm2d28/BZrqZNYM9R8GwQd2 DIvg= X-Google-Smtp-Source: AGHT+IGWCUdXjfzIuBI41BQvcj6OXU8JLNUZGUTBCWLpC17KE89nJLV+p3VsNLtSwAtfv1clWX26zA== X-Received: by 2002:a17:90a:3ea5:b0:2da:88b3:cff8 with SMTP id 98e67ed59e1d1-2e1e5b49169mr9471495a91.6.1728135900313; Sat, 05 Oct 2024 06:45:00 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e20b0f919csm1831617a91.44.2024.10.05.06.44.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 05 Oct 2024 06:45:00 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 1/4] wget: Make wget --passive-ftp option conditional on ftp/ftps Date: Sat, 5 Oct 2024 06:44:48 -0700 Message-Id: <477f90f9515cb7cd093920cee5518f662cef2e05.1728135724.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 05 Oct 2024 13:45:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16641 From: Rob Woolley Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. Signed-off-by: Rob Woolley Signed-off-by: Richard Purdie (cherry picked from commit f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Steve Sakoman --- lib/bb/fetch2/wget.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index b3a3de571..9bcb0d4b7 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -88,7 +88,10 @@ class Wget(FetchMethod): if not ud.localfile: ud.localfile = d.expand(urllib.parse.unquote(ud.host + ud.path).replace("/", ".")) - self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30 --passive-ftp" + self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30" + + if ud.type == 'ftp' or ud.type == 'ftps': + self.basecmd += " --passive-ftp" if not self.check_certs(d): self.basecmd += " --no-check-certificate" From patchwork Sat Oct 5 13:44:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 49972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 569FFCFA741 for ; Sat, 5 Oct 2024 13:45:08 +0000 (UTC) Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by mx.groups.io with SMTP id smtpd.web11.11809.1728135902633257515 for ; Sat, 05 Oct 2024 06:45:02 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=ZaYmliFL; spf=softfail (domain: sakoman.com, ip: 209.85.216.49, mailfrom: steve@sakoman.com) Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2e082bf1c7fso2187131a91.3 for ; Sat, 05 Oct 2024 06:45:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1728135902; x=1728740702; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=QzO3nwQqGxFP2DyvlxKQkMXVJaej6DjFda5/+UO1kKY=; b=ZaYmliFLctlnEwHw/AF32/XW7MbH08jHqGmj4bN/C91KrqayvnINVRxxf0dqm3IqgN YrkgPA3keU+9etxAm2nD2z0I3PNNRPBmJgzQZyZ1F/NADPgI0edBFmUj6aq93RErVJtM I98E7YetHo8vxrmom74MrIHsezEa0728my+QQA62ejSLXOVrqftFI8KjQK4RX/Oqkzrf aAlekUwmRyV4h+qAzZpZTX/vSS7NqtNUSa9EkuUorYgR5zlpJ4pgdlp/U8UEtPQZqv0K UTkb7eLUW/skKJByQ9ZNA4vy37XIc8J7gb9LNUsWwq2Xq8mRPrB8ANOyulcJmUy2F5sJ GCxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728135902; x=1728740702; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QzO3nwQqGxFP2DyvlxKQkMXVJaej6DjFda5/+UO1kKY=; b=S9g4RIfTJSEoom1XYjp+I6eQyIH2gxmQNz3R28d0dsrZ3L1mr8h+9NHY9pBRdgPiVs 7FyRyDhSosveRldsBZeBIWN8HrWTZN0Dvy7pI41SsYUqDayu1C0p6BvpK+/hGq6/nNrN xIDuqQCbNRwHRfpU3mzVdO350RrETEoyNfrP7atgAOaUwh4Sd3h5B4+w99c0y4fmsLH6 Qisu85JwSOeiZcQlKMqIsfo9A1O2QeunIJkNyoFg3thrwy+PAX80GUZC1fSX0mhjd6Xj TK/iH4YhAdFfM808EFutiJtfmKgl55lqr2eYnu5KHBbcnzoJATStBtaXVFTWIMe1KPNj 0znQ== X-Gm-Message-State: AOJu0Yy9sNaOz8y6Rr/H45Qn+V2palAcrDFPsKBm5GrcMpjbHA5K4s9Y 3cK1vulkSydOZ53AzDpcVULCkZ+s3ZhZAS1kd8OqMF/M6hYqn2ImRYCpqqAQ+JPsNrp+Yv2pon1 eYpY= X-Google-Smtp-Source: AGHT+IGpT7zBmzDHCmE1m/Npdjw+YWfDrfCez+L6vJtFc03ho0NhcfnA/MNz4eg0xDq5+yXIQ0S0kA== X-Received: by 2002:a17:90a:e2d5:b0:2e0:778f:d774 with SMTP id 98e67ed59e1d1-2e1e6213064mr7592468a91.2.1728135901851; Sat, 05 Oct 2024 06:45:01 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e20b0f919csm1831617a91.44.2024.10.05.06.45.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 05 Oct 2024 06:45:01 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 2/4] tests/fetch: Tweak to work on Fedora40 Date: Sat, 5 Oct 2024 06:44:49 -0700 Message-Id: <3147ebb0edcca8f7f850aabd86ef010ab593496f.1728135724.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 05 Oct 2024 13:45:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16642 From: Richard Purdie On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1 and python only binds to one of the addresses, leading to test failures. Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying to test things other than the host networking. Signed-off-by: Richard Purdie (cherry picked from commit 9adc6da42618f41bf0d6b558d62b2f3c13bedd61) Signed-off-by: Steve Sakoman --- lib/bb/tests/fetch.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index a7d9d5e88..efe5479a6 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -1417,7 +1417,7 @@ class FetchLatestVersionTest(FetcherTest): def test_wget_latest_versionstring(self): testdata = os.path.dirname(os.path.abspath(__file__)) + "/fetch-testdata" - server = HTTPService(testdata) + server = HTTPService(testdata, host="127.0.0.1") server.start() port = server.port try: @@ -1425,10 +1425,10 @@ class FetchLatestVersionTest(FetcherTest): self.d.setVar("PN", k[0]) checkuri = "" if k[2]: - checkuri = "http://localhost:%s/" % port + k[2] + checkuri = "http://127.0.0.1:%s/" % port + k[2] self.d.setVar("UPSTREAM_CHECK_URI", checkuri) self.d.setVar("UPSTREAM_CHECK_REGEX", k[3]) - url = "http://localhost:%s/" % port + k[1] + url = "http://127.0.0.1:%s/" % port + k[1] ud = bb.fetch2.FetchData(url, self.d) pupver = ud.method.latest_versionstring(ud, self.d) verstring = pupver[0] From patchwork Sat Oct 5 13:44:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 49970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56A57CFA742 for ; Sat, 5 Oct 2024 13:45:08 +0000 (UTC) Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by mx.groups.io with SMTP id smtpd.web10.11706.1728135904062699508 for ; Sat, 05 Oct 2024 06:45:04 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=pJopUML5; spf=softfail (domain: sakoman.com, ip: 209.85.215.173, mailfrom: steve@sakoman.com) Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-7e9f8d67df9so478717a12.1 for ; Sat, 05 Oct 2024 06:45:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1728135903; x=1728740703; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=i/VvMUnrpLzefVY0jP966Q2lBB8psSFaxQ7klS6H9VU=; b=pJopUML57uXtDV+q0aS1Cx8hxNepIib79CIrtRAqxfAk/YdZxGxiSsdRAkj7ZVNF2r 90kBypgYyyw1Sr9NZjVvZMp79VcIIL2Xb4oxdinqDMxvi1qPvYbjwKJ4atKdGYWZrW0+ za+h0LLgXy3u5pWzcLzP0tTaRgaH6Sz52p8S3orNvdIpyx8k3+IhdblNC8pLSWQbkLHv s65DoAtT76xAgdQcKioyl5NLRoYPggbq118OzHZzq5KJ6oTlrcAlUMhVyZdWaHxMYXiP QUJYNGX6ESpbQtU5DLCwOlaeLQOUA5wDHSCxmjpS4conhNEewsD8X33S4odpzIiBr5rl DkSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728135903; x=1728740703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i/VvMUnrpLzefVY0jP966Q2lBB8psSFaxQ7klS6H9VU=; b=v/Y+TesVTiTTx3IeMFlAjVW52g8GR683HZ9a+kkpjGc+9zJfIXFsJP2cb1+0g6UNb0 z/a96GryEixgYOkJKt/yDBau3awRi8Qjo5UZ0pOA1dn14pPOmmKwBI/24KduSJQdqbHl 9gvCJK8Tmm0YVdDmV4C49VzzW8OOIo6iywJrXiBBiMFvqIjfH8Br5I+WnYsLXB5d7QzG X9y5UjEUx+keeBGxC612swbcqd00kUJmjfWXUmL7sGE/mWb8p1+1IMyQq2iaMAvY/+zI yvnW2ihzEHAYxxtYdcpSaoVyypXptDrqmWamQ+GJHtyBy2WJw2byzD6/W+YSa3+J53yO /22A== X-Gm-Message-State: AOJu0Yyk8X/SPqhqX9Q+3r66qHmcjIaqi4VQEPRuuWyU9oSV9lsDWj0j KwdRM2cXZ7Hy0f6eJrJV7wvAEyenUWzJ+/zx1fSRTLAafDjb+cE/JJtQdjlJC4I9nYpzqxs49Q0 U2DU= X-Google-Smtp-Source: AGHT+IG5i6BhS7LPrD8sGPXs4G3HDIfE1tc5vcgJo3i1Qaaca5eswrGeFfARXnsBZ5umb3zNFK6kWg== X-Received: by 2002:a17:90a:b805:b0:2d8:7804:b3a with SMTP id 98e67ed59e1d1-2e1e631ef6emr6974022a91.26.1728135903299; Sat, 05 Oct 2024 06:45:03 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e20b0f919csm1831617a91.44.2024.10.05.06.45.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 05 Oct 2024 06:45:03 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 3/4] fetch/wget: Move files into place atomically Date: Sat, 5 Oct 2024 06:44:50 -0700 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 05 Oct 2024 13:45:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16643 From: Richard Purdie Signed-off-by: Richard Purdie (cherry picked from commit cd7cce4cf4be5c742d29671169354fe84220b47a) Signed-off-by: Steve Sakoman --- lib/bb/fetch2/wget.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index 9bcb0d4b7..708f3a42e 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -109,10 +109,9 @@ class Wget(FetchMethod): fetchcmd = self.basecmd - if 'downloadfilename' in ud.parm: - localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) - bb.utils.mkdirhier(os.path.dirname(localpath)) - fetchcmd += " -O %s" % shlex.quote(localpath) + localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) + ".tmp" + bb.utils.mkdirhier(os.path.dirname(localpath)) + fetchcmd += " -O %s" % shlex.quote(localpath) if ud.user and ud.pswd: fetchcmd += " --auth-no-challenge" @@ -136,6 +135,10 @@ class Wget(FetchMethod): self._runwget(ud, d, fetchcmd, False) + # Remove the ".tmp" and move the file into position atomically + # Our lock prevents multiple writers but mirroring code may grab incomplete files + os.rename(localpath, localpath[:-4]) + # Sanity check since wget can pretend it succeed when it didn't # Also, this used to happen if sourceforge sent us to the mirror page if not os.path.exists(ud.localpath): From patchwork Sat Oct 5 13:44:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 49968 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E423CF887F for ; Sat, 5 Oct 2024 13:45:08 +0000 (UTC) Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by mx.groups.io with SMTP id smtpd.web10.11707.1728135905494414693 for ; Sat, 05 Oct 2024 06:45:05 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=GZzyoBmv; spf=softfail (domain: sakoman.com, ip: 209.85.216.54, mailfrom: steve@sakoman.com) Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2e192a2fe6cso2305174a91.3 for ; Sat, 05 Oct 2024 06:45:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1728135905; x=1728740705; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=TwtMT9yS/5nlgHmQ4IME4Ml4eV8R3WBGi0rV80Mz3f4=; b=GZzyoBmvxc+0lmLTxTr2tp5kLui6KuNIYPXtlBMtbzK9ttD1FNiCO70Pj+rcvt2vRu /8WSGu1Wsyq9/QAWQGW2vvIS/ltyxV1JqZkwDcUXJ8FGhO/phXQg7cKUxkwAzzn4lR6j nB3NHOCMvt82fI14LGGVOd2U39cp28dhlNFdLkmJ2m7GS13MHHS6CY54UWXb2zuVWOLC xnYTJcPppBzzw8UiMO2PMBRqct7jF2CjBz0uZzoFtlWoFKeMJnsevHKia0kewjl/F1Yx jpEmusfkNBGZ3gQEvydIiovrDFIETF3x9HwlDUPewcjI9QcfSjkYbdKN70wWWvegCvnW p6xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728135905; x=1728740705; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TwtMT9yS/5nlgHmQ4IME4Ml4eV8R3WBGi0rV80Mz3f4=; b=ql1eT8BhpmYAfhgF0TDJyerQBeX66bmRgzKbsoJnZ9l/bp/8O11cXZqkmEdO2maOi5 zsqbnqNlr4kouecv+8v+tTOmhHjYIR94po1uPQfv4Ve/dCBFNDYuXzJuFfMa8IwUq4w4 EWwrVqpYlJIB5W8vmXWF3/4oS7RzW2YhLPHXF6wEPcWourEO7ptcW78hS92SGVPj7STU cvEb4fJ/wEXFxuEFZQ6SQzktasJGmQ7MbTxnslMEF6Es4U1uYVrT9SYKpH2kd5D0cERh iX/6QcQ7+b+NKuLvLeSUZp7FQpSY4h/8bmJ3nhbimhHl8v20tNBzX0/bbxReObb+R+7y llSA== X-Gm-Message-State: AOJu0YzpqUfmaBKL1LJ2TvkxdJRKYAzo5CKEoaMcOGR+kYDIYB7liOhO aYbvnQ3Qe7f9IEP8b2dG0SQCzFYvpz19unoqr9rezyAhb379/zPnnDQeRWKv0zYAYJFVIC4M38F szZg= X-Google-Smtp-Source: AGHT+IEXPrEp+sBLf0axxSgPcWY8b0IC2wcoc9by7IXIDpnoPO9jcLOz0wjxz8ob919TcUt+5R06pw== X-Received: by 2002:a17:90a:678d:b0:2e0:6c5f:4eb9 with SMTP id 98e67ed59e1d1-2e1e6227026mr7173739a91.12.1728135904702; Sat, 05 Oct 2024 06:45:04 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e20b0f919csm1831617a91.44.2024.10.05.06.45.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 05 Oct 2024 06:45:04 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][kirkstone][2.0][PATCH 4/4] fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility Date: Sat, 5 Oct 2024 06:44:51 -0700 Message-Id: <07081a94997142746f7d345c27bc6805231d025d.1728135724.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 05 Oct 2024 13:45:08 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16644 From: Rudolf J Streif Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. Signed-off-by: Rudolf J Streif Signed-off-by: Richard Purdie (cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Steve Sakoman (cherry picked from commit 47678142e26bb76d1351886060deff5e75039bc9) Signed-off-by: Steve Sakoman --- lib/bb/fetch2/wget.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index 708f3a42e..3849c8a5a 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -109,7 +109,8 @@ class Wget(FetchMethod): fetchcmd = self.basecmd - localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) + ".tmp" + dldir = os.path.realpath(d.getVar("DL_DIR")) + localpath = os.path.join(dldir, ud.localfile) + ".tmp" bb.utils.mkdirhier(os.path.dirname(localpath)) fetchcmd += " -O %s" % shlex.quote(localpath) @@ -129,9 +130,9 @@ class Wget(FetchMethod): uri = ud.url.split(";")[0] if os.path.exists(ud.localpath): # file exists, but we didnt complete it.. trying again.. - fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri) + fetchcmd += " -c -P " + dldir + " '" + uri + "'" else: - fetchcmd += d.expand(" -P ${DL_DIR} '%s'" % uri) + fetchcmd += " -P " + dldir + " '" + uri + "'" self._runwget(ud, d, fetchcmd, False)