From patchwork Mon Jun 17 12:23:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 45268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92E14C2BA18 for ; Mon, 17 Jun 2024 12:24:15 +0000 (UTC) Received: from mail-io1-f48.google.com (mail-io1-f48.google.com [209.85.166.48]) by mx.groups.io with SMTP id smtpd.web10.55383.1718627048961796561 for ; Mon, 17 Jun 2024 05:24:09 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=js+n38SZ; spf=softfail (domain: sakoman.com, ip: 209.85.166.48, mailfrom: steve@sakoman.com) Received: by mail-io1-f48.google.com with SMTP id ca18e2360f4ac-7ebc67365e3so177497739f.1 for ; Mon, 17 Jun 2024 05:24:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1718627048; x=1719231848; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=wEb3+/dsVp9s1EtByQmnzrZBdIUnEbdRrPYwGNfNrz8=; b=js+n38SZlHv/sesueJf0cikh91HpA9/WJn31qeeIRQouBI/h8SYPSe4tjL1ICrGDys sMVIgw4vMSD589TL6I2BDD4hyk2rCUYgkTxdOiTB4QcEdz0fIWmsXnYnbiu7wBz67uvp WwrQiBrhTFN1CEJid1AJvr8z+sEz7zYNvwX/Yk7wJ1HrEP65OmWgbuYp+YYrxE775PKg Yv/S8tTcpW0cKYulfI/DorR1gmJuUWF04xCKi481Fq4A9avVOqk2o1YjQ6FYqo02s3u1 fRGIK/hmq35Ug20YHB/Pu4/LoEvUWmOLArwj7mMq5XJ/mwcmcmVrFftNbjfsxG/26Vg6 mnHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718627048; x=1719231848; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wEb3+/dsVp9s1EtByQmnzrZBdIUnEbdRrPYwGNfNrz8=; b=wED42uuBL4nHquxhZaCvFUghSUcEiVLeuVf06yqLHsz/Kx5c/6WFMVsXB0pPXzJCS5 wWZ3GYLduwcMdXCaTcIwjtxpZMWHRbowilxzNvzyjXlAWV9AmjceG/0/HTaarcfWocKx VxeUOO7RabyrYGg4iceHLrdkGcTE+CVlfeABy6PmoepU/1s4pT0rg9mKTYVqH5nJpkmU lCJ/X+elI9M9s864ECZDRAyuOFVbSZzBMBL4kFNl+hcMzEy2O8ekLIVoiBs6lFclXkNE wXv8jaNb86JCpjsftOwOQEOG8OBolhllJznV77ug+9+B20npKs+lZNXLR3Xvn78lkzFq XJvA== X-Gm-Message-State: AOJu0Ywq1Imuht9bYUv01Q2lmp2V5LfGRTcchh8vACsgYCgFXq8NRiSl fsRTBwAZbtS7WYeizsSzPfhYSdlX0MU9xYWIKTFLr86aRS8K/75mYqBk7R1TH3NzhltLQk0tri5 01BM= X-Google-Smtp-Source: AGHT+IEH9yTrsuDMzfaqDHpBiNvC+5h2gecUmPwhslteCZnf/y3X8BGK1gVRS2lk44aZcN7MT46puw== X-Received: by 2002:a05:6602:158a:b0:7eb:7e0c:d186 with SMTP id ca18e2360f4ac-7ebeb62a060mr1081916339f.16.1718627048020; Mon, 17 Jun 2024 05:24:08 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-6fedf39b75csm6608259a12.45.2024.06.17.05.24.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Jun 2024 05:24:07 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 1/3] fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility Date: Mon, 17 Jun 2024 05:23:59 -0700 Message-Id: <47678142e26bb76d1351886060deff5e75039bc9.1718626914.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 17 Jun 2024 12:24:15 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16361 From: Rudolf J Streif Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. Signed-off-by: Rudolf J Streif Signed-off-by: Richard Purdie (cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Steve Sakoman --- lib/bb/fetch2/wget.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index fbfa6938a..b41f53a9e 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -108,7 +108,8 @@ class Wget(FetchMethod): fetchcmd = self.basecmd - localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) + ".tmp" + dldir = os.path.realpath(d.getVar("DL_DIR")) + localpath = os.path.join(dldir, ud.localfile) + ".tmp" bb.utils.mkdirhier(os.path.dirname(localpath)) fetchcmd += " -O %s" % shlex.quote(localpath) @@ -128,9 +129,9 @@ class Wget(FetchMethod): uri = ud.url.split(";")[0] if os.path.exists(ud.localpath): # file exists, but we didnt complete it.. trying again.. - fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri) + fetchcmd += " -c -P " + dldir + " '" + uri + "'" else: - fetchcmd += d.expand(" -P ${DL_DIR} '%s'" % uri) + fetchcmd += " -P " + dldir + " '" + uri + "'" self._runwget(ud, d, fetchcmd, False) From patchwork Mon Jun 17 12:24:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 45266 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84C3BC41513 for ; Mon, 17 Jun 2024 12:24:15 +0000 (UTC) Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) by mx.groups.io with SMTP id smtpd.web11.54899.1718627050989387824 for ; Mon, 17 Jun 2024 05:24:11 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=E90N2MZn; spf=softfail (domain: sakoman.com, ip: 209.85.166.42, mailfrom: steve@sakoman.com) Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-7eeea388a8eso38542439f.2 for ; Mon, 17 Jun 2024 05:24:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1718627050; x=1719231850; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=fvvYw9OP/t8K4TS6ZNiKvWNipFzBfh3QobbHP6kwhT8=; b=E90N2MZnV/Hx+kxKVapa5Qidnrxi7sZ1jAvQmZMFXguwckqZSojH3jgFy24t2ECzch pygqENSkv0zBLvkJV6Z4Sx2BRKtihTNkqSOAGr1032OwpzRKqDU1Soo1ARHowD3xkhHj Z59cdTc8PwYFSf45bqTt0Zzkay6fjmfioMyan/BEafELfMTHkBwP9QKT2bbt6+8ha3wH 47oY0qNyvaNj0xGvzRf+JiiAwLRRZg2dromXkfSBVA1fIfJbqCHsRUUznrUZnYHnUthm 5sRnJbqPi223MSmdm6P8UW0AGCiHbO7C8vK7JyfY7rlszEGKgnK6KtU9z1cFGkg8x3sy lSMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718627050; x=1719231850; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fvvYw9OP/t8K4TS6ZNiKvWNipFzBfh3QobbHP6kwhT8=; b=dmCVR87wD+O2KuAxFn+xuCZ+5UPKdWoGyIzN+TUH8UPC+vjJCv4ceY22mP/hsTy6gk 2n4zo60wBi9SSfNhmWcz/duSxhaoDiXULVGAnLLzoTRtZUPQZG3RUPBnUYHdxchvPz69 /xp3vc8ukcgGitpJNlGtmojiigl5hJzFe+drUOfGZB6ez0ge81p8s9CWwNcDg3LgorpO YLohkPwNO6bvZfjrgjZhGQtNzOl07crsazAeb9ECExXFmce5uZnv5w5Aj9BpWnrO0fWN wC5DJMdueZm1EZcOMrXcPZy9S6TksRhFACrNYANZejaeqibspK/Ey/EHKR93ddn6jrNX EomQ== X-Gm-Message-State: AOJu0YwBddIO5Hnoayj139VhoH1tZU5CgIDlfzq9Wp4jdKjRVsVTUjWK hL42u7GwiKLsI2hqi3LerpEDgvQKZ9nm568SgZV8YftKKonDKfaa9Qye5Qb7sn1pLLghp3IIisO tbbI= X-Google-Smtp-Source: AGHT+IH2L5QVWgBWidYytthVjhPfhBgjc71LJsgG8ZXhNmdMSyvdPR6UXunDEIdtG+gmxt1OVQcurg== X-Received: by 2002:a05:6602:2b92:b0:7e1:af90:44ee with SMTP id ca18e2360f4ac-7ebeb4c7167mr1124955339f.8.1718627049913; Mon, 17 Jun 2024 05:24:09 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-6fedf39b75csm6608259a12.45.2024.06.17.05.24.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Jun 2024 05:24:09 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 2/3] tests/fetch: Tweak to work on Fedora40 Date: Mon, 17 Jun 2024 05:24:00 -0700 Message-Id: <5fa596b3f2e49fbd144347b9800c76de72a29e28.1718626914.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 17 Jun 2024 12:24:15 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16362 From: Richard Purdie On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1 and python only binds to one of the addresses, leading to test failures. Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying to test things other than the host networking. Signed-off-by: Richard Purdie (cherry picked from commit 9adc6da42618f41bf0d6b558d62b2f3c13bedd61) Signed-off-by: Steve Sakoman --- lib/bb/tests/fetch.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 8a91f51d3..33cc9bcac 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -1511,7 +1511,7 @@ class FetchLatestVersionTest(FetcherTest): def test_wget_latest_versionstring(self): testdata = os.path.dirname(os.path.abspath(__file__)) + "/fetch-testdata" - server = HTTPService(testdata) + server = HTTPService(testdata, host="127.0.0.1") server.start() port = server.port try: @@ -1519,10 +1519,10 @@ class FetchLatestVersionTest(FetcherTest): self.d.setVar("PN", k[0]) checkuri = "" if k[2]: - checkuri = "http://localhost:%s/" % port + k[2] + checkuri = "http://127.0.0.1:%s/" % port + k[2] self.d.setVar("UPSTREAM_CHECK_URI", checkuri) self.d.setVar("UPSTREAM_CHECK_REGEX", k[3]) - url = "http://localhost:%s/" % port + k[1] + url = "http://127.0.0.1:%s/" % port + k[1] ud = bb.fetch2.FetchData(url, self.d) pupver = ud.method.latest_versionstring(ud, self.d) verstring = pupver[0] From patchwork Mon Jun 17 12:24:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 45267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4908C2BA15 for ; Mon, 17 Jun 2024 12:24:15 +0000 (UTC) Received: from mail-oo1-f54.google.com (mail-oo1-f54.google.com [209.85.161.54]) by mx.groups.io with SMTP id smtpd.web11.54900.1718627052719725230 for ; Mon, 17 Jun 2024 05:24:12 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20230601.gappssmtp.com header.s=20230601 header.b=yOfB/Lie; spf=softfail (domain: sakoman.com, ip: 209.85.161.54, mailfrom: steve@sakoman.com) Received: by mail-oo1-f54.google.com with SMTP id 006d021491bc7-5bad217c51aso2366409eaf.1 for ; Mon, 17 Jun 2024 05:24:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20230601.gappssmtp.com; s=20230601; t=1718627052; x=1719231852; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=QdqWBKmQVmS4eSqIPSRd8J5EdFOYnlxgVZ8vKFehHII=; b=yOfB/LiejIjVQHhP94jQEEfNYpSUs9y4z+rIWPS8WYGLF+mBU3JuI67wQogQPdq8LH 6IhEwmkLDmhrDKWlRW4NtDqW8oZOkaE/AstJNDso80q5u99a/OfzIcRDvHguipwx4UP5 +FBfF1+uZm+g6WL4q3SE7R+E9qcGbOeShBOA73V5D2AM+ohVUka6pUeau25iA3qpoY2S eJuB7I1rmkWfav7NMyve6CrUMOkIiijwPXzIcEL8jqcQ7psawhzfzMeEKas1iQjQmArH zVrEl3YKMtMYAQ2C7orLEL1eeSjaqav/q4j1AHgmvS/Rtsk1TGKt8hApgzmWqJCGidgV w/Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718627052; x=1719231852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QdqWBKmQVmS4eSqIPSRd8J5EdFOYnlxgVZ8vKFehHII=; b=iLK18b7D7k460cVULTKrzJEV32wnwNGkLoc1cB6p6twOQ56P8+QcZIigVIeY30FgFZ CNhCFZDpmXvHlLa7UrADRfSLbu0HvPXlqTKEYZ+a0cwBeKgoanZqb4KmWiEmuzc6Prhy u/JGwHckHa5CYW2uVeNCanggQRGKCQp3I9D0yJj4jhgAwtptTJbGryovdIvfnEkhahxX Xq//6SEOMYzKVYrL0AFsALQIVFiIj0Robmk243JVIM4shWGJGqg5K//Dj+/5xzUlEfV9 4be0K98DOhskYJeNOIAFX2LaHq/ZCVD5e59vDf7jei1zsblKrWZlBEEVjsFR6hRLYoWu +hSw== X-Gm-Message-State: AOJu0YzxK0nJ8G6T4twZp6O7s4whiTUqo+TLNMpxfeFsSVw8RFJf65WO LoZTnX/wOdLB15HPG/Mnf+AcKTJtbG4y1IPZl0+8cb9TAza3IakHASOlRqlDR59crOjck/Yv4yY jy+M= X-Google-Smtp-Source: AGHT+IGVnBGXNS/Sxs95eg7dFrcaCiPx0y79YA4zuXX7lJxdkDBWUveWDvcPrZhbmyKadaL+Sf8ydQ== X-Received: by 2002:a05:6358:704:b0:1a1:d24b:9a9f with SMTP id e5c5f4694b2df-1a1d24b9c36mr684599655d.1.1718627051579; Mon, 17 Jun 2024 05:24:11 -0700 (PDT) Received: from hexa.. ([98.142.47.158]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-6fedf39b75csm6608259a12.45.2024.06.17.05.24.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Jun 2024 05:24:11 -0700 (PDT) From: Steve Sakoman To: bitbake-devel@lists.openembedded.org Subject: [bitbake][scarthgap][2.8][PATCH 3/3] fetch2/wget: Fix failure path for files that are empty or don't exist Date: Mon, 17 Jun 2024 05:24:01 -0700 Message-Id: <8714a02e13477a9d97858b3642e05f28247454b5.1718626914.git.steve@sakoman.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 17 Jun 2024 12:24:15 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16363 From: Richard Purdie When we intercepted the file download to a temp file, we broke the exist/size checks which need to happen before the rename. Correct the ordering. For some reason, python 3.12 exposes this problem in the selftests differently to previous versions. Signed-off-by: Richard Purdie (cherry picked from commit c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d) Signed-off-by: Steve Sakoman --- lib/bb/fetch2/wget.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index b41f53a9e..2e9211763 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -135,6 +135,15 @@ class Wget(FetchMethod): self._runwget(ud, d, fetchcmd, False) + # Sanity check since wget can pretend it succeed when it didn't + # Also, this used to happen if sourceforge sent us to the mirror page + if not os.path.exists(localpath): + raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, localpath), uri) + + if os.path.getsize(localpath) == 0: + os.remove(localpath) + raise FetchError("The fetch of %s resulted in a zero size file?! Deleting and failing since this isn't right." % (uri), uri) + # Try and verify any checksum now, meaning if it isn't correct, we don't remove the # original file, which might be a race (imagine two recipes referencing the same # source, one with an incorrect checksum) @@ -144,15 +153,6 @@ class Wget(FetchMethod): # Our lock prevents multiple writers but mirroring code may grab incomplete files os.rename(localpath, localpath[:-4]) - # Sanity check since wget can pretend it succeed when it didn't - # Also, this used to happen if sourceforge sent us to the mirror page - if not os.path.exists(ud.localpath): - raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, ud.localpath), uri) - - if os.path.getsize(ud.localpath) == 0: - os.remove(ud.localpath) - raise FetchError("The fetch of %s resulted in a zero size file?! Deleting and failing since this isn't right." % (uri), uri) - return True def checkstatus(self, fetch, ud, d, try_again=True):