From patchwork Thu Dec 8 16:03:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 16540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A67B7C3A5A7 for ; Thu, 8 Dec 2022 16:03:27 +0000 (UTC) Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by mx.groups.io with SMTP id smtpd.web10.18113.1670515405882507772 for ; Thu, 08 Dec 2022 08:03:26 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=MctWeEqh; spf=pass (domain: gmail.com, ip: 209.85.221.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-wr1-f51.google.com with SMTP id w15so2136239wrl.9 for ; Thu, 08 Dec 2022 08:03:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=BLYxuU2JveJh+Ois2xgeuDtCWxYTV0JJkWjFoEd2T7U=; b=MctWeEqhKPE0ZVLN/iLWFzOvzpEcfNiowoKN5naIrXpQP+cGf5s1yWU7G0QtKP80L/ O+W/T2CuLz6YMO//xo6cDIilIyGmsEm1skvYRDqHhwcznwDqVYpcT0zvGiUCffHQomks yav7YKX+Pm3xYA43wJSzrBgq2UM6GGP+6hVdkFm3jjPJiJpr7wM4ejFqqCTGSC4nxFcM LEBWDqdDgaOzDFE33g2vvV5yaz65mthf3aYeKniTgGMRzFc9Ig7/NN7ro8NTB7Mju4Cr s4ZrOBAsGF/zSUasJlVOaG2CDkeTlXztOQ1Emq5jlfKayO3lZIlwRr5tefo0wVb0M/EF NCRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=BLYxuU2JveJh+Ois2xgeuDtCWxYTV0JJkWjFoEd2T7U=; b=HjwpJCtLNXRRHluM9gZNhBraz2jEqb8JihXVMhYV7Txq3mrQNd60GArNjmvRTlR98e +IFmlmtEJhLdi9alcOhsbayobeX91nOxJj3CYebO57TDnoGotKbk+e1jL8oclVB4Akwk j50QD8kYiSrWBFhYGpOypTVLHLRy/s309hXePtTivEdmdoXrPiZ2m0+hwI/AwBcrF4Ih GWGZTVA/1wl3ykp6J39r9gkCVT0Ol0eGP3zu0JoLA+Om5twPhMs9F+zhKqKaqnEpVePw /0wtFxxqn0Ho2YkX8VD6A3yXJXh2JAvLPVlpbhSoH/OQYXzF2/gqH26nQnFjkleftPH/ cV+w== X-Gm-Message-State: ANoB5pllZyi81+u6DZT7L9jJU/ghgmaOPoqLqjFji2CQC/s7/nUaqvbQ rzNEUiL+swPBmiXDUMhBw1wWrVVaDuQ= X-Google-Smtp-Source: AA0mqf6t1JzigPfDLkyggvaHvtT+vK7EtDIq+Jusp45tQKmP8jIhlncSsCI3oinCJP8HbBtOLqizzw== X-Received: by 2002:adf:f5c3:0:b0:242:3427:bb51 with SMTP id k3-20020adff5c3000000b002423427bb51mr18951550wrp.635.1670515404021; Thu, 08 Dec 2022 08:03:24 -0800 (PST) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id k10-20020a056000004a00b002365730eae8sm22634034wrx.55.2022.12.08.08.03.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Dec 2022 08:03:23 -0800 (PST) From: Alexander Kanavin X-Google-Original-From: Alexander Kanavin To: bitbake-devel@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH v2] fetch2/wget.py: correctly match versioned directories Date: Thu, 8 Dec 2022 17:03:20 +0100 Message-Id: <20221208160320.3306734-1-alex@linutronix.de> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 08 Dec 2022 16:03:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14152 When obtaining latest upstream versions, the code needs to check if the existing tarball is in a versioned directory (e.g. component-name/x.y/component-name-x.y.z.tar.gz) and if it is, it needs to first obtain the list of all such versioned directories and then check all of them by going one step up in the directory hierarchy. Existing code was returning a correct match when the component name did not have numbers, e.g. a check on 'source/epiphany/43/' would return 43, but was stopping too soon when the component name itself had numbers ('source/libxml2/2.10/' would return libxml2). This change ensures the last match is taken instead of the first. Also, adjust the fetcher tests to check that versioned directories are correctly traversed in this case (e.g. the step to go one level up is taken and a new tarball is discovered in a different versioned directory). Signed-off-by: Alexander Kanavin --- bitbake/lib/bb/fetch2/wget.py | 4 +- .../software/libxml2/2.10/index.html | 20 ++++++++++ .../software/libxml2/2.9/index.html | 40 +++++++++++++++++++ .../software/libxml2/index.html | 19 +++++++++ bitbake/lib/bb/tests/fetch.py | 3 ++ 5 files changed, 84 insertions(+), 2 deletions(-) create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py index 821afa5b58..7ccea3ad89 100644 --- a/bitbake/lib/bb/fetch2/wget.py +++ b/bitbake/lib/bb/fetch2/wget.py @@ -644,10 +644,10 @@ class Wget(FetchMethod): # search for version matches on folders inside the path, like: # "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz dirver_regex = re.compile(r"(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") - m = dirver_regex.search(path) + m = dirver_regex.findall(path) if m: pn = d.getVar('PN') - dirver = m.group('dirver') + dirver = m[-1][0] dirver_pn_regex = re.compile(r"%s\d?" % (re.escape(pn))) if not dirver_pn_regex.search(dirver): diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html new file mode 100644 index 0000000000..b1d12f6707 --- /dev/null +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html @@ -0,0 +1,20 @@ + + +Index of /sources/libxml2/2.10/ +

Index of /sources/libxml2/2.10/

+ + + + + + + + + + + + + + + +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
LATEST-IS-2.10.32.5 MiB2022-Oct-14 12:55
libxml2-2.10.0.news7.1 KiB2022-Aug-17 11:55
libxml2-2.10.0.sha256sum174 B2022-Aug-17 11:55
libxml2-2.10.0.tar.xz2.6 MiB2022-Aug-17 11:55
libxml2-2.10.1.news455 B2022-Aug-25 11:33
libxml2-2.10.1.sha256sum174 B2022-Aug-25 11:33
libxml2-2.10.1.tar.xz2.6 MiB2022-Aug-25 11:33
libxml2-2.10.2.news309 B2022-Aug-29 14:56
libxml2-2.10.2.sha256sum174 B2022-Aug-29 14:56
libxml2-2.10.2.tar.xz2.5 MiB2022-Aug-29 14:56
libxml2-2.10.3.news294 B2022-Oct-14 12:55
libxml2-2.10.3.sha256sum174 B2022-Oct-14 12:55
libxml2-2.10.3.tar.xz2.5 MiB2022-Oct-14 12:55
\ No newline at end of file diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html new file mode 100644 index 0000000000..d02dbbff67 --- /dev/null +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html @@ -0,0 +1,40 @@ + + +Index of /sources/libxml2/2.9/ +

Index of /sources/libxml2/2.9/

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
LATEST-IS-2.9.143.0 MiB2022-May-02 12:03
libxml2-2.9.0.sha256sum87 B2022-Feb-14 18:27
libxml2-2.9.0.tar.xz3.0 MiB2022-Feb-14 18:27
libxml2-2.9.1.sha256sum87 B2022-Feb-14 18:28
libxml2-2.9.1.tar.xz3.0 MiB2022-Feb-14 18:28
libxml2-2.9.10.sha256sum88 B2022-Feb-14 18:42
libxml2-2.9.10.tar.xz3.2 MiB2022-Feb-14 18:42
libxml2-2.9.11.sha256sum88 B2022-Feb-14 18:43
libxml2-2.9.11.tar.xz3.2 MiB2022-Feb-14 18:43
libxml2-2.9.12.sha256sum88 B2022-Feb-14 18:45
libxml2-2.9.12.tar.xz3.2 MiB2022-Feb-14 18:45
libxml2-2.9.13.news26.6 KiB2022-Feb-20 12:42
libxml2-2.9.13.sha256sum174 B2022-Feb-20 12:42
libxml2-2.9.13.tar.xz3.1 MiB2022-Feb-20 12:42
libxml2-2.9.14.news1.0 KiB2022-May-02 12:03
libxml2-2.9.14.sha256sum174 B2022-May-02 12:03
libxml2-2.9.14.tar.xz3.0 MiB2022-May-02 12:03
libxml2-2.9.2.sha256sum87 B2022-Feb-14 18:30
libxml2-2.9.2.tar.xz3.2 MiB2022-Feb-14 18:30
libxml2-2.9.3.sha256sum87 B2022-Feb-14 18:31
libxml2-2.9.3.tar.xz3.2 MiB2022-Feb-14 18:31
libxml2-2.9.4.sha256sum87 B2022-Feb-14 18:33
libxml2-2.9.4.tar.xz2.9 MiB2022-Feb-14 18:33
libxml2-2.9.5.sha256sum87 B2022-Feb-14 18:35
libxml2-2.9.5.tar.xz3.0 MiB2022-Feb-14 18:35
libxml2-2.9.6.sha256sum87 B2022-Feb-14 18:36
libxml2-2.9.6.tar.xz3.0 MiB2022-Feb-14 18:36
libxml2-2.9.7.sha256sum87 B2022-Feb-14 18:37
libxml2-2.9.7.tar.xz3.0 MiB2022-Feb-14 18:37
libxml2-2.9.8.sha256sum87 B2022-Feb-14 18:39
libxml2-2.9.8.tar.xz3.0 MiB2022-Feb-14 18:39
libxml2-2.9.9.sha256sum87 B2022-Feb-14 18:40
libxml2-2.9.9.tar.xz3.0 MiB2022-Feb-14 18:40
\ No newline at end of file diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html new file mode 100644 index 0000000000..b09341efa8 --- /dev/null +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html @@ -0,0 +1,19 @@ + + +Index of /sources/libxml2/ +

Index of /sources/libxml2/

+ + + + + + + + + + + + + + +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
2.0/-2009-Jul-14 13:04
2.1/-2009-Jul-14 13:04
2.10/-2022-Oct-14 12:55
2.2/-2009-Jul-14 13:04
2.3/-2009-Jul-14 13:05
2.4/-2009-Jul-14 13:05
2.5/-2009-Jul-14 13:05
2.6/-2009-Jul-14 13:05
2.7/-2022-Feb-14 18:24
2.8/-2022-Feb-14 18:26
2.9/-2022-May-02 12:04
cache.json22.8 KiB2022-Oct-14 12:55
\ No newline at end of file diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index 0af06e46e5..ad3d4dea7d 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -1401,6 +1401,9 @@ class FetchLatestVersionTest(FetcherTest): # http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz ("cmake", "/files/v2.8/cmake-2.8.12.1.tar.gz", "", "") : "2.8.12.1", + # https://download.gnome.org/sources/libxml2/2.9/libxml2-2.9.14.tar.xz + ("libxml2", "/software/libxml2/2.9/libxml2-2.9.14.tar.xz", "", "") + : "2.10.3", # # packages with versions only in current directory #