diff mbox series

fetch2: use downloadfilename to fetch from mirrors

Message ID 20250127144314.27990-1-stefan.herbrechtsmeier-oss@weidmueller.com
State New
Headers show
Series fetch2: use downloadfilename to fetch from mirrors | expand

Commit Message

Stefan Herbrechtsmeier Jan. 27, 2025, 2:43 p.m. UTC 
From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Use the downloadfilename to fetch from download mirrors. This allows
fetchers to store downloaded files inside a subfolder.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---
Alternative we should forbid folder inside the downloadfilename and we
should add a check to the fetcher to ensure plain filenames.


 lib/bb/fetch2/__init__.py | 4 +++-
 lib/bb/tests/fetch.py     | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)

Comments

Mathieu Dubois-Briand Jan. 28, 2025, 5:37 p.m. UTC | #1 
On Mon Jan 27, 2025 at 3:43 PM CET, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>
> Use the downloadfilename to fetch from download mirrors. This allows
> fetchers to store downloaded files inside a subfolder.
>
> Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
> ---

Hi Stefan,

Thanks for your patch.

It looks like this is causing some issues on the autobuilder, with the
oe-selftests:

2025-01-28 08:46:33,938 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 261, in test_sstate_creation_distro_nonspecific_pass
    self.run_test_sstate_creation(['linux-libc-headers'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
  File "/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 131, in run_test_sstate_creation
    self.assertTrue(badperms , msg="Found sstate directories with the wrong permissions: %s (found %s)" % (', '.join(map(str, targets)), str(badperms)))
  File "/usr/lib64/python3.12/unittest/case.py", line 727, in assertTrue
    raise self.failureException(msg)
AssertionError: [] is not true : Found sstate directories with the wrong permissions: linux-libc-headers (found [])

https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/815/steps/14/logs/stdio

Can you have a look at this issue please ?
Stefan Herbrechtsmeier Jan. 29, 2025, 12:20 p.m. UTC | #2 
Am 28.01.2025 um 18:37 schrieb Mathieu Dubois-Briand:
> On Mon Jan 27, 2025 at 3:43 PM CET, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
>> From: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com>
>>
>> Use the downloadfilename to fetch from download mirrors. This allows
>> fetchers to store downloaded files inside a subfolder.
>>
>> Signed-off-by: Stefan Herbrechtsmeier<stefan.herbrechtsmeier@weidmueller.com>
>> ---
> Hi Stefan,
>
> Thanks for your patch.
>
> It looks like this is causing some issues on the autobuilder, with the
> oe-selftests:
>
> 2025-01-28 08:46:33,938 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
>    File "/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 261, in test_sstate_creation_distro_nonspecific_pass
>      self.run_test_sstate_creation(['linux-libc-headers'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
>    File "/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 131, in run_test_sstate_creation
>      self.assertTrue(badperms , msg="Found sstate directories with the wrong permissions: %s (found %s)" % (', '.join(map(str, targets)), str(badperms)))
>    File "/usr/lib64/python3.12/unittest/case.py", line 727, in assertTrue
>      raise self.failureException(msg)
> AssertionError: [] is not true : Found sstate directories with the wrong permissions: linux-libc-headers (found [])
>
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/815/steps/14/logs/stdio
>
> Can you have a look at this issue please ?

I have found the problem and a simple fix but need to clarify the 
correct solution. Please drop the patch for now.
diff mbox series

Patch

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index f79e278b1..c0b39ea5c 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -472,7 +472,9 @@  def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
                     uri_decoded[5] = {}
                     uri_find_decoded[5] = {}
                 elif ud.localpath and ud.method.supports_checksum(ud):
-                    basename = os.path.basename(ud.localpath)
+                    basename = uri_decoded[5].get("downloadfilename")
+                    if not basename:
+                        basename = os.path.basename(ud.localpath)
                 if basename:
                     uri_basename = os.path.basename(uri_decoded[loc])
                     # Prefix with a slash as a sentinel in case
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 6dda0d381..50f047b4c 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -502,6 +502,8 @@  class MirrorUriTest(FetcherTest):
             : "file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz",
         ("https://somewhere.org/example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz", "https://.*/.*", "file:///mirror/some-example-1.0.0.tgz")
             : "file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz",
+        ("https://somewhere.org/scope/example/1.0.0/example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz", "https://.*/.*", "http://somewhere2.org/somedir3")
+            : "http://somewhere2.org/somedir3/subdir/scope-example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz",
 
         #Renaming files doesn't work
         #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz"