| Message ID | 20250902065507.35737-1-stefan.herbrechtsmeier-oss@weidmueller.com |
|---|---|
| Headers | show |
| Series | fetch2: add support for implicit urls | expand |
On Tue Sep 2, 2025 at 8:55 AM CEST, Stefan Herbrechtsmeier via lists.openembedded.org wrote: > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> > > The patch series add support for implicit URLs inside the fetcher. The > implicit URLs could be defined inside a source like a version control > system (git submodule) or a lock file (package-lock.json, cargo.lock or > go.sum). The integration of implicit URLs beside explicit URLs > simplifies the fetcher classes and avoid bugs because of iterations > between the Fetch and FetchMethod classes. > > The series remove most methods inside the gitsm fetcher and only leaves > the parsing of the git submodules and the unpack functionality. It > allows the gitsm fetcher to use the premirror only feature. The current > implementation leads to problems because the download of the git > submodules is triggered via the download method which is called deeply > inside the fetcher code. > > > Stefan Herbrechtsmeier (6): > fetch2: rename u to url in Fetch class > fetch2: call functions within loops of Fetch class > fetch2: add helper to get urldata in Fetch class > fetch2: add support for implicit urls > fetch2: gitsm: use implicit urls feature > tests: fetch: add test case for gitsm implicit local paths > > lib/bb/fetch2/__init__.py | 128 +++++++++++++++++++++++++++----------- > lib/bb/fetch2/gitsm.py | 46 ++------------ > lib/bb/tests/fetch.py | 12 ++++ > 3 files changed, 109 insertions(+), 77 deletions(-) Hi Stefan, I know it is just an RFC so far, but I did launch a build on the autobuilder. It was mostly correct but fails a selftest: ERROR: git-submodule-test-1.0-r0 do_ar_mirror: Error executing a python function in exec_func_python() autogenerated: ... File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/fetch2/__init__.py', lineno: 2102, function: expand_urldata ... Exception: UnboundLocalError: cannot access local variable 'urldata' where it is not associated with a value ... 2025-09-03 06:21:46,098 - oe-selftest - INFO - archiver.Archiver.test_archiver_mode_mirror_gitsm (subunit.RemotedTestCase) 2025-09-03 06:21:46,099 - oe-selftest - INFO - ... FAIL And a similar error with archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow. Thanks, Mathieu
Hi Mathieu, Thank you for testing. Until now I only run the bitbake tests. I will look into the error and will add a test case to bitbake to catch the problem early. Thanks Stefan Am 04.09.2025 um 08:00 schrieb Mathieu Dubois-Briand: > On Tue Sep 2, 2025 at 8:55 AM CEST, Stefan Herbrechtsmeier via lists.openembedded.org wrote: >> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> >> >> The patch series add support for implicit URLs inside the fetcher. The >> implicit URLs could be defined inside a source like a version control >> system (git submodule) or a lock file (package-lock.json, cargo.lock or >> go.sum). The integration of implicit URLs beside explicit URLs >> simplifies the fetcher classes and avoid bugs because of iterations >> between the Fetch and FetchMethod classes. >> >> The series remove most methods inside the gitsm fetcher and only leaves >> the parsing of the git submodules and the unpack functionality. It >> allows the gitsm fetcher to use the premirror only feature. The current >> implementation leads to problems because the download of the git >> submodules is triggered via the download method which is called deeply >> inside the fetcher code. >> >> >> Stefan Herbrechtsmeier (6): >> fetch2: rename u to url in Fetch class >> fetch2: call functions within loops of Fetch class >> fetch2: add helper to get urldata in Fetch class >> fetch2: add support for implicit urls >> fetch2: gitsm: use implicit urls feature >> tests: fetch: add test case for gitsm implicit local paths >> >> lib/bb/fetch2/__init__.py | 128 +++++++++++++++++++++++++++----------- >> lib/bb/fetch2/gitsm.py | 46 ++------------ >> lib/bb/tests/fetch.py | 12 ++++ >> 3 files changed, 109 insertions(+), 77 deletions(-) > Hi Stefan, > > I know it is just an RFC so far, but I did launch a build on the > autobuilder. It was mostly correct but fails a selftest: > > ERROR: git-submodule-test-1.0-r0 do_ar_mirror: Error executing a python function in exec_func_python() autogenerated: > ... > File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/fetch2/__init__.py', lineno: 2102, function: expand_urldata > ... > Exception: UnboundLocalError: cannot access local variable 'urldata' where it is not associated with a value > ... > 2025-09-03 06:21:46,098 - oe-selftest - INFO - archiver.Archiver.test_archiver_mode_mirror_gitsm (subunit.RemotedTestCase) > 2025-09-03 06:21:46,099 - oe-selftest - INFO - ... FAIL > > And a similar error with > archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow. > > Thanks, > Mathieu >
Am 04.09.2025 um 08:00 schrieb Mathieu Dubois-Briand: > On Tue Sep 2, 2025 at 8:55 AM CEST, Stefan Herbrechtsmeier via lists.openembedded.org wrote: >> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> >> >> The patch series add support for implicit URLs inside the fetcher. The >> implicit URLs could be defined inside a source like a version control >> system (git submodule) or a lock file (package-lock.json, cargo.lock or >> go.sum). The integration of implicit URLs beside explicit URLs >> simplifies the fetcher classes and avoid bugs because of iterations >> between the Fetch and FetchMethod classes. >> >> The series remove most methods inside the gitsm fetcher and only leaves >> the parsing of the git submodules and the unpack functionality. It >> allows the gitsm fetcher to use the premirror only feature. The current >> implementation leads to problems because the download of the git >> submodules is triggered via the download method which is called deeply >> inside the fetcher code. >> >> >> Stefan Herbrechtsmeier (6): >> fetch2: rename u to url in Fetch class >> fetch2: call functions within loops of Fetch class >> fetch2: add helper to get urldata in Fetch class >> fetch2: add support for implicit urls >> fetch2: gitsm: use implicit urls feature >> tests: fetch: add test case for gitsm implicit local paths >> >> lib/bb/fetch2/__init__.py | 128 +++++++++++++++++++++++++++----------- >> lib/bb/fetch2/gitsm.py | 46 ++------------ >> lib/bb/tests/fetch.py | 12 ++++ >> 3 files changed, 109 insertions(+), 77 deletions(-) > I know it is just an RFC so far, but I did launch a build on the > autobuilder. It was mostly correct but fails a selftest: > > ERROR: git-submodule-test-1.0-r0 do_ar_mirror: Error executing a python function in exec_func_python() autogenerated: > ... > File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/fetch2/__init__.py', lineno: 2102, function: expand_urldata > ... > Exception: UnboundLocalError: cannot access local variable 'urldata' where it is not associated with a value > ... > 2025-09-03 06:21:46,098 - oe-selftest - INFO - archiver.Archiver.test_archiver_mode_mirror_gitsm (subunit.RemotedTestCase) > 2025-09-03 06:21:46,099 - oe-selftest - INFO - ... FAIL > > And a similar error with > archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow. I've fixed the issue and added a BitBake test case. Perhaps you could test the second version of the two patch series. Regards Stefan
From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> The patch series add support for implicit URLs inside the fetcher. The implicit URLs could be defined inside a source like a version control system (git submodule) or a lock file (package-lock.json, cargo.lock or go.sum). The integration of implicit URLs beside explicit URLs simplifies the fetcher classes and avoid bugs because of iterations between the Fetch and FetchMethod classes. The series remove most methods inside the gitsm fetcher and only leaves the parsing of the git submodules and the unpack functionality. It allows the gitsm fetcher to use the premirror only feature. The current implementation leads to problems because the download of the git submodules is triggered via the download method which is called deeply inside the fetcher code. Stefan Herbrechtsmeier (6): fetch2: rename u to url in Fetch class fetch2: call functions within loops of Fetch class fetch2: add helper to get urldata in Fetch class fetch2: add support for implicit urls fetch2: gitsm: use implicit urls feature tests: fetch: add test case for gitsm implicit local paths lib/bb/fetch2/__init__.py | 128 +++++++++++++++++++++++++++----------- lib/bb/fetch2/gitsm.py | 46 ++------------ lib/bb/tests/fetch.py | 12 ++++ 3 files changed, 109 insertions(+), 77 deletions(-)